How Uber Manages 1 Million Daily Tasks Using Airflow, With Shobhit Shah And Sumit Maheshwari The Data Flowcast: Mastering Airflow For Data Engineering & AI podcast

İçerik The Data Flowcast tarafından sağlanmıştır. Bölümler, grafikler ve podcast açıklamaları dahil tüm podcast içeriği doğrudan The Data Flowcast veya podcast platform ortağı tarafından yüklenir ve sağlanır. Birinin telif hakkıyla korunan çalışmanızı izniniz olmadan kullandığını düşünüyorsanız burada https://tr.player.fm/legal özetlenen süreci takip edebilirsiniz.

The Data Flowcast: Mastering Airflow for Data Engineering & AI »
How Uber Manages 1 Million Daily Tasks Using Airflow, with Shobhit Shah and Sumit Maheshwari

7d ago 28:44

Paylaş

MP3•Bölüm sayfası

When data orchestration reaches Uber’s scale, innovation becomes a necessity, not a luxury. In this episode, we discuss the innovations behind Uber’s unique Airflow setup. With our guests Shobhit Shah and Sumit Maheshwari, both Staff Software Engineers at Uber, we explore how their team manages one of the largest data workflow systems in the world. Shobhit and Sumit walk us through the evolution of Uber’s Airflow implementation, detailing the custom solutions that support 200,000 daily pipelines. They discuss Uber's approach to tackling complex challenges in data orchestration, disaster recovery and scaling to meet the company’s extensive data needs.

Key Takeaways:

(02:03) Airflow as a service streamlines Uber’s data workflows.

(06:16) Serialization boosts security and reduces errors.

(10:05) Java-based scheduler improves system reliability.

(13:40) Custom recovery model supports emergency pipeline switching.

(15:58) No-code UI allows easy pipeline creation for non-coders.

(18:12) Backfill feature enables historical data processing.

(22:06) Regular updates keep Uber aligned with Airflow advancements.

(26:07) Plans to leverage Airflow’s latest features.

Resources Mentioned:

Shobhit Shah -

https://www.linkedin.com/in/shahshobhit/

Sumit Maheshwar -

https://www.linkedin.com/in/maheshwarisumit/

Uber -

https://www.linkedin.com/company/uber-com/

Apache Airflow -

https://airflow.apache.org/

Airflow Summit -

https://airflowsummit.org/

Uber -

https://www.uber.com/tw/en/

Apache Airflow Survey -

https://astronomer.typeform.com/airflowsurvey24

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

33 bölüm

#Tech #Airflow