Airflow Sucks for MLOps // Stephen Bailey // MLOps Podcast #141

MLOps.community

MLOps Coffee Sessions #141 with Stephen Bailey, Airflow Sucks for MLOps co-hosted by Joe Reis.
// Abstract
Stephen discusses his experience working with data platforms, particularly the challenges of training and sharing knowledge among different stakeholders. This talk highlights the importance of having clear priorities and a sense of practicality and mentions the use of modular job design and data classification to make it easier for end users to understand which data to use.
Stephen also mentions the importance of being able to move quickly and not getting bogged down in the quest for perfection. We recommend Stephen's blog post "Airflow's Problem" for further reading.
// Bio
Stephen has worked as a data scientist, analyst, manager, and engineer, and loves all the domains equally. He currently works at Whatnot, a collectibles marketplace that focuses on live shopping, and has previously worked in privacy tech at Immuta. He has his Ph.D. from Vanderbilt University in educational cognitive neuroscience, but it has yet to help him understand why his three children are so crazy.
// MLOps Jobs board
https://mlops.pallet.xyz/jobs

// MLOps Swag/Merch
https://mlops-community.myshopify.com/
// Related Links
Airflow's Problem blog post: https://stkbailey.substack.com/p/airflows-problem
Airflow's Problem and the reception it got on Hacker News: https://news.ycombinator.com/item?id=32317558
--------------- ✌️Connect With Us ✌️ -------------
Join our slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register
Catch all episodes, blogs, newsletters, and more: https://mlops.community/
Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with Joe on LinkedIn: https://www.linkedin.com/in/josephreis/
Connect with Stephen on LinkedIn: https://www.linkedin.com/in/stkbailey/
Timestamps:
[00:00] Stephen's preferred coffee
[00:19] Introduction to co-host Joe Reis
[01:40] Takeaways
[06:29] Subscribe to our newsletters!
[06:55] Shout out to our sponsor, Wallaroo!
[08:05] Whatnot
[10:47] Stephen's side hustle
[14:35] Stephen's work breakdown at Whatnot
[18:03] Fundamental tensions in the data world
[21:27] Initial questions to answer that you were on the right path
[24:06] Recommender systems
[28:15] Coordinating with ML teams
[29:43] Daxter
[31:38] Too advanced, more challenging
[34:37] Orchestration layer
[36:14] Decision criteria
[39:23] Human design aspect of Daxter
[40:53] Orchestration layer centralization and sharing knowledge with stakeholders
[46:18] Airflow's Problem and the reception it got on Hacker News
[51:00] Wrap up