Get the slides: https://www.datacouncil.ai/talks/argo-kubernetes-native-workflows-and-pipelines?hsLang=en

ABOUT THE TALK

Data orchestration and DAGs are something that most data teams need. There are many commercial and open source options available. Examples include Airflow, Luigi, Oozie and many others.

Airflow is very popular at the moment and rightly so; it is a very useful tool and is the backbone of very productive data teams. Argo is a relatively new challenger. It is a Kubernetes native workflow engine.

At Canva, we evaluated both Airflow and Argo and chose Argo as our primary data orchestration system. In this talk I’ll briefly compare Airflow and Argo, talk about the evaluation process we undertook and how we came to our decision. Finally, I’ll talk about our experience using it so far, the things that have been good and the things that have been not so good.

ABOUT THE SPEAKER

Greg is the Data Engineering Lead at Canva. He founded the team and quickly grew it to build out scalable systems that enable advanced analytics and machine learning features inside the Canva product.

Previously he was the Co-founder and CTO of AirHelp, a Y Combinator backed startup, where he built systems to process flight and booking data in real-time.

Greg is passionate about technology and is frequently involved with meetups, such as the Sydney Data Engineering meetup.

ABOUT DATA COUNCIL:
Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups.

FOLLOW DATA COUNCIL:
Twitter: https://twitter.com/DataCouncilAI
LinkedIn: https://www.linkedin.com/company/datacouncil-ai
Facebook: https://www.facebook.com/datacouncilai
Eventbrite: https://www.eventbrite.com/o/data-council-30357384520