WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE β€’ Dec 7, 2021

Fully Orchestrating Databricks from Airflow

Alan Mazankiewicz

Native Databricks scheduling often falls short for complex enterprise pipelines. Discover how integrating Apache Airflow unlocks dynamic, code-driven DAGs for highly scalable data orchestration.

Pause
Mute Enter Fullscreen
#1 about 1 min

Introduction to the speaker and IT solutions company

A machine learning engineer introduces their background and the IT solutions company Ovex.

#2 about 3 min

Overview of Databricks and interactive data processing

Managed Spark clusters enable interactive data analysis using Jupyter notebooks and parallel processing.

#3 about 3 min

Scheduling workflows with Databricks jobs and tasks

Multi-task job orchestration allows defining dependencies between scheduled notebooks and packages within a workspace.

#4 about 3 min

Introduction to Apache Airflow for advanced orchestration

The Airflow UI provides visibility into task schedules, success rates, and log outputs for complex dependencies.

#5 about 3 min

Writing dynamic Airflow DAGs with Python code

Python operators enable dynamic task generation and conditional branching based on external variables or execution dates.

#6 about 6 min

Executing Databricks jobs with built-in Airflow operators

The Databricks Run Now and Submit Run operators trigger predefined or dynamically configured Spark jobs via API calls.

#7 about 9 min

Building generic custom operators for Databricks APIs

Subclassing the base Airflow operator allows interacting with unsupported Databricks endpoints like file systems and custom clusters.

#8 about 2 min

Deploying and running Airflow in cloud environments

While local testing is possible, production Airflow typically runs on Kubernetes or managed services like Cloud Composer.

#9 about 3 min

Optimizing Spark performance for large data volumes

Analyzing the Spark UI helps identify bottlenecks and optimize processing times independently of the Airflow scheduler.

#10 about 1 min

Comparing Airflow orchestration with Azure Data Factory

Data Factory offers a UI-driven alternative for scheduling Spark workloads compared to code-first Airflow deployments.

#11 about 2 min

Selecting the ideal data stack for distinct workloads

Exploratory work benefits from interactive notebooks and Pandas, while production data products rely on Airflow and Databricks.

#12 about 2 min

Managing worker nodes and compute quotas in Databricks

Cluster sizes are primarily limited by cloud provider quotas rather than inherent Databricks constraints.

#13 about 3 min

Enhancing Databricks tooling for software engineering workflows

Future platform improvements should bridge the gap between data science notebook environments and standard application deployment practices.

#14 about 3 min

Learning Python and navigating a practical programming career

A logical approach to problem-solving helps beginners transition from theoretical statistics to practical software development.

#15 about 3 min

Daily routines and software engineering team collaboration

Effective engineering roles combine focused coding time with continuous communication and code reviews across teams.

Matching moments

1:32 min

Handling untyped ingestion and comparing dbt against Spark

Matthias Niehoff Matthias Niehoff Β· World Congress 2023

1:43 min

AWS infrastructure stack and data flow pipeline overview

Artem Volk Artem Volk +1 Β· World Congress 2024

2:15 min

Empowering domain teams with an open data platform

Sandhya Menon Sandhya Menon Β· World Congress 2026 Europe

5:19 min

Executing queries and scheduling pipeline jobs within DataWorks

Qiyang Duan Β· LIVE

3:33 min

Refactoring data science workflows using Rapids QDF and Pandas

Paul Graham Paul Graham Β· LIVE

2:38 min

Solving complex pipeline orchestration with Argo Workflows

Hauke Brammer Β· World Congress 2023

More from WeAreDevelopers LIVE

Fake or News - Claude's DNA Discovery, Quick Cloning & Crime on Insta thumbnail
6:07
1 day ago

Fake or News - Claude's DNA Discovery, Quick Cloning & Crime on Insta

Andrew MacLean, Daniel Cranney, Christian Heilmann

WeAreDevelopers LIVE - Playing in Production thumbnail
1:06:40
1 day ago

WeAreDevelopers LIVE - Playing in Production

Andrew MacLean, Daniel Cranney, Christian Heilmann

WeAreDevelopers LIVE - Is CSS Catching Up to Flash? thumbnail
1:03:33
15 days ago

WeAreDevelopers LIVE - Is CSS Catching Up to Flash?

Chris Heilmann, Daniel Cranney, Bramus Van Damme

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along thumbnail
1:21:17
22 days ago

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along

Chris Heilmann, Daniel Cranney, Krzysztof CieΕ›lak

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak thumbnail
4:23
24 days ago

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Open Source Agent Toolkits thumbnail
1:00:38
29 days ago

WeAreDevelopers LIVE - Open Source Agent Toolkits

Chris Heilmann, Daniel Cranney, Eddie Jaoude

Related videos

Enjoying SQL data pipelines with dbt thumbnail
26:53
World Congress 2023 β€’ almost 3 years ago

Enjoying SQL data pipelines with dbt

Matthias Niehoff

PySpark - Combining Machine Learning & Big Data thumbnail
43:26
LIVE β€’ almost 6 years ago

PySpark - Combining Machine Learning & Big Data

Ayon Roy

Alibaba Big Data and Machine Learning Technology thumbnail
43:57
LIVE β€’ almost 6 years ago

Alibaba Big Data and Machine Learning Technology

Qiyang Duan

Python-Based Data Streaming Pipelines Within Minutes thumbnail
39:04
LIVE β€’ almost 2 years ago

Python-Based Data Streaming Pipelines Within Minutes

Bobur Umurzokov

From Messy Queries to Scalable Systems - How Data Engineering actually works thumbnail
31:15
World Congress 2026 Europe β€’ 3 months ago

From Messy Queries to Scalable Systems - How Data Engineering actually works

Sandhya Menon

OLTP in the Lakehouse: Redefining Data for AI Workloads thumbnail
24:36
Europe 2026 Virtual β€’ 3 months ago

OLTP in the Lakehouse: Redefining Data for AI Workloads

Oleksandra Bovkun

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.