Data Engineer - Databricks / Scala
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Tech stack
+14 more
Job description
Design and develop scalable Databricks data engineering solutions.
Build and optimize batch and streaming data pipelines.
Implement data ingestion, transformation and Lakehouse solutions.
Troubleshoot production Databricks jobs and DevOps pipelines.
Optimize Spark workloads and ensure data processing performance.
Requirements
8+ years of Data Engineering experience.
Strong hands-on expertise in Azure Databricks, Scala and Apache Spark.
Scala / PySpark development and performance tuning.
Advanced SQL, ETL/ELT and data pipeline development.
Spark DataFrames, Datasets and Notebooks.
Large-scale data processing, streaming and transformations.
Delta Lake, Unity Catalog and Auto Loader.
Lakehouse and Medallion Architecture.
Spark Structured Streaming.
Databricks clusters, sizing, serverless and job orchestration.
Azure, DevOps pipelines and AKS troubleshooting.
Production monitoring, debugging and performance optimization.
Nice-to-Have Skills:
Apache Big Data ecosystem experience.
Java or R programming.
Docker and Kubernetes.
Containerization and cloud-native deployments.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦