Urgent Hiring for the Senior Databricks Engineer || 100 % Remote Role
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Tech stack
+11 more
Job description
Seeking an experienced Senior Databricks Engineer to support a greenfield Databricks implementation. The role involves rebuilding and optimizing existing AWS data pipelines in Databricks, developing scalable batch and streaming workflows, and ensuring data quality, governance, performance, and reliability. Key Responsibilities Design, develop, and deploy scalable Databricks data pipelines and transformation workflows. Reverse engineer existing AWS Glue, S3, Redshift, and related data processing solutions. Build and maintain Bronze, Silver, and Gold layers using Medallion Architecture. Develop ingestion and streaming pipelines using Auto Loader, Lakeflow Declarative Pipelines, Structured Streaming, PySpark, Spark SQL, and Delta Lake. Implement Unity Catalog governance, access controls, data lineage, and security. Optimize Spark workloads, Delta Lake tables, queries, and pipeline performance. Develop Databricks Workflows, CI/CD pipelines, and automated deployment solutions using Git, Databricks Asset Bundles, Terraform, or similar tools. Implement data quality checks, error handling, monitoring, testing, and pipeline recovery. Validate data parity between legacy AWS systems and new Databricks pipelines. Troubleshoot production issues, document solutions, conduct code reviews, and mentor client engineers.
Requirements
6+ years of data or software engineering experience. 3+ years of hands-on Databricks experience in production environments. Strong expertise in Databricks Auto Loader, Lakeflow Declarative Pipelines, Unity Catalog, Delta Lake, Apache Spark, PySpark, and Spark SQL. Experience with Medallion Architecture, batch and streaming processing, and high-volume data pipelines. Strong knowledge of Spark performance tuning, data quality, governance, monitoring, and production troubleshooting. Experience with AWS services, including S3, Glue, and Redshift. Hands-on experience with Databricks Workflows, CI/CD, Git, automated deployments, and infrastructure-as-code tools. Strong analytical, problem-solving, communication, and documentation skills.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦