Databricks Engineer

This individual will be responsible for building scalable, secure, and high-performing data environments from initial architecture and configuration through production deployment. They will design Databricks Lakehouse solutions, develop data pipelines, implement data governance, and help migrate legacy data systems into modern cloud platforms., * Architect and deploy enterprise Databricks Lakehouse platforms from the ground up.

• Lead environment configuration, data integration, migration, testing, and production deployment.
• Design data pipelines using Python, PySpark, SQL, Apache Spark, and Delta Lake.
• Implement Bronze, Silver, and Gold data layers.
• Establish governance and security using Unity Catalog.
• Modernize legacy data warehouses and ETL platforms.
• Optimize compute performance, scalability, reliability, and cloud costs.
• Implement CI/CD pipelines and infrastructure automation.
• Develop architecture diagrams, standards, and technical roadmaps.
• Provide hands-on technical leadership to engineering, DevOps, analytics, and BI teams.

The ideal candidate combines strategic architecture experience with strong hands-on engineering skills. They must be comfortable making technical decisions, writing code, configuring environments, troubleshooting complex issues, and leading engineering teams.

The biggest differentiator: Our client needs someone who has personally led a complete Databricks implementation-not just someone who has worked within an established Databricks environment.

Top 4 Required Skills

1. Databricks Architecture & End-to-End Implementation

3+ years of hands-on Databricks experience, including at least one enterprise implementation from initial architecture and environment setup through production.

1. Python, PySpark, Apache Spark & SQL

Advanced data engineering skills, including building and optimizing distributed data pipelines and ETL/ELT solutions.

1. Cloud Data Architecture & Lakehouse Design

Strong Azure, AWS, or GCP experience with Delta Lake, medallion architecture, cloud storage, and enterprise platform modernization.

1. Unity Catalog, Governance & Technical Leadership

Experience implementing governance, security, access controls, data quality, architectural standards, and guiding engineering teams., Qualification

Requirement

Overall experience

8+ years in data architecture, engineering, or warehousing

Databricks

3+ years hands-on

Implementation

At least one complete Databricks implementation

Programming

Advanced Python, PySpark, and SQL

Cloud

Azure, AWS, or GCP

Lakehouse

Delta Lake, Unity Catalog, Spark

Data engineering

ETL/ELT, integration, modeling, optimization

Modernization

Legacy-to-cloud migration experience

Leadership

Architecture ownership, technical mentoring, stakeholder communication

Preferred: Databricks certifications, Terraform, CI/CD, Azure Data Factory, AWS Glue, dbt, Airflow, Power BI, Tableau, streaming, and AI/ML workloads., * Databricks: 3 years (Required)

• Python: 3 years (Required)
• Delta Lake: 2 years (Required)
• Unity Catalog: 2 years (Required)

The successful candidate delivers a production-ready Databricks platform, establishes secure and reusable architecture patterns, improves data processing performance, and enables engineering teams to build reliable analytics solutions independently.

Pay: $100.00 - $130.00 per hour

Expected hours: 40.0 per week

Benefits:

• 401(k) matching
• Health insurance
• Vision insurance