Azure DataBricks Engineer

We are seeking a highly skilled Senior Azure Databricks Engineer to join our Data & Analytics team in Atlanta. The ideal candidate will have extensive experience designing, developing, and optimizing enterprise-scale data solutions using Azure Databricks, Apache Spark, and Microsoft Azure services.

This role will be responsible for building scalable data pipelines, implementing Lakehouse architectures, enabling real-time and batch data processing, and driving best practices for performance, governance, and security., * Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.

• Build and support ETL/ELT frameworks for large-scale structured and unstructured data.
• Develop batch and real-time data processing solutions.
• Implement and maintain Delta Lake and Medallion Architecture (Bronze, Silver, Gold).
• Integrate data solutions with Azure Data Factory, ADLS Gen2, Azure Synapse, Event Hub, and other Azure services.
• Optimize Spark workloads for performance, reliability, and cost efficiency.
• Implement data quality checks, monitoring, and alerting processes.
• Collaborate with business stakeholders, architects, analysts, and data scientists to deliver data solutions.
• Establish and enforce data governance, security, and compliance standards.
• Support CI/CD pipelines and automated deployment processes.
• Troubleshoot and resolve production issues related to data pipelines and Databricks workloads.
• Document technical solutions, architectural decisions, and operational procedures., * Azure Databricks
• Apache Spark
• PySpark
• Python
• SQL
• Azure Data Factory
• ADLS Gen2
• Delta Lake
• Git

• 6+ years of experience in Data Engineering.
• 3+ years of hands-on experience with Azure Databricks.

• Strong experience with: *

• Azure Databricks
• Apache Spark
• PySpark
• Python
• SQL
• Delta Lake
• Experience with Azure Data Factory (ADF).
• Experience with Azure Data Lake Storage Gen2 (ADLS Gen2).
• Strong understanding of Data Warehousing, Data Lakes, and Lakehouse architectures.
• Experience working with large-scale datasets and distributed computing environments.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication and collaboration abilities., * Experience with Azure Synapse Analytics.
• Experience working with Event Hub or Kafka.
• Knowledge of Unity Catalog and data governance frameworks.
• Experience with Delta Live Tables (DLT).
• Experience with Terraform, ARM Templates, or Infrastructure as Code (IaC).
• Experience with Azure DevOps or GitHub Actions.
• Knowledge of Power BI and modern analytics platforms.

• Cloud certifications such as: *

• Microsoft Azure Data Engineer Associate (DP-203)
• Databricks Certified Data Engineer Associate/Professional, The ideal candidate has hands-on experience building modern cloud-based data platforms and can demonstrate expertise in:
• Lakehouse Architecture
• Spark Performance Tuning
• Enterprise Data Engineering
• Data Pipeline Development
• Real-Time Data Processing
• Data Governance and Security
• CDC (Change Data Capture) Implementations
• Production Support and Troubleshooting
• Azure Cloud Data Services
• Cost Optimization and Scalability

• Collaborate with experienced data and cloud professionals.
• Exposure to cutting-edge Azure and Databricks technologies.
• Opportunity to drive impactful business solutions through data.
• Long-term growth and career advancement opportunities.