Azure Databricks Data Engineer

We are seeking a highly skilled Senior Azure Databricks Data Engineer to design, develop, and optimize enterprise-scale data platforms and pipelines. The ideal candidate will have extensive experience with Azure Databricks, PySpark, Delta Lake, Lakehouse Architecture, and modern data engineering practices. This role involves building scalable batch and real-time data solutions, implementing governance frameworks, and supporting business-critical analytics and reporting initiatives., * Design, develop, and maintain scalable, high-performance data engineering solutions on Azure Databricks.

• Build and support batch and real-time data pipelines using Apache Spark and Structured Streaming.
• Implement Lakehouse Architecture and Medallion Data Models (Bronze, Silver, Gold) using Delta Lake.
• Develop reusable ETL/ELT frameworks, data ingestion accelerators, and standardized processing components.
• Integrate and process data from multiple enterprise sources including databases, APIs, Kafka, Event Hubs, and file systems.
• Develop complex data transformation and enrichment logic using PySpark, Python, and SQL.
• Create trusted, governed, and analytics-ready datasets for BI, reporting, and downstream applications.
• Implement data quality, reconciliation, auditing, and governance controls throughout the data lifecycle.
• Manage and optimize Databricks Workflows, Delta Live Tables (DLT), Unity Catalog, and Databricks SQL.
• Collaborate with architects, business stakeholders, analysts, and engineering teams within Agile delivery environments.
• Monitor and optimize performance, scalability, and reliability of data platforms.

Required Skills Databricks & Big Data

• 5 years of experience in Data Engineering.
• Strong hands-on experience with Azure Databricks, Delta Lake, and Lakehouse Architecture.
• Expertise in PySpark, Python, and Advanced SQL.
• Experience with Apache Spark, Structured Streaming, Delta Live Tables (DLT), Unity Catalog, and Databricks Workflows.
• Strong experience designing Data Lakes, Data Warehouses, and Medallion Architecture.
• Experience building ETL/ELT frameworks and large-scale data processing solutions.
• Knowledge of streaming technologies such as Apache Kafka and Azure Event Hub.
• Strong understanding of data governance, lineage, security, and data quality frameworks.

Cloud Platforms

• Hands-on experience with Client Azure services.
• Exposure to AWS or GCP is a plus.

Application Integration

• Working knowledge of React and/or Angular for data-driven application integration.

Preferred Domain Experience

• Finance, Banking, Insurance, Regulatory Reporting, or US GAAP Reporting.
• Experience working with financial data models, compliance reporting, and governance frameworks.

Preferred Qualifications

• Bachelor’s or Master’s degree in Computer Science, Information Systems, Engineering, or related field.
• Azure Data Engineer Associate certification is preferred.
• Databricks certifications are highly desirable.