> Markdown version of [/jobs/ext/3727774-lead-data-engineer-azure-databricks](https://www.wearedevelopers.com/jobs/ext/3727774-lead-data-engineer-azure-databricks). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Lead Data Engineer - Azure Databricks

- **Company:** Conch Technologies
- **Location:** New York, NY, United States
- **Experience:** Expert
- **Contract:** Permanent contract
- **Skills:** Amazon Web Services, Data Analysis, Microsoft Azure, Big Data, Health Informatics, Cloud Database, Cloud Engineering, Computer Programming, Information Engineering, Data Governance, Data Integration, Extract Transform Load (ETL), Data Security, Data Systems, Data Warehousing, Relational Databases, Database Development, Dimensional Modeling, Apache Hadoop, Python (Programming Language), Machine Learning, Performance Tuning, Power BI, Cloud Services, SQL Databases, Tableau (Software), Unstructured Data, Data Processing, Google Cloud, Azure Data Factory, Snowflake, Apache Spark, Information Technology, Tools for Reporting, Azure Synapse Analytics, Data Pipelines, Databricks
- **Published:** October 14, 2026
- **Apply:** https://www.careerjet.com/jobad/us4417510b0bef7de4d3989a4bd4fbcf88

## About the Role

* 10+ years of overall IT experience.
 * 5+ years of experience in cloud leadership roles.
 * 5+ years of experience in Data Engineering or related data development roles.
 * Strong experience with SQL and relational databases.
 * Hands-on experience with ETL/ELT development.
 * Experience with cloud platforms such as Azure, AWS, or Google Cloud.
 * Strong programming skills in Python, Spark, or similar technologies.
 * Experience developing and optimizing large-scale data pipelines.
 * Knowledge of data warehousing, dimensional modeling, and data integration.
 * Understanding of data governance, security, and compliance practices.
 * Strong analytical, problem-solving, and communication skills.

Preferred Qualifications

 * Experience with Azure Databricks, Azure Data Factory, Azure Synapse, or Snowflake.
 * Experience with Hadoop or other big data technologies.
 * Healthcare or healthcare payer industry experience.
 * Experience supporting Medicare, Medicaid, or healthcare analytics projects.
 * Familiarity with Power BI, Tableau, or other reporting tools.
 * Exposure to machine learning and advanced analytics environments.
 * Experience working in regulated healthcare environments.

## Description

We are looking for an experienced Lead Data Engineer to support enterprise data engineering, analytics, and cloud initiatives. The ideal candidate will have strong expertise in designing, developing, and optimizing large-scale data pipelines, cloud data platforms, and ETL/ELT processes.
The candidate will work closely with business stakeholders, data analysts, data scientists, and engineering teams to deliver reliable, scalable, and secure data solutions that support reporting, analytics, and business decision-making., * Design, develop, and maintain scalable data pipelines and data processing solutions.

 * Build and optimize ETL/ELT workflows for large-scale data integration.
 * Work with Azure Databricks and other cloud-based data engineering technologies.
 * Develop data solutions using SQL, Python, Spark, and related technologies.
 * Work with structured and unstructured datasets to support analytics and reporting.
 * Design and maintain data warehouses and data models.
 * Ensure data quality, security, governance, and compliance.
 * Collaborate with business and technical teams to understand data requirements.
 * Troubleshoot performance issues and optimize data processing workflows.
 * Follow cloud engineering best practices to deliver reliable and scalable solutions.