> Markdown version of [/jobs/ext/3552161-lead-data-engineer-databricks-migration](https://www.wearedevelopers.com/jobs/ext/3552161-lead-data-engineer-databricks-migration). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Lead Data Engineer- Databricks Migration

- **Company:** Insight Global
- **Location:** Cleveland, OH, United States
- **Experience:** Expert
- **Contract:** Permanent contract
- **Skills:** Agile Methodology, Amazon Web Services, Automation of Tests, Microsoft Azure, Big Data, Cloud Computing, Code Review, Information Systems, Databases, Continuous Integration, Information Engineering, Data Governance, Data Integration, Extract Transform Load (ETL), Data Warehousing, Distributed Systems, Power BI, Software Deployment, Software Engineering, Software Organization, Azure Data Factory, Apache Spark, Data Lakes, Pyspark, Information Technology, Cloud Migration, Software Version Control, Data Pipelines, Databricks
- **Published:** October 2, 2026
- **Apply:** https://dejobs.org/x/x/F125E050C24846EC820FE940ABFEC9D6/job/

## About the Role

8+ years of software engineering experience with progressive technical leadership responsibilities. 3+ years of hands-on experience with Databricks. Proven experience leading large-scale data, application, or cloud migration projects. Strong experience developing solutions using Apache Spark (PySpark or Scala Spark). Experience building and optimizing ETL/ELT pipelines. Strong SQL development and database experience. Experience working with cloud platforms such as Azure, AWS, or GCP. Expertise with modern software development practices including CI/CD, source control, automated testing, and code reviews. Experience designing scalable, high-performance distributed systems. Strong stakeholder communication and leadership skills. Bachelor's degree in Computer Science, Engineering, Information Systems, or related field. Azure Databricks experience in a large enterprise environment. Experience migrating from traditional data warehouse platforms to Databricks. Experience with Delta Lake, Unity
Catalog, and Medallion architecture. Experience with Azure Data Factory, Synapse, or similar data integration tools. Knowledge of Power BI and enterprise analytics platforms. Experience leading Agile development teams. Exposure to data governance, security, and regulatory compliance frameworks. Databricks, Azure, or cloud certifications.

## Description

We are seeking a Lead Software Engineer to drive large-scale data platform modernization and migration initiatives. This individual will provide technical leadership for the migration of legacy data and application workloads to a modern cloud-based architecture built on Databricks. The ideal candidate will have deep expertise in data engineering, cloud technologies, platform migrations, and software development best practices. This role will partner with Product Owners, Architects, Data Engineers, and business stakeholders to design scalable solutions, establish engineering standards, and lead a team through the full migration lifecycle from assessment and planning through implementation and production deployment


## Related Videos

- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology)
- [Beyond Dashboards: Fixing Text-to-SQL with Semantic RAG](https://www.wearedevelopers.com/videos/2036-beyond-dashboards-fixing-text-to-sql-with-semantic-rag)
- [Fully Orchestrating Databricks from Airflow](https://www.wearedevelopers.com/videos/336-fully-orchestrating-databricks-from-airflow)
- [Data Analytics with Microsoft Fabric: End-to-End Use Case with Data Agents](https://www.wearedevelopers.com/videos/1547-data-analytics-with-microsoft-fabric-end-to-end-use-case-with-data-agents)
- [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail)

## Related Articles

- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [What Are The Top Skills Required For Azure Developers?](https://www.wearedevelopers.com/magazine/77-what-are-the-top-skills-required-for-azure-developers)
- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)