> Markdown version of [/jobs/ext/224952-lead-data-engineer](https://www.wearedevelopers.com/jobs/ext/224952-lead-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Lead Data Engineer

- **Company:** Insight Global
- **Location:** San Francisco, CA, United States
- **Experience:** Expert
- **Contract:** Temporary contract
- **Skills:** Application Programming Interfaces (APIs), Amazon Web Services, Business Analytics Applications, Microsoft Azure, Cloud Database, Computer Programming, Data Validation, Information Engineering, Extract Transform Load (ETL), Data Warehousing, IBM InfoSphere DataStage, Apache Hive, IBM Cognos Business Intelligence, Python (Programming Language), NoSQL, Raw Data, Software Engineering, SQL Databases, SQL Server Analysis Services, Data Streaming, Tableau (Software), Sql Optimization, Apache Spark, Apache Kafka, Data Pipelines, Databricks
- **Published:** May 28, 2026
- **Apply:** https://www.juju.com/job/00000000g3a0yi

## About the Role

Must Haves:

 * At least 5 years of strong hands-on expertise with Databricks and Datastage (or comparable ETL tools)

 * Experience with migrations

 * Exs: Migrating SQL to cloud, SQL to Azure Databricks, (on prem to the cloud)

 * 8-10 years of software development and data engineering experience with SQL, ADF, SSAS cubes, Cognos, Tableau, ThoughtSpot and other BI tools

 * Advanced SQL skills for processing raw data, Kafka ingestions, ADF pipelines, data validation and QA with experience across SQL and NoSQL databases

 * Programming experience with Python and/or Scala

 * Experience ingesting data via APIs and streaming sources

 * Experience building data pipelines, data warehouses, and analytics platforms

 * Proficiency with Spark, Hive, Kafka, and cloud data services

 * Experience with Azure, AWS or equivalent cloud platforms



## Related Videos

- [Data Governance in the Era of AI](https://www.wearedevelopers.com/videos/1622-data-governance-in-the-era-of-ai)
- [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt)
- [Leveraging Real time data in FSIs](https://www.wearedevelopers.com/videos/806-leveraging-real-time-data-in-fsis)
- [Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source](https://www.wearedevelopers.com/videos/100151-cutting-llm-costs-without-cutting-quality-how-to-beat-proprietary-llms-with-fine-tuned-open-source)
- [OLTP in the Lakehouse: Redefining Data for AI Workloads](https://www.wearedevelopers.com/videos/2038-oltp-in-the-lakehouse-redefining-data-for-ai-workloads)
- [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works)

## Related Articles

- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)
- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk)