> Markdown version of [/jobs/ext/641834-lead-data-engineer](https://www.wearedevelopers.com/jobs/ext/641834-lead-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Lead Data Engineer

- **Company:** SDH Systems LLC
- **Location:** San Jose, CA, United States
- **Experience:** Expert
- **Contract:** Permanent contract
- **Skills:** Artificial Intelligence, Microsoft Azure, Big Data, Cloud Computing, Data Architecture, Information Engineering, Extract Transform Load (ETL), Data Warehousing, Machine Learning, Performance Tuning, SQL Databases, Data Streaming, Azure Service Bus, Feature Engineering, Azure Data Factory, Snowflake, Apache Spark, Data Lakes, Pyspark, Apache Kafka, Azure Synapse Analytics, Stream Analytics, Data Pipelines, Databricks
- **Published:** June 25, 2026
- **Apply:** https://www.dice.com/job-detail/147d987b-8596-4819-9b4e-12539a1a259e

## About the Role

We are seeking a Lead Data Engineer with expertise in Databricks and Data Warehousing to drive data architecture, pipeline development, and optimization efforts. The ideal candidate will play a key role in designing scalable solutions, implementing best practices, and leading data initiatives within a dynamic and collaborative environment., * 8+ years of experience in data engineering, big data processing, and cloud-based solutions.

 * Strong expertise in Databricks, Spark (PySpark/SQL), and Delta Lake architecture.
 * Proven experience in designing and managing data warehouses using Snowflake, Azure Synapse, or equivalent technologies.
 * Deep understanding of data modeling, SQL, and performance optimization.
 * Hands-on experience with Azure Data Factory, Event Hubs, and cloud-based ETL processes.
 * Solid knowledge of real-time streaming technologies (Kafka, Azure Stream Analytics, or similar).
 * Familiarity with ML/AI data pipelines and feature engineering best practices.
 * Strong communication and collaboration skills, with experience working in fast-paced, enterprise environments.

## Description

* Design, build, and optimize scalable data pipelines using Databricks, Apache Spark, and Azure technologies.
 * Architect data warehousing solutions, ensuring seamless integration with cloud platforms and structured/unstructured data sources.
 * Collaborate with business stakeholders to understand data needs and develop high-performance analytical solutions.
 * Implement ETL/ELT processes leveraging cloud-based technologies such as Azure Data Factory, Snowflake, and Delta Lake.
 * Ensure data quality, governance, and security compliance while managing large datasets efficiently.
 * Drive performance tuning and optimization for data pipelines, ensuring efficiency across systems.
 * Work closely with cross-functional teams to support machine learning and advanced analytics initiatives.
 * Provide technical leadership and mentorship to junior data engineers, fostering a culture of innovation and continuous improvement.
 * Stay updated on emerging data technologies and recommend strategies to enhance existing architectures.


## Related Videos

- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology)
- [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works)
- [How Cisco embraced a DevOps culture within its network engineering team](https://www.wearedevelopers.com/videos/99-how-cisco-embraced-a-devops-culture-within-its-network-engineering-team)
- [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning)
- [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail)

## Related Articles

- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)
- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [What Are The Top Skills Required For Azure Developers?](https://www.wearedevelopers.com/magazine/77-what-are-the-top-skills-required-for-azure-developers)