> Markdown version of [/jobs/ext/1959444-big-data-engineer-lead](https://www.wearedevelopers.com/jobs/ext/1959444-big-data-engineer-lead). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Big Data Engineer Lead

- **Company:** Intersources Inc.
- **Location:** New York, NY, United States
- **Experience:** Expert
- **Contract:** Permanent contract
- **Skills:** Big Data, Extract Transform Load (ETL), Azure Data Lake, Data Processing, Data Ingestion, Apache Spark, Collibra, Databricks
- **Published:** August 6, 2026
- **Apply:** https://www2.jobdiva.com/portal/?a=62jdnw10t7d77ytz0lbaey9qxonk3b05a9tqw96rb7z6hlfo7xj79l9g6mp6aj2o&compid=0/jobs/25798012#/jobs/25798012

## About the Role

* Should be subject matter expert in building and designing the ETL pipelines in ADF or ADLS Gen2.
 * Should be able to use Spark, Synapse to process data from Azure data lake
 * Experience in downloading data from UDL & ingesting in PDS. Good experience with ADF.
 * Expert in Spark for structured, semi-structured and un-structured data processing
 * Hands-on knowledge in Azure Databricks or Azure Purview, Synapse
 * Should be clear on designing constructs for different data sets and come up with reusable templates for the same
 * Should have experience of implementing Data Quality methodologies to pre-empt errors in patterns and to identify Data Quality issues.

## Description

A Big Data Engineering lead handles the ETL jobs, Optimization, Automation of data processing for complex scenarios., * Data ingestion from Source

 * DQ checks
 * Handling ETL Automation
 * Building features / variable


## Related Videos

- [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology)
- [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works)
- [Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source](https://www.wearedevelopers.com/videos/100151-cutting-llm-costs-without-cutting-quality-how-to-beat-proprietary-llms-with-fine-tuned-open-source)
- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [OLTP in the Lakehouse: Redefining Data for AI Workloads](https://www.wearedevelopers.com/videos/2038-oltp-in-the-lakehouse-redefining-data-for-ai-workloads)
- [Why and when should we consider Stream Processing frameworks in our solutions](https://www.wearedevelopers.com/videos/1085-why-and-when-should-we-consider-stream-processing-frameworks-in-our-solutions)

## Related Articles

- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [What Are The Top Skills Required For Azure Developers?](https://www.wearedevelopers.com/magazine/77-what-are-the-top-skills-required-for-azure-developers)
- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)