> Markdown version of [/jobs/ext/1932647-data-engineer-lead](https://www.wearedevelopers.com/jobs/ext/1932647-data-engineer-lead). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Data Engineer Lead

- **Company:** Horizontal Talent
- **Location:** Chicago, IL, United States
- **Experience:** Expert
- **Salary:** $106,080.0 - $185,120.0
- **Contract:** Permanent contract
- **Skills:** Amazon Web Services, Cloud Database, Profiling, Data Discovery, Information Engineering, Data Infrastructure, Data Integration, Extract Transform Load (ETL), Data Structures, Data Systems, Document-Oriented Databases, Python (Programming Language), NoSQL, Performance Tuning, SQL Databases, Enterprise Data Management, Cloud Platform System, Azure Data Factory, Data Lakes, Pyspark, AWS Glue, Data Management, Data Pipelines, Databricks
- **Published:** August 5, 2026
- **Apply:** https://www.dice.com/job-detail/271d9dd5-abb3-4564-ad91-51261a77b14f

## About the Role

* 5+ years of experience in data engineering, data platforms, or a related field.
 * Strong background in data discovery, profiling, mapping, and engineering handoff work.
 * Hands-on experience with Python and PySpark.
 * Strong Databricks experience.
 * Experience with SQL and NoSQL databases.
 * Experience working in enterprise data environments.
 * Knowledge of ETL/ELT processes, data integration, and data modeling.
 * Hands-on experience with AWS or another major cloud platform.
 * Strong analytical, problem-solving, and communication skills.

Preferred Skills

 * Experience with cloud data tools such as AWS Glue or Azure Data Factory.
 * Familiarity with modern data architecture principles, including data lakes or data mesh.
 * Exposure to governance, security, and compliance practices in data environments.
 * Experience with performance optimization and scalable data solutions.
 * Certifications in cloud data technologies are a plus.
 * Ability to work collaboratively in a hybrid, fast-moving environment.

## Description

* Inventory priority source systems, data entities, identifiers, fields, relationships, and known data quality concerns.
 * Review source documentation, sample data structures, field inventories, and SME input to understand the current data landscape.
 * Assess data completeness, duplicates, freshness, conflicts, availability, lineage, and source reliability.
 * Support source-to-target mapping for a canonical marketing data model.
 * Translate identity rules, survivorship logic, source hierarchy, field ownership, account matching, and exception handling into engineering-ready inputs.
 * Document data gaps, dependencies, assumptions, risks, and open questions for the internal engineering team.
 * Help define acceptance criteria, data quality thresholds, and match-confidence considerations for the final handoff package.
 * Partner with strategy and architecture stakeholders to ensure outputs are actionable and ready for Databricks implementation.


## Related Videos

- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works)
- [How we built an AI-powered code reviewer in 80 hours](https://www.wearedevelopers.com/videos/1511-how-we-built-an-ai-powered-code-reviewer-in-80-hours)
- [Leveraging Real time data in FSIs](https://www.wearedevelopers.com/videos/806-leveraging-real-time-data-in-fsis)
- [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail)
- [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning)

## Related Articles

- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)
- [The Most Popular IT Jobs on the Market](https://www.wearedevelopers.com/magazine/376-the-most-popular-it-jobs-on-the-market)
- [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk)