> Markdown version of [/jobs/ext/1972249-lead-data-engineer](https://www.wearedevelopers.com/jobs/ext/1972249-lead-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Lead Data Engineer

- **Company:** Jobot
- **Location:** Chicago, IL, United States
- **Experience:** Expert
- **Salary:** $180,000.0 - $250,000.0
- **Contract:** Permanent contract
- **Skills:** Java (Programming Language), Agile Methodology, Amazon Web Services, Big Data, Cloud Engineering, Continuous Integration, Data Validation, Information Engineering, Data Files, Python (Programming Language), DataOps, Software Construction, Software Engineering, Data Streaming, System Availability, Apache Spark, HybridCloud, Build Management, Data Lakes, Kubernetes, Information Technology, Apache Kafka, Data Management, Presto, Databricks
- **Published:** August 7, 2026
- **Apply:** https://www.dice.com/job-detail/a14bf5ee-075c-41f9-bb5f-223947e965f0

## About the Role

Lead Data Engineer | 5+ Years Experience | Scalable Batch & Streaming Pipelines | Cloud Architecture (AWS, Databricks, Kubernetes), Service Level Agreements (SLAs) pertaining to data quality, data availability, and data correctness. Challenge the status quo and help push our organization forward as we grow beyond the limits of our current tech stack. Qualities that make great candidates: BS/MS/PhD in Computer Science, Engineering, Physics, Math, or equivalent technical field.5+ years of professional experience developing software applications. Java/Scala experience required; Python experience is a significant plus. Highly motivated and willing to take ownership of high-impact projects upon arrival. Prior hands-on experience with data platforms and technologies such as Delta Lake, Spark, Kubernetes, Kafka, Click House, and/or Presto/Trino. Experience building large-scale batch and streaming pipelines with strict SLA and data quality requirements. Must possess excellent communication, analytical, and problem-solving skills. Demonstrated experience working with diverse data sets and frameworks across multiple
domains-financial data experience is not required, but a strong plus. Recent hands-on experience with AWS Cloud development, deployment, and monitoring necessary. Demonstrated experience working on an Agile team employing software engineering best practices, such as GitOps and CI/CD, to deliver complex software projects. The ability to react quickly and accurately to rapidly changing market conditions, including the ability to quickly and accurately respond and/or solve math and coding problems, are essential functions of the role.

## Description

In this role, you will: Work within a growing Data Engineering division supporting the strategic role of data at the firm. Drive the ongoing design and expansion of our data platform across a wide variety of data sources, supporting an array of streaming, operational, and research workflows. Work closely with Trading, Quant, Technology, and Business Operations teams throughout the firm to identify how data is produced and consumed, helping to define and deliver high-impact projects. Build and deploy batch and streaming pipelines to collect and transform our rapidly growing Big Data set within our hybrid cloud architecture utilizing Kubernetes/EKS, Kafka/MSK, and Databricks/Spark. Mentor junior engineers in software and data engineering best practices. Produce clean, well-tested, and documented code with a clear design to support mission-critical applications. Build automated data validation test suites that ensure that data is processed and published in accordance with well-defined


## Related Videos

- [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology)
- [Modern Data Architectures need Software Engineering](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering)
- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [Why and when should we consider Stream Processing frameworks in our solutions](https://www.wearedevelopers.com/videos/1085-why-and-when-should-we-consider-stream-processing-frameworks-in-our-solutions)
- [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works)
- [Implementing continuous delivery in a data processing pipeline](https://www.wearedevelopers.com/videos/73-implementing-continuous-delivery-in-a-data-processing-pipeline)

## Related Articles

- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)
- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Top-Paying Tech Jobs (with Salaries)](https://www.wearedevelopers.com/magazine/372-top-paying-tech-jobs-with-salaries)