> Markdown version of [/jobs/ext/2696099-lead-data-engineer](https://www.wearedevelopers.com/jobs/ext/2696099-lead-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Lead Data Engineer

- **Company:** Sovereign Technologies
- **Location:** Eagan, MN, United States
- **Experience:** Expert
- **Contract:** Temporary contract
- **Skills:** Agile Methodology, Artificial Intelligence, Airflow, Amazon Web Services, Amazon Elastic Compute Cloud, Amazon S3, Big Data, Information Engineering, Extract Transform Load (ETL), Database Queries, Distributed Computing Environment, Distributed Systems, Healthcare Effectiveness Data and Information Set, Python (Programming Language), Scrum Methodology, Standard Sql, SAS (Software), Scripting, Freeform SQL, Generative AI, Git, Pyspark, Software Version Control, Data Pipelines, Databricks
- **Published:** September 3, 2026
- **Apply:** https://www.dice.com/job-detail/070b7295-5e6b-4bba-bcde-e810f0634d4a

## About the Role

* 8+ years of Data Engineering experience.
 * Strong Python programming and scripting skills.
 * Hands-on PySpark development.
 * Extensive Databricks experience (Notebooks, Workflows, Cluster Management).
 * Experience processing large datasets in distributed environments.
 * Strong SQL skills.
 * AWS experience with:
 + S3
 + Glue
 + Lambda
 + EC2
 * Experience building and automating ETL/ELT pipelines.
 * Git/version control experience.
 * Agile/Scrum experience.

Preferred Skills

 * Experience with SAS-to-Python migration.
 * AI/Automation experience.
 * Healthcare or HEDIS domain experience.
 * Lead/Principal Data Engineering experience.

Must-Have Skills

 * Python
 * PySpark
 * Databricks
 * SQL
 * AWS (S3, Glue, Lambda, EC2)
 * ETL/ELT Pipeline Development
 * Distributed Data Processing
 * Git
 * Agile/Scrum

Nice-to-Have Skills

 * SAS Modernization
 * AI/Generative AI
 * Healthcare/HEDIS
 * Airflow or other workflow orchestration tools
 * Leadership/Pipeline Ownership experience

## Description

We are seeking a Senior/Lead Data Engineer to modernize and own enterprise data pipelines by migrating legacy SAS-based workflows to Python/PySpark on Databricks. The engineer will partner with SMEs during the transition period and eventually take ownership of a critical healthcare analytics pipeline supporting HEDIS reporting., * Migrate legacy SAS pipelines to Python/PySpark on Databricks.

 * Design, build, and maintain scalable ETL/ELT pipelines.
 * Develop distributed data processing solutions using Databricks.
 * Create and optimize complex SQL queries.
 * Schedule, automate, and monitor data pipelines.
 * Work with AWS services including S3, Lambda, Glue, and EC2.
 * Manage Databricks notebooks, workflows, and clusters.
 * Collaborate with business stakeholders and SMEs to transition pipeline ownership.
 * Follow Agile methodologies and Git-based development practices.


## Related Videos

- [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works)
- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail)
- [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94)
- [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning)
- [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt)

## Related Articles

- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)
- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk)
- [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again)