> Markdown version of [/jobs/ext/2071362-lead-data-engineer-pierre-sd-hybrid](https://www.wearedevelopers.com/jobs/ext/2071362-lead-data-engineer-pierre-sd-hybrid). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Lead Data Engineer - Pierre, SD (Hybrid) - **Company:** MY3TECH INC - **Location:** Pierre, SD, United States - **Experience:** Expert - **Contract:** Temporary to permanent - **Skills:** Microsoft Excel, Application Programming Interfaces (APIs), Microsoft Azure, Continuous Integration, Data Validation, Data Control, Information Engineering, Extract Transform Load (ETL), Python (Programming Language), Metadata, SQL Azure, Cloud Services, Standard Sql, SAP Sales and Distribution, Software Deployment, Web Services, Parquet, Data Logging, Data Processing, Data Server Interface, Azure Data Factory, Fast Healthcare Interoperability Resources, Git, Data Layers, Build Management, Data Lakes, Health Level Seven International, Data Pipelines, Databricks - **Published:** August 15, 2026 - **Apply:** https://www.careerjet.com/jobad/us7bbf3b58de6ad1420cd2f675669261e4 ## About the Role * 7+ years of data engineering/ETL experience, with 3+ years on Azure or comparable cloud data platforms. * Strong SQL plus Python or equivalent data-engineering language. * Hands-on experience with ETL/ELT pipelines, APIs, file ingestion, data validation and reconciliation. * Experience with data-lake/lakehouse or layered/medallion architectures. * Ability to build production-grade error handling, logging, retry, quality and refresh processes. * Experience with CI/CD, Git and controlled deployments. * Ability to work with health/public-sector data and sensitive-data controls., * Azure Data Factory, Synapse, Fabric, Databricks, Azure SQL, Data Lake Storage or equivalent experience. * Healthcare/public-health/Medicaid data engineering. * Experience with metadata, lineage and data-quality tooling. * Experience with HL7/FHIR or other health-data interfaces is advantageous but not required. * Azure Data Engineer Associate or related certification. ## Description Job Summary: Lead the design and implementation of reusable data-ingestion, validation, transformation, curation and publication pipelines for the Atlas. The Lead Data Engineer will establish the pipeline pattern used for the MVP and future dataset onboarding., * Profile source datasets and document source-to-target mappings, refresh patterns and data-quality rules. * Design and build reusable batch, incremental, API, secure-file and near-real-time ingestion patterns where applicable. * Implement raw/validated/curated/published data layers consistent with the approved architecture. * Develop transformation, standardization, reconciliation, validation, exception, retry and quarantine controls. * Support CSV, Excel, delimited/fixed-width, Parquet, relational, API/web-service and cloud-storage sources. * Capture technical metadata and lineage and integrate quality results with governance processes. * Optimize data processing for reliability, performance and maintainability. * Implement CI/CD and configuration practices with the Architect and development team. * Support MVP data validation, UAT, production deployment and post-launch refresh operations. * Mentor the second Data Engineer and contribute to runbooks/documentation. ## Related Videos - [Parquet, Delta, Iceberg & Ducklake - An introduction for developers](https://www.wearedevelopers.com/videos/100075-parquet-delta-iceberg-ducklake-an-introduction-for-developers) - [A Data Mesh needs Open Metadata](https://www.wearedevelopers.com/videos/505-a-data-mesh-needs-open-metadata) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt) - [Modern Data Architectures need Software Engineering](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Top-Paying Tech Jobs (with Salaries)](https://www.wearedevelopers.com/magazine/372-top-paying-tech-jobs-with-salaries) - [The Most Popular IT Jobs on the Market](https://www.wearedevelopers.com/magazine/376-the-most-popular-it-jobs-on-the-market) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)