Lead Data Engineer - Pierre, SD (Hybrid)

Job Summary: Lead the design and implementation of reusable data-ingestion, validation, transformation, curation and publication pipelines for the Atlas. The Lead Data Engineer will establish the pipeline pattern used for the MVP and future dataset onboarding., * Profile source datasets and document source-to-target mappings, refresh patterns and data-quality rules.

• Design and build reusable batch, incremental, API, secure-file and near-real-time ingestion patterns where applicable.
• Implement raw/validated/curated/published data layers consistent with the approved architecture.
• Develop transformation, standardization, reconciliation, validation, exception, retry and quarantine controls.
• Support CSV, Excel, delimited/fixed-width, Parquet, relational, API/web-service and cloud-storage sources.
• Capture technical metadata and lineage and integrate quality results with governance processes.
• Optimize data processing for reliability, performance and maintainability.
• Implement CI/CD and configuration practices with the Architect and development team.
• Support MVP data validation, UAT, production deployment and post-launch refresh operations.
• Mentor the second Data Engineer and contribute to runbooks/documentation.

• 7+ years of data engineering/ETL experience, with 3+ years on Azure or comparable cloud data platforms.
• Strong SQL plus Python or equivalent data-engineering language.
• Hands-on experience with ETL/ELT pipelines, APIs, file ingestion, data validation and reconciliation.
• Experience with data-lake/lakehouse or layered/medallion architectures.
• Ability to build production-grade error handling, logging, retry, quality and refresh processes.
• Experience with CI/CD, Git and controlled deployments.
• Ability to work with health/public-sector data and sensitive-data controls., * Azure Data Factory, Synapse, Fabric, Databricks, Azure SQL, Data Lake Storage or equivalent experience.
• Healthcare/public-health/Medicaid data engineering.
• Experience with metadata, lineage and data-quality tooling.
• Experience with HL7/FHIR or other health-data interfaces is advantageous but not required.
• Azure Data Engineer Associate or related certification.