Lead Data Engineer - Pierre, SD (Hybrid)

MY3TECH INC

Pierre, SD, United States

22 days ago

Apply on www.careerjet.com

Prepare application

Role details

Contract type

Temporary to permanent

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

3 years minimum

Working hours

Regular working hours

Job source

www.careerjet.com

Tech stack

Microsoft Excel Application Programming Interfaces (APIs) Microsoft Azure Continuous Integration Data Validation Data Control Information Engineering Extract Transform Load (ETL) Python (Programming Language) Metadata SQL Azure Cloud Services

+17 more

Standard Sql SAP Sales and Distribution Software Deployment Web Services Parquet Data Logging Data Processing Data Server Interface Azure Data Factory Fast Healthcare Interoperability Resources Git Data Layers Build Management Data Lakes Health Level Seven International Data Pipelines Databricks

Job description

Job Summary: Lead the design and implementation of reusable data-ingestion, validation, transformation, curation and publication pipelines for the Atlas. The Lead Data Engineer will establish the pipeline pattern used for the MVP and future dataset onboarding., * Profile source datasets and document source-to-target mappings, refresh patterns and data-quality rules.

Design and build reusable batch, incremental, API, secure-file and near-real-time ingestion patterns where applicable.
Implement raw/validated/curated/published data layers consistent with the approved architecture.
Develop transformation, standardization, reconciliation, validation, exception, retry and quarantine controls.
Support CSV, Excel, delimited/fixed-width, Parquet, relational, API/web-service and cloud-storage sources.
Capture technical metadata and lineage and integrate quality results with governance processes.
Optimize data processing for reliability, performance and maintainability.
Implement CI/CD and configuration practices with the Architect and development team.
Support MVP data validation, UAT, production deployment and post-launch refresh operations.
Mentor the second Data Engineer and contribute to runbooks/documentation.

Requirements

7+ years of data engineering/ETL experience, with 3+ years on Azure or comparable cloud data platforms.
Strong SQL plus Python or equivalent data-engineering language.
Hands-on experience with ETL/ELT pipelines, APIs, file ingestion, data validation and reconciliation.
Experience with data-lake/lakehouse or layered/medallion architectures.
Ability to build production-grade error handling, logging, retry, quality and refresh processes.
Experience with CI/CD, Git and controlled deployments.
Ability to work with health/public-sector data and sensitive-data controls., * Azure Data Factory, Synapse, Fabric, Databricks, Azure SQL, Data Lake Storage or equivalent experience.
Healthcare/public-health/Medicaid data engineering.
Experience with metadata, lineage and data-quality tooling.
Experience with HL7/FHIR or other health-data interfaces is advantageous but not required.
Azure Data Engineer Associate or related certification.