Lead Data/ETL Engineer

DecisionPoint Corporation
Chicago, IL, United States
4 days ago
Apply on diversityjobs.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Working hours
Regular working hours

Tech stack

Training Data Application Programming Interfaces (APIs) Artificial Intelligence Audit Trail Business Intelligence Development CompTIA Security+ Information Systems Databases Continuous Integration Information Engineering Data Governance Data Infrastructure
+24 more
Data Integrity Extract Transform Load (ETL) Database Queries Python (Programming Language) Key Management Meta-Data Management Metadata Repositories Reference Data Regression Testing SQL Stored Procedures SQL Databases SQL Server Integration Services Systems Integration Data Logging Cloud Platform System Data Ingestion Data Layers Information Technology Data Lineage Deployment Automation Pagination Software Version Control Data Pipelines Api Management

Job description

DecisionPoint seeks a Lead Data/ETL Engineer to provide technical leadership and hands-on engineering for data ingestion, transformation, integration, and delivery under the USMEPCOM ENLIST program.

This SME-level position will maintain and optimize SQL Server Integration Services (SSIS), implement approved API integrations and incremental loading approaches, and build raw, standardized, and curated data layers. The engineer will establish reliable processes for reconciliation, retries, schema changes, identifier mappings, lineage, and failed-record handling.

The Lead Data/ETL Engineer will collaborate with database administrators, architects, governance personnel, AI/ML engineers, BI developers, testers, and Government stakeholders to deliver accurate, traceable, and maintainable data pipelines supporting mission analysis, AI capabilities, and reporting., The Lead Data/ETL Engineer will: Lead the design, implementation, maintenance, and optimization of enterprise data integration and ETL pipelines.

  • Maintain and optimize SSIS packages, configurations, schedules, dependencies, and execution processes.

  • Develop SQL queries, stored procedures, transformations, and reconciliation logic.

  • Use Python or another approved automation language to support ingestion, orchestration, validation, monitoring, and recovery.

  • Implement approved API integrations, including authentication, pagination, rate-limit handling, retries, and error processing.

  • Design incremental loading strategies and implement change-data capture where appropriate and approved.

  • Build and maintain raw, standardized, and curated data layers with documented transformation rules.

  • Establish source-to-target mappings, common identifier mappings, and consistent handling of reference data.

  • Implement reconciliation checks to verify record counts, totals, completeness, and transformation accuracy.

  • Develop restartable processing, controlled retries, checkpoints, and recovery procedures.

  • Implement failed-record handling, quarantine processes, error logging, and controlled reprocessing.

  • Assess and manage schema changes and their effects on pipelines, datasets, analytical models, and reports.

  • Maintain data lineage, transformation documentation, interface specifications, and pipeline dependencies.

  • Monitor pipeline execution, throughput, freshness, resource utilization, and operational failures.

  • Troubleshoot complex issues involving source systems, APIs, SSIS packages, SQL logic, data quality, and downstream integrations.

  • Optimize pipeline performance while maintaining data integrity, traceability, and approved security controls.

  • Coordinate with governance personnel on definitions, ownership, authoritative sources, metadata, and approved data uses.

  • Work with independent testing personnel to support reconciliation, regression testing, defect resolution, and release validation.

  • Coordinate pipeline changes with database, application, AI/ML, warehouse, and BI teams.

  • Maintain version-controlled code, deployment instructions, rollback procedures, and operational runbooks.

  • Review technical designs and code, guide engineering practices, and provide knowledge transfer to other engineers.

  • Communicate pipeline health, risks, dependencies, corrective actions, and improvement recommendations to technical leads and Government stakeholders.

Requirements

The following must-have qualifications are drawn from the supplied position requirements:

  • Active Secret clearance or higher.

  • Minimum of 12 years of relevant professional experience.

  • Bachelor’s degree in computer science, information systems, or a related field, or a permitted equivalent.

  • Experience maintaining and optimizing SQL Server Integration Services (SSIS).

  • Strong SQL skills.

  • Experience with Python and data engineering automation.

  • Experience implementing and supporting API integrations.

  • Experience developing incremental data loads.

  • Experience documenting and maintaining data lineage.

  • Experience troubleshooting complex data pipelines.

  • Ability to build raw, standardized, and curated data layers.

  • Ability to manage reconciliation, retries, schema changes, identifier mappings, and failed-record handling.

  • Ability to implement change-data capture where appropriate.

  • CompTIA Security+ CE or another approved IAT Level II certification when applicable to assigned duties.

Preferred Qualifications

The following qualifications are preferred but are not required:

  • Relevant data engineering or cloud platform certification.

  • Experience supporting federal or Department of Defense data environments.

  • Experience modernizing legacy SSIS integrations or transitioning pipelines to approved cloud platforms.

  • Experience with enterprise data warehouses, analytical datasets, BI platforms, or AI-ready data.

  • Familiarity with data catalogs, metadata management, data governance, and automated lineage tools.

  • Experience developing automated data-quality checks, reconciliation frameworks, and pipeline regression tests.

  • Experience integrating data engineering workflows with CI/CD, version control, and automated deployment.

  • Experience implementing secure service accounts, secrets management, encryption, and pipeline audit logging.

  • Strong technical mentoring, documentation, troubleshooting, and stakeholder communication skills.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on diversityjobs.com
Prepare application

Good distractions

Loading talks and stories from around this role…