Data Engineer

Professional Technology Integration (PTI)
Denver, CO, United States
1 day ago
Apply on professionaltechintegration.com
Prepare application

Role details

Contract type
Temporary contract
Employment type
Full-time (> 32 hours)
Experience required
1 year minimum
Working hours
Regular working hours

Tech stack

Data Analysis Application Release Automation Automation of Tests Microsoft Azure Code Review Cyber Security Continuous Integration Data Architecture Data Files Data Governance Data Integrity Extract Transform Load (ETL)
+16 more
Data Mining Data Sharing Data Systems Software Design Documents Object-Oriented Software Development Performance Tuning Release Management Tableau (Software) Data Ingestion Snowflake Informatica Cloud Git Entity Resolution Terraform Software Version Control Data Pipelines

Job description

We are seeking a Data Engineer to support a Statewide Longitudinal Data System (SLDS) initiative under a shared delivery model with a third-party implementation partner. Working closely with the Data Architect, you will design, build, test, and maintain automated ETL/ELT data pipelines using Snowflake and Informatica to move data from transactional and partner agency sources into analytical data platforms and longitudinal data models.

You will progress through a structured knowledge-transfer program (shadowing, co-running, and readiness checks), take ownership of specific new source ingestions, and transition into full production operations and upkeep of the SLDS data foundation and analytics layer.

Scope of Responsibilities

  • Data Ingestion & Pipeline Engineering: Design, develop, test, and deploy automated ETL/data pipelines in Informatica and Snowflake to ingest data from partner agencies and third parties into the common data model.
  • Transformation & Analytics Layer Support: Build transformation logic that converts raw agency feeds into harmonized, linked longitudinal records and analytics layer models following established naming and modeling standards.
  • Record Linkage & Identity Matching: Run internal matching routines to link newly ingested agency records to master longitudinal student and workforce profiles, managing match exceptions and cross-agency reconciliations.
  • Production Pipeline Operations & Incident Response: Own scheduled feed executions, monitor jobs, resolve failures, reload late or corrected files, reconcile row counts, and rebuild downstream objects.
  • Data Quality & Governance Compliance: Implement L1 (structural/format), L2 (completeness/de-duplication), and L3 (referential integrity/logic) validation checks, layout-change detection, and audit trails; enforce privacy protections, suppression rules, and regulatory compliance (FERPA, COPPA, ADA, and applicable statutes).
  • Performance Optimization & Platform Stewardship: Tune query execution, warehouse compute, and pipeline performance to optimize cost and latency, identifying and resolving bottlenecks in Snowflake.
  • Quality Assurance, Code Review & Promotion: Follow the release lifecycle across Development, Test, UAT, and Production using Azure DevOps and Azure Test Plans; write and execute test cases, perform peer code reviews, and maintain release change logs.

Ingestion & Pipeline Development

  • Courses Data Ingestion: Ingest and validate annual postsecondary course-level data files per data sharing agreements; build ETL mappings into the common data model based on implementation partner design specifications and pair on construction, validation, and promotion.
  • Postsecondary SES & Disability Data Ingestion (Conditional): Ingest and validate demographic, Pell Grant eligibility, and disability classification codes from postsecondary sources; apply privacy protections, cell suppression rules, and permissible-use governance.
  • PSAT/SAT Assessment Data Ingestion (Conditional): Ingest, validate, and map section/composite assessment scores and grade levels from assessment providers; run identity matching to link records to longitudinal student profiles.

Knowledge Transfer & Operational Transition

  • Shadowing & Co-running: Shadow third-party implementation engineers on daily operations of existing feeds (such as NSC, K-3, CDEC, and Phase 1 feeds); co-run live feed cycles, handle data loads, execute record matching, and troubleshoot exceptions.
  • Readiness Gate Execution: Execute two consecutive feed cycles completely unaided to achieve formal operational hand-over sign-off.
  • Full Production Operations: Run recurring feed schedules, rebuild downstream objects, reconcile row counts against source files, triage failed jobs, and fulfill ad-hoc data inquiries and export requests.

Data Stabilization & Analytics Layer Upkeep

  • Stabilization Backlog Support: Address data reliability defects, run variations, and de-duplication issues carried over from the Phase 1 stabilization backlog.
  • Analytics Layer Hardening & Maintenance: Apply object modeling standards, naming conventions, and layout-change detection in the RAW layer; reconcile analytics layer outputs across non-production and production environments; implement metric enhancements (program participation, completion, equitable access, and wage outcomes); and tune dashboard back-end queries.

Cross-Functional Collaboration & Testing

  • Testing & Release Management: Author manual and automated test cases in Azure Test Plans, document execution evidence, and resolve defects prior to promotion.
  • Cross-Role Support: Partner with the Data Architect on logical/physical models and pipeline designs; support Data Scientists and Visualization Designers with new data sources for Tableau refreshes and de-identified Institution of Higher Education (IHE) outcome files; coordinate with Operations Engineers on Snowflake compute management, auto-shutdown, role permissions, and CI/CD pipelines.

Key Deliverables

  • Technical Code & Pipeline Artifacts: Production ETL/ELT pipelines (Informatica workflows and Snowflake SQL/transformations), L1-L3 cleansing/validation modules and Informatica layout-change detection routines, record linkage/identity matching scripts, analytics layer transformation objects, and de-identified data extraction scripts for IHEs.
  • Documentation & Engineering Specifications: ETL pipeline design documents, source-to-target mapping specifications, lineage documentation and dependency maps, and operational runbooks/incident procedures.
  • Governance, Testing & Operational Reporting: Azure Test Plans and execution/defect records, release change logs, and feed operation run logs, reconciliation totals, and match summaries.

Additional Requirements

  • Residency & Location: Must currently reside within the State of Colorado for the duration of the contract (strong preference for candidates within 100 miles / Denver Metro).
  • Security Training & Background Check: Must complete required information security and acceptable use training within 30 days of start and annually thereafter and pass a multi-point background verification (including 7-year state, county, and federal criminal checks, SSN name search, and global monitor check).
  • Schedule: Monday - Friday, 8:00 AM - 5:00 PM MST (8 hours/day)
  • Work Arrangement: Remote / Telecommuting
  • Location: Must reside within the State of Colorado; strong preference for local / Denver Metro candidates within 100 miles

Requirements

  • Snowflake ETL/ELT Pipeline Development & Optimization (Required 5 Years)
  • Informatica Cloud Data Governance and Catalog (CDGC) (Required 1 Years)
  • Infrastructure as Code (Terraform) (Required 1 Years)
  • CI/CD & Release Automation (Azure DevOps) (Required 1 Years)
  • Version Control & Code Reviews (Git) (Required 1 Years)
  • Public Sector Longitudinal Education & Workforce Data Systems (e.g., K-12, higher education, workforce, clearinghouse feeds) (Desired 1 Years)
  • Record Linkage, Entity Resolution & Deterministic/Probabilistic Matching (Desired 1 Years)
  • Data Privacy, Cell Suppression & Compliance (FERPA, COPPA, ADA, state statutes) (Desired 1 Years)
  • Structured Test Planning & Execution (Azure Test Plans) (Nice to Have 1 Years)

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on professionaltechintegration.com
Prepare application

Good distractions

Loading talks and stories from around this role…