Data Engineer
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Tech stack
+16 more
Job description
We are seeking a Data Engineer to support a Statewide Longitudinal Data System (SLDS) initiative under a shared delivery model with a third-party implementation partner. Working closely with the Data Architect, you will design, build, test, and maintain automated ETL/ELT data pipelines using Snowflake and Informatica to move data from transactional and partner agency sources into analytical data platforms and longitudinal data models.
You will progress through a structured knowledge-transfer program (shadowing, co-running, and readiness checks), take ownership of specific new source ingestions, and transition into full production operations and upkeep of the SLDS data foundation and analytics layer.
Scope of Responsibilities
- Data Ingestion & Pipeline Engineering: Design, develop, test, and deploy automated ETL/data pipelines in Informatica and Snowflake to ingest data from partner agencies and third parties into the common data model.
- Transformation & Analytics Layer Support: Build transformation logic that converts raw agency feeds into harmonized, linked longitudinal records and analytics layer models following established naming and modeling standards.
- Record Linkage & Identity Matching: Run internal matching routines to link newly ingested agency records to master longitudinal student and workforce profiles, managing match exceptions and cross-agency reconciliations.
- Production Pipeline Operations & Incident Response: Own scheduled feed executions, monitor jobs, resolve failures, reload late or corrected files, reconcile row counts, and rebuild downstream objects.
- Data Quality & Governance Compliance: Implement L1 (structural/format), L2 (completeness/de-duplication), and L3 (referential integrity/logic) validation checks, layout-change detection, and audit trails; enforce privacy protections, suppression rules, and regulatory compliance (FERPA, COPPA, ADA, and applicable statutes).
- Performance Optimization & Platform Stewardship: Tune query execution, warehouse compute, and pipeline performance to optimize cost and latency, identifying and resolving bottlenecks in Snowflake.
- Quality Assurance, Code Review & Promotion: Follow the release lifecycle across Development, Test, UAT, and Production using Azure DevOps and Azure Test Plans; write and execute test cases, perform peer code reviews, and maintain release change logs.
Ingestion & Pipeline Development
- Courses Data Ingestion: Ingest and validate annual postsecondary course-level data files per data sharing agreements; build ETL mappings into the common data model based on implementation partner design specifications and pair on construction, validation, and promotion.
- Postsecondary SES & Disability Data Ingestion (Conditional): Ingest and validate demographic, Pell Grant eligibility, and disability classification codes from postsecondary sources; apply privacy protections, cell suppression rules, and permissible-use governance.
- PSAT/SAT Assessment Data Ingestion (Conditional): Ingest, validate, and map section/composite assessment scores and grade levels from assessment providers; run identity matching to link records to longitudinal student profiles.
Knowledge Transfer & Operational Transition
- Shadowing & Co-running: Shadow third-party implementation engineers on daily operations of existing feeds (such as NSC, K-3, CDEC, and Phase 1 feeds); co-run live feed cycles, handle data loads, execute record matching, and troubleshoot exceptions.
- Readiness Gate Execution: Execute two consecutive feed cycles completely unaided to achieve formal operational hand-over sign-off.
- Full Production Operations: Run recurring feed schedules, rebuild downstream objects, reconcile row counts against source files, triage failed jobs, and fulfill ad-hoc data inquiries and export requests.
Data Stabilization & Analytics Layer Upkeep
- Stabilization Backlog Support: Address data reliability defects, run variations, and de-duplication issues carried over from the Phase 1 stabilization backlog.
- Analytics Layer Hardening & Maintenance: Apply object modeling standards, naming conventions, and layout-change detection in the RAW layer; reconcile analytics layer outputs across non-production and production environments; implement metric enhancements (program participation, completion, equitable access, and wage outcomes); and tune dashboard back-end queries.
Cross-Functional Collaboration & Testing
- Testing & Release Management: Author manual and automated test cases in Azure Test Plans, document execution evidence, and resolve defects prior to promotion.
- Cross-Role Support: Partner with the Data Architect on logical/physical models and pipeline designs; support Data Scientists and Visualization Designers with new data sources for Tableau refreshes and de-identified Institution of Higher Education (IHE) outcome files; coordinate with Operations Engineers on Snowflake compute management, auto-shutdown, role permissions, and CI/CD pipelines.
Key Deliverables
- Technical Code & Pipeline Artifacts: Production ETL/ELT pipelines (Informatica workflows and Snowflake SQL/transformations), L1-L3 cleansing/validation modules and Informatica layout-change detection routines, record linkage/identity matching scripts, analytics layer transformation objects, and de-identified data extraction scripts for IHEs.
- Documentation & Engineering Specifications: ETL pipeline design documents, source-to-target mapping specifications, lineage documentation and dependency maps, and operational runbooks/incident procedures.
- Governance, Testing & Operational Reporting: Azure Test Plans and execution/defect records, release change logs, and feed operation run logs, reconciliation totals, and match summaries.
Additional Requirements
- Residency & Location: Must currently reside within the State of Colorado for the duration of the contract (strong preference for candidates within 100 miles / Denver Metro).
- Security Training & Background Check: Must complete required information security and acceptable use training within 30 days of start and annually thereafter and pass a multi-point background verification (including 7-year state, county, and federal criminal checks, SSN name search, and global monitor check).
- Schedule: Monday - Friday, 8:00 AM - 5:00 PM MST (8 hours/day)
- Work Arrangement: Remote / Telecommuting
- Location: Must reside within the State of Colorado; strong preference for local / Denver Metro candidates within 100 miles
Requirements
- Snowflake ETL/ELT Pipeline Development & Optimization (Required 5 Years)
- Informatica Cloud Data Governance and Catalog (CDGC) (Required 1 Years)
- Infrastructure as Code (Terraform) (Required 1 Years)
- CI/CD & Release Automation (Azure DevOps) (Required 1 Years)
- Version Control & Code Reviews (Git) (Required 1 Years)
- Public Sector Longitudinal Education & Workforce Data Systems (e.g., K-12, higher education, workforce, clearinghouse feeds) (Desired 1 Years)
- Record Linkage, Entity Resolution & Deterministic/Probabilistic Matching (Desired 1 Years)
- Data Privacy, Cell Suppression & Compliance (FERPA, COPPA, ADA, state statutes) (Desired 1 Years)
- Structured Test Planning & Execution (Azure Test Plans) (Nice to Have 1 Years)
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦