Data Architect / Data Engineer

Data Hub is a Databricks platform that brings transportation, safety, infrastructure and climate data from across CalSTA departments into one governed place. Analysts, dashboards, apps and business users rely on it. The first phase covers public state and federal data. The next phase adds internal, non-public data from the departments. We need a hands-on data architect and engineer to help build both phases.

What you will do

• Build and maintain automated pipelines that bring data from public portals, APIs and internal department systems into Delta tables.
• Assists the program manager in the platform’s architecture, naming standards, and documentation.
• Work with the data hub governance group to make decisions on architecture, data sharing, data contracts, data ownership and data quality standards.
• Advise the data hub governance group on how sensitive data can be protected, including row-level and column-level access rules, and keep it separate from public data.
• Make the data reliable through monitoring, alerts and data quality checks.
• Set up datasets and documentation so data experts and business users can explore the data in plain language with AI tools.
• Write and maintain skill files (instruction files that guide AI coding assistants) so pipelines are built consistently.
• Help other CalSTA departments build their own pipelines using shared templates.

• 7+ years in data engineering or a closely related role.
• Experience on large-scale, enterprise-wide data engineering or data architecture efforts.
• Strong Python and SQL.
• Experience with Apache Spark or a cloud data platform (Databricks, Snowflake, BigQuery or similar. Databricks is strongly preferred).
• Experience pulling data from REST APIs, databases or other source systems.
• Excellent communication skills, clear writing and good documentation habits.

Preferred

• Hands-on Databricks experience (Delta Lake, Unity Catalog, Jobs).
• Experience governing sensitive or confidential data, such as access controls, data masking and privacy requirements.
• Experience working within a data governance program, such as data contracts, data stewardship and cross-agency data sharing agreements.
• Experience writing skill files, prompts or instructions for AI coding assistants.
• Experience with GIS or location-based data.