Lead Data Architect / Data Engineer

We are seeking a hands-on Lead Data Architect/Data Engineer to define architecture and deliver production-grade pipelines across multiple phases of a Databricks platform.

Responsibilities include:

• Build and maintain automated ingestion and transformation pipelines that bring data from:

o public portals and open data sources

o REST APIs

o internal departmental databases and systems

into Delta tables in Databricks.

• Partner with the Program Manager to establish and maintain:

o platform architecture

o naming conventions and standards

o technical documentation

• Collaborate with the Data Hub governance group to support decisions on:

o platform and data architecture

o data sharing patterns

o data contracts and data ownership

o data quality and reliability standards

• Design recommendations for protecting sensitive data, including:

o row-level and column-level access controls

o separation of confidential/internal data from public datasets

• Improve reliability through:

o monitoring and alerting

o data quality checks and validation frameworks

• Enable discoverability and usability by publishing datasets and documentation that support plain-language exploration and AI-enabled analysis tools.

• Write and maintain skill files (instruction files for AI coding assistants) to enforce consistent pipeline patterns and development standards.

• Support and enable other departments to build pipelines using shared templates and best practices.

• Strong leadership experience, including executive-level stakeholder engagement

• 7+ years in data engineering, data architecture, or closely related roles

• Proven experience delivering large-scale, enterprise data engineering / architecture initiatives

• Strong hands-on expertise in Python and SQL

• Experience with Apache Spark or cloud data platforms such as:

o Databricks (strongly preferred)

o Snowflake, BigQuery, or similar

• Experience integrating data from REST APIs, databases, and other source systems

• Excellent communication skills with strong writing and documentation practices

Preferred

• Hands-on Databricks experience with:

o Delta Lake

o Unity Catalog

o Databricks Jobs

• Experience governing sensitive/confidential data (access control, masking, privacy requirements)

• Experience operating within a data governance program (stewardship, data contracts, cross-agency sharing agreements)

• Experience writing skill files, prompts, or instruction sets for AI coding assistants

• Experience with GIS / location-based datasets