Spark / Databricks Platform Engineer

The Databricks Engineer designs, develops, and optimizes scalable data solutions on the Databricks platform, leveraging PySpark or Scala for large-scale data processing. Operating with a high degree of creativity and latitude, this role involves constructing ingestion pipelines, implementing medallion architecture (Bronze, Silver, Gold layers), and designing robust dimensional data models. The engineer collaborates with cross-functional stakeholders to gather requirements, perform cost/benefit analyses of alternative solutions, and implement data governance, security, and quality frameworks to support enterprise public health and business intelligence initiatives., * Design, develop, and optimize highly scalable data solutions on Databricks using Apache Spark, PySpark, or Scala for large-scale data processing.

• Build, configure, and maintain robust ETL/ELT ingestion pipelines utilizing Lakeflow Declarative Pipelines (formerly Delta Live Tables / DLT).
• Implement and maintain Delta Lake architectures and medallion database structures across Bronze, Silver, and Gold data layers.
• Orchestrate, schedule, and monitor offline production jobs utilizing Lakeflow Jobs (formerly Databricks Workflows) or equivalent tools such as Apache Airflow.

Data Modeling & Quality Governance

• Design, develop, and support comprehensive enterprise data warehouses utilizing dimensional data modeling, including Star and Snowflake schemas.
• Implement rigorous data quality validation, data security controls, platform access models, and data governance frameworks.
• Analyze system specifications, evaluate operational limitations, and perform data audits to ensure scalability, security, and cost efficiency.

Business Analysis & Operational Insights

• Collaborate directly with business stakeholders, program managers, and technical peers to understand operational objectives, identify problems, and analyze current procedures.
• Translate high-level business goals into formal technical requirements, system design documentation, and cost/benefit analyses.
• Develop dynamic analytical dashboards and reporting solutions natively within Databricks, including Databricks SQL dashboards and Databricks Apps, to deliver actionable operational insights.

• 8 or more years of experience in IT, supporting the design, development, deployment, or delivery of technology solutions.
• 8 or more years of experience with Databricks, including building and optimizing ETL/ELT data pipelines using Apache Spark.
• 8 or more years of experience in data warehousing and dimensional data modeling, including Star and Snowflake schemas.
• 8 or more years of professional proficiency utilizing SQL and Python (or Scala) for large-scale data processing.
• 8 or more years of experience designing and developing dashboards and applications natively within Databricks, such as Databricks SQL dashboards or Databricks Apps.
• 8 or more years of experience implementing data governance, data quality metrics, and data security practices.
• 8 or more years of experience implementing Lakeflow Declarative Pipelines (formerly Delta Live Tables / DLT) to build and manage production pipelines.
• 8 or more years of experience with Delta Lake, medallion architecture (Bronze, Silver, Gold layers), data lakehouse design, and scheduling offline jobs using Lakeflow Jobs (formerly Databricks Workflows) or similar orchestration tools.
• 8 or more years of experience communicating technical specifications and presenting data-driven insights to technical and non-technical stakeholders., * 1 or more years of experience working within public sector or state government environments.
• 1 or more years of experience implementing CI/CD practices for data pipelines, including DevOps and Git-based version control workflows.
• Professional Databricks certification, such as Databricks Certified Data Engineer Associate or Professional., * Strong analytical, critical-thinking, and problem-solving skills with a high degree of creativity and latitude.
• Excellent written and verbal communication skills to write detailed descriptions of user needs and document system capabilities.
• Proactive, self-motivated approach to managing complex, cross-functional datasets and resolving workload issues.
• Strong collaborative capability to work effectively with data engineers, database administrators, and business partners.

Flexible work from home options available.

• Competitive salary