Databricks Architect

CEI LLC
Pittsburgh, PA, United States
1 day ago
Apply on www.thejobnetwork.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Experienced
Experience required
3 years minimum
Working hours
Regular working hours

Tech stack

Artificial Intelligence Amazon Web Services Data Analysis Microsoft Azure Continuous Integration Data Architecture Information Engineering Data Governance Extract Transform Load (ETL) Data Vault Modeling Distributed Systems Apache Hive
+18 more
Performance Tuning Power BI Azure Machine Learning Data Streaming Tableau (Software) Azure Service Bus Feature Store Apache Spark Git Event Driven Architecture Data Lakes Pyspark Optimization Algorithms Production Code Apache Kafka Machine Learning Operations Terraform Databricks

Job description

We are seeking a hands-on Databricks Technical Architect to design, build, and scale our modern data platform. This is a builder’s role: you will own the architecture of our Lakehouse while remaining deeply engaged in implementation - writing PySpark, tuning Delta tables, defining Unity Catalog policies, and reviewing production code.

\n

You will partner with data engineering, analytics, governance, and business teams to deliver a secure, performant, and cost-efficient platform that serves both analytical and AI/ML workloads

Requirements

  • 8-12+ years in data engineering, data architecture, or related roles, with 3+ years of production Databricks experience \n

  • Strong hands-on expertise in Apache Spark with PySpark and Spark SQL. Scala is a plus but not required \n

  • Production experience with Unity Catalog, including governance design, access control, and lineage \n

  • Deep experience designing Lakehouse architectures on Delta Lake using medallion patterns, with a strong grasp of partitioning, Z-ordering, liquid clustering, and Delta optimization techniques \n

  • Implementation experience on at least one major cloud platform: AWS, Azure, or GCP \n

  • Infrastructure-as-code experience with Terraform and/or Databricks Asset Bundles, plus CI/CD for data engineering workflows (Git-based, with environment promotion) \n

  • Strong understanding of data modeling (dimensional, Data Vault, or equivalent), ETL/ELT patterns, and distributed systems internals \n

  • Demonstrated experience with cost optimization and performance tuning on Databricks at scale \n

  • Working knowledge of data governance, security, and compliance principles, with the ability to translate policy into platform controls \n, * Databricks certifications: Data Engineer Professional and/or Solution Architect \n

  • Experience with Structured Streaming and event-driven architectures (Kafka, Event Hubs, Kinesis, or Pub/Sub) \n

  • Experience with Lakeflow Connect or equivalent ingestion frameworks \n

  • Familiarity with MLflow, feature stores, and ML platform integration patte \n

  • Experience designing serving layers for Power BI or Tableau at scale, including semantic modeling \n

  • Exposure to open table format interoperability (Delta UniForm, Iceberg) and Delta Sharing \n

  • Experience in a regulated industry (financial services, healthcare, public sector) with corresponding compliance frameworks. \n

Benefits & conditions

n \n

  • Architect end-to-end data solutions on Databricks, spanning ingestion, transformation, governance, and serving layers, using the medallion (Bronze/Silver/Gold) pattern on Delta Lakes \n

  • Lead hands-on implementation of production data pipelines in PySpark and SQL, with Scala where appropriate. Define and enforce engineering standards through code reviews, reference implementations, and reusable frameworks \n

  • Own the Unity Catalog design - including catalog/schema structure, access controls, row- and column-level security, data classification, lineage, and audit. Partner with the data governance function to operationalize policies at the platform layer \n

  • Define the serving layer strategy: Databricks SQL warehouses, materialized views, performance tuning for BI consumers (Power BI, Tableau), and semantic modeling patterns that bridge Gold tables to business consumption \n

  • Establish platform-as-code practices using Terraform and/or Databricks Asset Bundles. Design and operate CI/CD pipelines for notebooks, jobs, and infrastructure \n

  • Own cost and performance governance: cluster policies, serverless vs. classic compute decisions, Photon usage, job vs. all-purpose cluster strategy, and DBU budgeting and chargeback \n

  • Design orchestration using Databricks Workflows and Lakeflow Declarative Pipelines, integrating with Airflow where cross-platform orchestration is required \n

  • Partner with ML and AI teams on integration patterns for MLflow, feature engineering on Delta, and emerging Databricks AI capabilities (vector search, Mosaic, Agent Bricks). This role does not own model development but ensures the platform supports it \n

  • Provide technical leadership and mentorship to a team of [N] data engineers, and serve as the senior Databricks voice in cross-functional architecture forums \n

\n

\n

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.thejobnetwork.com
Prepare application

Good distractions

Loading talks and stories from around this role…