Databricks Developer (Unity Catalog Migration)

We are seeking a highly skilled Databricks Engineer with deep experience in Unity Catalog and enterprise-scale data platforms to support a major Oil & Gas Hydrocarbon Reporting initiative. The engineer will augment an existing delivery team and contribute across data engineering, governance, optimization, and analytics initiatives on the Databricks Lakehouse Platform., * Design, develop, and maintain scalable data pipelines on the Databricks Lakehouse Platform.

• Implement and administer Unity Catalog for enterprise data governance, access control, lineage, and data sharing.
• Support hydrocarbon reporting workflows, including ingestion, transformation, validation, and reconciliation of production and operational data.
• Develop ETL/ELT pipelines using Spark, Delta Lake, and Delta Live Tables (Lakeflow experience is a plus).
• Optimize Databricks workloads for performance, reliability, and cost efficiency.
• Collaborate with business stakeholders to translate reporting requirements into scalable technical solutions.
• Implement data quality checks, monitoring, and automated validation frameworks.
• Work closely with solution architects and data engineers on platform enhancements and production support.
• Document technical designs, governance standards, and operational procedures.
• Participate in code reviews, CI/CD implementation, and DevOps best practices.

The ideal candidate has hands-on experience implementing secure, governed data architectures using Unity Catalog and has worked on large-scale regulatory, production, or hydrocarbon reporting programs within the energy sector., * Bachelor’s or Master’s degree in Computer Science, Engineering, Information Systems, or a related discipline.

• 5+ years of data engineering experience with 3+ years of hands-on Databricks experience.
• Strong experience implementing and managing Unity Catalog in enterprise environments.
• Experience supporting Oil & Gas, Energy, or Hydrocarbon Reporting projects.
• Expertise with:
• Databricks Lakehouse
• Apache Spark (PySpark)
• Delta Lake
• SQL
• Python
• Experience building enterprise ETL/ELT pipelines.
• Experience with Azure Databricks (AWS Databricks also considered).
• Strong understanding of data governance, security, metadata management, and lineage, * Experience with Delta Live Tables (DLT) or Lakeflow Pipelines.
• Knowledge of Unity Catalog features including ABAC, row/column-level security, external locations, catalogs, schemas, and lineage.
• Experience integrating with Azure Data Factory, Event Hubs, Kafka, or similar ingestion frameworks.
• Familiarity with MLflow, Databricks Workflows, and Databricks Asset Bundles.
• Experience working with hydrocarbon production, drilling, refinery, pipeline, or regulatory reporting datasets.
• Understanding of DevOps, Git, CI/CD, and Infrastructure as Code (Terraform preferred).

DESIRED SKILLS:

• Databricks Lakehouse Platform
• Unity Catalog Administration
• Delta Lake & Delta Live Tables
• Apache Spark / PySpark
• SQL & Python
• Data Governance & Metadata Management
• Performance Tuning & Cost Optimization
• Azure Cloud Services
• CI/CD & Git
• Agile Delivery