Data Bricks Lead Engineer

Quantum Integrators
Manville, United States of America
2 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior

Job location

Manville, United States of America

Tech stack

Amazon Web Services (AWS)
Python
Power BI
Standard Sql
Data Ingestion
Data Lake
Star Schema
Databricks

Job description

We are seeking a DatabricksEngineer to build of a new Databricks solution that ingests source files and constructs a semantic/serving layer that powers a Power BI dashboard. A semantic layer already exists; the new solution must conform to the same structure so that a replicated Power BI solution works seamlessly against it with no rework to reports, measures, or relationships. The engineer owns the Databricks data and semantic layer and partners with a dedicated Power BI developer/team, who own the dashboard build, to deliver the end solution., * Lead the design and delivery of file-ingestion pipelines in Azure Databricks (batch and/or streaming) that land and process incoming files into Delta Lake using a Medallion architecture (Bronze ? Silver ? Gold).

  • Design the Gold/serving layer to replicate the existing semantic model exactly - matching table names, grain, column names, data types, keys, and relationships so the existing Power BI dataset can be repointed without redesign.
  • Expose the model through Databricks SQL endpoints (Direct Query) so the Power BI developer can connect and validate the dashboard with no redesign.
  • Collaborate with the Power BI developer/team as the data-side owner - align on the semantic contract (star schema, measures, RLS expectations), support their dashboard build, and resolve any data or model issues to land the end deliverable.
  • Document the semantic mapping and work with the Power BI team to reconcile the replicated dashboard's output against the existing solution to prove parity.

Requirements

  • AWS Databricks, Delta Lake, Medallion architecture, Unity Catalog
  • Python and strong SQL
  • Databricks SQL endpoints and a working understanding of Power BI semantic modeling (star schema, DAX, incremental refresh, RLS) sufficient to define and hand off the semantic contract
  • Databricks Certified Data Engineer (Professional preferred)

#Immediatehiring #DatabricksEngineer #Databricksjobs #DataEngineer #AzureDatabricks #AWSDatabricks #DeltaLake #MedallionArchitecture #UnityCatalog#PythonDeveloper #SQLDeveloper #hiringnow #ContractJobs #Raritan_NJ #QuantaSystemsTechnologies

Apply for this position