Data Engineer

Intone Networks
United States
2 days ago
Apply on www.careerbuilder.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
6 years minimum
Working hours
Regular working hours

Tech stack

Unity 3d Data Analysis Microsoft Azure Batch Processing Big Data Software Quality Continuous Integration Information Engineering Data Governance Data Warehousing Digital Architecture Dimensional Modeling
+20 more
Python (Programming Language) Operational Data Store Operational Databases Azure Data Lake SQL Databases Data Streaming Data Ingestion Azure Data Factory Sql Optimization Apache Spark Git Data Lakes Pyspark Data Lineage Low Latency Apache Kafka Data Management Data Pipelines Cisco Databricks

Job description

We are seeking a Senior Data Engineer for a full-time, remote position in the US to build, operate, and improve an enterprise data warehouse and production data pipelines in an Azure Databricks environment. This hands-on role will work across the Bronze, Silver, and Gold layers of a medallion architecture, with particular emphasis on data ingestion, transformation, modeling, quality, and reliability. You will own assigned data domains as production systems, help simplify legacy transformation layers, and support the transition from batch processing to event-driven, low-latency ingestion.

Responsibilities

  • Design, build, and support enterprise data warehouse pipelines and data models across Bronze, Silver, and Gold layers in Databricks.
  • Develop ingestion patterns into the Bronze layer and transformation logic that produces standardized, reliable Silver-layer data.
  • Create dimensional and harmonized data models, including conformance, survivorship, and Gold-layer promotion rules.
  • Take production ownership of assigned data domains, including model integrity, pipeline reliability, ingestion SLAs, incident response, and root-cause remediation.
  • Build and maintain Delta Lake pipelines while helping transition selected workloads from batch processing to event-driven, low-latency ingestion.
  • Use Zerobus Ingestion, Spark Declarative Pipelines, or supported event-streaming patterns for high-SLA data sources.
  • Implement row-level reconciliation, validation checkpoints, automated quality gates, and data-contract validation throughout the data lifecycle.
  • Build Gold-layer tables that support Unity Catalog metric views, lineage tracking, governed access, and downstream analytics.
  • Optimize pipeline performance, processing latency, reliability, and Databricks compute costs.
  • Consolidate legacy transformation layers into a simplified target architecture while validating that outputs remain accurate.
  • Monitor pipeline health, investigate anomalies, and resolve data-reliability incidents.
  • Document architecture decisions, data models, transformation logic, feed SQL, and operational procedures.
  • Review pull requests and maintain code quality through Azure DevOps and Git-based CI/CD practices.
  • Mentor Data Engineers in Databricks development, SQL, Python, PySpark, testing, and documentation.
  • Collaborate directly with data governance, quality assurance, analytics engineering, and business-facing teams to resolve requirements and delivery issues.

Requirements

Data Management, Data Warehousing, Microsoft Windows Azure, Production Systems, Data Modeling, Incident Response, Service Level Agreement (SLA), Reconciliation, Cisco Unity, Metrics, * Required

  • Six or more years of progressive data engineering experience.
  • Production ownership of an enterprise data warehouse or large-scale data transformation pipelines governed by defined SLAs.
  • Deep, hands-on Databricks experience in production environments.
  • Strong experience with Delta Lake, medallion architecture, Unity Catalog, and Databricks pipeline optimization.
  • Advanced SQL skills and strong proficiency with Python and PySpark.
  • Ability to independently build, review, troubleshoot, and optimize complex data transformations.
  • Strong data warehousing experience, including dimensional modeling, harmonized data models, conformance rules, and survivorship logic.
  • Hands-on experience with Azure Databricks, Azure Data Lake Storage, Azure Data Factory, and CI/CD practices using Azure DevOps.
  • Experience operating data pipelines as production systems, including monitoring, incident response, reliability, and performance management.
  • Ability to learn complex, under-documented data environments and convert that knowledge into clear, durable technical documentation.
  • Strong communication skills and the ability to collaborate directly with technical, governance, quality, analytics, and business stakeholders.
  • Preferred
  • Experience in healthcare, pharmacy, specialty pharmacy, or another regulated industry involving clinical or operational data.
  • Production experience with event-driven ingestion using Azure Event Hubs, Kafka, or equivalent technology through Structured Streaming.
  • Familiarity with Zerobus Ingestion and Spark Declarative Pipelines.
  • Experience implementing data contracts

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.careerbuilder.com
Prepare application

Good distractions

Loading talks and stories from around this role…