Principal Data Engineer or Lead Data Engineer

Pivotal Technologies Inc
Philadelphia, PA, United States
4 days ago
Apply on jobs.y-axis.com
Prepare application

Role details

Contract type
Temporary to permanent
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
5 years minimum
Compensation
$160,000
Working hours
Regular working hours

Tech stack

Application Programming Interfaces (APIs) Artificial Intelligence Data Analysis Microsoft Azure Software as a Service Continuous Integration Data as a Services Information Engineering Data Governance Extract Transform Load (ETL) Data Systems Apache Hive
+19 more
Python (Programming Language) Machine Learning Operational Data Store Operational Databases Commercial Software Software Engineering SQL Databases Data Streaming Delivery Pipeline Apache Spark Event Driven Architecture Data Lakes Pyspark Data Management Stream Processing Stream Analytics Data Pipelines Databricks Microservices

Job description

We are looking for a Principal Data Engineer to serve as a hands-on technical and people leader for data engineering, data platform architecture, real-time streaming, and production data services. This role will focus on designing, building, operating, and improving data pipelines and data products while also bringing principal-level judgment to architecture, stakeholder shaping, delivery priorities, team management, and production readiness., Hands-On Data Engineering and Platform Development

Design, build, optimize, and operate production-grade batch and streaming data pipelines on Azure and Databricks, with a primary focus on real-time IoT and telemetry use cases within a Medallion architecture.

Develop ETL/ELT workflows to ingest, transform, validate, and serve large volumes of structured, semi-structured, unstructured, and streaming data.

Build and maintain reliable data products, data services, APIs, and microservices that support operational applications, analytics, software engineering, and ML/AI teams.

Use Python, PySpark, Spark SQL, SQL, Delta Lake, Databricks Workflows, CI/CD, and related tools to build maintainable, testable, and observable data systems.

Troubleshoot complex production pipeline issues across Databricks, Azure, streaming systems, APIs, and source systems, including root cause analysis, corrective action, and prevention planning.

Move quickly from rough business need to prototype, pilot, and production-ready data capability while maintaining appropriate engineering discipline.

Real-Time Streaming, IoT Telemetry, and Operational Data Services

Lead the design and delivery of real-time streaming ingestion and processing patterns for connected medical device telemetry, event data, and operational data feeds.

Implement streaming solutions using Azure Event Hubs, Azure Stream Analytics, Databricks, Delta Lake, and related Azure integration patterns.

  • Design cost-effective throughput, partitioning, delivery, retention, and replay strategies for high-

Requirements

This is a hands-on engineering leadership role first. The ideal candidate should be comfortable spending significant time working directly with Databricks, Spark, SQL, Python/PySpark, Azure services, streaming architectures, data quality frameworks, pipeline automation, CI/CD, and production troubleshooting. They should also be able to operate with the maturity of a principal-level leader: shaping unclear requirements, making pragmatic technical decisions, managing and mentoring engineers, and driving work forward without waiting for perfect specifications.

This is a fast-moving, startup-like environment. Requirements may be incomplete, priorities may evolve, and the right candidate will help create clarity while building quickly. We need someone who can move from ambiguous business need to reliable data capability with urgency, discipline, and ownership.

Stakeholder shaping is a critical part of this role. The Lead Data Engineer should be able to work directly with business, product, software engineering, analytics, ML/AI, operations, and leadership stakeholders to define what data needs to exist, how it should be consumed, what production guarantees are required, and how success should be measured.

A background in commercial software, SaaS, digital products, healthtech, fintech, IoT, data platforms, or other product-driven environments is strongly preferred. We want someone who understands that data pipelines and data services are not just technical artifacts. They are product capabilities that support real users, real workflows, operational decisions, ML/AI systems, APIs, analytics, and measurable business outcomes.

About the company

We just got in this role: It is with Medical Guardian-this is a hybrid role onsite 3 days a week in Center City, Philadelphia.

This is a 4 month contract to hire with no conversion fee. Converting salary $160K annually. They can not sponsor so of US Citz.

Interview mode 1st round will be Virtual & Final will be onsite.

The Lead Data Engineer will own the design, delivery, and maturity of production-grade data pipelines and data platforms, with a primary emphasis on real-time streaming, IoT telemetry, Databricks, Azure, data services for APIs and microservices, and reliable data products for downstream consumption.

This role is focused on building and leading the data engineering foundation that powers real-time decisioning, operational applications, analytics, ML/AI model development, and data services across our Philadelphia-based customer.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on jobs.y-axis.com
Prepare application

Good distractions

Loading talks and stories from around this role…