Data Engineer - Remote

Experteer Overview As a Data Engineer, you design, build, and maintain scalable data pipelines on an Azure-based platform to enable batch and real-time analytics. You will collaborate with data engineers, data scientists, and reporting partners to meet evolving data needs and enforce governance, quality, and security across the data lifecycle. You’ll apply Medallion architecture (Bronze/Silver/Gold) and optimize Spark-based jobs for low latency and high throughput. This role offers remote work flexibility with in-office requirements in certain locations, and opportunities to impact large-scale health analytics. Compensation / Benefits * Design, develop, and maintain robust data pipelines for streaming and batch data using Azure Data Factory and PySpark via Databricks * Implement Medallion Architecture (Bronze/Silver/Gold) with data cleansing, transformation, and curated datasets * Optimize Spark jobs for performance, tune configurations, and scale clusters * Develop and maintain ETL processes; handle data from APIs, databases, files, and IoT streams * Implement data quality checks, automated alerts, and data governance practices * Leverage modern pipeline frameworks and declarative pipelines (e.g., Delta Live Tables, Lakehouse) * Monitor pipeline health, diagnose failures, and collaborate with teams to adjust pipelines * Document data workflows, lineage, and governance controls to ensure security and privacy Tasks * 3+ years in data engineering (designing and implementing data pipelines and ETL) * 2+ years in SQL for data manipulation and query optimization * 1+ years in Azure/Databricks or equivalent cloud data platform; experience with Delta Lake or Parquet * Proficiency in Python and Apache Spark (PySpark) for batch and streaming * Experience with streaming technologies (Spark Streaming, Kafka, Azure Event Hubs) * Agile team experience; strong communication with technical and non-technical stakeholders * Familiarity with data governance, security, and documentation practices * Ability to diagnose data issues and implement validation checks Key requirements * comprehensive benefits package * incentive and recognition programs * equity stock purchase * 401k contribution * remote work flexibility * telecommuter policy (for remote roles)

security handle data from APIs, databases, files, and IoT streams * Implement data quality checks, automated alerts, and data governance practices * Leverage modern pipeline frameworks and declarative pipelines (e.g., Delta Live Tables, Lakehouse) * Monitor pipeline health, diagnose failures, and collaborate with teams to adjust pipelines * Document data workflows, lineage, and governance controls to ensure security and privacy Tasks * 3+ years in data engineering (designing and implementing data pipelines and ETL) * 2+ years in SQL for data manipulation and query optimization * 1+ years in Azure/Databricks or equivalent cloud data platform; experience with Delta Lake or Parquet * Proficiency in Python and Apache Spark (PySpark) for batch and streaming * Experience with streaming technologies (Spark Streaming, Kafka, Azure Event Hubs) * Agile team experience; strong communication with technical and non-technical stakeholders * Familiarity with data governance, security, and documentation practices * Ability to diagnose data issues and implement validation checks Key requirements * comprehensive benefits package * incentive and recognition programs * equity stock purchase * 401k contribution * remote work flexibility * telecommuter policy (for remote roles)