> Markdown version of [/jobs/ext/2575922-sr-data-engineer](https://www.wearedevelopers.com/jobs/ext/2575922-sr-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Sr Data Engineer - **Company:** Healthfirst - **Location:** United States (Remote available) - **Experience:** Expert - **Salary:** $119,600.0 - $194,480.0 - **Contract:** Permanent contract - **Skills:** Application Programming Interfaces (APIs), Agile Methodology, Airflow, Amazon Web Services, Amazon S3, Apache HTTP Server, Application Frameworks, Automation of Tests, Microsoft Azure, Information Systems, Continuous Delivery, Continuous Integration, Data Governance, Extract Transform Load (ETL), Data Security, Data Systems, Data Warehousing, Relational Databases, Dimensional Modeling, Disaster Recovery, Distributed Computing Environment, Apache Hadoop, Identity and Access Management, Python (Programming Language), Meta-Data Management, NoSQL, Operational Databases, Cloud Services, DataOps, Data Streaming, Systems Integration, Google Cloud, Enterprise Software Applications, Cloud Platform System, Apache Spark, Git, Data Lakes, Pyspark, Infrastructure Automation Frameworks, Information Technology, AWS Glue, Integration Frameworks, Apache Kafka, Spark Streaming, Cloud Optimization, Cloudwatch, Restful APIs, Terraform, Software Version Control, Data Pipelines, Amazon Elastic Mapreduce (EMR), Amazon Redshift - **Published:** August 10, 2026 - **Apply:** https://www.indeed.com/viewjob?jk=f7642aa199d34181 ## About the Role * Bachelor's degree in Computer Science, Information Systems, Engineering, or a related technical field, or equivalent work experience. * 8+ years of overall IT experience. * 5+ years of hands-on experience designing and developing enterprise-scale data engineering solutions. * Strong experience developing scalable data pipelines and reusable frameworks using Python and PySpark. * Experience implementing enterprise data ingestion, integration, and transformation solutions using AWS Glue, dbt, Apache Spark, or comparable technologies. * Strong understanding of data warehousing concepts, dimensional modeling, and modern data lake / Lakehouse architectures. * Experience with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform (AWS preferred). * Strong SQL expertise with relational databases; familiarity with NoSQL databases is a plus. * Experience with distributed data processing technologies such as Apache Spark, Amazon EMR, or Hadoop-based platforms. * Experience using Git-based source control and Agile software development methodologies. * Strong analytical, problem-solving, and communication skills, with the ability to collaborate across technical and business teams., * Experience designing cloud-native data platforms using AWS services such as S3, Glue, EMR, Athena, Redshift, Lambda, Lake Formation, IAM, and CloudWatch. * Hands-on experience with Apache Iceberg (or similar open table formats) and governed Lakehouse patterns. * Experience implementing CI/CD pipelines, DataOps practices, continuous testing (CT), and infrastructure automation (e.g., Terraform). * Experience with workflow orchestration tools such as Apache Airflow, MWAA, AWS Step Functions, or similar platforms. * Experience developing RESTful APIs or other access layers and integrating with enterprise applications for data-product consumption. * Experience building and supporting real-time or streaming platforms using Kafka, Kinesis, or Spark Structured Streaming. * Strong understanding of data quality, observability, monitoring, and automated validation frameworks. * Knowledge of data governance, metadata management, lineage, and enterprise data catalog solutions. * Strong understanding of data security, encryption, access controls, and healthcare regulatory compliance (HIPAA/PHI). * Experience optimizing distributed workloads for scalability, reliability, and cloud cost efficiency. * Experience mentoring engineers, conducting design and code reviews, and establishing engineering best practices. ## Description The Senior Data Engineer designs, builds, and operates enterprise-scale data solutions on Healthfirst's AWS Lakehouse platform. They implement batch and streaming data ingestion, ELT/ETL, data quality, and transformation patterns that produce curated, trusted, operations/analytics ready data products. The role partners with product owners, data modelers, domain stakeholders, and platform teams to translate business requirements into scalable pipelines, reusable frameworks, and clear standards for storing, processing, and moving data. The Senior Data Engineer practices DataOps - CI/CD/CT, automated validation, observability, and Agile delivery - and provides technical leadership through design reviews, mentoring, and engineering best practices. They own reliable datasets that enable analytics and reporting - not ad-hoc analysis-and ensure pipelines support lifecycle management, resiliency, and governed access across the Lakehouse (raw refine publish)., * Design and implement ELT/ETL solutions for batch and streaming ingestion, integration, refinement, and publish patterns on the Lakehouse. * Develop reusable data processing frameworks and configuration-driven pipelines using Python and PySpark (EMR, Glue, or comparable Spark runtimes). * Build and maintain scalable orchestration workflows (e.g., Airflow) for production data delivery, including retries, historical loads, and operational runbooks. * Implement data quality checks, validation frameworks, and monitoring so data products meet defined contracts and SLAs. * Apply DataOps practices: Git-based development, CI/CD/CT for data pipelines, automated testing, and controlled promotion across environments. * Contribute to data lifecycle practices (retention, archival, disaster recovery / resiliency considerations) in partnership with platform and governance teams. * Support platform modernization and cloud migration of legacy data flows into Lakehouse patterns (Iceberg on S3, governed catalog access). * Collaborate with stakeholders to map technical designs to business processes, non-functional requirements, and consumption needs (Athena, Redshift, APIs, exports, streams). * Establish and document standards, naming/conventions, and engineering practices; participate in Agile ceremonies and cross-team delivery. * Provide technical leadership: mentor engineers, conduct design and code reviews, and continuously improve reliability, performance, and cost efficiency. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Leveraging Real time data in FSIs](https://www.wearedevelopers.com/videos/806-leveraging-real-time-data-in-fsis) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt) - [AI Model Management Life Circles: ML Ops For Generative AI Models From Research to Deployment](https://www.wearedevelopers.com/videos/1152-ai-model-management-life-circles-ml-ops-for-generative-ai-models-from-research-to-deployment) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again) - [Top-Paying Tech Jobs (with Salaries)](https://www.wearedevelopers.com/magazine/372-top-paying-tech-jobs-with-salaries)