> Markdown version of [/jobs/ext/1183078-software-engineer-spark-platform](https://www.wearedevelopers.com/jobs/ext/1183078-software-engineer-spark-platform). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Software Engineer, Spark Platform - **Company:** DOORDASH, INC. - **Location:** New York, NY, United States (Remote available) - **Salary:** $130,600.0 - $192,000.0 - **Contract:** Permanent contract - **Skills:** Java (Programming Language), Amazon Web Services, Distributed Systems, Python (Programming Language), Prometheus, SQL Databases, Data Logging, Autoscaling, Apache Spark, Amazon Virtual Private Cloud (VPC), Kubernetes, Information Technology, Dynatrace, Amazon Elastic Mapreduce (EMR), Databricks - **Published:** July 4, 2026 - **Apply:** https://www.indeed.com/viewjob?jk=6372b36a538baf83 ## About the Role * B.S., M.S., or PhD in Computer Science or equivalent. * 24+ years of industry experience operating production distributed systems. * Experience operating Apache Spark at scale on Amazon EMR, Databricks, or an in-house deployment - with a focus on platform operations (runtime upgrades, cluster lifecycle, shuffle, observability, multi-tenant scheduling) rather than authoring individual Spark jobs. * Hands-on experience operating production systems on Kubernetes - controllers, operators, custom resources, and the failure modes that show up in multi-tenant clusters. * Familiarity with batch or big-data schedulers (YuniKorn, Volcano, Kueue, or equivalent) and/or with the Spark-on-Kubernetes operator. * Familiarity with observability stacks (Prometheus, OpenTelemetry, distributed tracing, structured logging) and with defining SLOs and SLIs that change team behavior. * Comfort working in a cloud environment (AWS preferred) - VPC networking, instance lifecycle, spot/preemptible markets, and autoscaling primitives. * Professional experience with Python, Go, Scala, or Java; SQL fluency. * A bias toward incremental rollout, measurement, and reducing toil. * You are located or willing to relocate to the Bay Area, Seattle, or NYC. Notice to Applicants for Jobs Located in NYC or Remote Jobs Associated With Office in NYC Only ## Description As a Software Engineer on Spark Platform, you will execute across the surfaces of our in-house Spark deployment that serves the entire company. The work spans Spark runtime upgrades and performance, multi-tenant scheduling and executor bin-packing on Kubernetes, cluster lifecycle automation, and the observability and incident automation that keep the platform sustainable. You will move between layers as the work demands - picking up the next high-leverage problem regardless of where it sits - and partner closely with the rest of the team and with platform consumers across the company. You must be located in San Francisco, Sunnyvale, Seattle, or New York City for this hybrid position. You will report into the Engineering Manager on our Spark Platform team., * Build and operate an in-house Spark platform that runs at company-wide scale, spanning runtime, scheduler, reliability, and user-facing tooling. * Drive multi-tenant scheduling, executor bin-packing, and cost-aware placement that let a small team serve dozens of consumer teams. * Own pieces of cluster lifecycle automation - provisioning, upgrades, capacity changes, and node-failure handling - at a scale where these stop being manual events. * Build the observability and incident automation that make the platform debuggable end-to-end and keep on-call sustainable as the team and the workload grow. * Partner with senior engineers on shuffle, runtime, and architecture work, and grow into deeper ownership of those areas over time. ## Related Videos - [Flex your Energy: Building a Cloud-Native Platform for Renewable Energy Communities](https://www.wearedevelopers.com/videos/1990-flex-your-energy-building-a-cloud-native-platform-for-renewable-energy-communities) - [Understanding Kubernetes in a visual way](https://www.wearedevelopers.com/videos/100085-understanding-kubernetes-in-a-visual-way) - [Data binning and understanding histograms](https://www.wearedevelopers.com/videos/2086-data-binning-and-understanding-histograms) - [The Power of Purpose: Unlocking Potential and Innovation](https://www.wearedevelopers.com/videos/1110-the-power-of-purpose-unlocking-potential-and-innovation) - [Microservices: how to get started with Spring Boot and Kubernetes](https://www.wearedevelopers.com/videos/242-microservices-how-to-get-started-with-spring-boot-and-kubernetes) - [Remote Driving on Plant Grounds with State-of-the-Art Cloud Technologies](https://www.wearedevelopers.com/videos/251-remote-driving-on-plant-grounds-with-state-of-the-art-cloud-technologies) ## Related Articles - [Fully Remote Software Engineer Jobs](https://www.wearedevelopers.com/magazine/447-fully-remote-software-engineer-jobs) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Find a Developer Job: 12 Best Job Sites For Developers](https://www.wearedevelopers.com/magazine/165-find-a-developer-job-12-best-job-sites-for-developers) - [React Developer Salary [2023]](https://www.wearedevelopers.com/magazine/198-react-developer-salary-2023) - [DevOps Engineer Salary [2023]](https://www.wearedevelopers.com/magazine/203-devops-engineer-salary-2023) - [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers)