> Markdown version of [/jobs/ext/3023359-software-engineer-frontier-data-products](https://www.wearedevelopers.com/jobs/ext/3023359-software-engineer-frontier-data-products). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Software Engineer, Frontier Data Products - **Company:** Recruiting from Scratch - **Location:** San Francisco, CA, United States - **Experience:** Experienced - **Salary:** $130,000.0 - **Contract:** Permanent contract - **Skills:** Application Programming Interfaces (APIs), Artificial Intelligence, Airflow, Amazon Web Services, Cloud Computing, Software Debugging, Programming Tools, Distributed Systems, Fault Tolerance, Python (Programming Language), PostgreSQL, Machine Learning, Language Modeling, Software Architecture, Cadence Virtuoso, Software Engineering, Systems Architecture, Systems Integration, Workflow Management Systems, Scripting, System Availability, State Machines, Reliability of Systems, Backend, Build Management, Data Generation - **Published:** September 21, 2026 - **Apply:** https://www.careerbuilder.com/job-details/software-engineer-frontier-data-products-san-francisco-ca--2ec653bc-6556-4f11-9f2a-2820abc887dd ## About the Role * 4+ years of backend software engineering experience building production distributed systems. * Strong experience designing scalable backend services using Python and PostgreSQL. * Deep understanding of distributed systems concepts including asynchronous workflows, queues, retries, idempotency, and long-running processes. * Proven experience designing reliable systems with strong observability and operational excellence. * Strong system design skills with the ability to balance simplicity, scalability, and maintainability. * Experience owning production services from design through deployment and ongoing operation. * Comfortable working in highly ambiguous environments with significant ownership and autonomy. * Excellent communication skills and ability to collaborate across engineering, product, and operations teams. * Strong engineering judgment with an emphasis on building systems that remain maintainable as they scale. Preferred * Experience with workflow orchestration frameworks such as Temporal, Cadence, Airflow, or similar technologies. * Background building orchestration platforms, distributed state machines, or event-driven backend systems. * Experience integrating AI or machine learning inference into production applications. * Familiarity with AWS cloud infrastructure and modern backend architecture. * Experience building internal platforms, developer tooling, or infrastructure supporting high-throughput production systems. * Strong understanding of fault tolerance, observability, and debugging distributed production environments. * Startup experience building greenfield systems with significant architectural ownership. * Passion for solving difficult infrastructure problems that directly impact customer-facing AI products., Amazon Web Services (AWS), Application Programming Interface (API), Artificial Intelligence (AI), Auditing, Building Systems, Cadence, Candidate Sourcing, Cloud Computing, Communication Skills, Continuous Improvement, Customer Relations, Data Quality, Debugging Skills, Dental Insurance, Distributed Computing, Engineering, High Throughput, Laundry, Machine Learning, Machine Tool, Machining Operations, Modeling Languages, Needs Assessment, Operations Research, PostgreSQL, Problem Solving Skills, Production Control, Production Management, Production Systems, Programming Tools, Python Programming/Scripting Language, Reconciliation, Reimbursement, Software Engineering, Startup, System Architecture, Systems Reliability, Team Lead/Manager, Team Player, Training/Teaching, Vision Plan ## Description As AI systems become increasingly dependent on expert judgment, the company is building the orchestration platform that coordinates complex, long-running workflows between humans and AI models. This team owns the systems that sit directly between customer requests and production outputs, ensuring every stage of the data generation process is reliable, observable, and scalable. Engineers on this team will help define the architecture for one of the company's most strategic product areas while solving distributed systems challenges at the intersection of AI, workflow orchestration, and backend infrastructure. What You Will Do * Design and build backend services that orchestrate complex, multi-stage workflows involving AI models and human experts. * Develop distributed state management systems capable of coordinating long-running jobs that evolve over hours or days. * Build workflow orchestration primitives including retries, failure recovery, idempotency, auditing, and state reconciliation. * Integrate large language model inference into production systems while maintaining reliability, observability, and human oversight. * Design scalable APIs and internal tooling that enable Product, Operations, and Machine Learning teams to manage production workflows. * Build highly reliable backend infrastructure where correctness and recoverability are as important as system availability. * Debug complex distributed systems spanning asynchronous processing, model inference, and human review workflows. * Own services throughout their lifecycle, including architecture, implementation, production operations, monitoring, and continuous improvement. * Partner closely with engineering, product, operations, and AI research teams to translate ambiguous requirements into elegant technical systems. * Help define the long-term architecture for one of the company's core AI infrastructure platforms. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Shipping Faster with Less: Render on Cloud Hosting, AI Workloads, and the Future of DevOps](https://www.wearedevelopers.com/videos/1894-shipping-faster-with-less-render-on-cloud-hosting-ai-workloads-and-the-future-of-devops) - [JavaScript? No. Java Scripts! - Scripting with Java](https://www.wearedevelopers.com/videos/2094-javascript-no-java-scripts-scripting-with-java) - [Developing the Backend with Stefan Lingler, CTO at Shpock](https://www.wearedevelopers.com/videos/100360-developing-the-backend-with-stefan-lingler-cto-at-shpock) - [Intermediate Bitcoin Script](https://www.wearedevelopers.com/videos/25-intermediate-bitcoin-script) - [Nest.js - TypeScript in the backend can also be clean](https://www.wearedevelopers.com/videos/1033-nest-js-typescript-in-the-backend-can-also-be-clean) ## Related Articles - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again) - [Is Software Engineering Over-Saturated?](https://www.wearedevelopers.com/magazine/418-is-software-engineering-over-saturated) - [Why Upskilling And Reskilling is Important For Developers](https://www.wearedevelopers.com/magazine/428-why-upskilling-and-reskilling-is-important-for-developers) - [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)