Software Engineer-Data Optimization in San Mateo

In this role, you'll develop scalable data pipelines, backend services, and user-facing tools that support dataset creation, embedding search, and ML data optimization. While primarily focused on backend and data engineering, you'll also contribute to frontend features that improve workflow management and user experience.

The ideal candidate is a strong Python engineer who thrives in data-intensive environments and enjoys building scalable systems that support machine learning and large-scale data processing.

As a Software Engineer - Autonomy Behavior ML Data Optimization Team, you'll:

• Design, build, and maintain Airflow pipelines that orchestrate end-to-end dataset creation, refresh, and management workflows.
• Develop scalable Python/FastAPI services supporting embedding search, bulk operations, and dataset management.
• Build and optimize distributed data processing pipelines using Ray for embedding , clustering, FAISS index creation, cache , and large-scale dataset processing.
• Improve pipeline reliability, observability, and performance through monitoring, optimization, and operational tooling.
• Develop dashboards and visualization tools that provide insight into datasets, pipeline health, and system performance.
• Contribute to React-based administrative tools and workflow management features that improve dataset management and search capabilities.
• Partner with ML researchers and cross-functional engineering teams to integrate new embedding technologies and deliver scalable software solutions.

• 3+ years of professional software engineering experience with a focus on data processing and pipeline engineering.
• Experience designing, building, and optimizing distributed data processing pipelines at scale (e.g., ETL/ELT) using technologies like Spark, Databricks, AWS EMR, AWS Batch, or Ray Core/Data.
• Strong proficiency in Python with experience building production data pipelines and web services (FastAPI, Uvicorn, or similar async frameworks).
• Experience building and maintaining data visualization dashboards, with proficiency in SQL, and familiarity with PySpark/Scala for large-scale data manipulation.
• Experience with workflow orchestration tools (Airflow, Prefect, Dagster, or similar) for managing complex multi-step data processing pipelines.
• Familiarity with vector similarity search and indexing technologies (FAISS, Annoy, ScaNN, Milvus, or similar). Experience working with cloud infrastructure (AWS S3, EC2) and container orchestration (Kubernetes, Docker). Strong understanding of data structures, algorithms, and performance optimization for data-intensive workloads.
• Familiarity with React.js or a comparable modern frontend framework for building interactive data-driven applications.
• Excellent communication and collaboration skills; ability to work effectively with ML researchers, data engineers, and product stakeholders.

• Pre-tax commuter benefits
• Employer-subsidized healthcare benefits
• Flexible Spending Account for healthcare-related costs
• Employer covers all costs for short- and long-term and life insurance
• 401(k) package
• PTO and Sick Days