> Markdown version of [/jobs/ext/1376527-senior-data-engineer-ai-platform](https://www.wearedevelopers.com/jobs/ext/1376527-senior-data-engineer-ai-platform). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Data Engineer, AI Platform - **Company:** Inc. (kai) - **Location:** San Jose, CA, United States - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Amazon Web Services, Amazon S3, Automated Storage and Retrieval Systems, Microsoft Azure, BigQuery, Program Optimization, Encodings, Data Cleansing, Information Engineering, Data Infrastructure, Extract Transform Load (ETL), Data Security, Data Systems, Data Warehousing, Distributed Data Store, Distributed Systems, Information Retrieval, Python (Programming Language), Machine Learning, Recommender Systems, Cloud Services, Search Technologies, SQL Databases, Data Streaming, Management of Software Versions, AI Infrastructure, Parquet, Google Cloud, Real Time Systems, Large Language Models, Snowflake, Apache Spark, Caching, Generative AI, Indexer, Build Management, Data Lakes, AI Platforms, Kubernetes, Low Latency, Apache Flink, Apache Kafka, Spark Streaming, Machine Learning Operations, Stream Processing, Data Pipelines, Databricks - **Published:** July 22, 2026 - **Apply:** https://www.indeed.com/viewjob?jk=bda9f95c0610550f ## About the Role * 4+ years of experience in Data Engineering or related fields * Strong experience building large-scale distributed data pipelines * Proficiency in Python and SQL; experience with Spark or similar frameworks * Experience with both batch and streaming systems (e.g., Kafka, Flink, Spark Streaming) * Experience working with cloud data platforms (AWS, GCP, Azure) * Solid understanding of data modeling, storage systems, and distributed systems * Experience supporting AI/ML workloads through data infrastructure * Strong ownership mindset and ability to operate in fast-paced environments, * Experience working with LLM-powered systems and RAG pipelines * Familiarity with vector databases and ANN search systems * Experience in data systems for AI platforms or ML infrastructure * Background in search, recommendation systems, or information retrieval ## Description We are looking for a Senior Data Engineer (AI Platform) to design and build scalable data systems that power next-generation AI and Generative AI applications. This is a senior, hands-on technical role for someone who can operate across both classical data engineering and modern AI data infrastructure - including large-scale data pipelines, vector databases, and retrieval systems for LLM-powered applications. You will work at the intersection of data engineering, AI infrastructure, and LLM systems, enabling high-quality data flow, retrieval, and storage for production-grade intelligence systems. Key Responsibilities * Design and build scalable data pipelines for batch and real-time processing * Develop and maintain data infrastructure supporting AI/ML and Generative AI systems * Build and optimize retrieval pipelines for RAG and LLM-based applications * Design and manage vector data pipelines (embedding generation, indexing, storage, retrieval) * Implement hybrid retrieval systems (BM25 + vector search) * Work closely with AI/ML teams to enable training, evaluation, and inference workflows * Develop data models and storage systems optimized for large-scale AI applications * Ensure data quality, consistency, and reliability across pipelines * Optimize systems for performance, latency, scalability, and cost * Collaborate with product, engineering, and AI teams to translate requirements into data solutions, * Batch and streaming pipelines (Spark, Flink, Kafka) * ETL/ELT design, data modeling, and data warehousing * Data quality, validation, and observability AI Data Infrastructure * Data pipelines for ML training and inference * Feature stores and dataset versioning * Data preparation for LLM and GenAI systems Vector Databases & Retrieval Systems * Milvus, Pinecone, Databricks Vector Search, FAISS * ANN algorithms (HNSW, IVF, PQ) * Hybrid retrieval (BM25 + vector search) * Embedding pipelines (text, code, image) RAG & LLM Data Systems * Retrieval pipelines for LLM applications * Context construction and ranking * Data indexing and chunking strategies Storage & Distributed Systems * Data lakes (S3, GCS, ADLS), Parquet, Delta Lake, Iceberg * Distributed systems design and scalability * Caching and low-latency data access Platforms & Infrastructure * AWS, GCP, Azure * Databricks, BigQuery, Snowflake * Kubernetes, Ray (nice to have) ## Related Videos - [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology) - [Optimizing Discovery: PostgreSQL's Role in Transforming GetYourGuide's Search](https://www.wearedevelopers.com/videos/1647-optimizing-discovery-postgresql-s-role-in-transforming-getyourguide-s-search) - [HTTP headers that make your website go faster](https://www.wearedevelopers.com/videos/1676-http-headers-that-make-your-website-go-faster) - [Parquet, Delta, Iceberg & Ducklake - An introduction for developers](https://www.wearedevelopers.com/videos/100075-parquet-delta-iceberg-ducklake-an-introduction-for-developers) - [OLAP for AI Applications and why you should care](https://www.wearedevelopers.com/videos/100212-olap-for-ai-applications-and-why-you-should-care) - [Developer Experience, Platform Engineering and AI powered Apps](https://www.wearedevelopers.com/videos/990-developer-experience-platform-engineering-and-ai-powered-apps) ## Related Articles - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift) - [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud) - [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering)