Senior Data Engineer

Inc. (kai)
San Jose, CA, United States
20 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
7 years minimum
Working hours
Regular working hours
Job source

Tech stack

Artificial Intelligence Airflow Amazon Web Services Apache HTTP Server Microsoft Azure Batch Processing Cloud Database Code Review Information Engineering Data Infrastructure Data Systems DevOps
+21 more
Distributed Data Store Python (Programming Language) MongoDB NoSQL DataOps SQL Databases Management of Software Versions Data Processing Google Cloud Cloud Platform System Real Time Systems Apache Spark Build Management Kubernetes Low Latency Apache Flink Apache Kafka Machine Learning Operations Terraform Data Pipelines Docker

Job description

Kai is hiring a Sr. Data Engineer to join our data infrastructure team. This is a hands-on, high-ownership role at the core of what we do - we live on ingesting and processing data at very high speed, and this person owns the systems that make that possible.

You will design, build, and optimize the data pipelines and infrastructure that power Kai’s security platform across some of the largest enterprises in the world. This is not an advisory role. You are expected to architect and implement, to identify what needs to change, and start working on it.

We are building a world-class data function. The person who joins now will have real influence over how that function evolves.

WHAT YOU’LL DO

  • Design and build scalable data pipelines for batch and real-time processing across Kai’s agentic AI platform

  • Own and optimize high-volume data infrastructure handling hundreds of millions of entries with low latency and high reliability

  • Build and maintain data models and storage systems optimized for large-scale, high-throughput security data workloads

  • Identify bottlenecks in the current architecture and drive optimization - reduce processing time, improve reliability, and make the customer experience better

  • Lead the Terraformization of data pipelines to enable cloud-agnostic deployment across Azure, AWS, and GCP

  • Integrate and manage cloud data services, ensuring secure service principles, permissions, and cross-service connectivity

  • Collaborate closely with Backend Engineering teams on both the ingestion and consumption sides of the data pipeline

  • Ensure data quality, consistency, and reliability across all pipelines

  • Contribute to code reviews, technical documentation, and best practices

  • Bring a point of view - propose solutions, not just problems, and start building before you’re asked

Requirements

  • 7+ years of experience in data engineering or data platform engineering

  • Must have hands-on experience handling up to 200M+ entries in materialized views in an asynchronous manner

  • Strong proficiency in Python and SQL - these are how our systems are written

  • Strong data modeling skills - you can design schemas and storage systems that hold up at scale

  • Experience with NoSQL databases at scale - CosmosDB, MongoDB, or equivalent

  • Proven experience designing and building large-scale distributed data pipelines in both batch and streaming modes

  • Hands-on experience with Flink, Kafka, Spark, or similar stream and batch processing frameworks

  • Experience with data pipeline orchestration tools - Airflow, Temporal, or equivalent

  • Infrastructure experience - Terraform, Kubernetes, and Docker are expected, not aspirational

  • Cloud platform expertise - deep hands-on experience in at least one major cloud platform (Azure, AWS, or GCP); Azure experience strongly preferred

  • Strong communication skills - you work cross-functionally and can explain complex systems clearly

Preferred:

  • DataOps experience - ability to own data infrastructure decisions independently, reducing dependency on DevOps for pipeline deployment, permissions, and service integration

  • Experience with data systems supporting AI/ML workloads - feature stores, ML pipelines, or dataset versioning

  • Experience with DeltaLake, Apache Iceberg, or similar open table formats

  • Startup or high-growth experience - you have operated in a fast-paced environment where things change quickly, and ownership is expected

Benefits & conditions

  • Generous compensation: We offer highly competitive salaries, equity options, and a supportive work environment. Your contributions will be valued and rewarded as we grow together.

About the company

Kai is the AI company rebuilding cybersecurity for the machine-speed era. Founded by second time founders and trusted by Fortune 500 enterprises, Kai is building a future where security has no categories, no silos, and no human speed bottlenecks. The Kai Agentic AI Platform replaces fragmented, human-limited workflows with agentic AI systems that continuously contextualize, assess, reason, and execute security work at machine speed - making human defenders, superhuman.

Why Join Kai

  • Well-funded: With $125M raised, we have the capital, runway, and resolve to rebuild cybersecurity from first principles.

  • Proven: We’ve earned the trust of Fortune 500 and Global 1000 companies, and we’re just getting started. Their confidence in Kai reflects what we’ve built: an AI-powered cybersecurity platform that performs at the scale and speed the enterprise demands.

  • Experienced founders: Our founding team consists of second-time entrepreneurs, each with over 20 years of experience in the cybersecurity industry. Their proven expertise and vision drive our ambitious goals.

  • World-class leadership team: Our Heads of AI, Engineering, and Product bring extensive experience from some of the world’s most influential companies, ensuring top-tier mentorship, direction, and vision.

  • Frontier AI Applied Research Team: Our researchers operate at the leading edge of agentic AI systems, translating breakthrough capabilities into real-world cybersecurity applications.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.indeed.com

Good distractions

Talks and stories from around this role — technically off-topic, practically not.

3:03 min

Career evolution in data engineering and AI platforms

Maria Apazoglou · Coffee With Developers

2:37 min

Comparing traditional SQL tables versus NoSQL non-tabular databases

Stanimira Vlaeva · JS Congress

2:07 min

Inspecting default bridge architectures and custom Docker networks

Oliver Seitz Oliver Seitz · WWC 2025

2:17 min

Mapping the maturity roadmap for scaled devops adoption

Dominik Krichbaum Dominik Krichbaum · WWC Europe 2026

3:09 min

Balancing data science skillings alongside systems engineering rigor

Nico Schmidt · LIVE

3:16 min

Terminology differences between relational and NoSQL databases

Tim Faulkes · LIVE

Videos

See all

Related articles

See all