Senior Data Engineer
Role details
Job location
Tech stack
Job description
Kai is hiring a Sr. Data Engineer to join our data infrastructure team. This is a hands-on, high-ownership role at the core of what we do - we live on ingesting and processing data at very high speed, and this person owns the systems that make that possible.
You will design, build, and optimize the data pipelines and infrastructure that power Kai's security platform across some of the largest enterprises in the world. This is not an advisory role. You are expected to architect and implement, to identify what needs to change, and start working on it.
We are building a world-class data function. The person who joins now will have real influence over how that function evolves.
WHAT YOU'LL DO
-
Design and build scalable data pipelines for batch and real-time processing across Kai's agentic AI platform
-
Own and optimize high-volume data infrastructure handling hundreds of millions of entries with low latency and high reliability
-
Build and maintain data models and storage systems optimized for large-scale, high-throughput security data workloads
-
Identify bottlenecks in the current architecture and drive optimization - reduce processing time, improve reliability, and make the customer experience better
-
Lead the Terraformization of data pipelines to enable cloud-agnostic deployment across Azure, AWS, and GCP
-
Integrate and manage cloud data services, ensuring secure service principles, permissions, and cross-service connectivity
-
Collaborate closely with Backend Engineering teams on both the ingestion and consumption sides of the data pipeline
-
Ensure data quality, consistency, and reliability across all pipelines
-
Contribute to code reviews, technical documentation, and best practices
-
Bring a point of view - propose solutions, not just problems, and start building before you're asked
Requirements
-
7+ years of experience in data engineering or data platform engineering
-
Must have hands-on experience handling up to 200M+ entries in materialized views in an asynchronous manner
-
Strong proficiency in Python and SQL - these are how our systems are written
-
Strong data modeling skills - you can design schemas and storage systems that hold up at scale
-
Experience with NoSQL databases at scale - CosmosDB, MongoDB, or equivalent
-
Proven experience designing and building large-scale distributed data pipelines in both batch and streaming modes
-
Hands-on experience with Flink, Kafka, Spark, or similar stream and batch processing frameworks
-
Experience with data pipeline orchestration tools - Airflow, Temporal, or equivalent
-
Infrastructure experience - Terraform, Kubernetes, and Docker are expected, not aspirational
-
Cloud platform expertise - deep hands-on experience in at least one major cloud platform (Azure, AWS, or GCP); Azure experience strongly preferred
-
Strong communication skills - you work cross-functionally and can explain complex systems clearly
Preferred:
-
DataOps experience - ability to own data infrastructure decisions independently, reducing dependency on DevOps for pipeline deployment, permissions, and service integration
-
Experience with data systems supporting AI/ML workloads - feature stores, ML pipelines, or dataset versioning
-
Experience with DeltaLake, Apache Iceberg, or similar open table formats
-
Startup or high-growth experience - you have operated in a fast-paced environment where things change quickly, and ownership is expected
Benefits & conditions
- Generous compensation: We offer highly competitive salaries, equity options, and a supportive work environment. Your contributions will be valued and rewarded as we grow together.