Staff Data + DevOps Engineer - Pre-IPO Network Security Company

Working on data pipelines for various ML/AI products.

• Interfacing with DevOps teams to build out the pipelines.

• Interfacing with Data Scientists and Software engineers to implement new features.

• Improving monitoring and debuggability of the existing pipelines.

At least 5 years (preferably more) of professional experience as a Data Engineer.

• Experience in setting up multit-enant Spark & Kafka clusters.

• Experience running infrastructure automation using Jenkins/Ansible/Terraform or similar pipelining tools.

• Strong skills in Python.

• Experience in cleaning, processing, managing, analyzing & searching vast collections of data.

• Experience in Apache Spark, Apache Kafka and related toolset.

• Some experience with database technologies, knowledge on when to use a NoSQL, a regular SQL DB or a cloud SQL DB.

• Team player with the ability to handle multiple projects at the same time.

Nice to have:

• Experience with Go.

• Experience with Kubernetes.

• Experience with Scala.

The company has raised a large amount of funding from top investors and is expecting an IPO in the next 1-2 years. It has almost 1,000 employees (roughly double what it was 4 years ago), and 250 engineers, and is rapidly growing. In this position you would be spending about 50% of your time on data engineering and 50% on devops engineering. The role is focused on building and scaling ML/AI data infrastructure. You would design and improve large-scale data pipelines for AI/ML products, working closely with data scientists, software engineers, and DevOps teams to deliver new capabilities. Responsibilities include managing infrastructure automation, improving monitoring and debugging, and supporting technologies such as Spark, Kafka, Kubernetes, and cloud/database systems. You would primarily programming in Python, with some Go. The company will pay $200-250k in salary, plus equity.