Sr. Engineer, Software - Kafka

Sr. Engineer, Software - designs, develops, and delivers scalable software solutions that enable highly available and secure systems across the enterprise. Collaborating closely with data engineers, this role architect implements complex data pipelines, real-time event-driven messaging frameworks, and distributed microservices communication using Apache Kafka and messaging queue technologies. The Engineer brings deep expertise in Kafka, Python, and Java object-oriented programming, UNIX environments, and Bash shell scripting, along with hands-on experience building RESTful API endpoints, open-source messaging platforms, and end-to-end features spanning real-time, API, and data layers. Success is measured by the quality, scalability, and reliability of software delivered, and the ability to drive technical outcomes across teams. What You ll Do Build real-time data processing pipelines that support low-latency, high-throughput, and fault-tolerant distributed systems Develop and maintain event-driven messaging frameworks using Apache Kafka or similar technologies to enable microservices to communicate data through events Automate manual processes and operational workflows using Bash shell scripting in UNIX environments Design, develop, test, and deploy high-quality software solutions using Python and Java Build scalable, high-performing RESTful APIs and microservices aligned with fault-tolerant distributed systems Implement and support event-driven and message-based architectures using Kafka or similar technologies Build scalable ETL frameworks using object-oriented Python and Java programming languages Contribute to system and application architecture, including service design, data flows, and integration patterns Participate in code reviews, agile ceremonies, and technical design discussions Experience deploying and managing Kafka in containerized environments using Docker and Kubernetes Proficiency with Kafka Connect for data integration and Schema Registry for Avro/Protobuf/JSON schema management Troubleshoot and resolve complex software issues across development and production environments Experience developing Kafka Producer and Consumer APIs using Spring Boot and Spring Kafka Proficiency building and exposing RESTful API endpoints for seamless service integration Experience contributing to or building open-source messaging platforms Develop and maintain CI/CD pipelines, automation, and deployment workflows Write and maintain clear technical documentation, including design documents and system specifications Continuously evaluate and adopt modern technologies to improve system performance, reliability, and maintainability Daily use of AI productivity tools (Claude, Cursor) is required across the software development lifecycle including design, API and microservice development, code generation, code review, testing, debugging, deployment, documentation, and production support. Design, build, and operate AI agents that automate software engineering tasks such as generating boilerplate code for new microservices, creating unit and API tests, analyzing logs, triaging incidents, and supporting CI/CD pipelines. Build and integrate AI-enabled capabilities into backend services and APIs using foundation models, prompt engineering, and retrieval-augmented generation (RAG) patterns. Implement audit logging, observability, and human-in-the-loop controls for AI agents and AI-assisted workflows running in Tier-0 production environments. What You ll Bring

Bachelor s degree in computer science, Engineering, or a related field, or equivalent practical experience 6+ years of software engineering experience, building and supporting enterprise-grade applications Strong experience working in Agile or Scrum-based development environments Demonstrated ability to work independently on complex technical problems while collaborating across teams Strong analytical, problem-solving, and communication skills Expertise in Apache Kafka including brokers, topics, partitions, consumer groups, and Kafka Streams Experience implementing Kafka ACLs and role-based access control (RBAC) Familiarity with foundation models, prompt engineering, retrieval-augmented generation (RAG), and AI agent development applied to full-stack application development, including enterprise reusable solutions Must Have Skills Proficiency with messaging and event-driven systems such as Kafka Expert-level proficiency in Python or Java object-oriented programming Strong experience with Python or Java object-oriented programming Hands-on experience with UNIX/Linux environments and expert-level Bash shell scripting for automation and pipeline orchestration Automate manual processes and operational workflows using Bash shell scripting in UNIX environments Build responsive, accessible, and performant applications, delivering end-to-end features across API and data layers Experience building scalable REST APIs and microservices Knowledge of TLS/SSL encryption and SASL authentication for securing Kafka clusters Experience with Git, GitLab, and CI/CD automation Experience working with relational databases and writing complex SQL Experience with containerization and orchestration tools such as Docker and Kubernetes Experience using observability and telemetry tools such as Splunk, AppDynamics, or Grafana Hands-on experience with AI productivity tools (Claude and Cursor or similar IDE) and working knowledge of foundation models, prompt engineering, retrieval-augmented generation (RAG), and AI agent development. Nice to Have Experience with Databricks, Snowflake, or cloud-based data platforms Network Domain experience Experience working in cloud-native environments (Azure, AWS, or GCP) Experience contributing to system architecture or platform-level design decisions

• $95,540-127,380 per year Line of Business: Service & Support Pay Range: $95,540.00 - $127,379.99 About Heidelberg Materials Heidelberg Materials is one of the world's largest suppliers of building ma…

• 7 days ago

Corporate Mining Engineer Heidelberg Materials

• Allen, TX

• $95,540-127,380 per year Line of Business: Service & Support Pay Range: $95,540.00 - $127,379.99 About Heidelberg Materials Heidelberg Materials is one of the world's largest suppliers of building ma…

• 7 days ago

© 2026 Careerjet All rights reserved