Sr. Engineer, Software - Kafka

Sr. Engineer, Software designs, develops, and delivers scalable software solutions that enable highly available and secure systems across the enterprise. Collaborating closely with data engineers, this role architects and implements complex data pipelines, real-time event-driven messaging frameworks, and distributed microservices communication using Apache Kafka and messaging queue technologies. The Engineer brings deep expertise in Kafka, Python, and Java object-oriented programming, UNIX environments, and Bash shell scripting, along with hands-on experience building RESTful API endpoints, open-source messaging platforms, and end-to-end features spanning real-time, API, and data layers. Success is measured by the quality, scalability, and reliability of software delivered, and the ability to drive technical outcomes across teams. What You'll Do

• Build real-time data processing pipelines that support low-latency, high-throughput, and fault-tolerant distributed systems.
• Develop and maintain event-driven messaging frameworks using Apache Kafka or similar technologies to enable microservices to communicate data through events.
• Automate manual processes and operational workflows using Bash shell scripting in UNIX environments.
• Design, develop, test, and deploy high-quality software solutions using Python and Java.
• Build scalable, high-performing RESTful APIs and microservices aligned with fault-tolerant distributed systems.
• Implement and support event-driven and message-based architectures using Kafka or similar technologies.
• Build scalable ETL frameworks using object-oriented Python and Java programming languages.
• Contribute to system and application architecture, including service design, data flows, and integration patterns.
• Participate in code reviews, agile ceremonies, and technical design discussions.
• Experience deploying and managing Kafka in containerized environments using Docker and Kubernetes.
• Proficiency with Kafka Connect for data integration and Schema Registry for Avro/Protobuf/JSON schema management.
• Troubleshoot and resolve complex software issues across development and production environments.
• Experience developing Kafka Producer and Consumer APIs using Spring Boot and Spring Kafka.
• Proficiency building and exposing RESTful API endpoints for seamless service integration.
• Experience contributing to or building open-source messaging platforms.
• Develop and maintain CI/CD pipelines, automation, and deployment workflows.
• Write and maintain clear technical documentation, including design documents and system specifications.
• Continuously evaluate and adopt modern technologies to improve system performance, reliability, and maintainability.
• Daily use of AI productivity tools (Claude, Cursor) is required across the software development lifecycle including design, API and microservice development, code generation, code review, testing, debugging, deployment, documentation, and production support.
• Design, build, and operate AI agents that automate software engineering tasks such as generating boilerplate code for new microservices, creating unit and API tests, analyzing logs, triaging incidents, and supporting CI/CD pipelines.
• Build and integrate AI-enabled capabilities into backend services and APIs using foundation models, prompt engineering, and retrieval-augmented generation (RAG) patterns.
• Implement audit logging, observability, and human-in-the-loop controls for AI agents and AI-assisted workflows running in Tier-0 production environments., As a QA Automation Engineer, you will play a crucial role in shaping the future of AI systems by designing and implementing automated testing solutions. Your expertise will directl…

• 1 day ago, As a Software Engineer specializing in Clinical Data, you will play a crucial role in developing integration services that facilitate the movement of clinical data between electron…
• 3 days ago

• Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
• 6+ years of software engineering experience building and supporting enterprise-grade applications.
• Strong experience working in Agile or Scrum-based development environments.
• Demonstrated ability to work independently on complex technical problems while collaborating across teams.
• Strong analytical, problem-solving, and communication skills