Kafka/Flink Data Streaming Platform Engineer
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Tech stack
+44 more
Job description
We are seeking a talented and experienced Data Streaming Platform Engineer or DevOps Engineer with expertise in large cloud deployments, Kubernetes, Terraform/Ansible, Apache Kafka, and Apache Flink to join our team. In this role, you will be responsible for automating the build and maintenance scalable, high-performance data-streaming platforms that power real-time data processing and analytics.
You will work closely with data engineers, software developers, and infrastructure teams to ensure seamless data ingestion, processing, and delivery. The ideal candidate has a strong background in data-streaming technologies, cloud infrastructure, and DevOps practices, deep expertise with Kubernetes, and hands-on experience in Kafka and Flink., * Automate and Implement Data Streaming Solutions:
- Automate, deploy, and manage data streaming platforms using IaC tools such as Terraform and Ansible, onto Kubernetes or other cluster solutions
- Develop and optimize real-time data pipelines for event-driven applications and stream processing
- Configure and optimize Kafka Connectors for data pipelines
- Build and improve observability tools for large data processing pipelines, perform benchmark tests and optimize performance
- Ensure high availability, fault tolerance, and scalability of data streaming systems
- Stream Processing and Data Pipelines:
- Implement stream processing workflows using Apache Kafka, Apache Spark and Apache Flink for real-time analytics, transformations, and aggregations
- Integrate Kafka with Flink and other data processing frameworks to build end-to-end data pipelines
- Monitor and optimize the performance of streaming applications and data flows
- Infrastructure as Code (IaC) and Automation:
- Implement and maintain infrastructure as code using tools like Terraform, Ansible, or CloudFormation
- Automate deployment, scaling, and monitoring of Kafka clusters, Flink jobs, and related services
- Manage CI/CD pipelines for seamless integration and delivery of data streaming applications
- Monitoring and Performance Optimization:
- Build and configure monitoring, logging, and alerting systems (e.g., Prometheus, Grafana, ELK Stack) for Cloud, Kubernetes, Kafka and Flink ecosystems
- Troubleshoot and resolve issues related to data streaming platforms, including latency, throughput, and resource utilization
-
Optimize Kafka brokers, Flink job managers, and task managers for performance and scalability
- Collaboration and Support:
- Work closely with data engineers, software developers, and data scientists to understand data requirements and deliver robust solutions
- Provide technical guidance and support to teams using Kafka and Flink for real-time data processing
-
Document system architecture, processes, and best practices
- Cloud and On-Premise Infrastructure Management:
- Manage and optimize cloud-based infrastructure (AWS, GCP, Azure) or on-premise environments for Kafka and Flink deployments
- Ensure security, compliance, and cost-effectiveness of infrastructure
Requirements
Ansible, Software Engineering, Apache Kafka, Data Processing, Cloud Computing, DevOps, Data Analysis, Scalable System Development, Software Development, Streaming Technology, * Education:
- Bachelors or Masters degree in Computer Science, Engineering, or a related field
- Experience:
- 3+ years of hands-on experience building large scale solutions on Kubernetes
- 5+ years of hands-on experience with Unix/Linux systems and a variety of internet protocols
- 2+ years of hands-on experience with Apache Kafka (e.g., cluster management, topic design, Kafka Connect, Kafka Streams)
- 1+ years of experience with Apache Flink (e.g., stream processing, stateful computations, Flink SQL)
- 3+ years of experience in DevOps, cloud infrastructure, or platform engineering
- Proficiency in programming languages such as Java, Python, or Scala
-
Strong understanding of distributed systems, event-driven architectures, and microservices
- Technical Skills:
- Experience with Kafka Streams or KSQL for stream processing
- Knowledge of data serialization formats (e.g., Avro, Protobuf, JSON)
- Familiarity with cloud-based data streaming services (e.g., AWS MSK, Confluent Cloud, Google Pub/Sub)
- Experience with big data technologies (e.g., Hadoop, Cassandra, Elasticsearch)
-
Knowledge of infrastructure as code tools (e.g., Terraform, Ansible)
- Soft Skills:
- Strong problem-solving skills and attention to detail
- Excellent communication and collaboration skills
- Ability to work in a fast-paced, agile environment
- Passion for learning and staying up-to-date with emerging technologies, CI/CDCassandraComplianceCollaborationScalaApache KafkaHigh AvailabilityData ProcessingAzureElasticSearchOptionsHadoopPipelinesAgileAnsibleBig DataApacheScalabilityAttention To DetailAWSSalaryMicroservicesUnixDevOpsJSONProgramming LanguagesArchitectureKubernetesOptimizationDevelopersAnalyticsInfrastructureAvailabilityAutomationProgrammingIntegrationLinuxComputer ScienceSecurityEducationSoftwareJavaMaintenancePythonDesignSQLEngineeringScienceCommunicationManagement
Benefits & conditions
- Competitive salary and benefits package
- Opportunities for professional growth and development
- A collaborative and innovative work environment
- The opportunity to work with a variety of clients in various industries, solving hard problems in data & platform engineering
- Flexible working hours and remote work options
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦