ELASTIC DATA ENGINEER (APACHE KAFKA)

• The assignment is to set up and manage an event-driven data architecture in which Apache Kafka is used as a buffer and processing layer, and Elasticsearch as a data lake solution for structured and unstructured data.
• The focus is on setting up high-performance Kafka and Elastic stacks (schema management, topic management, indexing strategies) and ensuring data consistency, scalability and observability throughout the entire process from data ingestion to data consumption.
• The candidate designs, implements and documents the necessary components, makes the data available via a data service layer (REST APIs, microservices) and ensures error handling and optimization of the entire pipeline.

• For this assignment, we are looking for an experienced data engineer with a focus on event-driven systems and real-time data integration, with a thorough knowledge of Apache Kafka (Streams, Connect, Schema Registry, Avro, ksqlDB) and Elasticsearch (incl. Kibana, ingest pipelines, index lifecycle management).
• The assignment requires setting up a data service layer (REST APIs, microservices) that makes the data from Elasticsearch available for analysis, monitoring or integration with other systems, as well as experience with Docker, Kubernetes and Openshift for the deployment of the components.
• The candidate ensures data consistency and scalability throughout the entire process from data ingestion to data consumption, and is responsible for observability, error handling and optimization via CI/CD pipelines (GitLab CI) and Python scripting.

Soft skills

• Being methodical and precise in the work Precision is essential to ensuring data consistency and scalability within the Kafka and Elastic architectures.
• Autonomy and initiative The candidate can work independently, propose solutions and adapt quickly to changes in the situation.
• Documenting the work carried outThe analyses, pipelines, schedules, configurations and technical choices carried out must be documented in a clear, structured and transferable manner (reports, documentation).

Good communication

• The candidate possesses good written and verbal communication skills and can easily participate in meetings with specialists and user representatives.
• Being able to work in a group, flexibility of schedule
• The candidate can set priorities within a complex environment with multiple data pipelines, processes and technical dependencies, and has the necessary flexibility of schedule.
• Customer-oriented and solution-oriented attitude
• The candidate is stress-resistant, creative, constructive and solution-oriented, and works with an eye for the needs of the users., * Apache Kafka (Streams, Connect, Schema Registry, Avro, ksqlDB)
• CI/CD tools (GitLab CI)
• Data Service Layer (REST API’s, microservices)
• Docker, Kubernetes and Openshift
• Elasticsearch (Kibana, ingest pipelines, index lifecycle management)
• Indexing Strategies and Data Consistency (Elasticsearch)
• Observability, error handling and optimization of data pipelines
• PostgreSQL
• Python
• Schema management and topic management (Kafka)