Data Engineer

Onesource Consulting

Brussel, Belgium

2 days ago

Apply on www.adzuna.be

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Working hours

Regular working hours

Languages

Dutch

Job source

www.adzuna.be

Tech stack

Java (Programming Language) Application Programming Interfaces (APIs) Artificial Intelligence Big Data Databases Data Infrastructure Extract Transform Load (ETL) DevOps Python (Programming Language) Azure Data Lake Data Streaming Pulumi

+10 more

Data Processing Apache Spark Data Lakes Pyspark Data Analytics Restful APIs Terraform Data Pipelines Databricks Programming Languages

Job description

As a Data Engineer & DevOps Expert within the Data Platform department of Client, you will play a central role in developing and managing innovative data-driven solutions that contribute to a more efficient and transparent government. As a data engineer/DevOps, you will be responsible for designing, implementing, and maintaining scalable data processing solutions that meet the needs of AI projects. You optimize and manage data pipeline lifecycles and ensure data consistency, quality and reliability. You will work within the Central Team of the department in close collaboration with the AI & Analytics sub team of Data Insights within Client.

Requirements

The development of ready-to-use AI and analytics building blocks, as part of the new AI & Data Analytics Platform, enables users to extract valuable insights from data faster and easier. This forms the basis for a proactive, high-quality, efficient and data-driven service.
Exploring and validating new data and AI technologies through feasibility trajectories, where temporary, multidisciplinary teams work in short iterations according to the “fail fast” principle. The emphasis is on quickly testing hypotheses and demonstrating potential.
Demonstrable experience as a Data Engineer
Demonstrable experience with databases
Demonstrable experience with programming languages such as Python, Java or Scala for developing data streams and ETL processes
Demonstrable experience in designing and implementing Extract, Transform, Load (ETL) processes
Language requirement: Dutch at European CEFR - level C2., * Demonstrable experience as a Data Engineer
Demonstrable experience with programming languages such as Python, Java or Scala for developing data streams and ETL processes
Proven work experience with Databricks
Upload document NDA signed.
Language requirement: Dutch at European CEFR
Proven experience in designing and implementing Extract, Transform, Load (ETL) processes
Proven experience with data lake solutions (Azure Data Lake or Delta Lake).
Demonstrable experience with databases
Proven experience with DBT
Proven experience with ETL tools around Databricks to build reliable and scalable pipelines.
Demonstrable experience in linking data sources (databases, APIs, …) to the AI Analytics platform and setting up data pipelines using standardized building blocks
Demonstrable experience in developing IAC in terraform and/or pulumi
Proven experience with REST API
Proven experience with Spark or PySpark for processing large datasets