Senior Data Engineer Consultant

Keystone Solutions

Brussel, Belgium

8 days ago

Apply on be.indeed.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Working hours

Regular working hours

Job source

be.indeed.com

Tech stack

Microsoft Windows Application Programming Interfaces (APIs) Airflow Apache HTTP Server Business Intelligence Development Data Architecture Information Engineering Data Governance Data Infrastructure Extract Transform Load (ETL) Data Security Data Warehousing

+21 more

Relational Databases Linux Apache Hive Python (Programming Language) PostgreSQL Meta-Data Management Microsoft SQL Server OpenShift Windows PowerShell Power BI SQL Databases Macros Sql Optimization Beeline Apache Spark Pandas Data Lakes Debezium Real Time Data Apache Kafka Data Pipelines

Job description

Design and deploy data ingestion pipelines from heterogeneous sources: files, APIs, relational databases.
Implement pipelines for incremental and streaming ingestion.
Ensure reliability, idempotence, and traceability of each pipeline.
Develop transformation models on Bronze, Silver, and Gold layers (Medallion architecture).
Implement best practices for data historization (e.g., SCD2, upsert, append only).
Develop exports to Delta Lake.
Manage and maintain Delta Lake tables.
Guarantee transactional consistency of data in a multi-domain environment.
Implement and maintain orchestration assets.
Monitor runs, manage errors, and automate alerts.
Push technical and business metadata to the data platform.
Maintain end-to-end lineage.
Contribute to the drafting of DUA, SLA, and associated governance documents.
Develop endpoints to expose the Gold layer to BI dashboards.
Collaborate with reporting teams for BI dashboard feeding.

Requirements

Proficiency in Python.
Proficiency in SQL.
Proven experience with dbt-core (models, snapshots, macros, tests, profiles).
Good knowledge of DuckDB as an embedded analytical query engine.
Experience with Delta Lake (ACID transactions, time travel, OPTIMIZE/VACUUM).
Knowledge of relational databases: MSSQL, PostgreSQL (advanced SQL, CDC).
Experience with a data orchestrator: Dagster, Airflow, or equivalent.
Comfortable with Linux/OpenShift and using PowerShell (Windows dev environment).
Familiarity with governance tools: DataHub, OpenMetadata, or equivalent.
Experience in BI development.

Preferred Skills:

Experience with Kafka/Debezium for real-time data capture.
Knowledge of Spark (Spark SQL, Thrift Server, Beeline).
Experience with Power BI Report Server (PBIRS) or Apache Superset.
Awareness of data security in restricted access environments.
Knowledge of Lakehouse concepts (Medallion architecture), DWH, and Datalake.

Soft Skills:

Rigorous and autonomous in managing complex multi-domain pipelines.
Ability to document technical decisions and simplify for non-technical stakeholders.
Team spirit.
Curious and solution-oriented mindset., * Dagster - Level: Junior - Most recent: Any time
Data acquisition (ETL, ELT, …) - Level: Confirmed - Most recent: Any time
Data architecture - Level: Junior - Most recent: Any time
data engineering - Level: Confirmed - Most recent: Any time
Data Governance - Level: Confirmed - Most recent: Any time
DataHub - Level: Junior - Most recent: Any time
datalake - Level: Junior - Most recent: Any time
dbt - Level: Junior - Most recent: Any time
Kafka - Level: Junior - Most recent: Any time
Metadata management - Level: Confirmed - Most recent: Any time
POWER BI - Level: Junior - Most recent: Any time
Python (from a Data Engineer perspective), Pandas & Apache Spark - Level: Confirmed - Most recent: Any time
SPARK - Level: Junior - Most recent: Any time
SQL - Level: Confirmed - Most recent: Any time