Senior Data Engineer

Ubique Systems

Belgium

18 days ago

Apply on www.adzuna.be

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Working hours

Regular working hours

Job source

www.adzuna.be

Tech stack

Airflow Amazon Web Services Unit Testing Microsoft Azure C Sharp (Programming Language) Continuous Integration Data Architecture Information Engineering Extract Transform Load (ETL) Python (Programming Language) Team Foundation Server Operational Databases

+13 more

Systems Integration Unstructured Data Virtual Studio Data Processing Postman Apache Spark Pyspark Infrastructure Automation Frameworks Data Management Terraform Data Pipelines Api Management Databricks

Job description

We are seeking a Senior Data Engineer with strong hands-on experience in Python, PySpark, Databricks, and AWS. The candidate will be responsible for developing scalable data pipelines, integrating and transforming structured/unstructured data, ensuring data quality, and supporting production data platforms., * Design, develop, and maintain scalable data pipelines and ETL/ELT processes.

Ingest, integrate, map, cleanse, transform, and validate structured and unstructured data.
Develop data processing solutions using Python 3, PySpark, and Databricks.
Build and manage workflows using Apache Airflow.
Work with AWS cloud services for data engineering solutions.
Write technical specifications, unit tests, and maintain technical documentation.
Participate in data architecture and technical design discussions.
Implement data engineering and data management best practices.
Develop and maintain CI/CD pipelines using Azure DevOps/TFS/VSTS.
Use Terraform for infrastructure automation where required.
Perform API testing using Postman/Insomnia.
Monitor production pipelines, troubleshoot issues, and provide maintenance/support.
Ensure timely delivery and adherence to project deadlines.