Senior azure data engineer (id:3501)

Stafide

Amstelveen, Netherlands

1 day ago

Apply on clk.platform.jobster.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

6 years minimum

Working hours

Regular working hours

Job source

clk.platform.jobster.com

Tech stack

Airflow Microsoft Azure Cloud Database Information Engineering Data Governance Data Integration Extract Transform Load (ETL) Database Queries Distributed Computing Environment Python (Programming Language) Meta-Data Management Performance Tuning

+12 more

SQL Databases Workflow Management Systems Enterprise Data Management Data Processing Data Ingestion Azure Data Factory System Availability Pyspark Data Management Data Delivery Data Pipelines Databricks

Job description

Design, develop, and maintain scalable data pipelines and data processing solutions using Azure Databricks and Azure Data Factory.
Build, optimize, and support ETL workflows to ensure reliable and timely data ingestion and transformation across enterprise data platforms.
Manage run and operational activities, including monitoring data pipelines, identifying issues, and resolving incidents within defined SLAs.
Ensure timely and accurate data onboarding from source systems into enterprise data environments.
Develop and enhance data processing logic using Python, PySpark, and SQL to support large-scale analytics and reporting needs.
Implement and manage workflow orchestration and scheduling using Apache Airflow.
Support data governance and metadata management initiatives using the Atlas Framework.
Troubleshoot and resolve complex data pipeline, performance, and data quality issues in production environments.
Collaborate with cross-functional teams to ensure data availability, reliability, and operational stability.
Create and maintain technical documentation, operational procedures, and best practices for data engineering processes.

Requirements

6-8 years of overall experience in data engineering and enterprise data platform environments, with a strong focus on cloud-based data solutions.
Strong practical experience in building and managing data integration workflows using Azure Data Factory.
Advanced proficiency in Python for data processing, automation, and pipeline development.
Solid hands-on experience with PySpark for large-scale distributed data processing.
Strong command of SQL for data querying, transformation, and performance optimization.
Demonstrated experience in designing and supporting ETL pipelines in production environments.
Practical experience using Apache Airflow for workflow orchestration and scheduling.
Working knowledge of the Atlas Framework for data governance and metadata management.
Experience supporting data platforms in run and operations mode, including incident management and SLA adherence.
Strong analytical, troubleshooting, and problem-solving skills.
Effective communication skills and the ability to collaborate with cross-functional technical teams., * Design and implement scalable, reliable, and high-performance data engineering solutions on Azure.
Automate and optimize data processing workflows using Python and PySpark.
Proactively identify, analyze, and resolve data pipeline and performance issues.
Manage operational responsibilities while ensuring data accuracy and timely data delivery.
Work independently while taking ownership of end-to-end data engineering tasks.
Collaborate effectively with technical and non-technical stakeholders.
Develop and maintain clear technical documentation and operational runbooks.