Engineer - Data Ingestion

Community Of
Municipality of Madrid, Spain
2 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English

Job location

Municipality of Madrid, Spain

Tech stack

Application Integration Architecture
Azure
Continuous Integration
Data Dictionary
ETL
Minimum Data Set
Data Streaming
Transaction Data
Web Services
File Transfer Protocol (FTP)
Data Ingestion
Azure
System Availability
Spark
Data Layers
Data Lake
Software Version Control
Data Pipelines
Databricks

Requirements

Act in a hands-on capacity to implement, operate, and stabilize data ingestion and preparation processes, ensuring data is properly structured and ready for publication in the Common Semantic Layer (CSL), in alignment with the semantic data dictionary defined by the external provider. Role Mission * Design, implement, and operate data ingestion pipelines (batch and/or streaming) from multiple sources (e.g., SEP, NUEK, fraud systems, webservices). * Ensure data availability in the Stage layer as a faithful replica of source systems. * Prepare and transform datasets into Common and Refined layers, following CSL standards. * Apply business transformation rules aligned with data mappings (e.g., OneGMR, Churn). * Work closely with the external provider to ensure adherence to the semantic data dictionary and mapping requirements. * Identify, troubleshoot, and resolve ingestion issues (failures, delays, inconsistencies). * Ensure minimum data quality standards (completeness, consistency, standardization). * Document pipelines, transformations, and data sources. * Support dependency resolution with local teams and external providers. * Map technical dependencies across pipelines, sources, and deliverables. Technical Skills * Hands-on experience with Databricks / Apache Spark, Azure Data Factory (or similar ETL/ELT tools), and Advanced SQL. * Experience with data ingestion processes (ETL/ELT). * Knowledge of Data Lake / Lakehouse architectures. * Experience integrating APIs, file-based ingestion (SFTP/B2B), and third-party systems. * Basic knowledge of CI/CD and version control (Azure DevOps or similar). Nice to Have * Experience in multi-country / multi-source environments. * Understanding of data modeling and semantic layers. * Experience with financial / transactional data. * Familiarity with tools such as Databricks, Purview, or similar. Profile * Strong hands-on mindset. * Proactive problem solver. * Autonomous execution capability. * Effective communication with technical and functional teams. What we offer * Medical insurance. * Dental insurance. * Wellhub program. * Life insurance. * Annual healthcare. * Transportation allowance. * Meal voucher/food voucher. * Profit sharing program. * Access to self-development programmes. * Hybrid Collaborative environment: Successful candidate must be prepared to work 60% on site.

Apply for this position