Data Engineer - Azure Databricks / Python / Insurance - Outside IR35
gb
MYO Talent
Cardiff, United Kingdom
2 days ago
Role details
Contract type
Temporary to permanent Employment type
Full-time (> 32 hours) Working hours
Regular working hours Languages
EnglishJob location
Remote
Cardiff, United Kingdom
Tech stack
API
Unit Testing
Azure
Data Cleansing
Data Governance
Data Transformation
Python
Metadata
Data Logging
Data Ingestion
Data Lake
PySpark
Databricks
Job description
- Define and build a data lake architecture on Azure (ADLS + Databricks)
- Implement layered data model (bronze / silver / gold) for ingestion, processing, and reporting
- Develop ingestion patterns from API sources (batch and streaming)
- Establish data governance, metadata, and access controls
- Define storage structures, naming conventions, and schema design
- Develop API connectors and ingestion pipelines using Python
- Implement integration services feeding into the Databricks environment
- Build data transformation logic using PySpark / Python (data cleansing, standardisation, validation)
- Configure monitoring, logging, and alerting
- Execute unit and integration testing across pipelines
Requirements
- Azure
- Databricks
- Python
- Financial Services sector experience - Ideally Insurance / General or Life Insurance