Data Engineer

• We are looking for an experienced Azure Data Engineer to support the design, development and delivery of modern data platform solutions, with a particular focus on Microsoft Fabric migration and implementation .
• You will play a key role in delivering data engineering and migration activities, working closely with engineering teams and stakeholders to deliver high-quality, scalable and well-tested data solutions .
• The role requires strong technical expertise across Azure data technologies, alongside the ability to provide technical guidance, communicate effectively and contribute to successful delivery., * Configure data ingestion processes and solutions.
• Lead the execution of Microsoft Fabric migration activities.
• Lead and guide migration engineers and QA resources.
• Develop Python, PySpark and Spark SQL code for data validation and curation within notebooks.
• Create data integration test cases and support testing activities.
• Implement and amend data processing and worker pipelines.
• Implement data validation and curation rules.
• Translate data models and mappings into detailed technical data mappings.
• Design and implement data integrations.
• Support the implementation and execution of data migration activities.
• Perform data analysis and profiling.
• Configure Azure DevOps / GitHub environments and CI/CD processes for Azure Data Factory and Databricks code.

• Strong experience designing and delivering Azure-based data platform solutions.
• Experience with technologies including: o Microsoft Fabric o Power Apps o Azure Synapse o Azure Data Factory (ADF) o Databricks o Azure Functions o Azure App Service o Logic Apps
• o Azure Kubernetes Service (AKS) o Web Apps
• Hands-on expertise in PySpark, SQL and Python.
• Good knowledge of real-time streaming applications, preferably including Kafka, Azure Functions or Azure Service Bus.
• Strong understanding of Spark processing and performance tuning.
• Experience working with file formats and partitioning, including Parquet, JSON, XML and CSV.
• Experience with Azure DevOps / GitHub and CI/CD.
• Strong programming experience in Python, with knowledge of other programming languages beneficial.
• Experience with both synchronous and asynchronous interface approaches.
• Knowledge of data modelling, including 3NF, dimensional modelling and Data Vault 2.0.
• Experience working within Agile delivery environments.
• Ability to produce comprehensive technical documentation.
• Ability to set and manage realistic expectations around timescales, costs, benefits and measures of success.
• Power BI experience is desirable.

The role requires experience across the following skill areas:

• Data Engineering
• Software Design
• Analytics
• Data Management
• Data Modelling and Design
• Data Visualisation
• Database Design
• Release and Deployment

Nice to Have

• Experience integrating and implementing data cataloguing tools such as Azure Purview / Collibra.
• Experience implementing and integrating visualisation tools such as Power BI or Tableau.
• Experience with C# application development, including ASP.NET MVC.