Azure Data Engineer - SC Cleared

We are seeking a skilled Azure Data Engineer to join a public sector project focused on delivering a scalable, reusable, and collaborative data platform with the aim to eliminate repetition and maximise reuse across the enterprise., * Design, build, and maintain scalable data pipelines on Microsoft Azure

• Develop reusable data products and frameworks to reduce duplication across teams
• Implement and maintain medallion architecture (Bronze, Silver, Gold layers) for structured data processing
• Work with cross-functional teams to promote shared standards, patterns, and engineering best practices
• Contribute to open and transparent development practices, including code sharing and documentation
• Develop and optimise data workflows using notebooks and distributed processing frameworks
• Ensure data quality, reliability, and performance across the platform
• Support the evolution of a modern enterprise data platform, reducing siloed solutions

• Strong experience with Python for data engineering
• Hands-on expertise in PySpark and distributed data processing
• Advanced SQL skills for data transformation and optimisation
• Experience developing in notebook environments (e.g., Fabric, Databricks, or similar)
• Practical experience with Microsoft Fabric or Azure data services
• Solid understanding of medallion architecture and layered data design
• Experience building and maintaining ETL/ELT pipelines
• Familiarity with version control (e.g., Git) and collaborative development workflows

Desirable Skills

• Experience working in large-scale enterprise or government environments
• Knowledge of data governance, security, and compliance practices
• Familiarity with CI/CD pipelines for data engineering
• Experience with data modelling and performance tuning