SC Cleared Python Data Engineer - Azure & PySpark

We are looking for an experienced Data Engineer with strong Python skills to join a delivery-focused team working on complex data platforms in Azure. Although this is a data engineering role, the lead engineer is specifically looking for someone who is highly competent in Python, as a significant part of the work involves developing and maintaining complex, test-driven data flows. You will be responsible for building scalable data pipelines using Python and PySpark, optimizing Delta Lake storage, and deploying solutions in a containerised Azure environment. Quality, maintainability, and automated testing are key to this role.

What you'll be doing

• Designing and building robust data pipelines using Python and PySpark
• Developing complex data flows with a strong focus on clean, testable Python code
• Writing unit and BDD tests using Behave, including mocking and patching
• Creating and optimizing Delta Lake tables for performance and reliability
• Working with Docker to support consistent development and deployment
• Integrating Azure services such as Azure Functions, Key Vault, and Blob Storage
• Collaborating with engineering and DevOps teams on CI/CD pipelines
• Troubleshooting and tuning Spark jobs in production
• Following best practices around cloud security and data governance

• Strong Python development experience (essential)
• Solid data engineering background with PySpark
• Experience working with Delta Lake
• Hands-on experience with Docker
• Good knowledge of Azure data services
• Comfortable working in fast-paced, delivery-driven environments