Data Engineer (Python)

• Develop and maintain data ingestion, transformation, and validation pipelines using Python and PySpark.
• Implement unit and behavior-driven testing with Behave, ensuring robust mocking and patching of dependencies.
• Design and maintain Delta Lake tables for optimised query performance, ACID compliance, and incremental data loads.
• Build and manage containerised environments using Docker for consistent development, testing, and deployment.
• Develop configurable, parameter-driven codebases to support modular and reusable data solutions.
• Integrate Azure services, including:

• Azure Functions for serverless transformation logic
• Azure Key Vault for secure credential management
• Azure Blob Storage for data lake operations

About the RoleWe are looking for a Python Data Engineer with strong hands-on experience in Behave-based unit testing, PySpark development, Delta Lake optimisation, and Azure cloud services. This role focusses on designing and deploying scalable data processing solutions in a containerised environment, emphasising maintainable, configurable, and test-driven code delivery., * Proven experience in Python, PySpark, and Delta Lake.

• SC Cleared
• Strong knowledge of Behave for test-driven development.
• Experience with Docker and containerised deployments.
• Familiarity with Azure cloud services and data engineering best practices.
• Ability to deliver scalable, maintainable, and testable solutions in a fast-paced environment.