Python Data Engineer - Azure & PySpark - SC Cleared

We're looking for a Python Data Engineer skilled in PySpark, Delta Lake, Azure services, containerized development, and Behave-based testing. You'll design and build scalable data pipelines and maintain high-quality, test-driven code in a cloud environment. What you'll do

• Build and maintain Python/PySpark pipelines for data ingestion, processing, and validation.
• Write unit and BDD tests using Behave, including mocking and patching.
• Create and optimize Delta Lake tables for reliable, performant data storage.
• Use Docker to manage consistent development, testing, and deployment environments.
• Build configurable, parameter-driven code for modular data solutions.
• Work with Azure Functions, Key Vault, and Blob Storage for cloud-based workflows.
• Collaborate with architects, data scientists, and DevOps on CI/CD and deployment.
• Tune and troubleshoot Spark jobs in production.
• Document solutions and follow cloud security and governance best practices.

• Strong Python skills with a focus on clean, test-driven code.
• Experience writing Behave tests and using mocking/patching techniques.
• Solid PySpark knowledge and experience with distributed data processing.
• Understanding of Delta Lake (ACID, schema evolution, performance).
• Hands-on experience with Docker.
• Working knowledge of Azure Functions, Key Vault, Blob Storage, and Data Lake Gen2.
• Ability to build config-driven, environment-agnostic applications.
• Experience with CI/CD tools (Azure DevOps or similar) is a plus.
• Strong problem-solving and ability to work independently.