Data Engineer

The Data Engineering team is positioned in the Delivery business unit of the IT function within the Data & Automation department sitting alongside the Report Engineering, Data Governance, Data Migrations and Automation teams.

Data Engineering is a perfect blend of Data Engineers and QA Engineers that are renowned for their forward thinking and trail blazing. Exceptionally talented, supportive and personable, it's an area that's challenging, provides excellent growth opportunities and rewarding.

Role Expectations:

• Design and implement scalable data architectures using Databricks Unity Catalog, Delta Lake, and Apache Spark.
• Develop and maintain complex ETL/ELT pipelines processing terabytes of data daily.
• Develop medallion architecture (Bronze, Silver, Gold) data lakehouses with optimal performance patterns.
• Implement real-time and batch data processing solutions using Structured Streaming and Delta Live Tables.
• Design data mesh architectures with proper data governance and lineage tracking.
• Implement advanced Delta Lake features including time travel, vacuum operations, and Z-ordering
• Design and implement CI/CD pipelines for data workflows using Databricks Asset Bundles or similar tools

• 3+ years' experience in data engineering.
• Proven experience in designing and managing large-scale data solutions on Microsoft Azure.
• In-depth knowledge of setting up, configuring, and utilizing Unity Catalog for robust data governance, access control, and metadata management in a Databricks environment.
• Experience with implementing a Data Lakehouse solution with Azure Databricks, data modeling, warehousing, and real-time streaming.
• Expertise in Azure Databricks, including Delta Lake, Spark optimizations, and MLflow.
• Strong experience with Azure Data Factory (ADF) for data integration and orchestration.
• Hands-on experience with Azure DevOps, including pipelines, repos, and infrastructure as code (IaC).
• Experience working with big data technologies (Spark, Python, Scala, SQL).
• Excellent communication skills with the ability to collaborate with cross-functional teams to understand requirements, data solutions, data models and mapping documents.
• Hands-on with Azure DevOps pipelines (YAML, agents, service connections).
• Desirable: certification in Databricks Data Engineer and/or Azure Data Engineer are a plus.

• Collaborative office spaces designed for creative thinking and innovation
• Free on-site parking at our offices
• Generous holiday package - 25 days annual leave, 3 wellbeing days, and your birthday off (plus the option to buy up to 10 more days!)
• Private healthcare for all employees
• Competitive pension scheme and performance-related bonus opportunities
• Free broadband for all employees
• Life event gifts - celebrating milestones like marriages and births
• Inclusive employee networks - open to all, supporting peer connection and thought-provoking conversations
• Salary sacrifice scheme - save on dental, gym, and more
• Big retail and leisure discounts
• 3 paid volunteering days a year - because making a difference matters to us too

As a recognised Top 50 Inclusive Employer in the UK, we believe that diversity fuels innovation and success. We're committed to building a workplace that reflects the communities and customers we serve. At TalkTalk, inclusion is part of our DNA - we're all 100% human, and we've created a culture where you can truly be yourself.