Senior Data Engineer

We are looking for an experienced Senior Data Engineer to design, build, optimise, and maintain scalable data platforms and data pipelines across modern cloud and enterprise environments. The role will involve working with architects, analysts, data scientists, product owners, and client stakeholders to deliver robust, secure, and high-performing data solutions. The successful candidate will have strong hands-on engineering experience across cloud data platforms, data integration, data modelling, ETL/ELT pipelines, automation, DevOps, and data quality. They will be expected to take ownership of complex data engineering workstreams, provide technical leadership to junior engineers, and ensure solutions are delivered in line with agreed architecture, security, governance, and operational standards. This is a senior delivery role requiring both technical depth and practical delivery experience in complex, regulated, or enterprise environments.

RequirementsKey Responsibilities:

Data Engineering and Platform Delivery:

• Design, develop, test, deploy, and maintain scalable data pipelines using modern cloud-native and enterprise data engineering tools.
• Build robust ETL/ELT processes to ingest, transform, validate, and publish data from multiple structured and unstructured sources.
• Work with batch, near-real-time, and streaming data processing patterns where required.
• Develop reusable data engineering components, frameworks, templates, and automation scripts.
• Support the development of data lakes, lakehouses, data warehouses, operational data stores, and analytics platforms.
• Optimise data pipelines for performance, cost, reliability, scalability, and maintainability.
• Ensure data engineering solutions are production-ready, supportable, monitored, and documented.

Cloud and Technology Implementation:

• Build data solutions on cloud platforms such as Microsoft Azure, AWS, or Google Cloud, with strong preference for Azure experience.
• Work with technologies such as AWS Glue, Azure Data Factory, Synapse Analytics, Databricks, Fabric, Data Lake Storage, SQL, Python, Spark, Power BI, Snowflake, dbt, Airflow, Kafka, or equivalent tooling.
• Implement data ingestion from APIs, databases, files, SaaS platforms, event streams, and third-party systems.
• Use infrastructure-as-code, CI/CD pipelines, and automated deployment approaches where appropriate.
• Collaborate with DevOps and platform teams to ensure secure and reliable deployment of data workloads.

Data Modelling, Quality, and Governance:

• Design and implement appropriate data models, including dimensional models, data vault, star schemas, and curated analytical datasets.

• Apply data quality rules, validation checks, reconciliation controls, and exception handling.

• Support metadata management, lineage, data cataloguing, and governance requirements.

• Ensure solutions comply with data security, privacy, access control, retention, and audit requirements.

• Work with business and technical stakeholders to define data definitions, mapping rules, transformation logic, and acceptance criteria. Technical Leadership:

• Lead data engineering workstreams from discovery through to design, build, test, deployment, and support transition.

• Provide technical guidance, mentoring, and code reviews for junior and mid-level data engineers.

• Translate high-level architecture into practical engineering designs and delivery tasks.

• Contribute to technical decision-making, estimation, planning, and risk management.

• Identify engineering risks, dependencies, blockers, and improvement opportunities early.

• Promote engineering standards, reusable patterns, documentation, and good development practices.

Stakeholder and Delivery Management:

• Work closely with product owners, business analysts, architects, testers, data analysts, and client stakeholders.
• Participate in agile ceremonies including sprint planning, daily stand-ups, backlog refinement, reviews, and retrospectives.
• Support discovery workshops, requirements analysis, technical design sessions, and show-and-tell demonstrations.
• Produce clear technical documentation, data flow diagrams, mapping specifications, deployment guides, and support documentation.
• Support transition into live service, including knowledge transfer, runbooks, monitoring, incident response, and handover to support teams., * Data pipeline designs and implemented ETL/ELT workflows.
• Data ingestion, transformation, validation, and publishing components.
• Data models, schemas, mapping documents, and transformation specifications.
• Automated deployment pipelines and environment configuration.
• Data quality checks, reconciliation reports, and exception handling processes.
• Technical design documentation and data flow diagrams.
• Runbooks, operational guides, and support handover documentation.
• Performance optimisation recommendations and implemented improvements.
• Knowledge transfer sessions and mentoring for internal teams.

• Strong experience as a Data Engineer or Senior Data Engineer in enterprise or cloud environments.

• Strong SQL skills, including query optimisation, stored procedures, data modelling, and performance tuning.

• Strong Python or PySpark experience for data processing, automation, and transformation logic.

• Experience building ETL/ELT pipelines using tools such as AWS Glue, Azure Data Factory, Databricks, Synapse, Fabric, dbt, Airflow, Informatica, Talend, or similar.

• Experience working with cloud data platforms, preferably Microsoft Azure.

• Experience with data lake, lakehouse, data warehouse, and analytical platform architectures.

• Good understanding of batch processing, incremental loads, CDC, API ingestion, and file-based ingestion patterns.

• Experience with data validation, reconciliation, error handling, and data quality controls.

• Experience using Git-based source control and CI/CD practices.

• Understanding of security, access control, encryption, data privacy, and environment management. Essential Delivery Experience:

• Experience delivering production-grade data platforms or pipelines in complex organisations.

• Ability to work across the full delivery lifecycle from requirements and design through to build, test, release, and support.

• Experience working in agile delivery teams.

• Ability to produce clear technical documentation and explain technical concepts to non-technical stakeholders.

• Experience leading technical workstreams or mentoring other engineers.

• Strong analytical, problem-solving, and troubleshooting skills.

• Ability to work independently, manage priorities, and take ownership of outcomes.

Desirable Skills and Experience:

• Microsoft Azure certifications, such as Azure Data Engineer Associate or equivalent.
• Experience with AWS Glue, Microsoft Fabric, Azure Synapse Analytics, Azure Data Lake, Azure SQL, Azure Functions, Logic Apps, Event Hubs, or Azure Purview.
• Experience with Databricks, Delta Lake, Spark, Unity Catalog, MLflow, or lakehouse patterns.
• Experience with Snowflake, Redshift, BigQuery, or other cloud data warehouse platforms.
• Experience with dbt, data transformation frameworks, or analytics engineering practices.
• Experience with streaming technologies such as Kafka, Event Hubs, Kinesis, or Pub/Sub.
• Experience with Power BI semantic models, reporting datasets, or analytical consumption layers.
• Experience with data governance, data lineage, metadata management, master data management, or data cataloguing.
• Experience with Terraform, Bicep, ARM templates, Docker, Kubernetes, or other infrastructure and deployment tooling.

Behavioural Competencies:

• Strong ownership mindset with the ability to take accountability for technical delivery.

• Clear and confident communicator, able to engage with technical and business stakeholders.

• Pragmatic problem solver who balances engineering quality with delivery timelines.

• Collaborative team player who supports others and contributes to shared outcomes.

• Detail-oriented, with strong focus on data accuracy, quality, and operational reliability.

• Comfortable working in fast-paced, multi-disciplinary, and multi-supplier environments.

• Able to challenge constructively and recommend practical improvements.

• Committed to continuous learning and keeping up to date with modern data engineering practices. Typical Deliverables, * Degree in Computer Science, Data Engineering, Software Engineering, Information Systems, Mathematics, Statistics, or a related discipline, or equivalent professional experience.

• Relevant cloud or data engineering certifications are desirable but not mandatory.

Experience Level:

• 5+ years of experience in data engineering, software engineering, or data platform delivery.
• At least 2+ years of hands-on experience delivering cloud-based data engineering solutions.
• Prior experience in a senior, lead, or workstream ownership role is preferred.