> Markdown version of [/jobs/ext/3301815-principal-data-engineer](https://www.wearedevelopers.com/jobs/ext/3301815-principal-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Principal Data Engineer - **Company:** MEDIDATA - **Location:** London, UK - **Contract:** Permanent contract - **Skills:** Java (Programming Language), Artificial Intelligence, Amazon Web Services, Apache HTTP Server, Automation of Tests, Behavior-Driven Development, Code Generation, Program Optimization, Continuous Integration, Information Engineering, Data Integrity, Data Migration, Data Warehousing, Cursor (Graphical User Interface Elements), Software Design Patterns, Programming Tools, Distributed Data Store, Electronic Data Capture, Fault Tolerance, Interoperability, Online Analytical Processing, Open Data Protocol, Operational Data Store, Online Transaction Processing, Performance Tuning, Query Optimization, Software Engineering, SQL Databases, Data Streaming, Systems Integration, Test Case, Test-Driven Development (TDD), Data Ingestion, GitHub Copilot, Snowflake, Database Optimization, Reliability of Systems, Change Data Capture, Backend, Git, Microsoft Fabric, Data Lakes, Information Technology, Data Lineage, Apache Kafka, Data Management, Machine Learning Operations, Code Restructuring, Software Version Control, Data Pipelines, GXP - **Published:** September 4, 2026 - **Apply:** https://uk.indeed.com/viewjob?jk=218e74218638fa0b ## About the Role * Education & Experience: Bachelor's or Master's degree in Computer Science, Data Science, Software Engineering, or equivalent practical experience, alongside proven years of dedicated professional experience in enterprise data engineering and architecture. * Data Warehousing & Data Lake Mastery: Expert-level mastery of SQL (OLAP/OLTP) and enterprise cloud data platforms (specifically Snowflake), combined with practical experience architecting open data lake structures using Apache Iceberg. * Software Engineering & Streaming: Demonstrated expertise building production backend services in Java or Scala on AWS, alongside deep familiarity with real-time streaming architectures (e.g., Apache Kafka) and modern data engineering design patterns. * AI-Augmented Engineering Proficiency: Active, practical experience integrating AI developer tools (e.g., GitHub Copilot, Cursor) into daily workflows to accelerate SQL query generation, code refactoring, automated testing, and technical documentation drafting. * Engineering Rigor & Methodologies: Proven command of Git revision control, CI/CD pipeline automation, and TDD/BDD practices, augmented by AI-driven test case generation and quality checks. * Domain & Regulatory Awareness: Strong foundational understanding of clinical trial data workflows (e.g., EDC architectures), healthcare data models, and life science compliance standards (GxP, HIPAA, GDPR), with the ability to apply AI/ML tools for schema mapping and zero-loss data integrity verification. ## Description Reporting to a Director of Engineering, as a Principal Data Engineer / Architect, you will lead the strategic vision and hands-on execution of our next-generation Object-Centric Data Fabric. You will transition traditional application-centric architectures into a centralised semantic layer that seamlessly unifies multi-stream operational data-including Electronic Data Capture (EDC), patient telemetry, and real-world health datasets. In this role, AI augmentation is natively woven into your workflow. It acts as a force multiplier to automate routine mapping, query optimization, and regulatory documentation. This allows you to focus on driving high-impact platform architecture. * Data Fabric & Lake Architecture: Architect and evolve the enterprise semantic data fabric, converting multi-stream clinical execution datasets into an object-centric model. Design and execute a modern Data Lake strategy centred on Apache Iceberg as the core storage format, ensuring high-performance querying and seamless interoperability with Snowflake and heterogeneous compute engines. * AI-Accelerated Schema & Pipeline Engineering: Develop and maintain end-to-end multi-stream ingestion pipelines for complex clinical trial schemas. Use AI-driven schema inference and ontology alignment tools to auto-draft mapping artifacts, dramatically reducing integration timelines across different life science datasets. * High-Throughput Streaming & Backend Services: Build scale, fault-tolerant real-time ingestion pipelines using Kafka, AWS, and Snowflake. Write robust enterprise services in Java or Scala, leveraging AI coding assistants for rapid code generation, refactoring, and performance tuning. * Technical Strategy & Database Optimization: Promote technical direction and engineering best practices across teams for Change Data Capture (CDC), clustering, data migration, and aggregation. Use AI query-optimization tools to analyse execution plans, auto-tune complex Snowflake/Iceberg SQL workloads, and eliminate performance bottlenecks. * Intelligent Telemetry & Closed-Loop Reasoning: Integrate automated AI inferencing and reasoning layers directly into data pipelines to detect telemetry anomalies in real-time and automatically map safety signals back to operational trial datasets. * Quality & AI-Driven Compliance: Lead Test-Driven Development (TDD) and Behavior-Driven Development (BDD) initiatives. Use AI test generators to produce HIPAA/GxP-compliant synthetic clinical trial datasets for automated validation. Leverage AI tools to auto-draft validation artifacts, data lineage manifests, and audit documentation required under GxP, HIPAA, and GDPR standards. * System Resilience: Troubleshoot complex production issues across distributed data environments and implement resilient, self-healing pipeline architectures. Key Business Value & Strategic Impact Your leadership will directly advance life sciences technology. By pairing modern lakehouse architecture (Snowflake, Apache Iceberg, Kafka) with AI-augmented workflows, you will empower our platform to process critical safety signals in hours rather than weeks. This enables adaptive clinical trial execution, guarantees zero-loss data integrity, and significantly accelerates regulatory submission timelines for life-saving therapies. ## Related Videos - [Blueprints for Success: Steering a Global Data & AI Architecture](https://www.wearedevelopers.com/videos/1577-blueprints-for-success-steering-a-global-data-ai-architecture) - [Developing the Backend with Stefan Lingler, CTO at Shpock](https://www.wearedevelopers.com/videos/100360-developing-the-backend-with-stefan-lingler-cto-at-shpock) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Nest.js - TypeScript in the backend can also be clean](https://www.wearedevelopers.com/videos/1033-nest-js-typescript-in-the-backend-can-also-be-clean) - [Git for Code Reviews](https://www.wearedevelopers.com/videos/429-git-for-code-reviews) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)