Principal Data Engineer

• We set the medium- to long-term technical direction for our data infrastructure.
• We lead architectural design of complex, business-critical data systems that span multiple teams and affect the whole company.
• We shape how engineering teams work across the organisation, including data quality standards, tooling, and ways of working.
• We tackle the hardest data problems and deliver work with direct impact on company goals.
• We act as an interviewer, help improve the hiring process, and coach engineers across the organisation.
• We represent ComplyAdvantage at meet-ups, conferences, and industry forums.
• We architect petabyte-scale data platforms across batch, micro-batch, and streaming, making explicit trade-offs between latency, throughput, cost, and operational complexity.
• We design and own the lineage, quality, freshness, and observability of our financial crime knowledge graph and the pipelines that feed it.
• We build and evolve our foundational data infrastructure, including ingestion frameworks, the event bus, feature and serving stores, the lakehouse, orchestration, and the developer experience around them.
• We set the standard for event-sourced and streaming patterns across the company using Kafka and similar technologies.
• We design data services with scale and ease of operation in mind, and write maintainable, performant, well-tested Python code.
• We partner with ML engineers and data scientists so the platform supports feature engineering, training pipelines, and online inference at scale.
• We set the data quality, schema evolution, and contract-testing standards that other engineering teams adopt.
• We integrate the data platform with new and existing services by building and consuming APIs and event streams, and by producing documentation that engineers and analysts can self-serve from.
• We coach staff, senior, and mid-level engineers across our tribe and the wider engineering organisation, and help build the bench of future technical leaders.
• We own the technical architecture of the data platform behind our sanctions, PEP, adverse media, transaction monitoring, fraud, and customer risk products.
• We lead the architecture that helps ML, data science, and product teams ship new detection models and risk signals in days rather than quarters.
• We design the data foundations that make agentic AI work at scale, including retrieval pipelines, grounding sources, tool data, and event histories.
• We work closely with our Customer Risk, Fraud, Knowledge Graph, and Screening tribes so the data foundations keep pace with the product and AI roadmap.
• We set the technical direction for how we ingest, normalise, and merge entity, relationship, and event data from millions of public and private sources.
• We make company-wide data architecture decisions, including make-or-buy choices and our long-term vendor and tooling strategy for the data estate.

Technologies:

• Agentic AI
• AI
• Airflow
• AWS
• Architect
• ArgoCD
• Cloud
• Docker
• ETL
• Flink
• GCP
• Incident Management
• Java
• Kafka
• Kotlin
• Kubernetes
• LLM
• Python
• Spark
• dbt
• Backend
• ES6
• Frontend
• Grafana
• Support
• React
• TypeScript
• gRPC

• We are looking for an experienced Principal Data Engineer to lead the design and evolution of our data platform.
• You have substantial experience designing and operating production-grade data platforms at high scale.
• You have deep expertise in distributed data systems, including streaming technologies such as Kafka, and batch and ELT/ETL frameworks such as Spark, Flink, dbt, Airflow, or Argo Workflows.
• You have experience with modern lakehouse or warehouse technologies.
• You have strong production Python experience and enough familiarity with Java or Kotlin to set direction, review code, and coach others.
• You have experience designing for cloud environments such as AWS and GCP, and containerised infrastructure such as Kubernetes, Docker, and ArgoCD.
• You treat data quality, observability, and data contracts as first-class engineering concerns.
• You have a strong working understanding of logging, monitoring, alerting, and incident management tooling for data systems.
• You have excellent written and verbal communication skills and can produce technical documentation that senior leaders and engineers can act on.
• You have ownership of software and data products from inception through to production and long-term operation.
• You have a track record of coaching staff, senior, and mid-level engineers, and of helping Recruiting improve hiring and onboarding.
• Nice to have: experience in financial services, AML, KYC, fraud, regtech, or another regulated domain.
• Nice to have: familiarity with knowledge graph and entity resolution problems such as deduplication, linkage, hierarchies, and temporal relationships.
• Nice to have: experience supporting ML, LLM, and agentic AI workloads, including feature stores, vector stores, retrieval pipelines, tool data, and online/offline parity.
• Nice to have: experience representing engineering externally at conferences, meet-ups, or in technical publications.

We empower every business to eliminate financial crime. Using AI, a unified platform, and an extensive partner ecosystem, we help customers turn compliance into a catalyst for growth, operational resilience, and enduring regulatory trust. More than 3,000 enterprises across 75 countries rely on our end-to-end platform and financial crime risk intelligence. We are headquartered in London with global hubs in New York, Lisbon, Singapore, and Cluj-Napoca. We offer equity participation, unlimited time off, a hybrid working model with two office days per week, opportunities for collaboration and career development, an annual learning budget, a home office budget, enhanced parental leave and childcare benefits, life insurance and medical coverage through BUPA including pre-existing conditions, and pension contribution through The Peoples Pension.