Data Engineering & Agentic AI Architect (Data Architect II)

The successful candidate will shape the technical direction of large, complex Data & Digital Transformation programmes and delivering engineering Agentic AI systems at scale, to unlock Transformation pace, quality and impact - leaving behind something that stands the test of time?

You will be undertaking a monumental Transformation to unlock organisational pace and Data agility, moving to a scalable Cloud Data architecture centred around AWS Glue (Apache Spark) & Apache Airflow, with a key focus on re-engineering numerous complex Data pipelines to the new framework by leveraging Agentic AI, all whilst embedding great Data Management to build deep trust in Data.

We're looking for a Staff-level Data & Agentic AI engineer to join the programme - driving the agentic AI based conversion framework & Data pipeline modernisation, partnering across a Senior team of hands on Data Engineers, and driving the programme to completion with the quality and rigour that the Data engineering organisation can own and easily maintain into the future.

What you'll do

• Shape the technical direction for the Data pipeline modernisation and re-engineering programme:

• Define architecture, standards, engineering approach, and testing framework

• Ensure coherence across all workstreams from discovery through to validated delivery

Architect and build the Claude Code agentic workflow:

• Parse existing data movement pipelines and logic
• Generate equivalent PySpark transformations
• Produce Airflow DAGs
• Operate, iterate, and scale across the full data pipeline estate

Ensure rigorous creation and management of the PySpark data processing layer within AWS Glue:

• Support data pipeline re-engineering
• Ensure performance, standardisation, and cost efficiency

Design, write, and refine prompt architecture and context management logic:

• Govern Claude Code's output
• Ensure consistent conformance to the target framework across complex conversions

Build automated test harnesses:

• Validate functional equivalence between existing and new pipelines
• Ensure business logic is preserved throughout modernisation

Develop structural validation and static analysis tooling:

• Evaluate AI-generated output for correctness
• Minimise hallucination risks
• Ensure zero technical debt in modernised pipelines

Lead and coordinate engineering team efforts:

• Align technical direction
• Resolve architectural ambiguity
• Maintain delivery momentum in a fast-paced programme

Build and maintain stakeholder relationships:

• Engage senior leadership
• Clearly translate progress, AI output quality, and technical risks

Enable automated, high-quality documentation:

• Cover both platform and data pipelines
• Define SLOs and data producer/consumer agreements
• Leverage AI to improve documentation and delivery speed

Design for maintainability and transferability:

• Ensure frameworks, workflows, and pipelines are fully ownable post-programme
• Treat handover quality as a first-class engineering deliverable

• Proven experience setting technical direction and architectural standards on complex, multi-workstream programmes

• Hands-on capability to build solutions as well as design them

• Deep expertise in Python and PySpark:

• Strong track record delivering large-scale distributed data systems

Experience with Claude Code or similar agentic AI platforms:

• Prompt engineering
• Context management
• Workflow design for scalable automated code generation in regulated environments

Strong hands-on experience with AWS ecosystem:

• AWS Glue, S3, and related data platforms (e.g. Snowflake)

Strong experience with Apache Airflow and DAG-based orchestration

Experience designing configuration-driven frameworks:

• Using YAML or JSON

Experience building automated testing for data pipelines:

• Functional equivalence testing
• Regression testing

Proven technical leadership:

• Ability to influence without authority
• Align senior engineers across data and AI domains

Experience working in large, regulated enterprise environments:

• Navigate governance
• Build trust while maintaining pace

Strong business acumen:

• Ability to communicate technical concepts, risks, and delivery status to senior stakeholders

Experience designing for maintainability and handover:

• Build systems and documentation for long-term ownership by others

Skills

apache airflow,python,pyspark,claude code

UST are currently recruiting for a permanent Data Architect with experience working with Agentic AI to work with one of our clients on AI transformation project., About UST UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world's best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients' organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact-touching billions of lives in the process.