Senior QA Automation Engineer / SDET - GenAI & LLM Testing - Lorien Impellam

We’re supporting a leading banking client on a large-scale AI transformation programme and are looking for an experienced Senior QA Automation Engineer / SDET with a strong background in GenAI and LLM testing.

This role will focus on defining and implementing automated testing frameworks for AI-driven products, ensuring the quality, reliability and safety of LLM-powered solutions deployed within a highly regulated banking environment., * Design and develop scalable automated test frameworks using Python.

• Create test strategies from architecture and solution design documentation.
• Test RESTful APIs and microservice-based applications.
• Define and execute testing approaches for event-driven architectures and Kafka-based systems.
• Validate GenAI and LLM solutions through functional, non-functional and behavioural testing.
• Develop automated prompt testing and evaluation frameworks.
• Verify AI guardrails, safety controls and compliance requirements.
• Implement validation approaches for non-deterministic LLM outputs.
• Build and maintain contract tests using Pact Flow.
• Collaborate closely with Engineering, Architecture, Data Science and Product teams.
• Support CI/CD pipelines and quality engineering best practices.

Core Automation

• Strong Python 3.x development skills with experience building testing frameworks from the ground up.
• Extensive experience with Pytest.
• Strong API testing experience.
• Experience creating test strategies and quality approaches from architectural diagrams and solution designs.

Cloud & Platform

Working knowledge of AWS services including:

• Lambda
• Bedrock
• S3
• EKS

Event-Driven Systems

• Experience testing Kafka-based architectures.
• Strong understanding of asynchronous messaging and event-driven systems.

GenAI / LLM Testing (Essential)

• Proven experience testing LLM and GenAI applications.
• Prompt testing and prompt evaluation.
• Non-deterministic output validation.
• AI guardrail verification and safety testing.
• Hallucination and response quality assessment.
• Experience establishing quality standards for AI solutions.

LLM Evaluation Tooling

Experience with one or more of:

• DeepEval
• Promptfoo
• RAGAS
• LangSmith
• Similar LLM evaluation frameworks

Contract Testing

• Hands-on experience with Pact Flow.
• Consumer-driven contract testing within distributed systems.

Desirable

• Banking or Financial Services experience.
• Experience working in highly regulated environments.
• Exposure to RAG architectures.
• Knowledge of LLM observability and monitoring.
• Kubernetes experience.
• CI/CD pipeline integration.