Machine Learning Engineer / SME (LLM Eval & RAG Testing) - SME | TS/SCI

Prove which LLMs actually work for the mission. You’ll design RAG and LLM evaluation experiments, tune models on custom data, and run GPU-accelerated benchmarks so the team ships AI systems they can trust.

What you’ll do

• Test, evaluate and validate AI systems with a focus on LLM-based systems (including RAG testing)
• Design and run experiments that benchmark LLMs against defined measures
• Tune neural networks / LLMs on custom datasets and apply results to concrete use cases
• Build AI/ML models and ensembles in Python or R (library selection, preprocessing, model assessment)
• Use deep learning frameworks such as PyTorch, TensorFlow or Keras
• Leverage GPUs for accelerated training and evaluation
• Write solid SQL (CTEs, set operations, aggregates, nested subqueries) against analytical datasets
• Work in GitHub and Jenkins-style CI for reproducible experiments
• Communicate methodological choices and model results to technical and non-technical stakeholders

• Active TS/SCI
• Hands-on RAG testing experience (must-have)
• Tuning neural nets / LLMs on custom datasets for real use cases
• Building AI/ML models and ensembles in Python or R
• PyTorch, TensorFlow or Keras (key)
• GPU-accelerated computing (key)
• Testing, evaluating or validating LLM-based AI systems (key)
• Experiment design to benchmark LLMs (key)
• SQL including CTEs, set ops, aggregates and nested subqueries
• GitHub and Jenkins (or equivalent) version control / CI

Nice to have

• Hugging Face Transformers library and hub
• Building benchmarks that measure LLM performance
• Cloud computing development and architecture
• Flask or similar front-end/web frameworks