> Markdown version of [/jobs/ext/2095697-software-engineering-evaluation-specialist](https://www.wearedevelopers.com/jobs/ext/2095697-software-engineering-evaluation-specialist). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Software Engineering Evaluation Specialist - **Company:** Mindrift - **Location:** Edinburgh, UK - **Experience:** Starter - **Salary:** £54,207.0 - **Contract:** Permanent contract - **Skills:** Java (Programming Language), Artificial Intelligence, Computer Vision, JIRA, Automation of Tests, Bash Shell, Cron, Extract Transform Load (ETL), Cursor (Graphical User Interface Elements), Software Debugging, Linux, DevOps, Fuzz Testing, Python (Programming Language), Nginx, Node.Js, NumPy, Scientific Computating, SciPy, Software Engineering, Pytorch, Backend, Git, Pytest, Low-code, Front End Software Development, Virtual Agents, Docker, Golang - **Published:** August 17, 2026 - **Apply:** https://www.adzuna.co.uk/jobs/details/5845183202 ## About the Role * 3+ years of production software development in one backend stack - Python, Go, Node.js, Java, or Rust. Depth in one stack beats breadth. * Python + pytest fluency - required regardless of primary stack. The task harness is pytest-based even when the broken app is in another language. Fixtures, parametrize, monkeypatch, timeouts, conftest.py. * Docker authoring - reproducible Dockerfiles, pinned dependencies, multi-stage builds when needed, non-root user. * Linux & Bash - comfort debugging inside containers (strace, lsof, journalctl); shell beyond set -euo pipefail. * AI coding agent experience - Claude Code, Cursor, Roo Code, or similar, on non-trivial work. You can cite a specific time the AI was confidently wrong and how you caught it. * English - B2+ written., * Domain depth in Security, System Administration (nginx / systemd / cron), Scientific Computing (NumPy / PyTorch / SciPy), DevOps, or Git internals. * Modern Python tooling (uv, poetry, pyproject.toml). * Coverage tooling (pytest-cov, coverage.py, gcov, llvm-cov, kcov). * Fuzzing or property-based testing (Hypothesis). * Prior contribution to agent-evaluation benchmarks or related frameworks. ## Description You'll design coding tasks that challenge frontier AI coding agents. Each task is a self-contained Docker environment with a broken piece of software; an AI agent attempts the fix; automated tests verify the outcome. Your deliverable is the full task package: broken code, tests, instructions, and a reference solution proving the task is solvable., * Invent a realistic developer scenario - a real bug, a broken ETL, a missing feature - not a toy problem. * Build a reproducible Docker environment with pinned dependencies. * Write a pytest that verifies outcomes, not specific commands - deterministic, non-flaky, and does not leak the fix. * Write an instruction.md that reads like a Jira ticket a developer would receive. * Write a reference solve.sh proving the task is solvable. * Calibrate difficulty so current state-of-the-art agents solve the task 20-60% of the time. * Iterate based on feedback from expert QA reviewers. * Later: review other authors' tasks as a QA reviewer., * Data Science, ML, or Computer Vision engineers without backend-engineering output. * Manual QA testers without automation or test authoring. * Frontend-only, low-code / no-code, IT Support, or Business Analysts. * Engineers who have never written pytest from scratch. * Junior, intern, or assistant as the most recent role., * Steady state: ~5 hours per task, 2-4 parallel tasks per author. * Realistic weekly load: 8-20 hours. Higher volume available for top performers. * You choose when and how to contribute; tasks must be submitted by the deadline and meet acceptance criteria. ## Related Videos - [Beyond the Benchmark: How to Evaluate AI Agents in the Real World](https://www.wearedevelopers.com/videos/100269-beyond-the-benchmark-how-to-evaluate-ai-agents-in-the-real-world) - [From clicks to cribs - How to find your dream home with web scraping](https://www.wearedevelopers.com/videos/767-from-clicks-to-cribs-how-to-find-your-dream-home-with-web-scraping) - [Improving quality with Agentic AI with Rovo Dev and Xray](https://www.wearedevelopers.com/videos/2005-improving-quality-with-agentic-ai-with-rovo-dev-and-xray) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [Developer Experience in the Age of AI](https://www.wearedevelopers.com/videos/1118-developer-experience-in-the-age-of-ai) - [Is your backend a hodgepodge of queues, event stores and cron jobs? Durable Execution to the Rescue.](https://www.wearedevelopers.com/videos/744-is-your-backend-a-hodgepodge-of-queues-event-stores-and-cron-jobs-durable-execution-to-the-rescue) ## Related Articles - [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers) - [Dev Digest 137 - AI'm not sure about this](https://www.wearedevelopers.com/magazine/485-dev-digest-137-ai-m-not-sure-about-this) - [Dev Digest 131 - AI'm not sure about OSS](https://www.wearedevelopers.com/magazine/472-dev-digest-131-ai-m-not-sure-about-oss) - [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again) - [Dev Digest 121 - AI goes offline](https://www.wearedevelopers.com/magazine/456-dev-digest-121-ai-goes-offline) - [Dev Digest 138 - Are you secure about this?](https://www.wearedevelopers.com/magazine/486-dev-digest-138-are-you-secure-about-this)