> Markdown version of [/jobs/ext/1377291-senior-software-engineer-ai-benchmarking](https://www.wearedevelopers.com/jobs/ext/1377291-senior-software-engineer-ai-benchmarking). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Software Engineer, AI Benchmarking - **Company:** SecureBio, Inc. - **Location:** United States (Remote available) - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Clean Code Principles, Artificial Intelligence, Component-Based Software Engineering, Cloud Computing, Software Quality, Software Construction, Software Engineering, TypeScript, AWS Cdk, ReactJS, Large Language Models, Multi-Agent Systems, Infrastructure Automation Frameworks, Free and Open-Source Software, Terraform - **Published:** July 22, 2026 - **Apply:** https://www.indeed.com/viewjob?jk=5ba4b75c7b6251e9 ## About the Role * A proven track record in software engineering, including 5+ years of experience in industry roles and/or open-source contributions. * Fluency in Python. * Proficiency with Docker and AWS (particularly ECS). * Experience building software tools that use or evaluate LLMs or AI agents (for example, code that orchestrates agents for a specific application, or that evaluates model capabilities on a specific task). * Self-directed and comfortable working on small, fast-moving teams. * A commitment to good coding practices, including knowing when and how to use (or not use) coding agents. * Care about code quality, readability, and maintaining human oversight of human- and AI-written code. * Alignment with SecureBio's mission of preventing catastrophic pandemics., * Experience with AI evaluations (agent systems, scorer and grader design, agent trajectory analysis, jailbreaking or red-teaming), whether for security, safety, or model performance applications. * Experience with Terraform or other Infrastructure-As-Code tools (e.g. Tofu or AWS CDK). * Experience with modern, component-based web frontends (e.g. React with TypeScript). * Experience with our specific tech stack: + Evaluation frameworks: Inspect AI + LLM APIs: OpenAI-compatible, Anthropic, Together, etc. + Agent frameworks: Codex, Claude Code, etc. ## Description SecureBio is looking for a Senior Software Engineer to build, scale, and run biosecurity evaluations on frontier AI systems. The tools you build will directly inform decisions made by frontier AI companies, policymakers, and researchers to improve the safety of AI systems., * Perform pre-release and post-release assessments of frontier and open-source AI models using SecureBio's suite of biosecurity evaluations, which surface and measure dual-use biology capabilities in AI systems. + Build adapters for models, identify and troubleshoot unexpected agent behaviors, and ensure that our evaluations are methodologically rigorous. * Build, scale, maintain, and continuously improve SecureBio's suite of biosecurity evaluations. + Improve our evaluations to use state-of-the-art agent frameworks, jailbreaking methods, and elicitation strategies. * Develop and improve internal tools that enable our researchers to run capability assessments at scale. * Contribute to and maintain cloud infrastructure to run our evaluations and analyses. * Contribute analyses to our public dashboard of trends in AI biology capabilities (https://securebio.org/benchmarks/). ## Related Videos - [Infrastructure as Code: The Developer's Secret Weapon](https://www.wearedevelopers.com/videos/1221-infrastructure-as-code-the-developer-s-secret-weapon) - [Watch Tests Go Brrrr! : Getting Started with Cypress in ReactJS](https://www.wearedevelopers.com/videos/282-watch-tests-go-brrrr-getting-started-with-cypress-in-reactjs) - [Building Reliable Serverless Applications with AWS CDK and Testing](https://www.wearedevelopers.com/videos/812-building-reliable-serverless-applications-with-aws-cdk-and-testing) - [Agentic employees in world's most downloaded FinTech app](https://www.wearedevelopers.com/videos/100123-agentic-employees-in-world-s-most-downloaded-fintech-app) - [The power of Cloud Development Kit (CDK): How to get the most out of it](https://www.wearedevelopers.com/videos/740-the-power-of-cloud-development-kit-cdk-how-to-get-the-most-out-of-it) - [Implementing Feature Environments with AWS and Terraform](https://www.wearedevelopers.com/videos/531-implementing-feature-environments-with-aws-and-terraform) ## Related Articles - [Dev Digest 137 - AI'm not sure about this](https://www.wearedevelopers.com/magazine/485-dev-digest-137-ai-m-not-sure-about-this) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift) - [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers) - [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again) - [Dev Digest 121 - AI goes offline](https://www.wearedevelopers.com/magazine/456-dev-digest-121-ai-goes-offline) - [Dev Digest 132 - Binging WADFlix?](https://www.wearedevelopers.com/magazine/473-dev-digest-132-binging-wadflix)