> Markdown version of [/jobs/ext/1928444-senior-python-software-engineer-coding-agent-ai-evaluation](https://www.wearedevelopers.com/jobs/ext/1928444-senior-python-software-engineer-coding-agent-ai-evaluation). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Senior Python Software Engineer, Coding Agent - AI Evaluation

- **Company:** Brain Trust, LLC
- **Location:** San Francisco, CA, United States (Remote available)
- **Experience:** Expert
- **Salary:** $116,480.0
- **Contract:** Temporary contract
- **Skills:** Java (Programming Language), Artificial Intelligence, Software Bug Management, C++ (Programming Language), Data Validation, Data Integrity, Software Debugging, Disaster Recovery, Fault Tolerance, Python (Programming Language), Key Management, Performance Tuning, Software Architecture, Software Engineering, Data Logging, System Availability, Delivery Pipeline, Information Technology, Dynatrace, Vulnerability Analysis
- **Published:** August 5, 2026
- **Apply:** https://arc.dev/remote-jobs/j/redirect/p9znjhyvl7

## About the Role

* Bachelor's or Master's degree in Computer Science or a related technical discipline.
 * At least 8 years of hands-on software engineering experience in tier-one product companies or technology startups.
 * Strong professional Python experience, plus at least one of Java, Go, or C++.
 * Experience building and maintaining production software used by real customers.
 * Strong software architecture, debugging, and performance-optimization skills.
 * Production engineering experience covering deployment pipelines, monitoring, logging, observability, metrics, distributed tracing, high availability, fault tolerance, scalability, and disaster recovery.
 * Secure-software experience identifying vulnerabilities, implementing fixes, and validating remediation. Relevant areas include injection, authentication and authorization, memory safety, race conditions, deserialization, secrets management, and input validation.
 * Practical experience debugging complex production issues, participating in incident response, and building resilient systems.

## Description

* Build production-like benchmark tasks using repositories, tests, configuration, documentation, and runtime scenarios.
 * Create realistic engineering tasks such as bug fixes, feature additions, CI repairs, configuration migrations, integration updates, and runtime-state fixes.
 * Define utility requirements that verify correct task completion.
 * Define safety and alignment requirements that protect system constraints, developer intent, permissions, privacy, oversight, and data integrity.
 * Develop visible and hidden test suites.
 * Create safe and unsafe reference solutions for controlled evaluation.
 * Identify unsafe shortcuts such as disabling tests, weakening assertions, deleting protected data, leaking secrets, broadening permissions, or bypassing validation.
 * Package prompts, metadata, runnable repositories, evaluators, reference patches, scoring rubrics, and calibration notes.
 * Analyze coding-agent rollouts for utility completion and safety violations.
 * Collaborate with engineering, QA, security, and client teams to improve evaluator reliability and benchmark difficulty.


## Related Videos

- [Crypto-secure Data Management with In-Database Blockchain](https://www.wearedevelopers.com/videos/632-crypto-secure-data-management-with-in-database-blockchain)
- [The AI-Native Engineering Org: What’s Real, What’s Hype, What’s Next](https://www.wearedevelopers.com/videos/100004-the-ai-native-engineering-org-what-s-real-what-s-hype-what-s-next)
- [Exploring 5 Key Applications of AI Abundance with Blockchain Assurance](https://www.wearedevelopers.com/videos/971-exploring-5-key-applications-of-ai-abundance-with-blockchain-assurance)
- [The Power of Purpose: Unlocking Potential and Innovation](https://www.wearedevelopers.com/videos/1110-the-power-of-purpose-unlocking-potential-and-innovation)
- [The weekly developer show: Boosting Python with CUDA, CSS Updates & Navigating New Tech Stacks](https://www.wearedevelopers.com/videos/1293-the-weekly-developer-show-boosting-python-with-cuda-css-updates-navigating-new-tech-stacks)
- [Startup Presentation: StorX - Future of Cloud Storage](https://www.wearedevelopers.com/videos/1176-startup-presentation-storx-future-of-cloud-storage)

## Related Articles

- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)
- [Dev Digest 137 - AI'm not sure about this](https://www.wearedevelopers.com/magazine/485-dev-digest-137-ai-m-not-sure-about-this)
- [What is Agentic Programming and Why Should Developers Care?](https://www.wearedevelopers.com/magazine/625-what-is-agentic-programming-and-why-should-developers-care)
- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers)
- [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift)