Senior Python Software Engineer, Coding Agent - AI Evaluation

• Build production-like benchmark tasks using repositories, tests, configuration, documentation, and runtime scenarios.
• Create realistic engineering tasks such as bug fixes, feature additions, CI repairs, configuration migrations, integration updates, and runtime-state fixes.
• Define utility requirements that verify correct task completion.
• Define safety and alignment requirements that protect system constraints, developer intent, permissions, privacy, oversight, and data integrity.
• Develop visible and hidden test suites.
• Create safe and unsafe reference solutions for controlled evaluation.
• Identify unsafe shortcuts such as disabling tests, weakening assertions, deleting protected data, leaking secrets, broadening permissions, or bypassing validation.
• Package prompts, metadata, runnable repositories, evaluators, reference patches, scoring rubrics, and calibration notes.
• Analyze coding-agent rollouts for utility completion and safety violations.
• Collaborate with engineering, QA, security, and client teams to improve evaluator reliability and benchmark difficulty.

• Bachelor’s or Master’s degree in Computer Science or a related technical discipline.
• At least 8 years of hands-on software engineering experience in tier-one product companies or technology startups.
• Strong professional Python experience, plus at least one of Java, Go, or C++.
• Experience building and maintaining production software used by real customers.
• Strong software architecture, debugging, and performance-optimization skills.
• Production engineering experience covering deployment pipelines, monitoring, logging, observability, metrics, distributed tracing, high availability, fault tolerance, scalability, and disaster recovery.
• Secure-software experience identifying vulnerabilities, implementing fixes, and validating remediation. Relevant areas include injection, authentication and authorization, memory safety, race conditions, deserialization, secrets management, and input validation.
• Practical experience debugging complex production issues, participating in incident response, and building resilient systems.

• Expected start around September 7, 2026.
• Approximately 4-8 weeks.
• 8 hours per day with 6 hours of overlap with Pacific Time.

Compensation

$56 per hour

Braintrust enables organizations to pull together flexible teams of highly-skilled technical and design talent, no matter where they’re located.