AI Evaluation & Benchmark Engineer
THAMES, FREDERICK
San Francisco, CA, United States
2 days ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.juju.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Experienced
Experience required
4 years minimum
Working hours
Regular working hours
Job source
Tech stack
AI Evaluation
Application Programming Interfaces (APIs)
Unit Testing
Continuous Integration
Python (Programming Language)
Next.js
Tailwind
Evaluation of Large Language Models
Job description
- Build automated sandbox runners that execute and verify generated code across 10,000+ unit test suites.
- Create synthetic benchmark datasets for Next.js, Tailwind, and full-stack API generation.
- Identify model failure modes and partner with research teams on dataset filtering and RL feedback.
Requirements
- 4+ years experience in Python, automated testing frameworks, and LLM evaluation tooling.
- Strong knowledge of continuous integration (CI) and containerized execution.
Benefits & conditions
Python Docker PyTest PostgreSQL FastAPI GitHub Actions
100% Employer-Covered Health & Dental
Meaningful Equity Ownership
Remote-First Setup Stipend ($2,500)
Flexible Time Off & Mental Health Days
Annual Company Engineering Retreats
$2,000 / Year Continuous Learning Budget
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.juju.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this role…