AWS DevOps Engineer - AI Benchmarking (FT-WW

• Design intricate and adversarial AWS infrastructure tasks to evaluate AI capabilities.
• Develop realistic DevOps scenarios that include security constraints, dependencies, edge cases, and failure conditions.
• Write precise task specifications that define desired infrastructure outcomes.
• Create idempotent reference solutions using Terraform or other AWS Infrastructure-as-Code tools.
• Develop automated graders and validation scripts in Python to assess AI agent performance.
• Validate infrastructure through AWS APIs, CLI outputs, Terraform state, and system behavior.
• Craft tasks that detect incomplete, unsafe, or superficially correct solutions.
• Ensure tasks are challenging enough to thoroughly test advanced AI models.
• Review and quality-check tasks from other engineers for accuracy and difficulty.
• Document task intent, assumptions, expected outcomes, edge cases, and scoring rationale.

English・Professional, * 4+ years of experience in DevOps, cloud infrastructure, platform engineering, or site reliability engineering.

• Extensive hands-on AWS experience across networking, IAM, compute, storage, databases, security, and cloud operations.
• Advanced expertise in Infrastructure-as-Code, particularly with Terraform.
• Proficient in Python for building automated graders and validation tools.
• Experience in testing and validating infrastructure through APIs, CLIs, or automated frameworks.
• Strong understanding of secure, reliable, and reproducible infrastructure.
• Ability to design complex technical problems with ambiguity and edge cases.
• Attention to detail in identifying unsafe or incomplete solutions.
• Excellent written communication and technical documentation skills.
• Ability to work independently in a structured, task-based environment., * Experience with Pulumi, AWS CDK, or AWS CloudFormation.
• Familiarity with boto3, pytest, Terratest, LocalStack, or similar tools.
• Background in security engineering, chaos engineering, incident response, or SRE.
• Experience with AI evaluation, benchmarking, or red teaming.
• Knowledge of AI coding agents and common AI model failure modes.
• Experience designing technical assessments or infrastructure exercises.
• Experience reviewing engineers' work and maintaining quality standards.