> Markdown version of [/jobs/ext/4584390-sr-software-engineer-ai-evaluation-and-knowledge](https://www.wearedevelopers.com/jobs/ext/4584390-sr-software-engineer-ai-evaluation-and-knowledge). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Sr. Software Engineer - AI Evaluation and Knowledge - **Company:** Super Micro Computer, Inc. - **Location:** San Jose, CA, United States - **Experience:** Expert - **Salary:** $160,000 - $200,000 - **Contract:** Permanent contract - **Skills:** AI Evaluation, A/B Testing, Artificial Intelligence, Python (Programming Language), Regression Testing, Data Processing, Evaluation Pipelines, Retrieval-Augmented Generation, Large Language Models, Prompt Engineering, Generative AI, Knowledge Engineering, Virtual Agents, Network Server, Software Version Control - **Published:** November 5, 2026 - **Apply:** https://jobs.supermicro.com/talentcommunity/apply/1430201100/?locale=en_US ## About the Role * Experience in LLM / Generative AI applications, with practical experience in evaluation or quality (2+ years of experience is preferred) * Strong familiarity with Prompt Engineering and practical methods to reduce hallucinations * Experience systematically evaluating LLM or Agent outputs * Experience with RAG or Knowledge Base implementation / management * Strong analytical and cross-team communication skills * Ability to turn domain expert feedback into actionable improvements * Familiarity with Python and ability to implement basic evaluation or data processing scripts * Experience in technical support, troubleshooting, or vertical AI Agent applications is a plus * Experience with LLM-as-judge, automated evaluation pipelines, or A/B testing is a plus * Experience in knowledge engineering, data annotation, or case library building is a plus * Experience collaborating with non-AI domain experts is a plus * ## Description We are building an AI Agent-powered platform to handle customer issues related to AI Servers and Racks. This role focuses on the quality of agent outputs, evaluation systems, and continuous improvement of the Knowledge Base, ensuring the agent produces accurate and actionable troubleshooting steps., * Design and maintain the agent evaluation framework (test sets, automated scoring, human sampling review, regression testing) * Own Prompt Engineering and structured output optimization to improve step accuracy * Manage Knowledge Base quality processes (content review, version control, retirement of outdated content, writing back new cases) * Analyze agent errors and hallucinations, then propose and implement improvements * Work closely with Domain SMEs to validate the correctness and practicality of troubleshooting steps * Establish a feedback loop from on-site execution results back into agent improvements * Help define success metrics (e.g., first-time accuracy rate, step usability rate) and track them continuously * Support RAG retrieval quality tuning and knowledge presentation improvements