> Markdown version of [/videos/1536-bringing-ai-model-testing-and-prompt-management-to-your-codebase-with-github-models?t=1211](https://www.wearedevelopers.com/videos/1536-bringing-ai-model-testing-and-prompt-management-to-your-codebase-with-github-models?t=1211). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models

Stop relying on vibes for prompt engineering. Treat AI evaluation with strict determinism. GitHub Models brings CI/CD and version-controlled testing directly into your codebase.

- **Speakers:** [Kevin Lewis](https://www.wearedevelopers.com/@kevin-lewis), [Sandra Ahlgrimm](https://www.wearedevelopers.com/@sandra-ahlgrimm)
- **Event:** World Congress 2025
- **Published:** August 20, 2025
- **Duration:** 27:07
- **URL:** https://www.wearedevelopers.com/videos/1536-bringing-ai-model-testing-and-prompt-management-to-your-codebase-with-github-models

## Summary

As developers are increasingly tasked with implementing AI features, the rigorous methodologies applied to traditional software engineering are often abandoned. Prompt engineering currently relies heavily on "vibes," where developers deploy model combinations blindly rather than utilizing structured testing. Recognizing that flaky unit tests are unacceptable in traditional codebases, GitHub Models introduces a structured ecosystem that treats AI prompt management and evaluation with the same strict determinism as standard software development. By writing system prompts, variables, and expected test data into standard YAML files checked directly into repositories, teams can properly version-control their non-deterministic AI behaviors.

At the core of this toolkit is an experimentation interface that natively allows developers to run prompt configurations against over 40 distinct language models via a single API key, comparing outputs for latency, coherence, and structural precision. Utilizing an "LLM as a judge" framework alongside expected data mappings allows teams to explicitly validate model similarity and adherence to strict guidelines before writing any integration logic. Once baseline performance criteria are met, shifting from evaluation environments to production deployments requires only an endpoint and token swap via the Azure AI Inference SDK, leaving application code untouched.

Critically, this standardized workflow embeds AI directly into continuous integration workflows. Because evaluations can occur via the GitHub CLI, organizations can construct automated CI/CD pipelines through GitHub Actions that actively block pull requests if architectural changes cause a prompt file to fail its evaluation benchmarks. Beyond application deployment, the ease of access to inference endpoints enables rapid repository automation, allowing maintainers to reclaim compounding hours by using automated agents to generate changelogs from merged PRs, summarize weekly repository issues, and automatically triage incomplete user bug reports.

**Keywords:** github models, ai prompt management, ai model evaluation, azure ai inference sdk, llm as a judge, prompt version control, deterministic ai testing, ci/cd pipeline prompt testing, github actions automation, automated repository changelogs, github cli ai tools, system prompt configuration, automated bug issue triage, cross-model api comparison

## Chapters

1. **The challenge of testing AI prompts without tools** (00:31) — Why developers rely on vibe checks instead of systematic tests for non-deterministic AI outputs.
1. **Introduction to the GitHub Models testing toolkit** (03:12) — An overview of accessing over 40 models using a single API key for prompt experimentation.
1. **Managing and versioning system prompts as YAML files** (03:49) — How to configure system instructions and parameters using YAML configuration directly in your repository.
1. **Configuring generation parameters and committing prompt changes** (05:27) — Adjusting temperature and token limits before committing prompt configurations to feature branches.
1. **Evaluating prompt accuracy against expected test outcomes** (08:00) — Establishing test datasets and using LLM judges to ensure accuracy and coherence.
1. **Running simultaneous comparisons across different AI models** (10:36) — Experimenting with multiple models and prompt variants simultaneously to optimize latency and cost.
1. **Integrating GitHub Models into external enterprise applications** (13:35) — Consuming managed prompts using model API endpoints or the Azure AI Inference SDK.
1. **Building applications with Azure AI Inference SDK** (15:08) — Code walkthrough showing how a JavaScript application imports an SDK to consume repository configurations.
1. **Evaluating AI models locally using GitHub CLI** (18:24) — Running interactive sessions and local evaluations directly from the terminal before committing code changes.
1. **Automating repository issues and changelogs with AI actions** (20:11) — Leveraging inference integrations inside GitHub Actions to automate pull request summaries and triage bug reports.
1. **Running prompt evaluation tests in CI/CD pipelines** (24:00) — Running automated evaluations during pull requests prevents degraded prompts from reaching production pipelines.
1. **Migrating existing applications to managed prompt architectures** (24:53) — Converting plain text instructions to standard YAML files dramatically accelerates testing and deployment workflows.

## Related Moments

- [Evaluating AI prompts as code with automated CI pipelines](https://www.wearedevelopers.com/videos/2098-why-most-ai-features-fail-after-the-demo) (from "Why Most AI Features Fail After the Demo")
- [Introduction to GitHub Next and AI prototyping](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) (from "Innovating Developer Tools with AI: Insights from GitHub Next")
- [Testing AI models quickly using GitHub Models](https://www.wearedevelopers.com/videos/1532-agentic-ai-from-theory-to-practice-developing-multi-agent-ai-systems-on-azure) (from "Agentic AI - From Theory to Practice: Developing Multi-Agent AI Systems on Azure")
- [Exploring AI integrations in modern agile development workflows](https://www.wearedevelopers.com/videos/631-chatgpt-create-a-presentation) (from "ChatGPT: Create a Presentation!")
- [The shift from human prompt engineering to AI-generated prompts](https://www.wearedevelopers.com/videos/100255-design-patterns-for-ai-products-in-2026) (from "Design Patterns For AI Products in 2026")
- [Integrating version control and AI into productivity software](https://www.wearedevelopers.com/videos/919-coffee-with-developers-adam-wiggins) (from "Coffee with Developers - Adam Wiggins")

## Related Articles

- [Liuba Gonta and Yuliya Khadasevic - GitHub Copilot Beyond the Basics - 10 Ways to Elevate Your Coding](https://www.wearedevelopers.com/magazine/490-liuba-gonta-and-yuliya-khadasevic-github-copilot-beyond-the-basics-10-ways-to-elevate-your-coding)
- [GitHub Copilot: Beyond the Basics – 10 Ways to Elevate Your Coding](https://www.wearedevelopers.com/magazine/524-github-copilot-beyond-the-basics-10-ways-to-elevate-your-coding)
- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)
- [What is Agentic Programming and Why Should Developers Care?](https://www.wearedevelopers.com/magazine/625-what-is-agentic-programming-and-why-should-developers-care)

## Related Jobs

- [Staff Software Engineer, Copilot Experiences](https://www.wearedevelopers.com/jobs/ext/164361-staff-software-engineer-copilot-experiences) at **GitHub**
- [Principal Product Manager, Agent Platform](https://www.wearedevelopers.com/jobs/ext/277541-principal-product-manager-agent-platform) at **GitHub**
- [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**
- [Staff Developer Advocate, GitHub Security Lab](https://www.wearedevelopers.com/jobs/ext/1921051-staff-developer-advocate-github-security-lab) at **GitHub**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/588393-machine-learning-engineer) at **Twilio**
- [Principal Software Engineer, Enterprise AI Platform](https://www.wearedevelopers.com/jobs/ext/1467292-principal-software-engineer-enterprise-ai-platform) at **GitHub**