Sandra Ahlgrimm, Kevin Lewis
Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models
#1about 3 minutes
The challenge of testing non-deterministic AI features
Traditional development relies on rigorous testing, but AI features are often implemented based on intuition without a structured evaluation process.
#2about 5 minutes
Managing prompts as code with GitHub Models
GitHub Models integrates AI development into your repository by defining prompts, models, and parameters in a version-controlled YAML file.
#3about 6 minutes
Using evaluators to compare AI model variants
The platform allows you to run multiple prompt and model variations against a test dataset to compare outputs on metrics like latency, coherence, and similarity.
#4about 5 minutes
Consuming prompt files in your application code
Use the GitHub Models inference API or the Azure AI Inference SDK to load your version-controlled prompt files and integrate AI calls directly into your application.
#5about 2 minutes
Local development and testing with the CLI
The GitHub CLI extension allows you to run prompts and execute model evaluations directly from your terminal for rapid, local iteration before committing changes.
#6about 4 minutes
Automating repository tasks with AI-powered actions
Use GitHub Actions to automate common repository tasks like generating changelogs from pull requests, triaging bug reports, or creating weekly issue summaries.
#7about 1 minute
Implementing CI/CD for AI prompt changes
Integrate prompt evaluations into your CI/CD pipeline using GitHub Actions to automatically run tests and block pull requests that degrade model performance.
#8about 2 minutes
Adopting GitHub Models in existing projects
You can quickly convert existing prompt files to the GitHub Models format to gain access to powerful evaluation, comparison, and automation capabilities.
Related jobs
Jobs that call for the skills explored in this talk.
Featured Partners
Related Videos
25:17AI: Superhero or Supervillain? How and Why with Scott Hanselman
Scott Hanselman
31:39Livecoding with AI
Rainer Stropek
28:53Agentic DevOps: How AI-Powered Automation Transforms Software Delivery on GitHub and Azure
Mike
24:11You are not my model anymore - understanding LLM model behavior
Andreas Erben
33:28Prompt Engineering - an Art, a Science, or your next Job Title?
Maxim Salnikov
32:26Bringing the power of AI to your application.
Krzysztof Cieślak
42:38Innovating Developer Tools with AI: Insights from GitHub Next
Krzystof Czieslak
26:47The State of GenAI & Machine Learning in 2025
Alejandro Saucedo
From learning to earning
Jobs that call for the skills explored in this talk.
Senior Backend Engineer – AI Integration (m/w/x)
chatlyn GmbH
Vienna, Austria
Senior
JavaScript
AI-assisted coding tools