Sandra Ahlgrimm & Kevin Lewis
Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models
#1about 3 minutes
The challenge of testing non-deterministic AI features
Traditional development relies on rigorous testing, but AI features are often implemented based on intuition without a structured evaluation process.
#2about 5 minutes
Managing prompts as code with GitHub Models
GitHub Models integrates AI development into your repository by defining prompts, models, and parameters in a version-controlled YAML file.
#3about 6 minutes
Using evaluators to compare AI model variants
The platform allows you to run multiple prompt and model variations against a test dataset to compare outputs on metrics like latency, coherence, and similarity.
#4about 5 minutes
Consuming prompt files in your application code
Use the GitHub Models inference API or the Azure AI Inference SDK to load your version-controlled prompt files and integrate AI calls directly into your application.
#5about 2 minutes
Local development and testing with the CLI
The GitHub CLI extension allows you to run prompts and execute model evaluations directly from your terminal for rapid, local iteration before committing changes.
#6about 4 minutes
Automating repository tasks with AI-powered actions
Use GitHub Actions to automate common repository tasks like generating changelogs from pull requests, triaging bug reports, or creating weekly issue summaries.
#7about 1 minute
Implementing CI/CD for AI prompt changes
Integrate prompt evaluations into your CI/CD pipeline using GitHub Actions to automatically run tests and block pull requests that degrade model performance.
#8about 2 minutes
Adopting GitHub Models in existing projects
You can quickly convert existing prompt files to the GitHub Models format to gain access to powerful evaluation, comparison, and automation capabilities.
Related jobs
Jobs that call for the skills explored in this talk.
Wilken GmbH
Ulm, Germany
Senior
Kubernetes
AI Frameworks
+3
Eltemate
Amsterdam, Netherlands
Intermediate
Senior
TypeScript
Continuous Integration
+1
Matching moments
03:28 MIN
Why corporate AI adoption lags behind the hype
What 2025 Taught Us: A Year-End Special with Hung Lee
14:06 MIN
Exploring the role and ethics of AI in gaming
Devs vs. Marketers, COBOL and Copilot, Make Live Coding Easy and more - The Best of LIVE 2025 - Part 3
05:26 MIN
Using AI prompts to rebuild a classic 8-bit game
WeAreDevelopers LIVE – Frontend Inspirations, Web Standards and more
05:55 MIN
The security risks of AI-generated code and slopsquatting
Slopquatting, API Keys, Fun with Fonts, Recruiters vs AI and more - The Best of LIVE 2025 - Part 2
03:45 MIN
Preventing exposed API keys in AI-assisted development
Slopquatting, API Keys, Fun with Fonts, Recruiters vs AI and more - The Best of LIVE 2025 - Part 2
06:44 MIN
Using Chrome's built-in AI for on-device features
Devs vs. Marketers, COBOL and Copilot, Make Live Coding Easy and more - The Best of LIVE 2025 - Part 3
04:04 MIN
Shifting HR from standard products to AI-powered platforms
Turning People Strategy into a Transformation Engine
04:57 MIN
Increasing the value of talk recordings post-event
Cat Herding with Lions and Tigers - Christian Heilmann
Featured Partners
Related Videos
25:17AI: Superhero or Supervillain? How and Why with Scott Hanselman
Scott Hanselman
28:53Agentic DevOps: How AI-Powered Automation Transforms Software Delivery on GitHub and Azure
Mike
33:28Prompt Engineering - an Art, a Science, or your next Job Title?
Maxim Salnikov
24:11You are not my model anymore - understanding LLM model behavior
Andreas Erben
42:38Innovating Developer Tools with AI: Insights from GitHub Next
Krzystof Czieslak
32:26Bringing the power of AI to your application.
Krzysztof Cieślak
35:16How AI Models Get Smarter
Ankit Patel
26:47The State of GenAI & Machine Learning in 2025
Alejandro Saucedo
Related Articles
View all articles
From learning to earning
Jobs that call for the skills explored in this talk.
Forschungszentrum Jülich GmbH
Jülich, Germany
Intermediate
Senior
Linux
Docker
AI Frameworks
Machine Learning
Mindrift
Remote
£41K
Junior
JSON
Python
Data analysis
+1
Speechify
€70-90K
Azure
Python
Docker
Kubernetes
+1