WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2025 • Aug 20, 2025 • Session details

Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models

Kevin Lewis , Sandra Ahlgrimm

Stop relying on vibes for prompt engineering. Treat AI evaluation with strict determinism. GitHub Models brings CI/CD and version-controlled testing directly into your codebase.

Pause
Mute Enter Fullscreen
#1 about 3 min

The challenge of testing AI prompts without tools

Why developers rely on vibe checks instead of systematic tests for non-deterministic AI outputs.

#2 about 1 min

Introduction to the GitHub Models testing toolkit

An overview of accessing over 40 models using a single API key for prompt experimentation.

#3 about 2 min

Managing and versioning system prompts as YAML files

How to configure system instructions and parameters using YAML configuration directly in your repository.

#4 about 3 min

Configuring generation parameters and committing prompt changes

Adjusting temperature and token limits before committing prompt configurations to feature branches.

#5 about 3 min

Evaluating prompt accuracy against expected test outcomes

Establishing test datasets and using LLM judges to ensure accuracy and coherence.

#6 about 3 min

Running simultaneous comparisons across different AI models

Experimenting with multiple models and prompt variants simultaneously to optimize latency and cost.

#7 about 2 min

Integrating GitHub Models into external enterprise applications

Consuming managed prompts using model API endpoints or the Azure AI Inference SDK.

#8 about 4 min

Building applications with Azure AI Inference SDK

Code walkthrough showing how a JavaScript application imports an SDK to consume repository configurations.

#9 about 2 min

Evaluating AI models locally using GitHub CLI

Running interactive sessions and local evaluations directly from the terminal before committing code changes.

#10 about 4 min

Automating repository issues and changelogs with AI actions

Leveraging inference integrations inside GitHub Actions to automate pull request summaries and triage bug reports.

#11 about 1 min

Running prompt evaluation tests in CI/CD pipelines

Running automated evaluations during pull requests prevents degraded prompts from reaching production pipelines.

#12 about 3 min

Migrating existing applications to managed prompt architectures

Converting plain text instructions to standard YAML files dramatically accelerates testing and deployment workflows.

Matching moments

1:55 min

Evaluating AI prompts as code with automated CI pipelines

Akshay Nagpal Akshay Nagpal · Europe 2026 Virtual

4:04 min

Introduction to GitHub Next and AI prototyping

Krzystof Czieslak · LIVE

1:11 min

Testing AI models quickly using GitHub Models

Ricardo Ricardo · World Congress 2025

1:48 min

Exploring AI integrations in modern agile development workflows

Markus Walker Markus Walker · World Congress 2023

1:31 min

The shift from human prompt engineering to AI-generated prompts

Vitaly Friedman Vitaly Friedman · World Congress 2026 Europe

2:48 min

Integrating version control and AI into productivity software

More from World Congress 2025

GenAI Security: Navigating the Unseen Iceberg thumbnail
24:33
11 months ago

GenAI Security: Navigating the Unseen Iceberg

Maish Saidel-Keesing

Behind the Code: How Women Are Powering the Future of AI thumbnail
30:28
about 1 year ago

Behind the Code: How Women Are Powering the Future of AI

Alexandra Wudel, Laura Moritz, Madalina Florean

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise? thumbnail
28:12
about 1 year ago

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise?

Justus Lumpe, Konstantin Vinogradov, Nils Eiteneyer, Stephan Wirris

AI in Leadership: How Technology is Reshaping Executive Roles thumbnail
28:19
about 1 year ago

AI in Leadership: How Technology is Reshaping Executive Roles

Bernd Datler, Jasmin Kaiser, Jeff Hausmann, Sonja Alvarez

The AI-Ready Stack: Rethinking the Engineering Org of the Future thumbnail
28:09
about 1 year ago

The AI-Ready Stack: Rethinking the Engineering Org of the Future

Alex Laubscher, Jan Oberhauser, Keno Dreßel, Mirko Novakovic

AI Sovereignty: What Does It Take? thumbnail
29:13
about 1 year ago

AI Sovereignty: What Does It Take?

Boris Hecker, Clemens Wasner, Robin Hermann, Sonja Alvarez

Related videos

Bringing the power of AI to your application. thumbnail
32:26
World Congress 2024 • about 2 years ago

Bringing the power of AI to your application.

Krzysztof Cieślak

Validating Real-World Usefulness of AI Models - Julia Kasper thumbnail
22:00
Coffee With Developers • about 1 month ago

Validating Real-World Usefulness of AI Models - Julia Kasper

Julia Kasper

Innovating Developer Tools with AI: Insights from GitHub Next thumbnail
42:38
LIVE • almost 2 years ago

Innovating Developer Tools with AI: Insights from GitHub Next

Krzystof Czieslak

The Limits of Prompting: ArchitectingTrustworthy Coding Agents thumbnail
24:05
World Congress 2025 • about 1 year ago

The Limits of Prompting: ArchitectingTrustworthy Coding Agents

Nimrod Kor

The Power of Prompting with AI Native Development - Simon Maple thumbnail
43:08
Coffee With Developers • over 1 year ago

The Power of Prompting with AI Native Development - Simon Maple

Simon Maple

Beyond the IDE: A new era of agent collaboration thumbnail
27:12
World Congress 2025 • about 1 year ago

Beyond the IDE: A new era of agent collaboration

Ryan J. Salva

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.