WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2025 • Aug 20, 2025 • Session details

Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models

Kevin Lewis , Sandra Ahlgrimm

Stop relying on vibes for prompt engineering. Treat AI evaluation with strict determinism. GitHub Models brings CI/CD and version-controlled testing directly into your codebase.

Pause
Mute Enter Fullscreen
#1 about 3 min

The challenge of testing AI prompts without tools

Why developers rely on vibe checks instead of systematic tests for non-deterministic AI outputs.

#2 about 1 min

Introduction to the GitHub Models testing toolkit

An overview of accessing over 40 models using a single API key for prompt experimentation.

#3 about 2 min

Managing and versioning system prompts as YAML files

How to configure system instructions and parameters using YAML configuration directly in your repository.

#4 about 3 min

Configuring generation parameters and committing prompt changes

Adjusting temperature and token limits before committing prompt configurations to feature branches.

#5 about 3 min

Evaluating prompt accuracy against expected test outcomes

Establishing test datasets and using LLM judges to ensure accuracy and coherence.

#6 about 3 min

Running simultaneous comparisons across different AI models

Experimenting with multiple models and prompt variants simultaneously to optimize latency and cost.

#7 about 2 min

Integrating GitHub Models into external enterprise applications

Consuming managed prompts using model API endpoints or the Azure AI Inference SDK.

#8 about 4 min

Building applications with Azure AI Inference SDK

Code walkthrough showing how a JavaScript application imports an SDK to consume repository configurations.

#9 about 2 min

Evaluating AI models locally using GitHub CLI

Running interactive sessions and local evaluations directly from the terminal before committing code changes.

#10 about 4 min

Automating repository issues and changelogs with AI actions

Leveraging inference integrations inside GitHub Actions to automate pull request summaries and triage bug reports.

#11 about 1 min

Running prompt evaluation tests in CI/CD pipelines

Running automated evaluations during pull requests prevents degraded prompts from reaching production pipelines.

#12 about 3 min

Migrating existing applications to managed prompt architectures

Converting plain text instructions to standard YAML files dramatically accelerates testing and deployment workflows.

Matching moments

1:55 min

Evaluating AI prompts as code with automated CI pipelines

Akshay Nagpal Akshay Nagpal · Europe 2026 Virtual

4:04 min

Introduction to GitHub Next and AI prototyping

Krzystof Czieslak · LIVE

1:11 min

Testing AI models quickly using GitHub Models

Ricardo Ricardo · WWC 2025

1:48 min

Exploring AI integrations in modern agile development workflows

Markus Walker Markus Walker · WWC 2023

1:31 min

The shift from human prompt engineering to AI-generated prompts

Vitaly Friedman Vitaly Friedman · WWC Europe 2026

2:48 min

Integrating version control and AI into productivity software

More from World Congress 2025

GenAI Security: Navigating the Unseen Iceberg thumbnail
24:33
10 months ago

GenAI Security: Navigating the Unseen Iceberg

Maish Saidel-Keesing

Behind the Code: How Women Are Powering the Future of AI thumbnail
30:28
12 months ago

Behind the Code: How Women Are Powering the Future of AI

Alexandra Wudel, Laura Moritz, Madalina Florean

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise? thumbnail
28:12
12 months ago

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise?

Justus Lumpe, Konstantin Vinogradov, Nils Eiteneyer, Stephan Wirris

AI in Leadership: How Technology is Reshaping Executive Roles thumbnail
28:19
12 months ago

AI in Leadership: How Technology is Reshaping Executive Roles

Bernd Datler, Jasmin Kaiser, Jeff Hausmann, Sonja Alvarez

The AI-Ready Stack: Rethinking the Engineering Org of the Future thumbnail
28:09
12 months ago

The AI-Ready Stack: Rethinking the Engineering Org of the Future

Alex Laubscher, Jan Oberhauser, Keno Dreßel, Mirko Novakovic

AI Sovereignty: What Does It Take? thumbnail
29:13
12 months ago

AI Sovereignty: What Does It Take?

Boris Hecker, Clemens Wasner, Robin Hermann, Sonja Alvarez

Upcoming sessions on this topic

Open session

World Congress 2026 North America

SecurePrompt: Building a Pre-Flight Security Layer for Agentic AI

Ravi Sastry Kadali

AI/ML Engineer at General Motors

Ravi Sastry Kadali
Open session

World Congress 2026 North America

Reinventing Testing Practices in the AI Era

Eric Deandrea

Java Champion & Senior Principal Software Engineer, IBM

Eric Deandrea
Open session

World Congress 2026 North America

Beyond Vibe Coding: Using CLI Tools as Your AI Counterpart in Pair Programming

Ron Veen

Java enthusiast and Special agent for Team Rockstars IT

Ron Veen
Open session

World Congress 2026 North America

Vibe Coding Accessibility

Karl Groves

Focused on actively fixing accessibility

Karl Groves
Open session

World Congress 2026 North America

GitHub’s Team X-Ray: Your Repository Knows More About Your Team Than Your Team Does

Andrea Griffiths

Senior Developer Advocate

Andrea Griffiths
Open session

World Congress 2026 North America

Ship 10x Faster: AI-Powered Development with Claude Code and MCP Tools

Viktoria Semaan

Principal Technical Evangelist at Databricks

Viktoria Semaan

Related videos

Bringing the power of AI to your application. thumbnail
32:26
WWC 2024 • almost 2 years ago

Bringing the power of AI to your application.

Krzysztof Cieślak

Innovating Developer Tools with AI: Insights from GitHub Next thumbnail
42:38
LIVE • over 1 year ago

Innovating Developer Tools with AI: Insights from GitHub Next

Krzystof Czieslak

The Limits of Prompting: ArchitectingTrustworthy Coding Agents thumbnail
24:05
WWC 2025 • 12 months ago

The Limits of Prompting: ArchitectingTrustworthy Coding Agents

Nimrod Kor

The Power of Prompting with AI Native Development - Simon Maple thumbnail
43:08
Coffee With Developers • over 1 year ago

The Power of Prompting with AI Native Development - Simon Maple

Simon Maple

Beyond the IDE: A new era of agent collaboration thumbnail
27:12
WWC 2025 • 12 months ago

Beyond the IDE: A new era of agent collaboration

Ryan J. Salva

Building and Modernising Apps with Agentic AI - Julia Kordick thumbnail
38:01
Coffee With Developers • 5 months ago

Building and Modernising Apps with Agentic AI - Julia Kordick

Julia Kordick

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.