WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2025 • Aug 20, 2025 • Session details

Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models

Kevin Lewis , Sandra Ahlgrimm

Stop relying on vibes for prompt engineering. Treat AI evaluation with strict determinism. GitHub Models brings CI/CD and version-controlled testing directly into your codebase.

Pause
Mute Enter Fullscreen
#1 about 3 min

The challenge of testing AI prompts without tools

Why developers rely on vibe checks instead of systematic tests for non-deterministic AI outputs.

#2 about 1 min

Introduction to the GitHub Models testing toolkit

An overview of accessing over 40 models using a single API key for prompt experimentation.

#3 about 2 min

Managing and versioning system prompts as YAML files

How to configure system instructions and parameters using YAML configuration directly in your repository.

#4 about 3 min

Configuring generation parameters and committing prompt changes

Adjusting temperature and token limits before committing prompt configurations to feature branches.

#5 about 3 min

Evaluating prompt accuracy against expected test outcomes

Establishing test datasets and using LLM judges to ensure accuracy and coherence.

#6 about 3 min

Running simultaneous comparisons across different AI models

Experimenting with multiple models and prompt variants simultaneously to optimize latency and cost.

#7 about 2 min

Integrating GitHub Models into external enterprise applications

Consuming managed prompts using model API endpoints or the Azure AI Inference SDK.

#8 about 4 min

Building applications with Azure AI Inference SDK

Code walkthrough showing how a JavaScript application imports an SDK to consume repository configurations.

#9 about 2 min

Evaluating AI models locally using GitHub CLI

Running interactive sessions and local evaluations directly from the terminal before committing code changes.

#10 about 4 min

Automating repository issues and changelogs with AI actions

Leveraging inference integrations inside GitHub Actions to automate pull request summaries and triage bug reports.

#11 about 1 min

Running prompt evaluation tests in CI/CD pipelines

Running automated evaluations during pull requests prevents degraded prompts from reaching production pipelines.

#12 about 3 min

Migrating existing applications to managed prompt architectures

Converting plain text instructions to standard YAML files dramatically accelerates testing and deployment workflows.

Matching moments

1:55 min

Evaluating AI prompts as code with automated CI pipelines

Akshay Nagpal Akshay Nagpal · Europe 2026 Virtual

4:04 min

Introduction to GitHub Next and AI prototyping

Krzystof Czieslak · LIVE

1:11 min

Testing AI models quickly using GitHub Models

Ricardo Ricardo · World Congress 2025

1:48 min

Exploring AI integrations in modern agile development workflows

Markus Walker Markus Walker · World Congress 2023

1:31 min

The shift from human prompt engineering to AI-generated prompts

Vitaly Friedman Vitaly Friedman · World Congress 2026 Europe

2:48 min

Integrating version control and AI into productivity software

More from World Congress 2025

GenAI Security: Navigating the Unseen Iceberg thumbnail
24:33
11 months ago

GenAI Security: Navigating the Unseen Iceberg

Maish Saidel-Keesing

Behind the Code: How Women Are Powering the Future of AI thumbnail
30:28
about 1 year ago

Behind the Code: How Women Are Powering the Future of AI

Alexandra Wudel, Laura Moritz, Madalina Florean

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise? thumbnail
28:12
about 1 year ago

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise?

Justus Lumpe, Konstantin Vinogradov, Nils Eiteneyer, Stephan Wirris

AI in Leadership: How Technology is Reshaping Executive Roles thumbnail
28:19
about 1 year ago

AI in Leadership: How Technology is Reshaping Executive Roles

Bernd Datler, Jasmin Kaiser, Jeff Hausmann, Sonja Alvarez

The AI-Ready Stack: Rethinking the Engineering Org of the Future thumbnail
28:09
about 1 year ago

The AI-Ready Stack: Rethinking the Engineering Org of the Future

Alex Laubscher, Jan Oberhauser, Keno Dreßel, Mirko Novakovic

AI Sovereignty: What Does It Take? thumbnail
29:13
about 1 year ago

AI Sovereignty: What Does It Take?

Boris Hecker, Clemens Wasner, Robin Hermann, Sonja Alvarez

Upcoming sessions on this topic

Open session

World Congress 2026 North America

September 24, 2026 · 10:20–10:50

Stage 6

Practices, Not Prompts: Scale GitHub Copilot with AI-Ready Repositories

Luis Pujols

Staff Customer Success Architect, GitHub

Luis Pujols
Open session

World Congress 2026 North America

September 24, 2026 · 16:10–16:40

Stage 1

Sandboxing the Swarm: Building Secure, Serverless AI Agents with Wasm

Lena Hall, Thorsten Hans

Lena Hall
Thorsten Hans
Open session

World Congress 2026 North America

September 23, 2026 · 15:40–16:10

Stage 2

Lean Intelligence: Lessons from GitHub Copilot Data Science Efforts

Rahul Pandita

Researcher and Technical Advisor at Microsoft

Rahul Pandita
Open session

World Congress 2026 North America

September 23, 2026 · 13:40–14:10

Stage 2

Building an AI-Native Development Workflow

Vanessa Minik

Senior Service Delivery Engineer, GitHub

Vanessa Minik
Open session

World Congress 2026 North America

September 24, 2026 · 14:25–14:35

Outdoor Stage

SecurePrompt: Building a Pre-Flight Security Layer for Agentic AI

Ravi Sastry Kadali

AI/ML Engineer at General Motors

Ravi Sastry Kadali
Open session

World Congress 2026 North America

September 24, 2026 · 11:00–11:30

Stage 6

Accelerating development and managing agents with the GitHub Copilot app

Christopher Harrison

Senior Developer Advocate at GitHub

Christopher Harrison

Related videos

Bringing the power of AI to your application. thumbnail
32:26
World Congress 2024 • about 2 years ago

Bringing the power of AI to your application.

Krzysztof Cieślak

Validating Real-World Usefulness of AI Models - Julia Kasper thumbnail
22:00
Coffee With Developers • 24 days ago

Validating Real-World Usefulness of AI Models - Julia Kasper

Julia Kasper

Innovating Developer Tools with AI: Insights from GitHub Next thumbnail
42:38
LIVE • over 1 year ago

Innovating Developer Tools with AI: Insights from GitHub Next

Krzystof Czieslak

The Limits of Prompting: ArchitectingTrustworthy Coding Agents thumbnail
24:05
World Congress 2025 • about 1 year ago

The Limits of Prompting: ArchitectingTrustworthy Coding Agents

Nimrod Kor

The Power of Prompting with AI Native Development - Simon Maple thumbnail
43:08
Coffee With Developers • over 1 year ago

The Power of Prompting with AI Native Development - Simon Maple

Simon Maple

Beyond the IDE: A new era of agent collaboration thumbnail
27:12
World Congress 2025 • about 1 year ago

Beyond the IDE: A new era of agent collaboration

Ryan J. Salva

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.