WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
Coffee With Developers • Aug 17, 2026

Validating Real-World Usefulness of AI Models - Julia Kasper

Julia Kasper

There is no universally superior AI model. Frontend developers prefer Claude, while backend engineers favor GPT. Julia Kasper explains how to rigorously evaluate AI for your IDE.

Pause
Mute Enter Fullscreen
#1 about 2 min

Introduction to AI features in Visual Studio Code

An overview of launching AI models and Copilot capabilities inside the code editor.

#2 about 2 min

Developer adoption of integrated AI chat features

How traditional developers and AI-first coders utilize new integrated chat interfaces differently.

#3 about 3 min

Integrating new third-party AI models into environments

How the team manages quick turnarounds and policy restrictions for incoming AI model releases.

#4 about 3 min

Evaluating non-deterministic AI models with offline testing

Using traditional assertion-based test scripts to measure the reliability of non-deterministic AI responses.

#5 about 3 min

Choosing AI models based on task and personality

Balancing token costs and task efficiencies by evaluating benchmarks and specific model characteristics.

#6 about 2 min

Tracking model quality improvements and potential saturation

Observing incremental performance gains across updates despite a sense of overall AI capability saturation.

#7 about 2 min

Connecting custom local models versus optimized built-in providers

How developers can bring their own API keys while native models benefit from specialized system prompt optimization.

#8 about 3 min

Preparing system prompts and evaluating models before launch

Refining prompts and running internal dogfooding sessions to validate model quality before public release.

#9 about 1 min

Performing local A/B testing across multiple AI agents

Running parallel sessions with different local and remote agents to directly compare model outputs.

#10 about 2 min

Inspecting open-source system prompts to understand model behavior

Reviewing public prompts allows external contributors to suggest tweaks and better understand AI nuance.

#11 about 2 min

Navigating user questions on selecting the optimal model

Addressing developer inquiries by recommending the latest updates and switching models based on task context.

#12 about 2 min

Maintaining an open-source core amidst proprietary editor forks

Navigating the challenges of third-party forks while continuing to value the open-source community mindset.

#13 about 3 min

Leveraging AI to lower the barrier for external contributions

Using AI tools to explain large codebases empowers external developers to make their first pull requests.

Matching moments

1:48 min

Exploring AI integrations in modern agile development workflows

Markus Walker Markus Walker · World Congress 2023

2:41 min

Leveraging AI tooling for accelerated learning and productivity

Dan Mindru · Coffee With Developers

1:56 min

Analyzing cloud-based AI code completion architectures

Daniel Savenkov Daniel Savenkov · World Congress 2024

5:16 min

Motivations for adopting AI to enhance developer productivity

1:55 min

Shifting developer workloads and realistic AI productivity gains

Chris Heilmann Chris Heilmann +2 · LIVE

2:26 min

Customizing developer tools and integrating domain-specific AI agents

Francisco Carneiro Francisco Carneiro · World Congress 2025

More from Coffee With Developers

Using Agents to Curate the Content You See Online - Dan Gish thumbnail
12:47
about 1 month ago

Using Agents to Curate the Content You See Online - Dan Gish

Dan Gish

AI Vector Search at Scale - Ewa Szyszka - Ewa Szyszka thumbnail
11:14
about 1 month ago

AI Vector Search at Scale - Ewa Szyszka - Ewa Szyszka

Ewa Szyszka

What is Cross-Origin Storage - Thomas Steiner thumbnail
34:51
about 1 month ago

What is Cross-Origin Storage - Thomas Steiner

Thomas Steiner

Women in AI - Nalini Garg thumbnail
42:19
about 1 month ago

Women in AI - Nalini Garg

Nalini Garg

Agents and People Dynamics in Shared Chats - Sam Liu thumbnail
37:41
about 1 month ago

Agents and People Dynamics in Shared Chats - Sam Liu

Sam Liu

Why Token Reduction Isn’t Cost Reduction - Dave Anderson & Sarel Weinberger, PhD. thumbnail
41:09
about 2 months ago

Why Token Reduction Isn’t Cost Reduction - Dave Anderson & Sarel Weinberger, PhD.

Dave Anderson, Sarel Weinberger, Phd

Related videos

WeAreDevelopers LIVE - Is AI replacing developers?, Stopping bots, AI on device & more thumbnail
1:01:03
LIVE • over 1 year ago

WeAreDevelopers LIVE - Is AI replacing developers?, Stopping bots, AI on device & more

Chris Heilmann, Daniel Cranney, Sebastian Gingter

Innovating Developer Tools with AI: Insights from GitHub Next thumbnail
42:38
LIVE • almost 2 years ago

Innovating Developer Tools with AI: Insights from GitHub Next

Krzystof Czieslak

Developer Productivity Using AI Tools and Services - Ryan J Salva thumbnail
59:16
Coffee With Developers • over 1 year ago

Developer Productivity Using AI Tools and Services - Ryan J Salva

Ryan J Salva

Boost your coding productivity with Github Copilot Agent thumbnail
28:40
World Congress 2025 • about 1 year ago

Boost your coding productivity with Github Copilot Agent

Dr. Alexander Wachtel, Julia Kordick

Bringing the power of AI to your application. thumbnail
32:26
World Congress 2024 • about 2 years ago

Bringing the power of AI to your application.

Krzysztof Cieślak

Azure AI Foundry for Developers: Open Tools, Scalable Agents, Real Impact thumbnail
23:33
World Congress 2025 • about 1 year ago

Azure AI Foundry for Developers: Open Tools, Scalable Agents, Real Impact

Oliver Will

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.