WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
Coffee With Developers • Aug 17, 2026

Validating Real-World Usefulness of AI Models - Julia Kasper

Julia Kasper

There is no universally superior AI model. Frontend developers prefer Claude, while backend engineers favor GPT. Julia Kasper explains how to rigorously evaluate AI for your IDE.

Pause
Mute Enter Fullscreen
#1 about 2 min

Introduction to AI features in Visual Studio Code

An overview of launching AI models and Copilot capabilities inside the code editor.

#2 about 2 min

Developer adoption of integrated AI chat features

How traditional developers and AI-first coders utilize new integrated chat interfaces differently.

#3 about 3 min

Integrating new third-party AI models into environments

How the team manages quick turnarounds and policy restrictions for incoming AI model releases.

#4 about 3 min

Evaluating non-deterministic AI models with offline testing

Using traditional assertion-based test scripts to measure the reliability of non-deterministic AI responses.

#5 about 3 min

Choosing AI models based on task and personality

Balancing token costs and task efficiencies by evaluating benchmarks and specific model characteristics.

#6 about 2 min

Tracking model quality improvements and potential saturation

Observing incremental performance gains across updates despite a sense of overall AI capability saturation.

#7 about 2 min

Connecting custom local models versus optimized built-in providers

How developers can bring their own API keys while native models benefit from specialized system prompt optimization.

#8 about 3 min

Preparing system prompts and evaluating models before launch

Refining prompts and running internal dogfooding sessions to validate model quality before public release.

#9 about 1 min

Performing local A/B testing across multiple AI agents

Running parallel sessions with different local and remote agents to directly compare model outputs.

#10 about 2 min

Inspecting open-source system prompts to understand model behavior

Reviewing public prompts allows external contributors to suggest tweaks and better understand AI nuance.

#11 about 2 min

Navigating user questions on selecting the optimal model

Addressing developer inquiries by recommending the latest updates and switching models based on task context.

#12 about 2 min

Maintaining an open-source core amidst proprietary editor forks

Navigating the challenges of third-party forks while continuing to value the open-source community mindset.

#13 about 3 min

Leveraging AI to lower the barrier for external contributions

Using AI tools to explain large codebases empowers external developers to make their first pull requests.

Matching moments

1:48 min

Exploring AI integrations in modern agile development workflows

Markus Walker Markus Walker · World Congress 2023

2:41 min

Leveraging AI tooling for accelerated learning and productivity

Dan Mindru · Coffee With Developers

1:56 min

Analyzing cloud-based AI code completion architectures

Daniel Savenkov Daniel Savenkov · World Congress 2024

5:16 min

Motivations for adopting AI to enhance developer productivity

1:55 min

Shifting developer workloads and realistic AI productivity gains

Chris Heilmann +2 · LIVE

2:26 min

Customizing developer tools and integrating domain-specific AI agents

Francisco Carneiro Francisco Carneiro · World Congress 2025

More from Coffee With Developers

What to Do About Hackathons in the Time of Agents - Mike Swift thumbnail
46:03
17 days ago

What to Do About Hackathons in the Time of Agents - Mike Swift

Mike Swift

Moving from Playing with AI to Implementing AI - Lee Stott thumbnail
18:43
21 days ago

Moving from Playing with AI to Implementing AI - Lee Stott

Lee Stott

Evals vs. Evil - AI and Package Security - Laurie Voss thumbnail
16:57
30 days ago

Evals vs. Evil - AI and Package Security - Laurie Voss

Laurie Voss

Burning Satellites, Talking Plants and Agents Hiring Each Other - Fake or News with Matt Zeunert thumbnail
4:57
about 1 month ago

Burning Satellites, Talking Plants and Agents Hiring Each Other - Fake or News with Matt Zeunert

Matt Zeunert

The Open-source Java SDK for Multi-Cloud Development - Sandeep Pal thumbnail
18:11
about 2 months ago

The Open-source Java SDK for Multi-Cloud Development - Sandeep Pal

Sandeep Pal

Web Scraping, AI Agents, and the Future of Open Source - Kevin Lewis (Apify) thumbnail
40:29
about 2 months ago

Web Scraping, AI Agents, and the Future of Open Source - Kevin Lewis (Apify)

Upcoming sessions on this topic

Open session

World Congress 2026 North America

The Reality of AI Adoption in Enterprises

Priyanka Vergadia

Cloud and AI expert @Microsoft, previously lead Devrel @Google

Priyanka Vergadia
Open session

World Congress 2026 North America

Beyond Vibe Coding: Using CLI Tools as Your AI Counterpart in Pair Programming

Ron Veen

Java enthusiast and Special agent for Team Rockstars IT

Ron Veen
Open session

World Congress 2026 North America

Vibe Coding Accessibility

Karl Groves

Focused on actively fixing accessibility

Karl Groves
Open session

World Congress 2026 North America

From Model Selection to Smart Routing: How to Use the Right LLM for Every Task

Viktoria Semaan

Principal Technical Evangelist at Databricks

Viktoria Semaan
Open session

World Congress 2026 North America

RoboCoders: Judgment Day: AI-Assisted Engineering Applied - The Battle of Agents

Baruch Sadogursky, Viktor Gamov

Baruch Sadogursky
Viktor Gamov
Open session

World Congress 2026 North America

Building Stuff with GenAI - The Open Minded Workshop beyond OpenAI

Andreas Erben

CTO for Applied AI and Metaverse at daenet

Andreas Erben

Related videos

WeAreDevelopers LIVE - Is AI replacing developers?, Stopping bots, AI on device & more thumbnail
1:01:03
LIVE • over 1 year ago

WeAreDevelopers LIVE - Is AI replacing developers?, Stopping bots, AI on device & more

Chris Heilmann, Daniel Cranney, Sebastian Gingter

Innovating Developer Tools with AI: Insights from GitHub Next thumbnail
42:38
LIVE • over 1 year ago

Innovating Developer Tools with AI: Insights from GitHub Next

Krzystof Czieslak

Developer Productivity Using AI Tools and Services - Ryan J Salva thumbnail
59:16
Coffee With Developers • about 1 year ago

Developer Productivity Using AI Tools and Services - Ryan J Salva

Ryan J Salva

Boost your coding productivity with Github Copilot Agent thumbnail
28:40
World Congress 2025 • 12 months ago

Boost your coding productivity with Github Copilot Agent

Dr. Alexander Wachtel, Julia Kordick

Bringing the power of AI to your application. thumbnail
32:26
World Congress 2024 • almost 2 years ago

Bringing the power of AI to your application.

Krzysztof Cieślak

Azure AI Foundry for Developers: Open Tools, Scalable Agents, Real Impact thumbnail
23:33
World Congress 2025 • 12 months ago

Azure AI Foundry for Developers: Open Tools, Scalable Agents, Real Impact

Oliver Will

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.