How we built an AI-powered code reviewer in 80 hours

Our AI prototype was built in a weekend. But productionizing it meant fighting collapsing context windows, high latency, and massive costs.

#1about 2 minutes

An overview of an AI-powered code reviewer

The tool reviews pull requests, leaves inline comments, and provides a dashboard, with the talk focusing on lessons learned from building it.

#2about 3 minutes

The high-level serverless architecture for the application

The system uses GitHub webhooks, EventBridge, Lambda, Amazon Bedrock, and DynamoDB, with Clerk for auth and Stripe for payments.

#3about 4 minutes

Choosing Amazon Bedrock for security and privacy

Amazon Bedrock was selected for its strong security guarantees, data privacy policies, and serverless, token-based pricing model suitable for sensitive customer code.

#4about 5 minutes

The truth about LLM context window size and reasoning

Large context window sizes are misleading because a model's ability to reason over content collapses long before the advertised limit, forcing a one-prompt-per-file strategy.

#5about 3 minutes

Managing API rate limits and model availability

To overcome low default API rate limits, strategies include requesting limit increases, using cross-region inference, and implementing fallbacks to other models for reliability.

#6about 3 minutes

Strategies for controlling high LLM costs

The most effective cost control measure is to analyze only the changed lines in a pull request rather than the entire file, which also improves user experience.

#7about 4 minutes

Handling timeouts with durable execution in Lambda

A lightweight durable execution mechanism using checkpoints in DynamoDB prevents reprocessing files on Lambda retries, which are common due to slow LLM response times.

#8about 4 minutes

Dealing with different types of LLM hallucinations

Hallucinations range from simple invalid JSON to complex errors like suggesting fixes for outdated libraries, which can be mitigated with RAG but at a significant cost.