The core concept is to use a large language model to generate and execute commands within an application, with a human always in the loop for safety.
The journey began with a proprietary model before shifting to GPT-3.5 and building a vendor-agnostic layer to support providers like Azure OpenAI for data residency.
Different LLM providers implement the OpenAI standard inconsistently, requiring model-specific feature flags and custom logic to handle variations in JSON strict mode.
To handle chatty or malformed LLM responses, a custom parser was built to find and extract the first valid JSON block before attempting validation.
A customer syncing their entire Confluence knowledge base led to a massive, unexpected bill, forcing the implementation of token limits and a user-facing credit system.
To meet enterprise security and data residency requirements, customers can configure their own LLM endpoints, ensuring their data never leaves their environment.
The system proposes operations for human approval rather than executing them directly, providing a safety harness to prevent unintended actions like data deletion.
Complex tasks are broken down into a two-step process where one LLM first creates a plan and subsequent, focused LLM calls execute each item individually.
Instead of testing for exact output, tests validate the intent of the LLM's action and are designed to pass if they succeed a majority of the time.
A control plane allows enterprise customers to customize system prompts for their specific needs, and a research project explored using specialized prompts for automated GDPR reviews.
Treat LLMs as unreliable APIs, always validate their output, manage costs proactively, avoid vendor lock-in, and keep a human in the loop for all critical operations.
Jobs that call for the skills explored in this talk.