> Markdown version of [/videos/1522-adding-knowledge-to-open-source-llms](https://www.wearedevelopers.com/videos/1522-adding-knowledge-to-open-source-llms). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Adding knowledge to open-source LLMs

How do you inject proprietary knowledge into open-source LLMs? Master the pipeline of continued pre-training, supervised fine-tuning, and modern preference alignment to build tailored, domain-specific AI.

- **Speakers:** [Harshita Seth](https://www.wearedevelopers.com/@harshita-seth), [Sergio Perez](https://www.wearedevelopers.com/@sergio-perez)
- **Event:** World Congress 2025
- **Published:** August 20, 2025
- **Duration:** 25:35
- **URL:** https://www.wearedevelopers.com/videos/1522-adding-knowledge-to-open-source-llms

## Summary

Adding new knowledge to open-source Large Language Models (LLMs) is a critical step in overcoming knowledge cutoffs, teaching domain-specific vocabulary like proprietary chip design, and incorporating regional nuances. The lifecycle of embedding this intelligence requires a robust pipeline traversing continued pre-training, supervised fine-tuning, and preference alignment. Continued pre-training utilizes self-supervised next-token prediction to adapt generic foundation models to specialized verticals, producing highly tailored assets like NVIDIA's ChipNeMo. By leveraging end-to-end orchestration tools such as the NVIDIA NeMo framework, developers can utilize granular pipeline and tensor parallelism to scale these massive domain-specific training workflows. <br><br>Moving beyond raw data ingestion, supervised fine-tuning reshapes next-word predictors into capable problem solvers. Historically, architectures required separate neural heads for distinct functional tasks, but modern instruction tuning enables generalized, zero-shot capabilities across varied demands. This conversational methodology expanded significantly when chat-specific tasks were introduced, empowering models to empathize and follow continuous dialogue. Fine-tuning increasingly injects chain of thought reasoning traces into datasets, forcing an LLM to generate internal logic tokens before finalizing an answer to build robust problem-solving behaviors for advanced math and software engineering. <br><br>Ultimately, maximizing model utility demands aligning outputs with human intent—a phase traditionally reliant on reinforcement learning with human feedback. While reinforcement learning has historically been considered "a bit of a dark science" due to volatile reward models and intensive value function evaluations, the AI industry is quickly adopting streamlined alternatives. Algorithms like group relative policy optimization mathematically bypass expensive value constraints, while rule-based rewards eliminate the need for secondary LLM reward judges in objectively verifiable domains. Furthermore, direct preference optimization skips complex policy updating altogether by learning purely from binary good-versus-bad feedback loops, radically democratizing the model alignment process.

**Keywords:** open-source LLM customization, continued pre-training adaptation, domain-specific knowledge injection, supervised fine-tuning methodology, instruction tuning datasets, chain of thought reasoning, self-supervised next-token prediction, reinforcement learning with human feedback, proximal policy optimization, direct preference optimization, group relative policy optimization, open-source NVIDIA NeMo framework, zero-shot instruction generalization, rule-based reward verification, LLM preference alignment

## Chapters

1. **The core stages of large language model training** (00:05) — Understanding the pre-training and alignment stages establishes a foundation for how language models process and refine information.
1. **Why open-source language models need continuous knowledge updates** (02:03) — Updating models with current affairs and domain-specific knowledge prevents information degradation and improves task relevance.
1. **Using continued pre-training for domain-specific model adaptation** (03:50) — Applying self-supervised learning during continued pre-training ensures foundational models adapt accurately to highly specialized industry domains.
1. **Adding reasoning capabilities through supervised fine-tuning tasks** (08:29) — Structuring instruction datasets with chain-of-thought traces teaches neural networks to formulate explainable reasoning steps before answering.
1. **Aligning behavior and human preferences with reinforcement learning** (15:01) — Implementing comparative reward models and direct preference optimization aligns generative textual outputs with human behavioral expectations.
1. **Orchestrating model alignment with the NeMo RL framework** (23:40) — Adopting specialized automation frameworks simplifies the integration of complex reinforcement learning loops without engineering custom training layouts.

## Related Moments

- [Leveraging large language models for code optimization and development](https://www.wearedevelopers.com/videos/1106-the-future-of-computing-ai-technologies-in-the-exascale-era) (from "The Future of Computing: AI Technologies in the Exascale Era")
- [Understanding core parameters and mechanics of large language models](https://www.wearedevelopers.com/videos/1512-building-ai-applications-with-langchain-and-node-js) (from "Building AI Applications with LangChain and Node.js")
- [Applying large language models to infrastructure tasks](https://www.wearedevelopers.com/videos/2084-your-infrastructure-is-not-a-playground-ai-agents-for-infra-done-right) (from "Your Infrastructure Is Not a Playground: AI Agents for Infra Done Right")
- [Enhancing language models with retrieval-augmented generation](https://www.wearedevelopers.com/videos/844-enter-the-brave-new-world-of-genai-with-vector-search) (from "Enter the Brave New World of GenAI with Vector Search")
- [Extending large language models without expensive retraining](https://www.wearedevelopers.com/videos/1513-best-practices-building-enterprise-applications-that-leverage-genai) (from "Best practices: Building Enterprise Applications that leverage GenAI")
- [Maturing through the machine learning development lifecycle](https://www.wearedevelopers.com/videos/587-creating-bots-with-dialogflow-cx) (from "Creating bots with Dialogflow CX")

## Related Articles

- [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models)
- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)
- [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production)
- [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it)

## Related Jobs

- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/588393-machine-learning-engineer) at **Twilio**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1355348-machine-learning-engineer) at **TWILIO**
- [Staff, Machine Learning Engineer (L4)](https://www.wearedevelopers.com/jobs/ext/1202639-staff-machine-learning-engineer-l4) at **Twilio**
- [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**
- [Principal Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1706410-principal-machine-learning-engineer) at **Almedia**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1377841-machine-learning-engineer) at **Almedia**