> Markdown version of [/videos/158-serverless-deployment-of-large-nlp-models](https://www.wearedevelopers.com/videos/158-serverless-deployment-of-large-nlp-models). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Serverless deployment of (large) NLP models 

Can you run massive NLP models within AWS Lambda's strict 250MB limit? Discover how Slido used knowledge distillation and ONNX to achieve sub-100ms serverless inference.

- **Speakers:** Marek Suppa
- **Event:** WeAreDevelopers LIVE
- **Published:** May 26, 2021
- **Duration:** 39:02
- **URL:** https://www.wearedevelopers.com/videos/158-serverless-deployment-of-large-nlp-models

## Summary

Integrating machine learning capabilities into a TypeScript-dominated infrastructure often introduces intense operational friction due to the Python-centric nature of most NLP frameworks. To bypass heavy DevOps overhead and server maintenance, the data team at Slido engineered a pipeline to run complex natural language processing tasks—like sentiment analysis and semantic similarity detection—directly on serverless architecture using AWS Lambda. However, serverless environments impose strict constraints, notably a 250MB deployment package limit and the absence of GPU acceleration. Standard pre-trained models like bert-base, combined with massive inference libraries like PyTorch, grossly exceed these boundaries.

To compress the footprint without significantly degrading accuracy, the team leveraged a robust knowledge distillation strategy. By fine-tuning a massive teacher model on highly specific domain data and using it to generate a vast silver dataset, they effectively trained highly compressed student models, such as TinyBERT and MobileBERT. This reduced overall model sizes from roughly 400MB to under 100MB. To completely resolve framework bloat, PyTorch was entirely stripped from the deployment artifact. Instead, models were converted to the Open Neural Network Exchange format and served using ONNX Runtime, which requires a mere 14MB operational footprint.

Beyond raw size reduction, optimizing serverless execution speed proved vital for ensuring real-time user experiences. Because AWS Lambda proportionally links virtual CPU allocation to provisioned memory, increasing a function's RAM directly accelerates execution. Bumping the allocation to 1GB slashed inference durations to under 100 milliseconds, comfortably meeting production latency thresholds. Ultimately, this architecture makes advanced text classification highly accessible, demonstrating that serverless NLP is not only viable but highly cost-effective when analyzing irregular or event-driven inference workloads.

**Keywords:** serverless nlp deployment, aws lambda cold starts, knowledge distillation strategy, bert model compression, tinybert and mobilebert, onnx runtime inference, removing pytorch dependencies, semantic similarity algorithms, sentiment analysis models, aws lambda memory scaling, cross-stack ml development, natural language processing pipelines, event-driven architecture, audience interaction analytics

## Chapters

1. **Machine learning opportunities within audience interaction platforms** (00:18) — Assessing large text datasets from audience interactions enables automated insights and predictive event intelligence.
1. **Extracting key phrases and analyzing sentiment from audience questions** (02:53) — Applying text preprocessing algorithms reveals event themes and participant emotions without manual oversight.
1. **Detecting similar questions and generating automated quiz responses** (05:12) — Algorithmic grouping of participant questions reduces host redundancy while generative capabilities populate alternative quiz options dynamically.
1. **Moving to serverless deployments for Python machine learning models** (08:53) — Transitioning to serverless architecture bridges integration gaps between a TypeScript platform environment and a Python machine learning stack.
1. **Addressing constraints and limitations of serverless model deployments** (11:45) — Working within serverless functions requires mitigating cold start delays, operating without hardware acceleration, and adapting to fixed file sizes.
1. **Fine-tuning BERT models for audience sentiment analysis classification** (14:24) — Training standard language representations on domain-specific event text improves sentiment scoring accuracy for non-standard sentences.
1. **Reducing model size using knowledge distillation with teacher-student models** (17:50) — Using heavy teacher models to label intermediate datasets trains compact student networks to achieve comparable predictive accuracy strictly inside serverless memory constraints.
1. **Optimizing similar question detection with linear complexity sentence encoders** (22:24) — Replacing standard dual-input classification with isolated pooled vectors drastically decreases computational complexity when comparing live text logs.
1. **Distilling cross-encoder models into smaller efficient sentence embedding models** (26:22) — Leveraging resource-heavy cross-encoder outputs as training subsets for bi-encoders yields micro models capable of continuous semantic matching.
1. **Replacing PyTorch with ONNX runtime for AWS Lambda deployments** (30:34) — Swapping comprehensive deep learning frameworks for a minimalist neural exchange runtime forces overall application payloads below strict Lambda limits.
1. **Evaluating cost effectiveness and summarizing serverless machine learning strategies** (35:16) — Deploying optimized machine learning models via reactive functions proves highly economical against maintaining hosting environments for sporadic traffic.

## Related Moments

- [Scaling model inference leveraging serverless edge functions](https://www.wearedevelopers.com/videos/100355-running-secure-life-science-research-at-scale-using-hybrid-gpu-hpc-and-kubernetes) (from "Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬")
- [Reducing infrastructure capacity requirements by deploying small language models](https://www.wearedevelopers.com/videos/1096-architecting-the-future-leveraging-ai-cloud-and-data-for-business-success) (from "Architecting the Future: Leveraging AI, Cloud, and Data for Business Success")
- [Executing LoRA fine-tuning using serverless Databricks AI runtimes](https://www.wearedevelopers.com/videos/100151-cutting-llm-costs-without-cutting-quality-how-to-beat-proprietary-llms-with-fine-tuned-open-source) (from "Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source")
- [Introduction to serving large language models locally](https://www.wearedevelopers.com/videos/1619-unveiling-the-magic-scaling-large-language-models-to-serve-millions) (from "Unveiling the Magic: Scaling Large Language Models to Serve Millions")
- [Running lightweight large language models on local hardware](https://www.wearedevelopers.com/videos/829-multimodal-generative-ai-demystified) (from "Multimodal Generative AI Demystified")
- [Exploring hybrid models, monoliths, and serverless computing](https://www.wearedevelopers.com/videos/261-why-you-shouldn-t-build-a-microservice-architecture) (from "Why you shouldn’t build a microservice architecture ")

## Related Articles

- [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production)
- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)
- [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models)
- [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it)

## Related Jobs

- [Staff, Machine Learning Engineer (L4)](https://www.wearedevelopers.com/jobs/ext/1202639-staff-machine-learning-engineer-l4) at **Twilio**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/588393-machine-learning-engineer) at **Twilio**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1355348-machine-learning-engineer) at **TWILIO**
- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**
- [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**
- [Software Engineer, Platform Engineering (L2)](https://www.wearedevelopers.com/jobs/ext/1956829-software-engineer-platform-engineering-l2) at **Twilio**