Kateryna Hrytsaienko

Multilingual NLP pipeline up and running from scratch

Translating text can reduce your NLP model's accuracy by 20%. Learn how to build a single, unified pipeline that handles multiple languages without the loss.

Multilingual NLP pipeline up and running from scratch
#1about 3 minutes

The challenge of building end-to-end NLP pipelines

There is a lack of comprehensive guides for integrating multilingual NLP models into applications with proper CI/CD practices, especially for non-English languages.

#2about 5 minutes

Understanding the core components of an NLP pipeline

A typical NLP pipeline consists of three key stages: pre-processing, feature extraction, and modeling, with pre-processing being critical for handling unstructured data.

#3about 8 minutes

Why simply translating everything to English is not enough

Translating all text to English for NLP analysis can decrease accuracy by up to 20% due to lost semantic nuance and dialectical differences.

#4about 10 minutes

Generalizing languages with stemming and bag-of-words

Handle similar languages by using stemming to find common root words and a bag-of-words model with a similarity index to treat them as a single language.

#5about 5 minutes

Achieving high accuracy with a unified language model

By training classifiers on stemmed and normalized vectors from multiple similar languages, it's possible to achieve high accuracy of around 90% in tasks like topic classification.

#6about 8 minutes

Choosing the right deployment strategy for your model

Decide between embedding your model or exposing it as an API, considering options like serverless for simple cases or Kubernetes for scalable, cloud-agnostic deployments.

#7about 7 minutes

Implementing a CI/CD pipeline for your NLP model

Establish an MLOps workflow with continuous training, integration, and delivery by containerizing your model with Docker and automating builds with tools like GitHub Actions.

#8about 6 minutes

Q&A on slang processing, debugging, and transformers

The Q&A covers practical advice on handling slang with dictionaries, debugging with robust logging, and understanding the complexity gap between traditional methods and transformers like BERT.

Related jobs
Jobs that call for the skills explored in this talk.

Featured Partners

Related Videos

See all videos
DevOps for AI: running LLMs in production with Kubernetes and KubeFlow34:21

DevOps for AI: running LLMs in production with Kubernetes and KubeFlow

Aarno Aukia

Creating Industry ready solutions with LLM Models58:00

Creating Industry ready solutions with LLM Models

Vijay Krishan Gupta & Gauravdeep Singh Lotey

What do language models really learn47:28

What do language models really learn

Tanmay Bakshi

Multimodal Generative AI Demystified56:55

Multimodal Generative AI Demystified

Ekaterina Sirazitdinova

Overview of Machine Learning in Python57:46

Overview of Machine Learning in Python

Adrian Schmitt

From Traction to Production: Maturing your LLMOps step by step41:45

From Traction to Production: Maturing your LLMOps step by step

Maxim Salnikov

The state of MLOps - machine learning in production at enterprise scale47:19

The state of MLOps - machine learning in production at enterprise scale

Bas Geerdink 

The best of both worlds: Combining Python and Kotlin for Machine Learning26:28

The best of both worlds: Combining Python and Kotlin for Machine Learning

Nils Kasseckert

From learning to earning

Jobs that call for the skills explored in this talk.

Rust and GoLang

Rust and GoLang

NHe4a GmbH
Karlsruhe, Germany

Remote
€55-65K
Intermediate
Senior
Go
Rust