Michele Riva

Writing a full-text search engine in TypeScript

Can TypeScript power a search engine that queries a million titles in 41 microseconds? Discover the data structures and algorithms that make it possible.

Writing a full-text search engine in TypeScript
#1about 2 minutes

Why build a full-text search engine from scratch

Building a search engine from scratch is the best way to understand the underlying data structures and algorithms that power it.

#2about 2 minutes

An overview of existing full-text search solutions

Full-text search uses text indexes to quickly find terms, with established solutions like Elasticsearch, Algolia, and newer ones like Meilisearch.

#3about 6 minutes

Preparing text data with tokenization and stemming

Raw text is processed through tokenization, lowercasing, stop-word removal, and stemming to create a clean set of searchable terms.

#4about 6 minutes

Using hash maps to create an inverted index

An inverted index, implemented with a hash map, provides constant-time (O(1)) lookups by mapping search tokens directly to the documents that contain them.

#5about 8 minutes

Optimizing storage space with prefix trees (tries)

Prefix trees, or tries, optimize memory usage by storing common prefixes of words only once, avoiding redundant data storage.

#6about 9 minutes

Implementing typo tolerance with Levenshtein distance

The Levenshtein distance algorithm uses dynamic programming to calculate the "edit distance" between two strings, enabling effective typo tolerance in search queries.

#7about 2 minutes

Introducing Lyra, a fast TypeScript search engine

Lyra is a new, open-source full-text search engine written in TypeScript that achieves microsecond search times by leveraging efficient data structures.

#8about 3 minutes

Q&A on hash functions and memory constraints

The Q&A covers the educational value of custom hash functions, handling acronyms versus stop words, and Lyra's current in-memory architecture.

Related jobs
Jobs that call for the skills explored in this talk.

Matching moments

Q&A on TypeScript, clean code, and algorithms
02:08 MIN

Q&A on TypeScript, clean code, and algorithms

Things I learned while writing high-performance JavaScript applications

The four pillars of high-performance JavaScript
04:53 MIN

The four pillars of high-performance JavaScript

Things I learned while writing high-performance JavaScript applications

Q&A on performance, interfaces, and advanced learning
04:23 MIN

Q&A on performance, interfaces, and advanced learning

Where we're going we don't need JavaScript - Programming with Type Annotations

Uncovering truths about TypeScript, tracking, and data weight
01:03 MIN

Uncovering truths about TypeScript, tracking, and data weight

Heavy Hard Drives, Car Tyre Trackers and One Big $44 Billion Mistake - Kitty Giraudel

Q&A on indexing, aggregations, and OpenSearch vs Elasticsearch
03:31 MIN

Q&A on indexing, aggregations, and OpenSearch vs Elasticsearch

Search and aggregations made easy with OpenSearch and NodeJS

Introducing the core principles of Elasticsearch
04:29 MIN

Introducing the core principles of Elasticsearch

Distributed search under the hood

Reflecting on 30 years of JavaScript and the rise of TypeScript
05:02 MIN

Reflecting on 30 years of JavaScript and the rise of TypeScript

WeAreDevelopers LIVE – Guten TAG, Web Standards, AI and more

Understanding TypeScript's origins and role in scalability
02:37 MIN

Understanding TypeScript's origins and role in scalability

All you need is types

Featured Partners

Related Videos

See all videos
Things I learned while writing high-performance JavaScript applications
27:27

Things I learned while writing high-performance JavaScript applications

Michele Riva

Building software that scales with Typescript
37:18

Building software that scales with Typescript

Tal Joffe

Advanced Typing in TypeScript
28:44

Advanced Typing in TypeScript

Lars Hupel

Lies we Tell Ourselves As Developers
31:13

Lies we Tell Ourselves As Developers

Stefan Baumgartner

Don't compromise on speedy delivery nor type-safety by choosing TypeScript
29:44

Don't compromise on speedy delivery nor type-safety by choosing TypeScript

Jens Claes

From clicks to cribs - How to find your dream home with web scraping
26:08

From clicks to cribs - How to find your dream home with web scraping

Alexander Lichter

Where we're going we don't need JavaScript - Programming with Type Annotations
29:30

Where we're going we don't need JavaScript - Programming with Type Annotations

Peter Kröner

All you need is types
48:56

All you need is types

Tal Joffe

Related Articles

View all articles
Daniel Cranney
Exploring TypeScript: Benefits for Large-Scale JavaScript Projects
JavaScript is the backbone of web development, powering everything from small websites to large-scale enterprise applications. However, as projects grow in complexity, maintaining JavaScript code can become increasingly difficult. This is where TypeS...
Exploring TypeScript: Benefits for Large-Scale JavaScript Projects
Daniel Cranney
Dev Digest 159: AI Pipelines, 10x Faster TypeScript, How to Interview
Inside last week’s Dev Digest 159 . 🤖 How to use LLMs to help you write code ⚡ How much electricity does AI need? 🔒 Is your API secure? Learn all about hardening it… 🟦 TypeScript switches to go and gets 10 times faster 🖼️ An image cropper in your ap...
Dev Digest 159: AI Pipelines, 10x Faster TypeScript, How to Interview
Chris Heilmann
Dev Digest 151: SEO in an AI world, security fixes and Doomed PDFs
Inside last week’s Dev Digest 151 . 🔎 How ChatGPT compares to search and what that means for SEO ✂️ Job cuts across the board as companies curb DEI programs 🟨 @Microsoft releases 161 Windows security updates ⚠️ @Google’s OAuth bug endangers million...
Dev Digest 151: SEO in an AI world, security fixes and Doomed PDFs

From learning to earning

Jobs that call for the skills explored in this talk.