> Markdown version of [/videos/1213-vision-for-websites-training-your-frontend-to-see](https://www.wearedevelopers.com/videos/1213-vision-for-websites-training-your-frontend-to-see). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Vision for Websites: Training Your Frontend to See

Ditch rigid keyword metadata and teach your frontend to see. Use multimodal AI and vector databases to build context-aware apps that intuitively understand images, text, and audio.

- **Speakers:** Daniel Madalitso Phiri
- **Event:** WeAreDevelopers LIVE
- **Published:** September 18, 2024
- **Duration:** 34:34
- **URL:** https://www.wearedevelopers.com/videos/1213-vision-for-websites-training-your-frontend-to-see

## Summary

Developers traditionally rely on rigid keyword metadata to build search experiences, inherently limiting the natural interaction users expect. By bringing multimodal AI and vector search directly to the frontend, applications can be trained to "see"—meaning they can deduce, reflect, and understand rich media inputs like text, images, or audio seamlessly. This evolution shifts web development away from literal text matching toward building context-aware interfaces that more accurately mirror human connection and perception.

Unlike generative AI models that create new content, embedding models structure existing knowledge by converting data into numerical arrays plotted within a multidimensional vector space. This works similarly to how RGB mathematically represents color, but scales across hundreds or thousands of dimensions. When routing mixed-media inputs through API models like Google Vertex AI and storing them in a Weaviate vector database, frontend frameworks such as Nuxt.js can intuitively execute cross-modal queries purely based on vector distance metrics. Utilizing query structures like `nearText` or `nearImage` allows systems to immediately return contextually related images or documents without relying on explicit tags.

Implementing these multimodal capabilities handles complex workflows with surprisingly concise code. A critical technique for maintaining precise query relevance involves utilizing dynamic distance cutoffs—such as Weaviate's `autoLimit`—which clusters responses by proximity and automatically strips away out-of-context search outliers. Whether designing visual e-commerce product catalogs or experimental recommendation engines based on uploaded photos, democratized access to multimodal embeddings empowers developers to rapidly build resilient frontends that genuinely comprehend complex, multi-format user intent.

**Keywords:** multimodal ai, vector embedding models, multidimensional vector space, frontend vision architectures, weaviate vector database, google vertex ai, image similarity search, nuxt.js frontend, dynamic distance grouping, e-commerce visual search, multimodal user intent, generative vs embedding models, neartext search queries, vector distance metrics, typescript search clients

## Chapters

1. **Defining machine vision for modern web frontends** (00:54) — Redefining digital vision as data deduction provides clear strategies for frontend data interpretation.
1. **Demonstrating multimodal search in an enterprise storefront** (02:01) — Retrieving catalog items without specific metadata requires multimodality to dynamically match text and visual queries.
1. **Improving user interactions via multimodal context gathering** (06:04) — Transitioning traditional keyword searches into natural multimodal queries significantly improves user interface interactions.
1. **Encapsulating complex frontend data using embedding models** (10:41) — Storing distinct multi-format data in multidimensional mathematical spaces enables the direct comparison of non-uniform assets.
1. **Executing vector search queries using spatial distance** (14:01) — Plotting queries and data into numbered vector coordinates measures physical distance to determine search relevance.
1. **Unifying distinct media formats across shared vector spaces** (18:30) — Processing varying media through specialized encoders deposits uniform embeddings into a shared retrieval environment.
1. **Architecting a multimodal storefront interface using Next.js** (20:52) — Wiring frontend UI components to remote API routes securely hands text and image queries off to backend machine learning tools.
1. **Filtering text search queries with TypeScript and Weaviate** (23:18) — Interrogating vector clusters with text values relies on dynamic API limits to trim results by statistical distance jumps.
1. **Processing base64 image queries against remote vector collections** (29:32) — Submitting user photos to backend machine learning functions requires encoding visuals into processable base64 strings.
1. **Expanding vision applications beyond standard ecommerce features** (31:54) — Deploying camera-based cross-referencing inside user applications bridges digital product catalogs with real-world physical objects.

## Related Moments

- [Building neural search applications across diverse data modalities](https://www.wearedevelopers.com/videos/389-dall-e-flow-when-neural-search-meets-generative-art) (from "DALL·E Flow: when neural search meets generative art")
- [Substituting traditional classification models with search-based AI architecture](https://www.wearedevelopers.com/videos/830-what-comes-after-chatgpt-vector-databases-the-simple-and-powerful-future-of-ml) (from "What comes after ChatGPT? Vector Databases - the Simple and powerful future of ML?")
- [Introduction to generative AI and vector search](https://www.wearedevelopers.com/videos/844-enter-the-brave-new-world-of-genai-with-vector-search) (from "Enter the Brave New World of GenAI with Vector Search")
- [Implementing multimodal image processing and visual capabilities on the edge](https://www.wearedevelopers.com/videos/100157-ai-in-your-browser-exploring-chrome-s-built-in-llm) (from "AI in Your Browser: Exploring Chrome's Built-In LLM")
- [Demonstrating Viro React capabilities for interactive applications](https://www.wearedevelopers.com/videos/160-cross-platform-augmented-reality-development-with-react-native) (from "Cross platform Augmented Reality development with React Native")
- [Visualizing semantic similarity within a multi-dimensional vector space](https://www.wearedevelopers.com/videos/830-what-comes-after-chatgpt-vector-databases-the-simple-and-powerful-future-of-ml) (from "What comes after ChatGPT? Vector Databases - the Simple and powerful future of ML?")

## Related Articles

- [Dev Digest 133 - Back to Front](https://www.wearedevelopers.com/magazine/474-dev-digest-133-back-to-front)
- [SEO in an AI world - Google vs. ChatGPT and survival tips for content creators](https://www.wearedevelopers.com/magazine/534-seo-in-an-ai-world-google-vs-chatgpt-and-survival-tips-for-content-creators)
- [WeAreDevelopers Dev Digest Issue 116 - The new search wars…](https://www.wearedevelopers.com/magazine/445-wearedevelopers-dev-digest-issue-116-the-new-search-wars)
- [Dev Digest 132 - Binging WADFlix?](https://www.wearedevelopers.com/magazine/473-dev-digest-132-binging-wadflix)

## Related Jobs

- [Staff Frontend Engineer](https://www.wearedevelopers.com/jobs/48313-staff-frontend-engineer) at **SMG Swiss Marketplace Group**
- [Staff Frontend Engineer (Expert+/Lead equivalent) - Hybrid working model, 100%, Ho Chi Minh City](https://www.wearedevelopers.com/jobs/48314-staff-frontend-engineer-expert-lead-equivalent-hybrid-working-model-100-ho-chi-minh-city) at **SMG Swiss Marketplace Group**
- [Principal Engineer - AI Search & Vector Infrastructure](https://www.wearedevelopers.com/jobs/ext/319507-principal-engineer-ai-search-vector-infrastructure) at **Redis**
- [Principal Engineer - AI Search & Vector Infrastructure](https://www.wearedevelopers.com/jobs/ext/353953-principal-engineer-ai-search-vector-infrastructure) at **Redis**
- [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**
- [Principal Engineer - AI Search & Vector Infrastructure](https://www.wearedevelopers.com/jobs/ext/381484-principal-engineer-ai-search-vector-infrastructure) at **Redis**