How Mixed Reality, Azure AI and Drones turned me into a Magician?

You don't need magic to command a physical drone with your hands. Just mixed reality, Azure AI, and a bit of code.

#1about 4 minutes

Fusing mixed reality, AI, and drones for magical experiences

The talk introduces a project that combines Azure AI, drones, and mixed reality to create immersive, interactive experiences.

#2about 2 minutes

Exploring the core technology stack for the project

An overview of the key technologies used, including the Unity game engine, Meta Quest 3 for mixed reality, and Azure AI Foundry for intelligence.

#3about 1 minute

Getting started with the Azure AI Foundry platform

Azure AI Foundry is presented as a centralized platform for accessing various AI models, developer tools, and services to build generative AI applications.

#4about 4 minutes

A practical walkthrough of the Azure AI Foundry playground

A live demonstration shows how to create a project, deploy a GPT-4o model, and use the chat playground to test the AI before integration.

#5about 4 minutes

Using Unity and the Meta SDK for XR development

The process of building the mixed reality application involves using the Meta SDK in Unity for camera rigs, controller systems, and crucial hand tracking capabilities.

#6about 11 minutes

Programming drone controls with GitHub Copilot

The Tello drone's UDP-based communication protocol is implemented by feeding its technical architecture to GitHub Copilot, which generates the necessary C# code.

#7about 3 minutes

Live demo of the immersive AI assistant Anna

A live demonstration showcases the AI assistant "Anna" in mixed reality, featuring natural language conversation, image recognition, and function calling to manipulate virtual objects.

#8about 4 minutes

Flying a real drone using hand gestures in MR

The final demo shows a real Tello drone being controlled directly through hand movements tracked by the Meta Quest 3 headset, including takeoff and flips.