Reality and Beyond: Coding a Drone Using {Unity 3D .NET} and ChatGPT AI!

This drone has no formal API, just documentation. So we asked ChatGPT to write the C# code to fly it with a HoloLens.

#1about 4 minutes

Combining spatial computing and drones for new interactions

The speaker shares his vision for controlling a Tello drone with a HoloLens to create more engaging and exciting coding experiences.

#2about 3 minutes

Exploring the tools for drone and MR development

An overview of the key technologies used, including the Unity 3D game engine, ChatGPT for code inspiration, and the Mixed Reality Toolkit (MRTK) for HoloLens.

#3about 2 minutes

A closer look at the programmable Tello drone

The Tello drone is introduced as a lightweight, programmable device comparable to a flying Raspberry Pi, controllable via its SDK.

#4about 4 minutes

Understanding the spectrum of immersive technologies

A breakdown of the reality-virtuality continuum defines augmented reality (AR), virtual reality (VR), and how mixed reality (MR) blends the physical and digital worlds.

#5about 5 minutes

How the HoloLens enables hands-free enterprise solutions

The Microsoft HoloLens is presented as an enterprise-focused MR headset enabling natural, hands-free interaction for use cases like remote assistance and healthcare.

#6about 5 minutes

Leveraging ChatGPT to interpret documentation and write code

ChatGPT is used as a tool to understand the drone's documentation and generate initial C# code for UDP communication, including IP addresses and ports.

#7about 5 minutes

A walkthrough of the Unity project and UDP client

A look inside the Unity project and the C# script for the UDP client, which handles connecting to the drone, entering SDK mode, and sending commands.

#8about 8 minutes

Demonstrating drone control with the HoloLens

A live demo shows the Tello drone taking off, moving, and flipping, all controlled by commands from the mixed reality application on the HoloLens.

#9about 3 minutes

Answering questions about Unreal Engine and HoloLens

The speaker addresses audience questions on Unreal Engine as an alternative, the development status of HoloLens, and using mixed reality on mobile devices.