WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE β€’ Apr 28, 2021

Detect Hand Pose with Vision

Milan Todorovic

Want to build touchless interfaces without training custom ML models? Discover how the iOS Vision framework lets you track 21 specific hand landmarks in real-time.

Pause
Mute Enter Fullscreen
#1 about 3 min

Introduction to the Apple Vision framework

The Vision framework provides native support for complex computer vision tasks like image classification and real-time activity detection.

#2 about 3 min

Recognizing hand landmarks and finger joints

The tracking system correctly identifies twenty-one specific coordinates on a human hand including individualized joints and wrist positioning.

#3 about 2 min

Addressing limitations in real-world hand tracking

Environmental edge cases like screen boundaries, irregular hand angles, and physical obstructions can temporarily disrupt accurate camera recognition.

#4 about 3 min

Overview of the sample hand tracking project

The sample tracking application relies on an architectural pattern involving camera views, controllers, and specialized gesture processing files.

#5 about 3 min

Live demonstration of drawing with hand gestures

Running the application on a physical device demonstrates how continuous pinch gestures translate directly into rendering visible screen paths.

#6 about 3 min

Configuring the camera view and request handler

The view controller initializes the camera feed and configures the tracking requests required to process incoming video frame data.

#7 about 7 min

Processing finger tip observations and drawing paths

The application extracts precise landmark coordinates from handler observations and converts them into core graphics points for rendering.

#8 about 3 min

Calculating distance thresholds for pinch detection states

Developers use numeric distance thresholds and state counters to cleanly distinguish between active pinch gestures and resting hand states.

#9 about 5 min

Expanding recognition to human body pose tracking

The underlying mechanisms used for small appendage tracking also apply to mapping structural body joints and overall human posture.

Matching moments

3:56 min

Introduction to text recognition with the Vision framework

Milan Todorovic Β· WWC 2021

4:13 min

Implementing machine learning with Core ML and Vision

MIlan Todorović MIlan Todorović · WWC 2025

3:58 min

Controlling hardware cameras and analyzing visual frames

Marc Rousavy Marc Rousavy Β· WWC 2025

3:43 min

Exploring Unity AR Foundation for advanced tracking logic

Vladimir Novick Β· WWC 2021

2:51 min

Building real-time object tracking with smart glasses

Sasha Denisov Sasha Denisov +1 Β· WWC Europe 2026

3:00 min

Creating intuitive onboarding for controllerless hand tracking

Mario Petricevic Β· WWC 2024

More from WeAreDevelopers LIVE

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz thumbnail
5:43
5 days ago

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Node and Package Security thumbnail
1:08:30
5 days ago

WeAreDevelopers LIVE - Node and Package Security

Chris Heilmann, Daniel Cranney, Zbyszek Tenerowicz

WeAreDevelopers LIVE - Streaming HTML thumbnail
1:10:26
12 days ago

WeAreDevelopers LIVE - Streaming HTML

Chris Heilmann, Daniel Cranney, Jad Joubran

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe thumbnail
1:02:34
19 days ago

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe

Chris Heilmann, Daniel Cranney, Justin Halsall, Werner Jeschko, Nicole Jeschko, Mirco Tomić

WeAreDevelopers LIVE - Web Performance in an AI World thumbnail
1:07:59
26 days ago

WeAreDevelopers LIVE - Web Performance in an AI World

Chris Heilmann, Daniel Cranney

FIFA Hack, Midjourney Does Ultrasounds, and Trampoline Traffic Lights - Nicole Jeschko and Werner Jeschko thumbnail
5:55
about 2 months ago

FIFA Hack, Midjourney Does Ultrasounds, and Trampoline Traffic Lights - Nicole Jeschko and Werner Jeschko

Chris Heilmann, Daniel Cranney

Upcoming sessions on this topic

Open session

World Congress 2026 North America

Point. Ask. Answer. Building Vision into AI Live on Stage.

Kavya Sri Chennoju

Staff AI Engineer at Arm

Kavya Sri Chennoju
Open session

World Congress 2026 North America

From Software Agents to Physical Devices: Inside the Agentic Hardware Stack

Michael Yuan, Vivian Hu

Michael Yuan
Vivian Hu
Open session

World Congress 2026 North America

Vibe Coding Accessibility

Karl Groves

Focused on actively fixing accessibility

Karl Groves
Open session

World Congress 2026 North America

GitHub’s Team X-Ray: Your Repository Knows More About Your Team Than Your Team Does

Andrea Griffiths

Senior Developer Advocate

Andrea Griffiths
Open session

World Congress 2026 North America

Building Stuff with GenAI - The Open Minded Workshop beyond OpenAI

Andreas Erben

CTO for Applied AI and Metaverse at daenet

Andreas Erben
Open session

World Congress 2026 North America

Beyond Vibe Coding: Using CLI Tools as Your AI Counterpart in Pair Programming

Ron Veen

Java enthusiast and Special agent for Team Rockstars IT

Ron Veen

Related videos

Let your iOS app read texts thumbnail
36:34
WWC 2021 β€’ about 5 years ago

Let your iOS app read texts

Milan Todorovic

Unsolved mysteries between Usability and Accessibility in iOS thumbnail
25:54
WWC 2023 β€’ almost 3 years ago

Unsolved mysteries between Usability and Accessibility in iOS

Moh’d Abualhuda, Stephan Lemnitzer

Apple Vision Pro: Proven Development Methods Meet the Latest Technology thumbnail
28:48
WWC 2024 β€’ almost 2 years ago

Apple Vision Pro: Proven Development Methods Meet the Latest Technology

Mario Petricevic

Harnessing Apple Intelligence: Live Coding with Swift for iOS thumbnail
28:54
WWC 2025 β€’ 12 months ago

Harnessing Apple Intelligence: Live Coding with Swift for iOS

MIlan Todorović

Edge AI on iOS: Beyond the Cloud, Designing the Next Generation of Intelligent On-Device Apps thumbnail
26:30
WWC Europe 2026 β€’ about 1 month ago

Edge AI on iOS: Beyond the Cloud, Designing the Next Generation of Intelligent On-Device Apps

Precious Osaro

AR Kit intro - placing 3D objects in a scene and interacting with them in real-time thumbnail
48:21
LIVE β€’ over 5 years ago

AR Kit intro - placing 3D objects in a scene and interacting with them in real-time

Nermin Sehic

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.