WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Apr 28, 2021

Detect Hand Pose with Vision

Milan Todorovic

Want to build touchless interfaces without training custom ML models? Discover how the iOS Vision framework lets you track 21 specific hand landmarks in real-time.

Pause
Mute Enter Fullscreen
#1 about 3 min

Introduction to the Apple Vision framework

The Vision framework provides native support for complex computer vision tasks like image classification and real-time activity detection.

#2 about 3 min

Recognizing hand landmarks and finger joints

The tracking system correctly identifies twenty-one specific coordinates on a human hand including individualized joints and wrist positioning.

#3 about 2 min

Addressing limitations in real-world hand tracking

Environmental edge cases like screen boundaries, irregular hand angles, and physical obstructions can temporarily disrupt accurate camera recognition.

#4 about 3 min

Overview of the sample hand tracking project

The sample tracking application relies on an architectural pattern involving camera views, controllers, and specialized gesture processing files.

#5 about 3 min

Live demonstration of drawing with hand gestures

Running the application on a physical device demonstrates how continuous pinch gestures translate directly into rendering visible screen paths.

#6 about 3 min

Configuring the camera view and request handler

The view controller initializes the camera feed and configures the tracking requests required to process incoming video frame data.

#7 about 7 min

Processing finger tip observations and drawing paths

The application extracts precise landmark coordinates from handler observations and converts them into core graphics points for rendering.

#8 about 3 min

Calculating distance thresholds for pinch detection states

Developers use numeric distance thresholds and state counters to cleanly distinguish between active pinch gestures and resting hand states.

#9 about 5 min

Expanding recognition to human body pose tracking

The underlying mechanisms used for small appendage tracking also apply to mapping structural body joints and overall human posture.

Matching moments

3:56 min

Introduction to text recognition with the Vision framework

Milan Todorovic · World Congress 2021

4:13 min

Implementing machine learning with Core ML and Vision

MIlan Todorović MIlan Todorović · World Congress 2025

3:58 min

Controlling hardware cameras and analyzing visual frames

Marc Rousavy Marc Rousavy · World Congress 2025

3:43 min

Exploring Unity AR Foundation for advanced tracking logic

Vladimir Novick · World Congress 2021

2:51 min

Building real-time object tracking with smart glasses

Sasha Denisov Sasha Denisov +1 · World Congress 2026 Europe

3:00 min

Creating intuitive onboarding for controllerless hand tracking

Mario Petricevic · World Congress 2024

More from WeAreDevelopers LIVE

WeAreDevelopers LIVE - Is CSS Catching Up to Flash? thumbnail
1:03:33
12 days ago

WeAreDevelopers LIVE - Is CSS Catching Up to Flash?

Chris Heilmann, Daniel Cranney, Bramus Van Damme

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along thumbnail
1:21:17
19 days ago

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along

Chris Heilmann, Daniel Cranney, Krzysztof Cieślak

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak thumbnail
4:23
20 days ago

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Open Source Agent Toolkits thumbnail
1:00:38
26 days ago

WeAreDevelopers LIVE - Open Source Agent Toolkits

Chris Heilmann, Daniel Cranney, Eddie Jaoude

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude thumbnail
6:16
27 days ago

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude

Chris Heilmann, Daniel Cranney

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas thumbnail
5:46
about 1 month ago

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas

Chris Heilmann, Daniel Cranney, Marc Plogas

Related videos

Let your iOS app read texts thumbnail
36:34
World Congress 2021 • about 5 years ago

Let your iOS app read texts

Milan Todorovic

Unsolved mysteries between Usability and Accessibility in iOS thumbnail
25:54
World Congress 2023 • almost 3 years ago

Unsolved mysteries between Usability and Accessibility in iOS

Moh’d Abualhuda, Stephan Lemnitzer

Apple Vision Pro: Proven Development Methods Meet the Latest Technology thumbnail
28:48
World Congress 2024 • about 2 years ago

Apple Vision Pro: Proven Development Methods Meet the Latest Technology

Mario Petricevic

Harnessing Apple Intelligence: Live Coding with Swift for iOS thumbnail
28:54
World Congress 2025 • about 1 year ago

Harnessing Apple Intelligence: Live Coding with Swift for iOS

MIlan Todorović

Edge AI on iOS: Beyond the Cloud, Designing the Next Generation of Intelligent On-Device Apps thumbnail
26:30
World Congress 2026 Europe • 3 months ago

Edge AI on iOS: Beyond the Cloud, Designing the Next Generation of Intelligent On-Device Apps

Precious Osaro

AR Kit intro - placing 3D objects in a scene and interacting with them in real-time thumbnail
48:21
LIVE • over 5 years ago

AR Kit intro - placing 3D objects in a scene and interacting with them in real-time

Nermin Sehic

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.