> Markdown version of [/videos/135-detect-hand-pose-with-vision](https://www.wearedevelopers.com/videos/135-detect-hand-pose-with-vision). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Detect Hand Pose with Vision

Want to build touchless interfaces without training custom ML models? Discover how the iOS Vision framework lets you track 21 specific hand landmarks in real-time.

- **Speakers:** Milan Todorovic
- **Event:** WeAreDevelopers LIVE
- **Published:** April 28, 2021
- **Duration:** 28:51
- **URL:** https://www.wearedevelopers.com/videos/135-detect-hand-pose-with-vision

## Summary

The iOS Vision framework significantly reduces time to market for computer vision applications by providing out-of-the-box machine learning capabilities. By leveraging the `VNDetectHumanHandPoseRequest` class, developers can seamlessly detect and track 21 specific hand landmarks in real-time, mapping four distinct joints per finger alongside the wrist. This foundational capability enables mobile devices to extract complex gesture data directly from live camera streams.

Implementing hand pose recognition inside Xcode involves setting up a camera view, processing video frame buffers, and passing the data to an image request handler. By translating internal Vision coordinates into Core Graphics space using `UIBezierPath`, applications can visually overlay responsive tracking features, such as drawing tools that follow a user's fingertips. Tracking specific anatomical points—like the thumb tip and index metacarpophalangeal (MCP) joint—allows developers to calculate relative distances and define custom interaction states like "pinched" or "apart."

While tracking accuracy can temporarily diminish when hands reach the edge of the screen, fold at obscure angles, or wear gloves, the underlying framework reliably estimates joint roots even during partial occlusion. The identical architecture easily pivots to full-body tracking via `VNDetectHumanBodyPoseRequest`, laying the groundwork for spatial analytics, sign language translation apps, and accessibility-driven touchless interface development.

**Keywords:** ios vision framework, swift computer vision, hand pose detection, vndetecthumanhandposerequest, realtime landmark tracking, xcode gesture recognition, video frame buffer processing, core graphics coordinate mapping, uibezierpath drawing, pinch gesture logic, machine learning camera integration, touchless interface development, vndetecthumanbodyposerequest, sign language translation apps

## Chapters

1. **Introduction to the Apple Vision framework** (01:54) — The Vision framework provides native support for complex computer vision tasks like image classification and real-time activity detection.
1. **Recognizing hand landmarks and finger joints** (04:15) — The tracking system correctly identifies twenty-one specific coordinates on a human hand including individualized joints and wrist positioning.
1. **Addressing limitations in real-world hand tracking** (06:48) — Environmental edge cases like screen boundaries, irregular hand angles, and physical obstructions can temporarily disrupt accurate camera recognition.
1. **Overview of the sample hand tracking project** (08:14) — The sample tracking application relies on an architectural pattern involving camera views, controllers, and specialized gesture processing files.
1. **Live demonstration of drawing with hand gestures** (11:06) — Running the application on a physical device demonstrates how continuous pinch gestures translate directly into rendering visible screen paths.
1. **Configuring the camera view and request handler** (13:34) — The view controller initializes the camera feed and configures the tracking requests required to process incoming video frame data.
1. **Processing finger tip observations and drawing paths** (15:52) — The application extracts precise landmark coordinates from handler observations and converts them into core graphics points for rendering.
1. **Calculating distance thresholds for pinch detection states** (22:18) — Developers use numeric distance thresholds and state counters to cleanly distinguish between active pinch gestures and resting hand states.
1. **Expanding recognition to human body pose tracking** (24:21) — The underlying mechanisms used for small appendage tracking also apply to mapping structural body joints and overall human posture.

## Related Moments

- [Introduction to text recognition with the Vision framework](https://www.wearedevelopers.com/videos/167-let-your-ios-app-read-texts) (from "Let your iOS app read texts")
- [Implementing machine learning with Core ML and Vision](https://www.wearedevelopers.com/videos/1515-harnessing-apple-intelligence-live-coding-with-swift-for-ios) (from "Harnessing Apple Intelligence: Live Coding with Swift for iOS")
- [Controlling hardware cameras and analyzing visual frames](https://www.wearedevelopers.com/videos/1466-building-better-apps-with-react-native) (from "Building Better Apps with React Native")
- [Exploring Unity AR Foundation for advanced tracking logic](https://www.wearedevelopers.com/videos/160-cross-platform-augmented-reality-development-with-react-native) (from "Cross platform Augmented Reality development with React Native")
- [Building real-time object tracking with smart glasses](https://www.wearedevelopers.com/videos/100033-the-retrieval-layer-for-edge-ai) (from "The Retrieval Layer for Edge AI")
- [Creating intuitive onboarding for controllerless hand tracking](https://www.wearedevelopers.com/videos/1190-apple-vision-pro-proven-development-methods-meet-the-latest-technology) (from "Apple Vision Pro: Proven Development Methods Meet the Latest Technology")

## Related Articles

- [What’s in The box? – Unboxing The DeepFace](https://www.wearedevelopers.com/magazine/117-what-s-in-the-box-unboxing-the-deepface)
- [Dev Digest 178: VPNs to remove, Coder > AI, Your Trackpad as a Scale](https://www.wearedevelopers.com/magazine/608-dev-digest-178-vpns-to-remove-coder-ai-your-trackpad-as-a-scale)
- [Dev Digest 154: Responsible AI? Mistakes of CSS & track all the things!](https://www.wearedevelopers.com/magazine/548-dev-digest-154-responsible-ai-mistakes-of-css-track-all-the-things)
- [Dev Digest 112 - The True Crime of AI Development](https://www.wearedevelopers.com/magazine/421-dev-digest-112-the-true-crime-of-ai-development)

## Related Jobs

- [Founding Mobile Engineer (iOS)](https://www.wearedevelopers.com/jobs/ext/1648532-founding-mobile-engineer-ios) at **Almedia**
- [iOS Engineer, Swift/SwiftUI/Combine (B2C Broker)](https://www.wearedevelopers.com/jobs/ext/1611832-ios-engineer-swift-swiftui-combine-b2c-broker) at **Bitpanda**
- [Digital Forensic Analyst](https://www.wearedevelopers.com/jobs/ext/1271949-digital-forensic-analyst) at **ZEISS Group**
- [Digital Forensic Analyst](https://www.wearedevelopers.com/jobs/ext/1982266-digital-forensic-analyst) at **ZEISS Group**
- [AI Operations Manager (all genders)](https://www.wearedevelopers.com/jobs/48263-ai-operations-manager-all-genders) at **envelio**
- [Hardware-naher Algorithmenentwickler](https://www.wearedevelopers.com/jobs/ext/1684535-hardware-naher-algorithmenentwickler) at **ZEISS Group**