Pose estimation is the computer vision task of detecting the positions of body joints, called keypoints, to determine posture and movement. A model locates points such as shoulders, elbows, wrists, hips, knees and ankles, then connects them into a skeleton. It is used in fitness apps, physiotherapy, sports analysis, workplace safety, animation and gesture control.
Key takeaways
- Output is a set of keypoints, each with coordinates and a confidence score.
- 2D pose gives image coordinates; 3D pose estimates depth as well.
- Top-down methods detect people first, then their keypoints; bottom-up methods find all keypoints and group them.
- The common COCO format uses 17 body keypoints.
- Occlusion and unusual viewpoints are the main sources of error.
Pose estimation explained in simple terms
Watching someone lift a box, you can tell whether they bent their knees or their back. Pose estimation gives a computer that ability: it tracks where the joints are, so software can judge posture, count repetitions or flag unsafe movement, without any wearable sensors.
How it works
- Detect people in the frame (object detection), for top-down methods.
- For each person, predict a heatmap per keypoint showing likely locations.
- Take the peak of each heatmap as the joint position, with a confidence score.
- Connect keypoints into a skeleton using known body structure.
- For video, smooth positions across frames and track identities over time.
Bottom-up approaches, such as the OpenPose work from 2017, instead find all keypoints in the image at once and then group them into people, which scales better in crowded scenes.
2D vs 3D pose
| Aspect | 2D pose | 3D pose |
|---|---|---|
| Output | x, y in the image | x, y, z in space |
| Hardware | Ordinary camera | Camera plus depth, multiple views, or model inference |
| Use | Counting reps, basic posture | Biomechanics, animation, precise angles |
| Accuracy | Higher | Harder; depth is ambiguous from one view |
A worked example
A gym app counts squats from a phone camera.
- Pose estimation returns 17 keypoints per frame.
- The app computes the knee angle from hip, knee and ankle positions.
- A repetition is counted when the angle drops below 90 degrees and returns above 160.
- If the hip-shoulder line leans beyond a threshold, it warns about form.
- Low-confidence keypoints, for example when the user leaves frame, pause counting rather than producing false reps.
The vision model supplies coordinates; the application supplies the logic and the judgement about what counts as good form.
Metrics
- PCK (percentage of correct keypoints): share of keypoints within a tolerance of the true position.
- OKS-based average precision: the standard benchmark measure, accounting for keypoint scale and difficulty.
- Temporal smoothness: jitter across frames, which matters for video applications.
Applications
Fitness and rehabilitation feedback, sports performance analysis, ergonomics and workplace safety monitoring, motion capture for animation and games, gesture-based interfaces, retail and crowd analytics, and animal pose estimation in research and livestock management.
Practical considerations
- Camera placement matters more than model choice for accuracy.
- On-device models allow private, low-latency processing without uploading video.
- Occlusion: hidden limbs produce low-confidence or missing keypoints; handle this explicitly.
- Privacy: pose data is less identifying than video, so processing on-device and discarding frames is often a good design.
Limitations
- Occlusion, loose clothing and unusual angles degrade accuracy.
- Crowded scenes cause keypoint grouping errors.
- 3D depth from a single camera is inherently ambiguous.
- Pose alone does not capture intent; interpretation belongs to the application.