APD-Week8

video link:https://youtu.be/MBe2xFr1-Ms

Theme: Play & Body

Context:

My first response to this theme was the body game “Twister” I played as a kid, a game of physical skill produced by Milton Bradley Company . however，After watching peer presentations, I was inspired by the example of “hand shadow play” which was given by Yuxin, and also inspired by the American game show called Hole in the Wall in which players need to contort themselves to fit through cutouts in a large wall moving towards them.

My initial idea is to make a project that detects human poses in real time, and players need to use their bodies to pose as shown on the screen.

Method：

Learn about “human pose detection and tracking” technology based on OpenPose/Mediapipe/…

Build world in Unity

Response：

This is my new exploration in the field of real-time human pose detection and tracking with Unity, Python and Barracuda. It needs a USB camera and GPU to keep it running. Players are advised to stand two meters in front of the camera to ensure all parts of the body are captured. After detecting and tracking human motion, the character in game scene will behave same as the player on real time. Then patterns will be shown on the screen in turn, and players need to shape themselves according to the patterns on time.

Reflection：

Durning the production, I spent a lot of time on researching articles and tutorials of human motion detection, and I found that this technology relates to the fields of Python and computer graphics. Since I have basic knowledge of computer graphics and coding, the process of learning was difficult but smooth.

The article "Detailed explanation of human gesture recognition scheme" summarizes five human motion detection methods in detail: Mediapipe, OpenPose, OpenMMD, MoveNet, PoseNet. After reading article "OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields." And "MediaPipe: A Framework for Building Perception Pipelines.", I found that these methods are all implemented by python or MATLAB. Sadly, I did not learn them before, and it is impossible to master them in a week.

Fortunately, I finally found a 3D pose capture method based on Barracuda(a plugin of Unity). By importing the onnx model with Barracuda and using USB camera, it can do 3D pose estimation on Unity. People will have a good interactive experience without using extra device or sensors.

This project also has a lot of room for improvement:

-mechanics

This project only has one simple game mechanics. Players will be interested at first, but get bored quickly. Thus, an unique mechanics is needed, such as adding battle elements, or increase the speed and difficult of game.

-multiplayer

It's a good idea to add online function and multiplayer detection. It allows players invite their friends to play together, even if their friends are in different countries, they can play together online.

Overall, this project gives me a lot of confidence of exploring the field of human motion detection, and I would like to continue to explore its possibilities further in the future.

Reference：

Cao, Zhe et al. “OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields.” IEEE transactions on pattern analysis and machine intelligence 43.1 (2021): 172–186. Web.

Lugaresi, Camillo et al. “MediaPipe: A Framework for Building Perception Pipelines.” (2019): n. pag. Print.

Yukihiko. ”ThreeDposeUnityBarracuda”(2021)