Testing Kinect with Unity (Studio 2 - Week 3)

At the beginning of this week, I felt quite lost. I was struggling with all the technical issues I was running into, and felt overwhelmed by the various technologies that could work with Kinect (TouchDesigner, Processing and Unity) as well as the webcam (OpenCV which also works in Processing, TouchDesigner and Unity) and was unsure which one to go with. TouchDesigner allows for Kinect integration, but is mainly used for interactive installation work. Processing is great for beginners, and quick to code in order to get something working, but I'm unsure how it'll do with managing a whole game. While Unity is great for all types of interactive work, especially games, but I haven't used it before for my own work. After talking to Jadd during my Week 3 consultation, he suggested going with Unity as it is widely used for making games. There is also a lot more Unity support from other students and lecturers that I could tap into should I run into issues learning the program. 

I was also unsure which idea/concept to go with for my studio project. In the end, I decided to go with my musical jump rope idea, as that seemed the simplest and most resolved. As I'm completely new to using the Kinect and Unity, this will be a good beginner's project for me. Skipping was voted the fourth most popular childhood street/playground game in the survey I conducted on social media, and the most popular playground game in Britain remembered by adults according to a 2018 survey conducted. Jumping is also a simple action that most people can do, and a discrete gesture that hopefully will be easier to detect using the Kinect.

After deciding on my concept and technology, I got to work trying to find ways to integrate the Kinect with Unity. I ended up going with the Kinect 2 with MS-SDK asset, and had a chat with Hizi in regards to using the Koreographer Unity plugin for the music beat detection. I borrowed a Kinect 2 from a friend, and bought the relevant power adapter from eBay. The attached video is of me playing around with the demo scenes that came with the Kinect 2 asset, just so I can familiarise myself with the technical possibilities of the Kinect device:

• Controlling an avatar
• Face tracking and head orientation
• Avatar guided by game object
• Detecting multiple players and gestures
• Silhouette interacting with virtual 3D objects
• Hand swipe gestures
• Wheel and zoom in/out gestures

Video music credits: Happy Rock from Bensound.com

Reflection:

There were still many more demo scenes which I haven't played around with, but I ran out of time. But spending an afternoon just playing around with the technology made me feel less fearful, and that my idea may be feasible after all. It was a good start on knowing what the capabilities of the Kinect are, and how I can utilise the various gestures for my concept. Although the Kinect 2 has been discontinued, I was amazed at its range of functionalities and reliability in gesture detection. Considering this is my first foray into full body motion sensing, I found this exercise to be fun and fruitful.