Movement Capture

Motion capture in the BlackBox

There are currently two major systems for movement capture and recognition in the Blackbox: The Vicon system and the Kinect. There are also a set of accelerometers that you can use.

The systems

• Vicon
• Kinect
• EffortDetect Wearable Accelerometers

What type of sensor is best?

What kinds of sensors best fit the motion phenomena you wish to capture. Motion capture systems use different kinds of sensors. To find out which type of sensor best suits your application, see the section on Sensor Selection. In general:

• Low-frequency movement is best captured using measurements of position (the Vicon or the Kinect will be your best)
• Intermediate-frequency movement is best captured using measurements of velocity
• High-frequency movement is best captured using measurements of acceleration (accelerometers will be appropriate) Generally, human movement tends to be slow enougho that intermediate

Kinect versus Vicon

The Vicon is a powerful motion capture tool and can be used to capture an enormous variety of movement. However, using the Vicon isn't always ideal:

What kind of movement do you need to capture?

What kind of movement do you need to capture? Consider the following.

Do you need to identify individual body parts or is it sufficient to treat the body as a blob?

If all you need to do is treat the body as a blob (for example, tracking the position of a body in space in order to project an image around that body), the Kinect is a more convenient choice than the Vicon. Recognizing human skeleton data using the Vicon data takes more work than processing Kinect data. While many of the Kinect APIs map the cloud of points to a human skeleton, for the Vicon, you will need to set up the skeleton yourself.

Are you capturing movements of the entire body or only some body parts (only finger gestures, or only full arm movements, or only the torso, or only leg bending and swinging)? Do you need to track both large movements (e.g., lunging, jumping, turning around) and small gestures (e.g., foot tapping, fingers drumming, head slightly tilting)?

In his seminal book on acting, An Acrobat of the Heart, theater theorist and director Jerzy Grotowski distinguishes the two kinds of movements as plastiques and corporeals. If you are tracking movement that happens on similar scales (either mostly plastiques or mostly corporeals) and for body parts that close to each other, then you can use the Kinect effectively by using the appropriate Kinect API.Otherwise, you are better off using the Vicon.

Is the entire body going to be moving through a large volume of space (greater than about 5 feet by 5 feet) (e.g., walking, running, many forms of dancing) or will the body stay more or less in one spot (within a space of about 5 feet by 5 feet)?

If so, then you must use the Vicon.

Are you going to be tracking movement in the transverse plane of the body?

These movements include

• supination and pronation of the legs or arms?
• left or right rotation of the head?
• twisting of the spine?

These movements cannot be tracked by the Kinect APIs available. You will have to search for APIs that do this, or build it yourself using the cloud point data. The Vicon is a better choice for this task.

Do you need to track movements of the separate parts of the torso (the cervical, lumbar, thoracic, and sacral areas)?

These movements cannot be tracked by the Kinect APIs available. You will have to search for APIs that do this, or build it yourself using the cloud point data. The Vicon is a better choice for this task. If you don't need to track precise movements of the parts of the torso, the Kinect is fine.

Will the mover be changing their frontal orientation (e.g., by turning around or spinning)?
The change in front cannot be automatically tracked by the Kinect APIs available. You will have to search for APIs that do this, or build it yourself using the cloud point data. The Vicon is a better choice for this task.

Are you going to be tracking very fast movements?

The sampling rate of the Vicon is 200 Hz, which is sufficiently large to capture many nuances of human movement. The Kinect's, however, is (as most) 30 Hz. Is this fast enough? The answer is: in many instances, yes, but it really depends on what you want to do. See the section on Errors and limits in using the Kinect for motion tracking.

Are you going to be tracking very small movements?

See the section on Errors and limits in using the Kinect for motion tracking to learn about considerations you need to take into account if you intend to track small movements.

How much time do you have to plan and set up your motion capture session?

If you can prepare adequately for it, use the Vicon. But if you want to quickly capture movement data and don't have a lot of time on your hands, use the Kinect.

Do you need to process the movement data in real-time?

The current Vicon setup in the Blackbox prevents you from using it for real-time motion capture. A kludge was devised a long time ago by a former SIAT student, Rob Lovell, but the technical details on how to do this has been lost. Plans for upgrading the Vicon system are underway. In the meantime, the Kinect will allow you to stream real-time movement data very easily.

Movement recognition in the Blackbox

The Blackbox currently supports Laban Basic Effort recognition through the EffortDetect system.

In general, a variety of heuristic techniques are applicable to movement information derived from sensor data. The following techniques below represent some that have been reported in the literature:

• Frequency-domain analysis (Yang & Hsu, 2010)
  • Analysis of variance
  • Analysis of frequency peaks
  • Discrete wavelet transform(Sekine, Tamura, Togawa, & Fukui, 2000)
  • Signal magnitude area (Karantonis, Narayanan, M. Mathie, Lovell, & Celler, 2006)
• Statistical approaches (Yang & Hsu, 2010)
  • Decision trees (M. J. Mathie, Celler, Lovell, & Coster, 2004)
  • k-nearest neighbor
  • support vector machines
  • Naïve Bayes classifier
  • Gaussian mixture model
  • Hidden Markov models
  • Dynamic Conditional Random Field (Morency, Quattoni, & Darrell, 2007)
  • Boltzmann machines (Taylor & Hinton, 2009)

(Note: Dynamic conditional random fields and Boltzmann machines were suggested by AAAI reviewers for any future versions of EffortDetect.)

References

• Karantonis, D. M., Narayanan, M. R., Mathie, M., Lovell, N. H., & Celler, B. G. (2006). Implementation of a real-time human movement classifier using a triaxial accelerometer for ambulatory monitoring. Information Technology in Biomedicine, IEEE Transactions on, 10(1), 156–167.
• Mathie, M. J., Celler, B. G., Lovell, N. H., & Coster, A. C. F. (2004). Classification of basic daily movements using a triaxial accelerometer. Medical & Biological Engineering & Computing, 42(5), 679-687. doi:10.1007/BF02347551
• Morency, L. P., Quattoni, A., & Darrell, T. (2007). Latent-dynamic discriminative models for continuous gesture recognition. 2007 IEEE Conference on Computer Vision and Pattern Recognition (pp. 1–8).
• Sekine, M., Tamura, T., Togawa, T., & Fukui, Y. (2000). Classification of waist-acceleration signals in a continuous walking record. Medical Engineering & Physics, 22(4), 285-291. doi:16/S1350-4533(00)00041-2
• Taylor, G. W., & Hinton, G. E. (2009). Factored conditional restricted boltzmann machines for modeling motion style. Proceedings of the 26th annual international conference on machine learning (pp. 1025–1032).
• Yang, C. C., & Hsu, Y. L. (2010). A review of accelerometry-based wearable motion detectors for physical activity monitoring. Sensors, 10(8), 7772–7788.