Blog

Real-Time Motion Capture using Kinect and XNA (Part 2)

Apr 3

Written by:
Tuesday, April 03, 2012  RssIcon

Our last post about real-time motion capture using Kinect seems to be pretty popular. And for the small amount of time (only a few hours) that we put into this example, the results are satisfying – but we can do better! We have updated the project and included a brand new sample. The new sample uses skeleton mapping to animate 3D models using Kinect. The code is much simpler, and exchanging the 3D model is a lot easier. We have also updated the example application to use Kinect SDK v1.0 instead of Kinect SDK beta2. You can download the sample project (including the source code) at the end of this post.

Video

Here are the results of the new sample application:

Description

The new sample project demonstrates 2 different methods to animate 3D models using Kinect in real-time:

The Skeleton Mapping Approach

The first sample uses Skeleton Mapping to animate the XNA Dude model and a Space Marine model. There are three different skeletons involved: The Kinect player skeleton, the Dude skeleton and the Space Marine skeleton. All three skeletons are different, i.e. have different bone names and a different number of bones. DigitalRune Animation contains a SkeletonMapper which can be used to transfer an animation from one skeleton to another skeleton with a different structure. (Take a look at the documentation to learn more about Skeleton Mapping.)

Setting up the bone mapping for a model is not very difficult. This is the setup for the Dude model:

var ks = KinectWrapper.SkeletonPoseA.Skeleton;
var ms = _skeletonPoseA.Skeleton;

_skeletonMapperA.BoneMappers.Add(new DirectBoneMapper(ks.GetIndex("HipCenter"), ms.GetIndex("Root"))
{
  MapTranslations = true,
  ScaleAToB = 1f,          
});

_skeletonMapperA.BoneMappers.Add(new UpperBackBoneMapper(
  ks.GetIndex("Spine"), ks.GetIndex("ShoulderCenter"), ks.GetIndex("ShoulderLeft"), ks.GetIndex("ShoulderRight"),
  ms.GetIndex("Spine"), ms.GetIndex("Neck"), ms.GetIndex("R_UpperArm"), ms.GetIndex("L_UpperArm")));

_skeletonMapperA.BoneMappers.Add(new ChainBoneMapper(ks.GetIndex("ShoulderLeft"), ks.GetIndex("ElbowLeft"), ms.GetIndex("R_UpperArm"), ms.GetIndex("R_Forearm")));
_skeletonMapperA.BoneMappers.Add(new ChainBoneMapper(ks.GetIndex("ShoulderRight"), ks.GetIndex("ElbowRight"), ms.GetIndex("L_UpperArm"), ms.GetIndex("L_Forearm")));
_skeletonMapperA.BoneMappers.Add(new ChainBoneMapper(ks.GetIndex("ElbowLeft"), ks.GetIndex("WristLeft"), ms.GetIndex("R_Forearm"), ms.GetIndex("R_Hand")));
_skeletonMapperA.BoneMappers.Add(new ChainBoneMapper(ks.GetIndex("ElbowRight"), ks.GetIndex("WristRight"), ms.GetIndex("L_Forearm"), ms.GetIndex("L_Hand")));
_skeletonMapperA.BoneMappers.Add(new ChainBoneMapper(ks.GetIndex("HipLeft"), ks.GetIndex("KneeLeft"), ms.GetIndex("R_Thigh"), ms.GetIndex("R_Knee")));
_skeletonMapperA.BoneMappers.Add(new ChainBoneMapper(ks.GetIndex("HipRight"), ks.GetIndex("KneeRight"), ms.GetIndex("L_Thigh1"), ms.GetIndex("L_Knee2")));
_skeletonMapperA.BoneMappers.Add(new ChainBoneMapper(ks.GetIndex("KneeLeft"), ks.GetIndex("AnkleLeft"), ms.GetIndex("R_Knee"), ms.GetIndex("R_Ankle")));
_skeletonMapperA.BoneMappers.Add(new ChainBoneMapper(ks.GetIndex("KneeRight"), ks.GetIndex("AnkleRight"), ms.GetIndex("L_Knee2"), ms.GetIndex("L_Ankle1")));
_skeletonMapperA.BoneMappers.Add(new ChainBoneMapper(ks.GetIndex("ShoulderCenter"), ks.GetIndex("Head"), ms.GetIndex("Neck"), ms.GetIndex("Head")));
_skeletonMapperA.BoneMappers.Add(new ChainBoneMapper(ks.GetIndex("WristLeft"), ks.GetIndex("HandLeft"), ms.GetIndex("R_Hand"), ms.GetIndex("R_Middle1")));
_skeletonMapperA.BoneMappers.Add(new ChainBoneMapper(ks.GetIndex("WristRight"), ks.GetIndex("HandRight"), ms.GetIndex("L_Hand"), ms.GetIndex("L_Middle1")));

The source code, which can be downloaded below, contains additional comments and explanations.

The setup for the Space Marine model looks similar – only the bone names are different.

In each frame the SkeletonMapper is called to transfer the current pose of the Kinect player skeleton to the skeleton of the 3D model:

_skeletonMapperA.MapAToB();

Kinect sensor input is not perfect and causes jittering. Jittering can be reduced by applying a low-pass filter to the bone orientations.

The Marionette Ragdoll Approach

The second example uses the approach that we have shown in the previous post: A physics ragdoll is created for the Dude, which is used to animate the model. Certain joints of the Kinect player skeleton are used as target positions (e.g. hands, elbows, head, knees, etc.). The ragdoll is attached to the target positions using weak ball joint constraints. – This is like a marionette: The ragdoll is the puppet and the ball joints are strings that pull on the marionette.

This approach is more complex: A suitable ragdoll needs to be created for each 3D model. It can be difficult to make this approach stable. A lot of parameter tuning is required.

Comparison

The Skeleton Mapping approach is a lot simpler and in most cases yields better results. But the direct mapping does not prevent unrealistic poses, for example, when the Kinect reports wrong data: Kinect regularly thinks my office chair is a player and the 3D model ends up in an unrealistic, twisted pose.

The Marionette approach is difficult to use, but has the advantage of configurable joint limits: The ragdoll can be set up with proper limits to prevent unrealistic rotations. In addition, the ragdoll can interact with other objects. It can push other bodies – and it can itself be pushed by other objects.

I think in practice the Skeleton Mapping approach will satisfy most users’ needs. (The Marionette approach was mainly created as a proof-of-concept for a discussion in our forums.)

Download

Here you can download this XNA sample project including source code:

>>> Download new KinectAnimationSample <<<

Requirements
Instructions
  • Extract the zip into the Samples/DigitalRune.Animation folder of the DigitalRune Engine.

This sample will also be included in the next release of the DigitalRune Engine. More information can be found in the source code. Have fun!

45 comment(s) so far...


Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

Thank you for your publications, this might really help me, but the problem is that the demo says my trial has expired. Do you have any kind of license for students, Im an Engineer student from Brazil, and we are developing a kinect software, we wanted a dude moving just as yours example to make our pragramms intercace more user friendly.

By Lucas on   Tuesday, April 24, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

We offer free licenses for non-commercial and educational use. See page Buy > Non-Commercial on this website.

By HelmutG on   Tuesday, April 24, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

Very useful
But why when i move right part of my body the model moves the left part?
example: When i move right shoulder the model moves left shoulder?Like in the mirror.
How do i correct it?

Thanks...
(Sorry for bad english ^_^ )

By Keyvan on   Wednesday, April 25, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

@Keyvan: The mirror effect is by design. To un-mirror: In the KinectWrapper mirror the x values of the joints (use -x instead of +x). In the code that creates the bone mappings update the mappings so that all right Kinect bones map to right model bones (currently we map right to left).

By HelmutG on   Wednesday, April 25, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

Great work!
Is it possible that the environment where the Dude/ Marine are, to be what the Kinect RGB camera sees?
And put the 3D models on top?
It probably won't look as realistic, but it might be fun.

By mircea on   Tuesday, May 01, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

Sorry My question isn't related to this section.
Is it possible to get bones position after transform?

By Saturn on   Monday, May 07, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

@Saturn: Yes, no problem. Have a look at the SkeletonPose class in the documentation. You can read and change the bone poses relative to the parent bone, bone poses relative to the whole model, and more.

By HelmutG on   Monday, May 07, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

@mircea: I haven't tried overlaying the 3d model over the RGB image - but one of our users did something similar - see the video in the next blog post.

By HelmutG on   Monday, May 07, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

For example i want to get the position of head
Is that code Correct?

_skeletonPoseA.GetBonePoseAbsolute(_skeletonPoseA.Skeleton.GetIndex("Head")).ToPose().ToWorldPosition(DudePosition);

Sorry for beginner questions,I am new to XNA. :D

By Saturn on   Tuesday, May 08, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

To get the position of the head in the local space of the model:
_skeletonPoseA.GetBonePoseAbsolute(_skeletonPoseA.Skeleton.GetIndex("Head")).Translation

GetBonePoseAbsolute() returns an SrtTransform, which is very similar to a Pose. In this blog we have two introductory articles about Pose (see "Pose" in the tag list of the blog). These articles might be helpful if you are new to this. For a thorough discussion you can also ask in our forum.

By HelmutG on   Tuesday, May 08, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

Is it possible to use this software with other Depth sensors like Asus Xtion Pro?

By Keyvan on   Tuesday, May 15, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

@Keyvan: Interesting, I haven't seen this Xtion Pro device before.
This sample only works with Kinect. But our libraries are Kinect independent. If the device supports skeleton tracking (e.g. can return the position of the human joints), then I don't see any problems.

By HelmutG on   Tuesday, May 15, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

I have studied this example in detail and I enjoyed the good code comments. I really like the simplicity and accuracy of the Skeleton Mapping approach. However I would think many including myself need it to react with objects and have joint limits. Can I somehow have a mapped skeleton that also has a RagDoll underneath for interaction/joint limits?

By aidesigner on   Thursday, May 17, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

As you mentioned creating a RagDoll can be quite a manual effort. Can I import a RagDoll from other another tool that has RagDoll creation wizard?

By aidesigner on   Thursday, May 17, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

@aidesigner: Currently you can use the marionette approach if you need limits and dynamic interaction - but I know that this is a bit painful to use. For one-way interaction without limits you can use the skeleton mapping approach and a "kinematic ragdoll" (see our samples). Creating a user-controlled, fully dynamic ragdoll with limits that is easy to use is a research project on its own, which we hope to solve in the future.

At the moment we do not have any tools that help to import and convert ragdolls from other application to the format that we use. But perhaps some of our users will jump in and create something?

By HelmutG on   Thursday, May 17, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

hi
im developing a kinect virtual dreesing room in XNA.
what i have done is: captured the depth,RGB,Skeleton From Kinect and mach them together.
i load the dress 3d model(.fbx) but i cant adapt the dress to the user Body Based On User Skeleton.

my project goal is: www.fitnect.com/

Please Help.
thanks

By MRJM on   Sunday, May 20, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

I have some problems changing the model of this example. I built a model and I copied the dude's skeleton for that model but when I exported it to the aplication the model is very deformed and i don't know why.

By david on   Thursday, May 24, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

@david: This is a common XNA problem. Your model must be exported using the right settings.
Test your model with the normal XNA Skinned Model Sample from AppHub. If it does not work, check out the AppHub forums.
If it works in the XNA Skinned Model Sample - but not in our sample, then please let us know in the DigitalRune support forum.

By HelmutG on   Friday, May 25, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

hi, I'm working in a project with kinect and I like your work. I want to know what are the characteristics that should be a model to work in this example to make it work with any skinned model. I tried load a skinned model with the same bone structure but doesn't work.
please help.
thanks

By Brian on   Thursday, May 31, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

@Brian: The skinned model must work with XNA. It helps if the skeleton hierarchy is similar to the Kinect skeleton - but this is not required.
See also my answer to david above.

By HelmutG on   Friday, June 01, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

Hi,
How can i rotate some of model bones without kinect data? with a custom 3d vector?
Thanks.

By Keyvan on   Monday, July 02, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

@Keyvan: Have a look at the CharacterAnimationSample project that comes with the DigitalRune Engine. It contains a sample for that (SkeletonManipulationSample.cs).

By HelmutG on   Tuesday, July 03, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

Hi, How can i change that background on the sample? i want to put the RGB camera from kinect but with the 3d model instead of me, is that possible? Thanks and sorry poor english =D

By Lucas on   Monday, July 23, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

This probably won't be a popular comment, considering the location, but I'm interested in doing this *without* digitalrune.

I've managed to find this:
www.youtube.com/watch?v=9u8d_BZ0cN0

...but not having much luck implementing it. Has anyone else attempted this? If it matters, I'm using XNAnimation libraries for animated models, instead of the popular SkinnedAnimation xna project.

By Sam on   Wednesday, September 19, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

I tried to run the project but I had this error The type or namespace name 'Ragdoll' could not be found (are you missing a using directive or an assembly reference?)
so I need help so that i can rub it successfully

By Mohammed on   Friday, September 28, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

In the last release the Ragdoll class was moved to the assembly DigitalRune.Physics.Specialized to clean up dependencies. Check out the KinectAnimationSample that comes with the DigitalRune Engine download.

By HelmutG on   Friday, September 28, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

I have a question about the Offset variable in your Kinect wrapper. From the way it's implemented I assume this is just a normal translation of the model around the world space. But it seems there are some rotation effect as well.

For example, set an Offset value to (2,0,0) will not only move the model to the right, but also turn the model for about 45 decree. Is there a reason for this behavior?

By Merc on   Saturday, September 29, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

@Merc: The Offset is added to the Kinect joint positions in KinectWrapper.UpdateKinectSkeletonPose(). I am not sure why it would rotate the model. Usually I only use an Y offset to move the model up or down.
Does it rotate the model in the SkeletonMappingSample and the RagdollMarionetteSample?

By HelmutG on   Saturday, September 29, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

Yes, I know where the off set go in the code, that's why I kinda assumed it does what it should (a translation). It rotates both the model and the skeleton in the SKeleton mapping Sample. I haven't tested the Ragdoll Marionette Sample yet, but I don't know how to active it. Can you tell me how to run that mode?

By Merc on   Sunday, September 30, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

You can use the left and right arrow keys to switch between the skeleton mapping sample and the ragdoll marionette sample.

By HelmutG on   Monday, October 01, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

I'm trying to replace the Dude model in this sample with one from another FBX file. The ChainMapping seems to work fine because the model's limbs move with mine, but the replacement models do not move from the center of the world, making me think the DirectMapping is not working correctly? I'm new to XNA so I'm not sure if there's something about my FBX files that are off but if anyone knows what might be causing this I'd really appreciate the help.

By Brian on   Tuesday, October 02, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

Sorry for the late response. I tested out the ragdoll and it's the same thing, they both got rotate with an X offset. I think this has something to do with the way the model is rig. I notice that it seems the program does not support twisting, as in when only the top body twist, the whole body got twisted when it's not supposed to. Can you explain this?

Also a small question, how can I disable the mouse camera control as in undocked it when the program is run? I'm running data analysis between several programs so I need to freely move around the desktop, and the mouse docking make it hard to do so. I looked at your camera code and set both condition to false but it doesn't seem to do anything.

Thanks HelmutG.

By Merc on   Thursday, October 04, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

@Merc: I assume following happens: The top body twist rotates the spine bone. The hip bones are attached to the spine bones, so they move when the spine bone is rotated. Perhaps it helps to set MapFromBindPose to false for the ChainBoneMappers of the hip bones.

The mouse cursor is set to invisible in the Game class. The mouse centering in the camera must be disabled.

For a thorough discussion please use our support forum.

By HelmutG on   Friday, October 05, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

hi! i am using visual studio 2010 and i can not star debugging becouse the Debug Folder is empty. :( please how can i solve that?

thanks too much

By juankol19 on   Friday, October 12, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

Interesting, I want to know more about the differnt Skeleton mappers used during retargeting process. What do you think is the best reference for this info.? The help documentation gives some detail but it is not detailed enough.
Please help.

By Mobeen on   Monday, October 22, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

@Mobeen: You can use our support forum to ask for more details.

By HelmutG on   Monday, October 22, 2012
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

Awesome sample! This is an xna question and not a digital rune one, but is there an easy way to build a menu system off of this sample? Thanks for the consideration!

By Tee on   Thursday, April 04, 2013
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

@Tee: Depends on what you mean by "menu system". Microsoft has added "Kinect Interactions" in the latest release of the Kinect for Windows SDK. Perhaps that's what you are looking for: dotneteers.net/blogs/vbandi/archive/2013/03/25/kinect-interactions-with-wpf-part-i-getting-started.aspx

By MartinG on   Friday, April 05, 2013
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

Seeing as how this project is dated and a lot of things have upgraded since then, have you, perhaps considered doing a part 3 using the latest version of both your engine and the SDK?

By Nikola on   Friday, April 12, 2013
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

@Nikola: We would like to update the sample, but at the moment we are working on other issues with higher priority. If you have any difficulties using the new Kinect SDK and the new DigitalRune Engine, please let us know in our forum. - We like to help.

By HelmutG on   Monday, April 15, 2013
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

Could you, perhaps, give an estimate on when you might be able to look into creating such a sample?

I know the question is very rhetorical, but I plan to work on the kinect a lot this summer (in about 2.5 months), and if I had a newer sample from your team I know that both my learning and my project would be far better. That being said, my real question is: Do you think you'll get the chance to create something for the kinect by August?

By Nikola on   Saturday, April 27, 2013
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

Chances are good that we have an updated sample by August - but I cannot make any promises.

By HelmutG on   Monday, April 29, 2013
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

That's good enough for me.

Thank you for creating such a fantastic product, for making it available for students everywhere and for being generally helpful. You guys are awesome! :)

By Nikola on   Saturday, May 04, 2013
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

Hello.

I'm just checking in to see if a new sample will be released any time soon (by which I mean within a month or so). I'm planning on returning to my kinect projects in two or so weeks and I'd love to have a new sample from you guys, seeing as how the first two were great.

Could you give a rough estimate when and if you'll have something ready soon?

Here's hoping :)

By Nikola on   Wednesday, July 10, 2013
Gravatar

Re: Real-Time Motion Capture using Kinect and XNA (Part 2)

Hi Nikola, we try to update the sample next week and include them in the next DigitalRune Engine update. If you send an email to our office email address, we can send you the new code as soon as it is available.

By HelmutG on   Thursday, July 11, 2013

Your name:
Gravatar Preview
Your email:
(Optional) Email used only to show Gravatar.
Your website:
Title:
Comment:
Security Code
CAPTCHA image
Enter the code shown above in the box below
Add Comment   Cancel