Skeleton Keypoints
We explain how you can label and interpolate keypoint skeletons and custom polygons.
In this Darwin Fundamentals session, we explore the Keypoint Skeleton feature, a particularly useful tool for 2D image and video annotation. In simple terms, Keypoint Skeletons are sets of predefined points, joints, and shapes that can be adjusted to match an object’s pose. They’re an efficient way to define object poses or create custom polygons for reuse in projects.
Eager to explore keypoint annotation tools and tricks in more detail? Explore our guide to keypoints and skeletons.
In this video, we start by demonstrating how to create a Keypoint Skeleton for a human hand, mapping each point and joint to the fingers. From here, we explain how to change the Instance Tracking ID of each point, to make use of the Keypoint Skeleton simple for your use case.
Next, we explore how to use the Keypoint Skeleton within your datasets once it's been set up. We showcase how to drag and drop the Keypoint Skeleton onto your frame, manipulate it to an object, and adapt it per frame. We also explain how the Keypoint Skeleton tool behaves in video, showcasing how the tool interpolates between varying frames so you only have to label a few frames to get a smooth transition that represents ground truth.
We also note one important thing: the Keypoint Skeleton is versatile, and not limited to specific object types. For example, it can be used to create custom polygons, such as cuboids or pentagons, to represent different objects.
You’ll come away with a clear understanding of the Keypoint Skeleton tool, how it works, and how you can use it within your project to save time and bolster your overall annotation process.
Keypoint skeletons are a series of points, joints, and shapes that you can predefine apply to an object, and then adjust to meet its pose. You can use them to define the pose and shape of an object, or as a handy way of creating some custom polygons that you can reuse throughout your project. Let's take a look at how you use them in image and video annotation.
In your class creation modal, you'll notice a skeleton option at the bottom right, we'll select this option and give it a name. In this case, we'll create a human hand. We'll pick a nice color for it. Let's go for teal. And once we pick this skeleton class type, we'll be presented with a skeleton editor window.
This is where we'll draw our shape and it's quite simple. Just like drawing a regular polygon. We'll click to create some points. In this case, I'll make a series so that I can create a whole human hand. Note that each point is unique and automatically given a name in this case. And number next up, I can choose to connect them.
In this case, this will create the bones and the skeleton, if you will. Use this link button to connect them with one another. You can connect the point with as many other points as you wish or also leave them alone. And finally, when you're happy with your shape, you can also rename these points. For example, this one here is the thumb tip, so it'll be useful to recognize it when I draw it.
This one is the wrist and so on. This will be quite useful while you're drawing your skeleton as its shape may morph significantly. Especially in video. You may rename and move these origin points later, but you cannot add new ones. So make sure your schema is correct. When you're done, you can add a description as usual and select a thumbnail.
In this case, we have a skeleton hand here that I can use to represent it. When you're using the skeleton tool, simply drag the shape like a bounding box and adapt it to your object. In this case, we'll move these joints of a person so it matches their skeleton. Starting with a template like this makes it much faster and easier to adapt objects to a new shape, especially if you have many types of skeletons in a dataset.
You'll also encounter cases in which your objects are flipped in the vertical axis. In which case, simply draw the box the other way around and the points will be mirrored. And even though it's called a skeleton tool, it can be used for any type of custom polygon. For example, you may have a weird type of cuboid such as this one, or you need to apply a pentagon to represent an object.
All you have to do is draw it in the skeleton editor and then apply to your images. Now let's take a look at how skeletons behave in video. We'll create a new annotation class. In this case, a right eyebrow. Select the skeleton tool and we'll define five points and connect them together. We'll also define an eyebrow tip so we know it's orientation.
Keypoint skeletons can interpolate between frames on V7 Darwin, so we'll only have to label a few frames of this video to get a smooth transition that looks realistic and represents ground truth. I am playing and pausing the video here, and every time I pause, I move the whole shape or some of the points.
This will automatically generate key frames for me, and now I can pass on to the other eyebrow, which I drew the other way around. I'm going to flip this over and start with this left eyebrow. You'll notice whenever I hover over a point to move it, it tells me the name of that point. Those are the names that we defined during the skeleton editing phase.
You'll also notice that I can move the whole shape if I drag one of the vectors or bones, if you will, which will also generate a new key frame. V7 Darwin allows you to play 4K videos like this one, even when zoomed in, so you can focus on a specific point of action and make sure that your ground truth is accurate.
Okay, I'm sure you have better things to do than watch me label the side brow, so I'm going to speed this up significantly. When you feel like you're done, you can drag the play head left and right just to make sure that every frame is sticking to the right points of your object. And now let's take a look at the final result.
You can continue to pan around and zoom in and out while the video is displaying or toggle display options, for example, brightness, contrast, or the display label that you're seeing over here. These are looking quite good for a few minutes of work. If you're happy with this, you can press the center review button on the top right.
Okay. One last example, I hope you haven't forgotten about the hand that we drew at the beginning of this video. We're going to use it to define the position and pose of this human hand over here. Otherwise, how else will we teach robots to pet dogs? One day we're going to move these joints to match the position of the this hand, even though it's a bit occluded.
You'll see how useful the point names are in this example in which there are so many types of joints in this hand, and we have to remember which pose is where between frames. As per the previous example, we'll play and pause the video and move the points to match their new position, after which we'll trace back, play the video again and make sure that they're sticking correctly.
And after a couple of minutes of work, we have our results. I'm going to leave you with this. You can try this skeleton editing tool now at v7labs.com.
