Introducing the Consensus Stage

The Consensus Stage measures the degree of agreement (overlap of the annotation area) among multiple annotators and/or models and highlights any areas of disagreement. This enables the selection of the best possible annotation and reduces the amount of time required for manual review.

This new feature:

• Allows multiple users or models to label the same task in a blind test

• Calculates any differences in the applied labels between the users or models once the task is complete

• Surfaces any disagreements and allows them to be reviewed and resolved

After all annotators and/or models have added their annotations, the disagreements can be reviewed in a review stage. Head over to the Datasets tab to access files that were sent to the review stage. Once you are in the workview, you’ll notice the annotations created by both parties and the IoU (Intersection Over Union) value, indicating the degree of the overlap between the annotations.