[How To] Upload Label Predictions with Python

Apr 8, 2020 | How To

Upload Label Predictions with our Python API

 

In our latest Blog post we introduced the “Import Label” feature which allows DataGym users to import their annotated image data directly into their DataGym Projects. The feature enables users to inspect and evaluate the results of their prediction models from within DataGym. This article aims to give a real-life example of how to use the “Import Labels” feature with our Python API. The code samples in this guide are also available as a Jupyter Notebook on GitHub.

Check it out and register your DataGym account hereit’s free!

Use Case

In their cutting-edge paper, Wang et al. [1] presented a new chest X-ray database, namely “ChestX-ray8”, which comprises 108,948 frontal-view X-ray images of 32,717 unique patients with the text-mined eight disease image labels (where each image can have multi-labels). They demonstrated that commonly occurring thoracic diseases can be spatially-located via their unified weakly-supervised multi-label image classification and disease localization framework.

Wang et al. made their findings and resources publicly available to other researchers. Their datasets contain frontal-view chest X-ray PNG images as well as coordinates for bounding boxes that identify the location of the detected thoracic diseases. In this tutorial, we use a subset of their ímages and annotated label-data to create a project in DataGym that imports and combines these resources. We chose this use case to demonstrate that DataGym can be used to import pre-labeled images that are the outcome of a prediction model. After the data is imported into a Project, DataGym allows its users to easily inspect and re-evaluate the predicted labels. Thereby, there is no more need to start labeling images from scratch, labelers have now the reduced task of correcting pre-labeled images.

In total, we will upload a subset of 880 images and 984 annotations into a DataGym Project.

Starting Point

We start with the resources provided by Wang et al.: 

  • A .csv file that contains the bounding boxes, which identify the location of thorax diseases in X-ray images. Preview:

 

  • A set of 880 X-ray images. In this example image we already marked the predicted location of the disease with a red circle:

Goal

Our goal is to import the images and the labeled data into a DataGym Project. This allows the labelers to view the predicted diseases and to correct the size and location of the labels if necessary. Instead of labeling images from scratch, labelers have now the reduced task of correcting pre-labeled images. As you can see in the image below, we imported both the image and its respective label. Thereby, we directly depict the bounding box from the .csv file to identify the affected region in the X-ray image.

 

 

References 

[1] Wang, Xiaosong, et al. “Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

Import the Data

We use Pandas’ DataFrames to read and process the prepared data from the .csv file.

Image Index Finding Label Bbox [x y w h] Unnamed: 6 Unnamed: 7 Unnamed: 8
0 00013118_008.png Atelectasis 225.084746 547.019217 86.779661 79.186441 NaN NaN NaN
1 00014716_007.png Atelectasis 686.101695 131.543498 185.491525 313.491525 NaN NaN NaN
2 00029817_009.png Atelectasis 221.830508 317.053115 155.118644 216.949153 NaN NaN NaN
3 00014687_001.png Atelectasis 726.237288 494.951420 141.016949 55.322034 NaN NaN NaN
4 00017877_001.png Atelectasis 660.067797 569.780787 200.677966 78.101695 NaN NaN NaN

As you can see, this is a bit messy. Therefore, we start by removing the unnecessary columns and changing the column names into a more readable format.

image_name label x y w h
0 00013118_008.png Atelectasis 225.084746 547.019217 86.779661 79.186441
1 00014716_007.png Atelectasis 686.101695 131.543498 185.491525 313.491525
2 00029817_009.png Atelectasis 221.830508 317.053115 155.118644 216.949153
3 00014687_001.png Atelectasis 726.237288 494.951420 141.016949 55.322034
4 00017877_001.png Atelectasis 660.067797 569.780787 200.677966 78.101695

This looks much better!

Prepare the Resources in DataGym

 

Before we can import our labeled data into DataGym, we need to create a Project and set a Label Configuration. If you have any trouble following these steps, please visit DataGym’s documentation.

Create a Project

Creating a Project in DataGym is rather simple. Sign in to your account and create a Project named ‘Research Project

Create a Label Configuration

In order to import annotated data into your Project, you need to define a Label Configuration first. The Label Configuration defines which labels and classifications are available in a Project. You can define geometries for labeling, attach classifications to them or create global classifications for more general questions.

In our case, we will create a geometry for each label from our .csv file. These labels are:

  • Atelectasis
  • Cardiomegaly
  • Effusion
  • Infiltrate
  •  Mass
  • Nodule
  • Pneumonia
  • Pneumothorax

You can also print these labels to check if the list is complete.

Follow the steps below to create a Label configuration 

Step 1: Navigate to the “Label configuration” tab within your “Research Project”

Step 2: Add a Geometry to your Label configuration and choose a rectangle

 

Step 3: Enter the first label name as key in your new Geometry. Then, click on “Save edits” to save you label entry.

You can choose a color and shortcut to better distinguish between your annotations.

Step 4: Click on “Save config” to save your Label Configuration.

Step 5: Add the remaining labels from the list. The result should look like this:

Don’t forget to save your configuration!

Create a Dataset

In order to upload images to our Project, we need to create a Dataset that can hold the images from our annotated data. This time we can use the Python API to quickly generate a Dataset directly from a Jupyter Notebook. Therefore, we need the following Client methods:

If this is your first time using the Python API, please visit our Getting Started Guide.

We easily created a Dataset via Python. But there aren’t any images in this Dataset yet.

Upload Images to the Dataset

Prepare a list with image URLs

To upload images, we need a list of URLs that reference all of the annotated images. The images are stored on our website: “https://media.datagym.ai/blog/chestxray/images/”. Since we already know the image names from our .csv file, we can easily combine the “image_name” column with our image server path. Thereby, we can generate a URL list that links all the images from our .csv file.

Upload the URL list

Now we can upload the images to our Dataset. Therefore, we use the Client class of the Python API.

DataGym returns a success message for each uploaded image. The response contains the internal_image_IDs which are later needed as a reference to upload the annotated data into DataGym’s Datasets.

Create an internal image ID reference Dictionary

In the next step, we create a Dictionary that maps the internal image ID to the image name from our .csv file. This is needed to identify the images in our DataGym Project when we upload our annotated image data.

There are two ways to generate this Dictionary.

1. By using the response data from the image upload:

2. By using the Images from your Dataset

Connect the Dataset to the Project

We already created a Dataset and filled it with our images. The only thing that’s left is to connect this dataset to our Research Project. The Python API provides a simple method to add a Dataset:

Prepare the upload of annotated image data

 

We want to upload the annotated image data from our .csv file to our DataGym Project. DataGym uses a specific format for annotated data imports. Therefore, we have to convert our rows from the .csv file into this specific form.

Understand the schema

Before we start, let’s have a look at our .csv again.

image_name label x y w h
0 00013118_008.png Atelectasis 225.084746 547.019217 86.779661 79.186441
1 00014716_007.png Atelectasis 686.101695 131.543498 185.491525 313.491525
2 00029817_009.png Atelectasis 221.830508 317.053115 155.118644 216.949153
3 00014687_001.png Atelectasis 726.237288 494.951420 141.016949 55.322034
4 00017877_001.png Atelectasis 660.067797 569.780787 200.677966 78.101695

Every row in this table represents an annotated segment in an image. DataGym can add these annotations to its Project images when a specific JSON format is used. As an example we take the first row of the table above and convert it to valid DataGym JSON:

Property Description
internal_image_ID The internal ID to identify the image.

In order to address the correct image we have to replace the image name from our .csv with the internal image ID of our Dataset. For this case, we already prepared the image_ids_dict to get these internal image IDs by their image name.

keepData If keepData is equal to false, all already existing labels for the current Image will be deleted after the labels upload. If keepData is equal to true, all new labels will be added to the already existing labels for the current Image.
Default value for keepData is true
global_classifications It can be left empty because we haven’t defined any additional global classifications in our Label configuration.
labels A label describes a geometry within an image.

As you can see, the annotated image segments end up in the labels attribute. Remember that we defined a Geometry in our DataGym Project for every label of our .csv file. One of these labels is ‘atelectasis’, which we created as a rectangle.

geometry The geometry attribute contains the coordinates of the annotated segment.
classification The attribute named classifications can be left empty because we haven’t defined any additional classifications in our Label configuration.

Convert the .csv into DataGym JSON

Follow this step-by-step guide to create the DataGym JSON from the .csv file

1. Create a Dictionary template for annotated data

First, we need to create a Dictionary that can hold all labels for every image in our .csv. Therefore, we define a nested Dictionary based on the image name and label. Since there can be multiple instances of annotated data (aka geometries) for each label, we initialize a (yet) empty list per label. The code snippet below results in a template Dictionary called __labels_per_image__ which has the following form:

 

2. Create and Add Label Entries

We iterate a second time over the DataFrame to generate a Label Entry for every row in our .csv. A label_entry has the following form:

Then we add the Label Entries to our template Dictionary labels_per_image:

3. Generate and fill the final template

The final step is to create a list of Dictionaries that hold the image data and label data we defined above. At this point, we can recreate the schema introduced at the beginning of this section. The label_data Dictionary is created for each image in our DataFrame. It also uses the image_ids_dict to set the internal image IDs. The already formatted labels of the image can be set via the labels_per_image Dictionary.

The only thing left to do is to create a list with all the labeled data. This List is now in a valid DataGym JSON format and contains all annotated image segments from the .csv file.

Look at the last label_data object to get an example of the final form after the conversion:

Import the Annotated Data into the DataGym Project

Now we can finally import the annotated data from the .csv file to our DataGym Project. We only need to pass the label_data_list to the Python API Client.

Inspect your Results

Visit DataGym to inspect your labeled images. As you can see in the image below, we imported both the image and its respective label. Thereby, we directly depict the bounding box from the .csv file to identify the affected region in the X-ray image. Feel free to browse through the images within your Project to see more examples. There are also multiple ways to design your own Label configuration for your own Projects. We recommend using this real-life example to further explore the functionalities of DataGym and its Python API.

 

This is it! You imported 880 images and 984 labels into your DataGym Project using our Python API. Now you can evaluate the labeled data by adjusting misplaced labels or adding labels that were missing. There is no more need to start labeling new images from scratch, labelers now have the reduced task of correcting pre-labeled images. Thereby, you can save time labeling and quickly improve your prediction models. DataGym also supports team workflows, so that many labelers are able to work on the same project at the same time.

Check it out and register your DataGym account hereit’s free!

We hope you enjoyed our article. Please contact us if you have any suggestions for future articles or if there are any open questions.