Preparing the train and validation data in Cell-DETR

[guanyichen]

Hi Lisa,

Thank you so much for providing the code and the explanations. It is really brilliant work.
I would like to use your modified Cell-DETR model for the detection and segmentation of a single object in order to study the DETR.
However, I met some problems when I prepared the train and validation data.
As your custom dataloader (dataset.py) indicated, you prepare the data in images_dir and masks_dir.
Now I have almost 100 images that have hyperbolic objects to be annotated.
I'm really confused about how to prepare the train and validation data.
Would you please share the details of how you prepare the train and validation data?
Thanks in advance.

Best regards,
Simon

[LSnyd]

Hi Simon aka @guanyichen,
thank you, I am very happy to hear that you like the PartSegmentationToolbox.

Just to clarify this, the PartSegmentationToolbox is independent of the Cell-DETR model. Cell-DETR works with images and masks in the 2D space, while the PartSegmentationToolbox is made for 3D Meshes.

Within Cell-DETR, I typically have a directory for images (images_dir) as well as a directory for masks (masks_dir). The mask directory has one folder for each image name and in this folder the corresponding collection of masks. I am storing in each folder one mask per existing class. In dataset.py, these masks are loaded and stacked as channels. So a segmentation task with 5 classes, ends up with 5 channels for the mask. There is no approach to creating segmentation masks, if this is what you are looking for.

[guanyichen]

Hi Lisa,
Thanks for your quick reply.
Would you please check the following steps for creating segmentation masks?

First step
I completed image annotations by the tool Labelme.
View the original image here
View the corresponding JSON file here

Second step
This tool can convert instance segmentation annotations to VOC-format.Labelme convert format details
After complete annotations, I convert all the related files to VOC-format in the 'data_dataset_voc' folder.
View all converted files here
It generates:

• data_dataset_voc/JPEGImages
• data_dataset_voc/SegmentationClass
• data_dataset_voc/SegmentationClassVisualization
• data_dataset_voc/SegmentationObject
• data_dataset_voc/SegmentationObjectVisualization

The Original Image

The Segmentation Mask Image

The Segmentation Mask Visualization Image

Thus, I think maybe creating segmentation masks can be done in this way.
With this method, does it possible to adapt all my data to your modified Cell-DETR model for the detection and segmentation of a hyperbolic object?

Thanks in advance.
Best regards,
Simon

[LSnyd]

Hey Simon,
your approach sounds good to me.

I think you just need to save the segmentation mask with the hyperbolic object in the correct directory structure.

• images
  • train
    • image123.jpg
  • test
  • validate
• masks
  • image123
    • 0.jpg

With image123 being the name of the corresponding image directory and 0 being the class label of the segmentation mask.
The background class is added automatically by the dataset.py class.

Let me know if that works for you. I never expected anyone else to use my modification actually, so I didn't prepare a readme or so. There are also a couple hard-coded values that fit to my use-case specifically. I can help you with those later on, if the data loading works.

[guanyichen]

Hi Lisa,

Thanks for your great support.
I save the segmentation mask with the hyperbolic object in the correct directory structure as you described.
(View the images folder)
(View the masks folder)
It seems like the data loading (dataset.py) works fine.

Thanks in advance.

Best regards,
Simon

[LSnyd]

Hi Simon,
perfect. I believe this error comes from the initialisation of the ModelWrapper.
When you initilize it in the main.py you need to add a list of class labels (class_labels) and a list of colors (colors) for each class label. Seems like I've missed that part when pushing.

You could use this function:

def create_class_colors(classes): 
    """This function creates color values for the list of classes.
    :param classes: (list) list of all existing classes"""
    
    colors = dict.fromkeys(classes)

    for key in colors.keys():
        r = np.random.randint(0, 255) / 255
        g = np.random.randint(0, 255) / 255
        b = np.random.randint(0, 255) / 255
        colors[key] = [r, g, b]

    return list(colors.values())
        
    

so you would end up with

classes = ["1","2","3"]
colors =  create_class_colors(classes)

and call the ModelWrapper with

model_wrapper = ModelWrapper(detr=detr,
                               detr_optimizer=detr_optimizer,
                               detr_segmentation_optimizer=detr_segmentation_optimizer,
                               training_dataset=training_dataset,
                               validation_dataset=validation_dataset,
                               test_dataset=test_dataset,
                               class_labels=classes,
                               colors = colors,
                               experiment = "test1",
                               loss_function=InstanceSegmentationLoss(
                                   classification_loss=ClassificationLoss(class_weights=torch.tensor(np.ones(3), dtype=torch.float)),
                                   segmentation_loss=SegmentationLoss(),
                                   ohem=args.ohem,
                                   ohem_faction=args.ohem_fraction),
                               device=device)

Best,
Lisa

[guanyichen]

Hi Lisa,

Thank you for your hints.
I modified the main.py according to your description. (View the modified main.py here)
It seems like some problems occurred. I am working on this.

Best regards,
Simon

[LSnyd]

Hi Simon,
seems like there are still some problems with loading your masks, as mask_collection seems to be empty.
Have you checked whether the masks are loaded properly?

Also in your main.py you are are still using my classes (CLASSES) to create your dataset.