cpsam direct method

[yw254]

Hi, thanks for the great tool!
I was trying to use cpsam (manually installed) + uSegment3D with the ovule image in the example data. I managed to use the indirect method after a bit of transposing the cpsam masks. But with the direct method, the aggregated labels did not follow the cell boundaries (it worked nicely in another uSegment3D env with 'cyto3' specified). I can use the indirect method for cpsam as the model works better for some of my images, but I would love to know if there is currently any flow incompatibility between cpsam and uSegment3D? Or I could be making some mistakes?

[fyz11]

Thanks for reaching out! I've actually never played around with the CPSAM flow outputs before, since they got rid of the model zoo. And since, we just use indirect method to operate on the masks. Conceptually, I don't see why there should be any discrepancy unless they changed the ordering convention of xy, xz, yz between CPSAM and cellpose3. Another possibility is differences in probability output?

Would you be able to share an example with:

1. CPSAM flow outputs for xy/xz/yz slices,
2. your code
3. input image (.tif)

and i will take a look into this issue. It does seem weird to me.

[yw254]

Thanks for the quick response. The image I used for testing is the Arabidopsis ovule image provided in the paper, and I referred to the [segment_ovules_3D.py] code. And here comes the full code I ran for the direct method: https://gist.github.com/yw254/e3bbcafeeadef8ab730ed16c16e7a230. The raw image and flows can also be found here: https://sid.erda.dk/sharelink/BtrZ9Hmkq3

[fyz11]

here you go, a modified version of the code that should now work (giving the output in the screenshot): https://gist.github.com/fyz11/4fe1af641e6fea214e0ed4b01fe5950f

there were a couple of things in your code that i reworked:

1. xz: is derived by img.transpose(1,0,2), the reverse transformation for probability is also (1,0,2)
2. yz: is derived by img.transpose(2,0,1), the reverse transformation for probability is then (1,2,0)
3. your code injected extra zeros to get 3D gradients. actually the direct aggregation function does this step internally, so there was potentially some cross-talk. you only needed to pass the reordered 2D gradients.

In light of 1, and 2, maybe you also want to check your code for the indirect method

For reference, the way i apply the 2D cellpose model is in the function Cellpose2D_model_auto in https://github.com/DanuserLab/u-segment3D/blob/master/segment3D/usegment3d.py - you can see how the array ordering is done.

[yw254]

I see! Thanks for helping me out of the trouble. It works perfectly.

There was a subtle thing I noticed I'd better discuss with you. Let me call what was done above the 'manual direct method'. I went on digging a bit more to compare it with the indirect method and the Cellpose2D_model_auto (with cpsam specified). They all work pretty well. For some cells with imperfect 2D segmentations, the manual direct method still did a perfect job. While the direct method, with flows from Cellpose2D_model_auto (with cpsam specified) and indirect method did not aggregate labels as good for these cells. See the snapshot below of slice254 (from top left on: indirect method, direct method with flows from Cellpose2D_model_auto, manual direct method, raw image). What could be causing the difference? And here is my code: https://gist.github.com/yw254/93ce0fd9feab20da0ffc8dffa11ff7eb, in case I made some mistakes again.

[fyz11]

That's a good question. Thanks for asking. I definitely need to pick a full FAQ at some point.

I don't have the bandwidth this week or so to run the code and show you actual data. So for now, I will just post a textual answer based on what i see in your script. It will go a little into the nitty-gritty details. It also helps to note that all the default parameter values is specified in this file: https://github.com/DanuserLab/u-segment3D/blob/master/segment3D/parameters.py

There's a couple of reasons and all have to do with the parameters of the aggregation_params which varies in your code for all cases, presumably because you copied it from different example scripts which were being applied to different input images. It so happens that they differ in some key parameters that affect the 3D result.

• Indirect Method only reasons: dtform_method = 'cellpose_improve' corresponds to the type of flow predicted by cellpose i.e. direct method. The others do not. 'cellpose_skel' can improve on cellpose results if you have branched shapes, but you do need to then tune 'gradient_decay'. For your test case of ovules, since all cells are convex-shaped (roughly circular), results will be best if you use 'gradient_decay'=0. This ensures best separation of closely packed cells. Otherwise, the number of iterations=250, might not be sufficient and you get cells still joined together.

• Indirect Method + direct Method reasons: the probability of cellpose is thresholded and this binary mask determines what is foreground pixels and what is background pixels (i.e. not cell). In the manual direct method you specified ['combine_cell_probs']['prob_thresh'] = 0.5 . This value corresponds to cellpose's cell_probability=0, and is actually the threshold cellpose models are trained with. So i would expect this to be 'best'. In u-Segment3D, because we find this value is not always the best (particularly on out-of-distribution data), unless you explicitly set this, it uses multi-level Otsu (specifically 3 levels) on the (combined) cellpose probability from xy, xz, and yz to automatically infer a cutoff (c.f. ['combine_cell_probs']['threshold_n_levels'] and ['combine_cell_probs']['threshold_level'] parameters). This cutoff is not necessarily going to be 0.5. I think in this case it might have been higher? because i see the bottom layer of cells in the top right image were not segmented. The cutoff shouldn't matter in the case of the indirect method but what you might find is that sometimes cellpose is not very consistent in segmenting consecutive slices ..,.. therefore leaving 'gaps'. This is less detrimental in direct method, because you only make the binary at the end. So for indirect method, you can play a little with adding a little smoothing sigma i.e. ['combine_cell_probs']['smooth_sigma']=1

• gradient_decay: this parameter is pretty important in controlling under and over segmentation. Generally, if all your cells are roughly convex use 0.0. If you have some branching shaped cells or vesselly / highly ellipsoidal cells, you can try increasing this in increments of 0.05 to find a balance of preserving the branch/ellipsoid vs cellpose wanting to split them into two convex.

• last consideration: did you set ['postprocess_binary']['binary_fill_holes']=True? By default it is False, which would mean it would not fill in the central hole, and therefore you see no segmentation in the middle in the top two panels, but i see 'cell' in the bottom left. I assume binary fill holes was set in this case. Setting it means it will perform fill holes on thresholded binary of the cellpose probability. This is good when you know you have a densely cell packed object with no holes.

in short: if you have fine-tuned the 2d segmentation model to work well on the input data, go with the manual direct, or if you need to improve on the cellpose model to better handle branches, use a 'manual indirect' where you can switch the distance transform.

hope this makes sense. I'll post back when i have actual results where i get them to be identical (or close to). cheers.

[yw254]

Thanks for the detailed explanation. Guess I didnt understand the parameters well.

The 'binary_fill_holes' was always on. I played a bit with the [gradient_decay], ['combine_cell_probs']['prob_thresh'], and ['combine_cell_probs']['smooth_sigma']. Generally when I lower the ['combine_cell_probs']['prob_thresh'] to 0.2 to 0.5, the gaps in somes cells disappeared. The central hole was also filled. A gradient_decay of 0.05 fixed the over-segmentation in a few elongated cells. This is kind of close to the 'manual direct' method.

The 'unsegmented cells at the bottom of the top right pannel' actually belong to the raw image at the bottom right. My bad for the cropping.

I am trying to fine-tune the cpsam model on tissues with some elongated cells. I will go for the manual direct later and remember to play with the mentioned parameters.

Thanks again, and looking forward to hearing if you have more suggestions after testing.

[fyz11]

maybe a bit late, but what we have found in the lab is that for 3D microscopy data where typically the 'z' resolution is worse than the 'xy', we first resize so that it is isotropic volume, then train separate segmentation models for i) 'xy' slices and ii) 'yz' and 'xz' slices. If we don't train separate models, we find cellpose 'averages' the predictions so that xy is worse ...

[yw254]

I see. That is a very useful point. And for some cells, the shapes look distinct from different directions. It's probably better to train models separately in different directions. I will try it. Thanks a lot!