Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft PR that tracks the work towards TrackMate v8 #309

Draft
wants to merge 67 commits into
base: master
Choose a base branch
from
Draft

Conversation

tinevez
Copy link
Member

@tinevez tinevez commented Nov 7, 2024

TrackMate v8 - better integration of external segmentation algorithms and interactive segmentation editor

A LabKit launcher made to edit spots in the current frame

This launcher catches what time-point is currently displayed, and wrap all information needed for LabKit to edit the spot it contains, as labels.

TrackMate-LabKit_01 TrackMate-LabKit_02 TrackMate-LabKit_03

While the user edits the labels in LabKit, the TrackMate UI is disabled.
Once the user closes the LabKit window, the modifications they made are inspected. This class compares the new labels with the previous ones, and can determine whether a spot has been added, removed or modified. In the last case it updates the model with the modified spots, and reintroduces it in the tracks as it should.

TrackMate-LabKit_04

All features of modified spots and their edges are recomputed.

If a label has several connected components, they are added as separate spots. The one closest to the original spot is reintroduced in the tracks.

For developers: tools to integrate algorithms run from a command line interface

There is now a small framework to make it simpler and faster to integrate in TrackMate segmentation or tracking algorithms that can be run from a command line interface (CLI). Examples of such CLI components are cellpose, omnipose, YOLO, Trackastra, etc.

The CLI framework is in the fiji.plugin.trackmate.util.cli package. A developer that wants to integrate a CLI component needs to extend the class CLIConfigurator, declaring the parameters that the CLI will accept. The syntax is made to resemble that of Python argparse
For instance for Trackastra (another module):

public class TrackastraCLI extends CondaCLIConfigurator
{
// ...

	public TrackastraCLI( final int nChannels )
	{
		this.modelPretrained = addChoiceArgument()
				.name( "Model pretrained" )
				.help( "Name of pretrained Trackastra model." )
				.argument( "--model-pretrained" )
				.addChoice( "general_2d" )
				.addChoice( "ctc" )
				.defaultValue( DEFAULT_TRACKASTRA_MODEL )
				.key( KEY_TRACKASTRA_MODEL )
				.get();

		this.customModelPath = addPathArgument()
				.name( "Path to a custom model" )
				.argument( "--model-custom" )
				.help( "Local folder with custom model." )
				.defaultValue( DEFAULT_TRACKASTRA_CUSTOM_MODEL_FOLDER )
				.key( KEY_TRACKASTRA_CUSTOM_MODEL_FOLDER )
				.get();

		// ...

The addChoiceArgument() method will declare an argument that can accept a series of discrete values (specified by addChoice). The addPathArgument() method will declare an argument that is actually a path to a file. They take the shape of a Builder pattern, where:

  • name() specifies a user-friendly name for the argument.
  • argument() specifies what is the switch to use in the CLI
  • help() specifies a help message to be used in UIs.
  • defaultValue() specifies a default value. If the user does not specify a value, the default value will be used in the CLI.
  • key() is used for the TrackMate integration only. Here you can specify the String key that will be used to save parameter values in the TrackMate file.

Let's carry on with example from YOLO integration:

		this.conf = addDoubleArgument()
				.name( "Confidence threshold" )
				.argument( "conf=" )
				.defaultValue( DEFAULT_YOLO_CONF )
				.min( 0. )
				.max( 1. )
				.help( "Sets the minimum confidence threshold for detections. Objects detected "
						+ "with confidence below this threshold will be disregarded. Adjusting "
						+ "this value can help reduce false positives." )
				.key( KEY_YOLO_CONF )
				.get();

This builder adds an argument that accept floating point argument, as doubles.

  • min() and max() set limits for acceptable values. If the user tries to set values outside of these limits, an error is thrown.
		addFlag()
				.name( "Save detections to text files" )
				.help( "Whether the detections will be saved as a text files." )
				.argument( "save_txt=" )
				.defaultValue( true )
				.required( true )
				.visible( false )
				.key( null )
				.get();

The addFlag() builder adds a boolean switch argument. In the CLI tools we have been trying to implement, they exist in two flavors.

  • if the argument starts with a double-dash --, as in Python argparse syntax, then it is understood that setting this flag to true makes it appear in the CLI. For instance: --use-gpu.
  • if the arguments ends with a = sign, then it expects to receive a 'true' or 'false' value.

In this specific case, we want to tell YOLO to save results to text files so that TrackMate can read them and import detection results from them. We want to set the value of this argument to always true and don't let the user change this value in TrackMate. To this aim:

  • reguired() determines whether setting a value for this argument is required. If the user does not set one, and if there is no defaultValue, an error will be thrown. Here it is set to true and we have a default value.
  • The visible method states whether this argument will appear in the TrackMate UI for the module, letting the user configures its value. By settings it to false, we prevent it from appearing.
  • Here the value for key() is null, which specifies that TrackMate should not save this parameter to the TrackMate XML file. No need, it is always true.

We use the same strategy for the arguments that say where to load the input images from. This parameter is set by TrackMate when it exports the image to disk for YOLO, and must not be set by the user. This is how it is done:

		this.imageFolder = addPathArgument()
				.name( "Input image folder path" )
				.help( "Directory with series of .tif files." )
				.argument( "source=" )
				.visible( false )
				.required( true )
				.key( null )
				.get();

visible( false ) and key( null ) make sure the argument won't be visible in the UI, and won't be serialized in the TrackMate XML file, but because there is a argument, the CLI will have it, configured by the TrackMate detector.

tinevez and others added 30 commits July 7, 2024 12:34
And remove Gson from now. Apparently there are incompatibilities
from the (same) declaration in LabKit pom, which triggers the
errors I complained about today on the forum.
This launcher catches what time-point is currently diplayed,
and wrap all information needed for LabKit to edit the spot
it contains, as labels.

While the user edits the labels in LabKit, the TrackMate UI is
disabled.

Once the user closes the LabKit window, the modifications they
made are inspected. This class compares the new labels with the
previous ones, and can determine whether a spot has
been added, removed or modified. In the last case it updates the
model with the modified spots, and reintroduces it in the tracks
as it should.

All features of modified spots and their edges are recomputed.

If a label has several connected components, they are added as
separate spots. The one closest to the original spot is reintroduced
in the tracks.

The label names are important: they are used to retrieve the original
spot id and the original spot shape for comparison.
If the user modifies a label, it will be perceived as a
new spot instead of a modified one.

Tested in 2D so far.
…ard.

But because the other table, trackscheme and bvv buttons take
too much place, we don't see it without resizing the window.
Otherwise the button for the editor is not visible without
resizing the window. I also had to programmatically resize the
main TrackMate frame after display so that components are
properly aligned. Such a hack for something simple...

Also give the editor button a proper name and a temporary icon.
- We don't depend on labels anymore, but directly operate and compare
the index images (before modification and after). Because the index
is directly related to the spot ID, we can get a match from previous
spot to novel spot in an easy manner.

- The spots from the edited version are created directly from the
novel index image, using something adapted from the label image
detector code, so again, just one pass. We use the fact that we can
provide it with a 'quality' image, and read the index of the label
image 'under' the spot and write it into its quality value.
This way we can retrieve the id of the matching previous
spot in an easy manner.

- The price to pay for not working with labels anymore
is that we don't have access to the label name, but that's life.

- We make only one pass over the image to collect the ids of the spots
that have been modified, instead of one pass per spot. Also, this
pass is multithreaded (thanks LoopBuilder).

- I have also learned that I should not use weakListeners() if I
am doing something with threads inside the listener. Using listeners()
instead works, but I do not know why the other one does not.
Probably something arcane with Java WeakReferences being collected.

- As a result of all this the performance is much better than before
and the 'return to TrackMate' should happen without the user noticing
the process.
I wanted to use it to check whether the user has activated the
'LabKit' update site, but apparently the labkit jars are included
with vanilla Fiji.
Let them choose to discard or commit the changes.
So that the image in the LabKit window opens with the same
display settings than in the ImagePlus main view.
We don't need the Labeling wrapper.
The LabKit editor worked fine foe us but only in 2D.
In 3D the labels imported from the spots into LabKit were off along
the Z axis, as if the Z caibration was not handled properly. This
was caused by the custom BDV showable I made missing some important
preparation steps. I simply copied these steps from the working
version of BDVShowabble in the core LabKit code.
And remove debug code.
I am not so sure it is a good idea for the color...
and use the white LUT when the imp is not displayed as a Composite.
Otherwise we get crashes when we have more than 4k labels. Which is not
what Labkit is optimized for but we will see that in a second time.
In case we change our minds on the backing integer type, right now
the labkit launcher and importer classes are generic.
The re-importing of labels from Tabkit to TrackMate could fail for
2D images and labels with a large index. For instance, it failed
consistently when trying to re-import labels with an index larger
than 65643.

This problem roots in the getSpots() method of LabkitImporter. It
relies on a trick: We get the new label image, and create spots from
this label image. But we want the new spots to keep track of the index
in the label image they were generated from.

For this, in 2D, we use the MaskUtils.fromLabelingWithRoi()
method. These methods accept an image as last argument used to
read a value in the label image within the spot, that is normally
used for the quality value of the new spot.

But the SpotRoiUtils.from2DLabelingWithRoi() method converted the
extra image to ImagePlus (because I was lazy). So the label image
was effectively cast on ushort for an IntegerType image, hence
the problem with the max label being 65453.

The solution is to rewrite the from2DLabelingWithRoi() so that
it does not rely on converting to ImagePlus, but on good old
iteration with imglib2.
Provided that the detector that is called is cancelable.
tinevez and others added 23 commits July 11, 2024 15:11
Also better message when closing the editor.
If a ROI exists in the input image when launching the LabKit editor,
only the image and the spots present in the ROI are sent to the
editor. Spots touching the borders are not imported.
Normally pressing space and dragging should move the view, as
for normal ImageJ tools. The removed line was preventing it.
When editing a sub-portion of the image, the spots outside the ROI
are not imported in the editor. It is possible that the user creates
a new label in LabKit that will have the same id that an existing
spot outside the ROI.
Before this fix, the spots in that case were replaced by the new ones,
creating a mess.
The fix consists in reminding what spots are imported in the LabKit
editor, then compariing to this list the new ids when reimporting
from the editor.
Simplified, removing the menu, the segmentation features and the
segmenter features.
Somply done by copy-pasting Matthias classes and removing the lines
with the features to prune.
This time it was due to confusion between a map from spot ID to spots,
and a map from label value to spot.
I fixed it by rewriting everything in terms of label values in the
labeling. So we do not need to keep track of the spot ID anymore. The
labels could be anything. They are still equal to spot ID + 1, because
it is convenient to debug.
When editing the whole movie, if the label of a new spot was using
the label of an existing spot in another time-point, the existing
one was removed. Because the new spot was identified as a modification
of the existing one.
The solution is to pass to the importer only the list of existing
spots in the current time-frame.
This could happen after loading a TrackMate file with specific
display settings but no tracks.
Mother class for CLI configurator that relies on an executable
installed in a conda environment. This happens when the Python code
we want to call does not have a Python module (that we could call
with 'python -m') or is simply not a Python program. In that case:

on Mac we resolve the env path, and append the executable name to
build an absolute path to the executable.
on Windows we simply activate the conda environment and call the
executable assuming it is on the path.
A null key means that the argument should not appear in the
settings map when the CLI will be deserialized to the map. This is
useful e.g. for arguments of the CLI that must not be configured
by the user, nor should not be serialized to disk. Such as the path
where to save temp images.
0 is good enough as we require the quality to be a positive
value.
Then removes the specific imglib2 and labkit version specs.
Also named this future version the true v8. The existing v8 branch,
that focuses on 3D, shall be renamed v9.
Focuses on the rewrite of KDtree. The left and right fields of
KDTreeNode are not visible anymore. The implementation changed to be
proxy-based, so the corresponding methods, used in this commit to fix
the compile errors, are deprectaed.

We therefore shall also full rewrite the KDtree NN tracker using
the proxy approach.
- Now RAIs are iterable.
- You can get its type with getType() method.
TrackMate model requires the graph to be undirected (weird choice
be hey, it enforces the time-directivity 'by hand'). But sometimes
it is advantageous for trackers to manipulate a directed graph. This
method bridges the two, because there are no method for that in the
JGraphT lib.
Get rid of the custom classes, simply rely on the imglib2 implementation
and on a directed graph to check whether a target node is free or not.
Much simplier, shorter.
Before this commit, the labkit import "did not work".
The image appeared black, the UI was super slow etc.
Because I do not know what is really going on, and because
I already have been changed these two lines, I am keeping
them in comments, to see if I need to revert this later.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant