In the past, I used LabVIEW’s template matching function to locate zebrafish larvae in an arena for behavioral studies. However, LabVIEW is very expensive, and I can no longer afford to maintain the license.
Now is the time to switch to Python. It’s free, and ChatGPT is here to help me write Python scripts.
There are many different approaches to template matching. One method is correlation: comparing the correlation between the target image and the template image while rotating the template. This accounts for the movement of the larval fish, which can appear at various angles.
Here is the template. It is created by averaging several frames from the recording to be analyzed. Since pigmentation varies between individual larvae, the best results are obtained when the same fish is used as the template. Lighting conditions affect the fish’s appearance, and eye movements also affect how they look. This is why I use an averaged image.
The process involves selecting several frames, cropping the area containing the larva, aligning them to the same orientation, using TurboReg (in ImageJ) for image registration, and then averaging them. Here is the result:
The tail can bend and change shape, but the head and thorax regions remain relatively stable, making them suitable as a template image.
（AI-generated image (DALL·E), created via ChatGPT.）
When I used LabVIEW for template matching, I included the background area in the template image, which was not a good approach. This time, I will use a mask instead.
Creating a Mask in ImageJ – Step-by-Step Guide
- Open the Image
- Open the fish image using File > Open
- Select Only the Fish Using Thresholding
- Open Image > Adjust > Threshold
- Adjust the slider until the fish area turns red
- Click Apply
- Create the Mask
- Edit > Selection > Create Mask
- Process > Binary > Make Binary
- Process > Binary > Fill Holes (fills any gaps)
- Remove the Background
- Edit > Clear Outside (makes the area outside the selection black)
- Save the Mask Image
- File > Save As > PNG
Here is what I’ve got—the mask.
ChatGPT wrote a script for me, and here are the results after running it. The bounding box successfully locates the zebrafish larva in the behavioral arena. However, since the larva is near the edge, the dark background outside the arena seems to influence the bounding box’s position, causing it to be slightly off-center. Preprocessing the target image—such as subtracting or dividing by the background—may help improve accuracy.
Anyway, this result is more satisfactory than I expected.
（DALL·E AI-generated）
How good is it? The template is rotaed by 1 dgree to find the best fit. In the midway, the correlation can be as high as 0.5. So 0.6 may not be safe. I also noticed that the templae should have been square. If it was rectangle, the som of the image can be lost when rotated as seen below.
So, we need to use a square-shaped template image for this python script. Here is the result with square-shaped template. It was a bit better with the correlation=0.6702.
To show the orientation of the bounding box, I made a slight change.
This script works for most frames, but not all. Here is an example of a failed frame. I recall encountering a similar type of failure when using LabVIEW for template matching. This is not surprising if the same method (i.e., correlation) is being used. However, it is frustrating because we have to manually remove these failed cases. One possible solution is to set a correlation threshold to determine whether to include or exclude a frame from the data.
Example: Histogram of correlation values from a image processing of a 100 frame movie segment.
Maybe that’s not the best idea, because sometimes template matching succeeds, with low correlation values, even when the fish image is blurred due to rapid movement.
（DALL·E AI-generated）
Masking the target image or normalizing it (by subtracting or dividing by the background image) may help.
Let’s tray subtraction. I made a MEDIAN image using Image J (In the menu, Image > Stacks > Z-project… > Median).
Now l subtract this Median image from all the movie stack (On the ImageJ menu, Process > Iamge Calculator… > Operation:Subtract), and apply template matching in the same way shown above.
Now the edge of the arena is gone, which makes the script work much better.
