Prepare a .avi file
The file format should be something that can be opened by ImageJ. Some of my recordings are stored in camera-specific formats.
Background subtraction
I usually use imageJ for this process. As the edge of the behavioral areana gives strong contrast that affects image registration, first make a median or averaged iamge of the movie as a background where larval fish should disappear as it moves around, and subtract that reference image from the movie frames. Be sure that fish or any objects of interest are gone in the background image.
This is a background image made by “median” in ImageJ. Note that fish are gone because of taking “median” of pixel values through the frames of the movie.
Background-subtracted image using ImageJ
I found that performing background subtraction is critical for successful template matching in the next step. This is because the edge of the behavioral arena (i.e., the wall) features a high-contrast, noisy region that can be mistakenly identified as a fish larva during template matching.
Preparation of a template image of the larval fish
We employ template matching in our method. First, we require a template image of the fish larva along with its corresponding mask image. Since the fish’s appearance may vary slightly in each frame, I suggest using an averaged image.
- Prepare some frames in the movie to be analysed that gives defference of the larval fish in appearance (lighting in different directions for example).
- In ImageJ, using line/measuring angle tool, obtain the angle of the fish and rotate it to make it up straight and crop it. Larva should be roughly in the same position in each frame. Have an enough space around the fish so that we can crop it in a sqaure shpae, at later step.
- Make the cropped frames to a stack. (In the ImageJ menu, Image > Stacks > Images to Stack)
(Here to show frames in the stack, ImageJ:Image > Stacks > Make Montage… was used) - Z-project the stack and make an averaged image that is used for image registration of the template fish in the next step.
I found that the use of this averaged image gives better results than use of a single frame from the stack as a target image. - Use TurboReg plugin for image registration with the above averaged image as a reference image (i.e., target image) (https://bigwww.epfl.ch/thevenaz/turboreg/).
(stack was montaged) - Crop the head and thorax area.
- Z-project the TurboReg-ed stack and make a averaged image.
This can be as an template in template matching. The shape should be square because the template is going to be rotated 360 degrees during template matching to look for the bet match. - Also make a mask from this image. (Process > Binary > Convert to Mask) Use thresholding before this to get the thresholded image.
Template matching of the larval fish
I use the pythone script “Template matching (larval fish tracking) parallel ThreadPool.jpynb” (ChatGPT wrote it for me). The script does image registration while changine the angle of the template image one degree at a step (i.g., working on 360 degrees) and gives the best match. Then it repeats that process for all the frames in the movie.
This step is kind of slow. Thescript utilize multi cores. Even so, it takes about 1.3 sec on my Laptop PC (Panasonic Let’s Note).
Input and output are the following.
#input
avi_path = “20120324_1839-(DV)(2)100frames(justfortest)MED-subtracted.avi”
template_path = “1frameFish38x38.png”
mask_path = “1frameFishMask38x38.png”
#output
output_txt = “output_results.tsv” # bounding boxの描きこみ用
out = cv2.VideoWriter(“output.avi”, fourcc, 30.0, (480, 480))
The out.avi can be an option, in which the bounding box are drawn. The output_txt.tsv (tab-separated) saves
- Frame
- X_center
- Y_center
- Angle
- Correlation
in each column.
Image registration
No using the result from the above template matching (X_center, Y_center,Angle), we do translation and rotation of the oritinal movie so that the larval fish can be registered. Here we use a python script “Image_Registration_XYAngle.ipynb” that I (ChatGPT) made.
Maybe the raw image (before background subtraction) works better for the further analysis like eye movement measurements, I guess.
Further Image registration with TurboReg
The image registration using the python script give good but not perfect results. If we want to do more detailed analysis (like eye movements measurement), we want to have better image registration. I found that Turbo Reg (ImageJ)(“Rigid Body”) works pretty well now. TurboReg cannot be in the first step.
Left:target image, middle:movie, right:registered iamge.
