idtrackerai

Main idtrackerai objects

Video

Blob

class blob.Blob(centroid, contour, area, bounding_box_in_frame_coordinates, bounding_box_image=None, bounding_box_images_path=None, estimated_body_length=None, pixels=None, number_of_animals=None, frame_number=None, frame_number_in_video_path=None, in_frame_index=None, pixels_path=None, video_height=None, video_width=None, video_path=None, pixels_are_from_eroded_blob=False, resolution_reduction=1.0)[source]

Represents a segmented blob (collection of pixels) from a given frame.

A blob can represent a single animal or multiple animals during an occlusion or crossing.

Parameters
centroidtuple

Tuple (float, float) with the pixels coordinates of the blob center of mass.

contournumpy array

Array with the points that define the contour of the blob. The array is of the form [[[x1,y1]],[[x2,y2]],…,[[xn,yn]]].

areaint

Number of pixels that conform the blob.

bounding_box_in_frame_coordinateslist

Coordinates of the bounding box that encloses the segmented blob [(x, y), (x + bounding_box_width, y + bounding_box_height)]. Note that this bouding box is expanded some pixels with respect to the original bounding box obtained with OpenCV.

bounding_box_imagenumpy array or None, optional

Bonding box image that encloses the blob, by default None.

bounding_box_images_pathstr, optional

Path to the file where the bounding box images are stored, by default None.

estimated_body_lengthfloat, optional

Body length of the animal estimated from the diagonal of the original bounding box, by default None.

pixelslist, optional

List of pixels that define the blob, by default None.

number_of_animalsint, optional

Number of animals in the video as defined by the user, by default None.

frame_numberint, optional

Frame number in the video from which the blob was segmented, by default None.

frame_number_in_video_pathint, optional

Frame number in the video path from which the blob was segmented, by default None.

in_frame_indexint, optional

Index of the blob in the frame where it was segmented, by default None. This index comes from OpenCV and is defined by the hierarchy of the countours found in the frame.

pixels_pathstr, optional

Path to the file were the pixels of the blob are stored, by default None.

video_heightint, optional

Height of the video considering the resolution reduction factor, by default None.

video_width[type], optional
Width of the video considering the resolution reduction factor,

by default None.

video_pathstr, optional

Path to the video file from which the blob was segmented, by default None.

pixels_are_from_eroded_blobbool, optional

Flag to indicate if the pixels of the blobs come from from an eroded blob, by default False.

resolution_reductionfloat, optional

Resolution reductio factor as defined by the user, by default 1.0.

Attributes
accumulation_step

Integer indicating the accumulation step in which the blob

assigned_centroids

Centroids assigned to the blob during the tracking process.

assigned_identities

Identities assigned to the blob during the tracking process

blob_index

Blob index at the segmentation step (comes from the find contours

bounding_box_full_resolution

Bounding box cordinates without considering the resolution reduction factor, i.e.

bounding_box_image

Image cropped from the original video that contains the blob.

contour_full_resolution

Blob contour coordinates in pixels without considering the resolution reduction factor, i.e.

eroded_pixels

Pixels of the blob after erosion.

final_centroids

List of the animal/s centroid/s in the blob, considering the potential centroids that might have been aded by the user during the validation.

final_centroids_full_resolution

Returns the centroids considering the full resolution of the frame.

final_identities

Identities of the blob after the tracking process and after potential modifications by the users during the validation procedure.

fragment_identifier

Integer indicating the index of the fragment that contains the blob

identities_corrected_closing_gaps

Identity of the blob after crossings interpolation

identity

Identity of the blob assigned during the identification process

identity_corrected_solving_jumps

Identity of the blob after correcting impossible velocity jumps

is_a_crossing

Flag indicating whether the blob represents two or more animals together.

is_a_generated_blob

Flag indicating whether the blob was created by the user.

is_an_individual

Flag indicating whether the blob represent a single animal.

is_an_individual_in_a_fragment

Boolean indicating if the blob is in an individual fragment

is_in_a_fragment

Boolean indicating if the blob is in a fragment, i.e.

pixels

List of pixels that define the blob as determined from the segmentation parameters defined by the user.

pixels_set

A set() version of blob.pixels, used in blob.overlaps_with()

resolution_reduction

Resolution reduction factor defined by the user

summary

[Validation] Returns a summary string for some blob attributes.

used_for_training

Flag indicating if the blob has been used to train the

user_generated_centroids

List of centroids generated by the user during the validation processes.

user_generated_identities

List of identities of the blob some of which might have been give

was_a_crossing

Flag indicating whether the blob was created after splitting a

Methods

add_centroid(video, centroid, identity[, ...])

[Validation] Adds a centroid with a given identity to the blob.

check_for_crossing_in_next_or_previous([...])

Flag indicating if the blob has a crossing in its past or future overlapping history

check_for_multiple_next_or_previous([direction])

Flag indicating if the blob has multiple blobs in its past or future overlapping history

delete_centroid(video, identity, centroid, ...)

[Validation] Deletes a centroid of the blob.

distance_from_countour_to(point)

Returns the distance between point and the closest point in the contour of the blob.

draw(frame[, colors_lst, selected_id, ...])

[Validation] Draw the blob in a given frame of the video.

get_image_for_identification(...)

Gets the image used to train and evaluate the crossing detector CNN and the identification CNN.

in_a_global_fragment_core(blobs_in_frame)

Boolean indicating if a blob is in the core of a globalfragment.

is_a_sure_crossing()

Flag indicating that the blob is a sure crossing according to some heuristics and it can be used to train the crossing detector CNN.

is_a_sure_individual()

Flag indicating that the blob is a sure individual according to some heuristics and it can be used to train the crossing detector CNN.

now_points_to(other)

Given two consecutive blob objects updates their respective overlapping histories

overlaps_with(other)

Computes whether the pixels in self intersect with the pixels in other

propagate_identity(old_identity, ...)

[Validation] Propagates the new identity to next and previous blobs.

removable_identity(identity_to_remove, ...)

[Validation] Checks if the identity can be removed.

save_image_for_identification(...)

Saves in disk the image that will be used to train and evaluate the crossing detector CNN and the identification CNN.

squared_distance_to(other)

Returns the squared distance from the centroid of self to the centroid of other

update_centroid(video, old_centroid, ...)

[Validation] Updates the centroid of the blob.

update_identity(old_identity, new_identity, ...)

[Validation] Updates the identity of the blob.

property accumulation_step

Integer indicating the accumulation step in which the blob was assign by the cascade of training and identification protocols

add_centroid(video, centroid, identity, apply_resolution_reduction=True)[source]

[Validation] Adds a centroid with a given identity to the blob.

This method is used in the validation GUI. It is useful to add centroids for crossing blobs that are missing some centroids, or to individual blobs that should have been classified as crossings and are also missing some centroids.

Parameters
videoidtrackerai.video.Video

Instance of Video object

centroidtuple

Centroid to be added to the blob

identityint

Identity of the centroid to be added

apply_resolution_reductionbool, optional

Whether to consider the resolution reduction factor when adding the centroid, by default True. Note that the video is showed as full resolution in the validation GUI, but all centroids of the blobs consider the resolution reduction factor.

Raises
Exception

If the centroid is not a tuple of length 2

Exception

If the identity is not an integer between 1 and number of animals

Exception

If there is already another centroid with the same identity

property assigned_centroids

Centroids assigned to the blob during the tracking process.

It considers the default centroid of the blob at segmentation time or new centroids added to the blob during the interpolation of the crossings.

Returns
list

List of pairs (x, y) indicating the position of each individual in the blob.

property assigned_identities

Identities assigned to the blob during the tracking process

property blob_index

Blob index at the segmentation step (comes from the find contours function of OpenCV)

property bounding_box_full_resolution
Bounding box cordinates without considering the resolution reduction

factor, i.e. in the full resolution of the video.

Returns
numpy array, or None

Bounding box coordinates of the blob in full resolution of the video, [(x, y), (x + bounding_box_width, y + bounding_box_height)].

property bounding_box_image

Image cropped from the original video that contains the blob.

This image is used later to extract the image_for_identification that will be used to train and evaluate the crossing detector CNN and the identification CNN. This image can either be stored in the object (in RAM), or be stored in in a file (in DISK), or not be stored at all, in which case it is recomupted from the original video.

Returns
numpy array (uint8)

Image cropped from the video containing the pixels that represent the blob.

check_for_crossing_in_next_or_previous(direction=None)[source]

Flag indicating if the blob has a crossing in its past or future overlapping history

This method is used to check whether the blob is an individual.

Parameters
directionstr

“previous” or “next”. If “previous” the past overlapping history will be checked in order to find out if the blob ends up in a crossing. Symmetrically, if “next” the future overlapping history of the blob will be checked.

Returns
bool

If True the blob has a crossing in its “past” or “future” history, depending on the parameter direction.

check_for_multiple_next_or_previous(direction=None)[source]

Flag indicating if the blob has multiple blobs in its past or future overlapping history

This method is used to check whether the blob is a crossing.

Parameters
directionstr

“previous” or “next”. If “previous” the past overlapping history will be checked in order to find out if the blob will split in the past. Symmetrically, if “next” the future overlapping history of the blob will be checked.

Returns
bool

If True the blob splits into two or multiple overlapping blobs in its “past” or “future” history, depending on the parameter “direction”.

property contour_full_resolution

Blob contour coordinates in pixels without considering the resolution reduction factor, i.e. in the full resolution of the video.

Returns
numpy array.

Coordinates in pixels of the points in the blob contour

delete_centroid(video, identity, centroid, blobs_in_frame, apply_resolution_reduction=True)[source]

[Validation] Deletes a centroid of the blob.

Parameters
videoidtrackerai.video.Video

Instance of Video object

identityint

Identity of the centroid to be deleted

centroidtuple

Centroid to be deleted from the blob

blobs_in_frameList

List of the blobs in the frame

apply_resolution_reductionbool, optional

Whether to consider the resolution reduction factor when adding the centroid, by default True. Note that the video is showed as full resolution in the validation GUI, but all centroids of the blobs consider the resolution reduction factor.

Raises
Exception

If the centroid is not a tuple of length 2

Exception

If the identity is unique in the frame

Exception

If it is the last centroid of the blob

distance_from_countour_to(point)[source]

Returns the distance between point and the closest point in the contour of the blob.

Parameters
pointtuple

(x,y)

Returns
float

Smallest distance between point and the contour of the blob.

draw(frame, colors_lst=None, selected_id=None, is_selected=False)[source]

[Validation] Draw the blob in a given frame of the video.

Parameters
framenumpy.array

Image where the blob should be draw.

colors_lst[type], optional

List of colors used to draw the blobs, by default None

selected_id[type], optional

Identity of the selected blob., by default None

is_selectedbool, optional

Flag indicated if the blob has been selected by the user, by default False

property eroded_pixels

Pixels of the blob after erosion.

The erosion is performed during the crossings interpolation process.

The pixels can either be stored in the object (in RAM), or be stored in a file (in DISK), or not be stored at all, in which case they are computed

Returns
List

List of integers indicating the linarized indices of the pixels that represent the blob after the erosion process.

property final_centroids

List of the animal/s centroid/s in the blob, considering the potential centroids that might have been aded by the user during the validation.

By default the centroid will be the center of mass of the blob of pixels defined by the blob. It can be different if the user modified the default centroid during validation or generated more centroids.

Returns
list

List of tuples (x, y) indicating the centroids of the blob.

property final_centroids_full_resolution

Returns the centroids considering the full resolution of the frame.

Note that after validation a blob can have more than one centroid. For example, this can happen in a crossing blob with missing centroids.

Returns
List of tuples (x, y), where x, and y are coordinates in the full
resolution of the video frame.
property final_identities

Identities of the blob after the tracking process and after potential modifications by the users during the validation procedure.

property fragment_identifier

Integer indicating the index of the fragment that contains the blob

See also

Fragment
get_image_for_identification(identification_image_size, height, width)[source]

Gets the image used to train and evaluate the crossing detector CNN and the identification CNN.

Parameters
identification_image_sizetuple

Dimensions of the identification image (height, widht, channels). Channels is always 1 as images in color are still not considered.

heightint

Video height considering resolution reduction factor.

widthint

Video width considering resolution reduction factor.

Returns
ndarray

Square image with black background used to train the crossings detector CNN and the identifiactio CNN.

property identities_corrected_closing_gaps

Identity of the blob after crossings interpolation

property identity

Identity of the blob assigned during the identification process

property identity_corrected_solving_jumps

Identity of the blob after correcting impossible velocity jumps

in_a_global_fragment_core(blobs_in_frame)[source]

Boolean indicating if a blob is in the core of a globalfragment.

A blob in a frame is in the core of a global fragment if in that frame there are as many blobs as number of animals to track

Parameters
blobs_in_framelist

List of Blob objects representing the animals segmented in the frame self.frame_number

Returns
bool

True if the blob is in the core of a global fragment

property is_a_crossing

Flag indicating whether the blob represents two or more animals together.

This attribute is the negative of is_an_individual and is set at the same time as is an individual

property is_a_generated_blob

Flag indicating whether the blob was created by the user.

Blobs created by the users during the validation process do not have a contour as only the centroid is created

is_a_sure_crossing()[source]

Flag indicating that the blob is a sure crossing according to some heuristics and it can be used to train the crossing detector CNN.

Returns
bool
is_a_sure_individual()[source]

Flag indicating that the blob is a sure individual according to some heuristics and it can be used to train the crossing detector CNN.

Returns
bool
property is_an_individual

Flag indicating whether the blob represent a single animal.

The private attribute _is_an_individual is assigned during the crossings detection process.

property is_an_individual_in_a_fragment

Boolean indicating if the blob is in an individual fragment

property is_in_a_fragment

Boolean indicating if the blob is in a fragment, i.e. is not an isolated blob or is in the extreme of a fragment

now_points_to(other)[source]

Given two consecutive blob objects updates their respective overlapping histories

Parameters
other<Blob object>

An instance of the class Blob

overlaps_with(other)[source]

Computes whether the pixels in self intersect with the pixels in other

Parameters
other<Blob object>

An instance of the class Blob

Returns
bool

True if the lists of pixels of both blobs have non-empty intersection

property pixels

List of pixels that define the blob as determined from the segmentation parameters defined by the user.

Note that pixels are represented as linearized coordinates of the frame (considering the resolution reduction factor), i.e. each pixel is a single integer.

The pixels can either be stored in the object (in RAM), or be stored in a file (in DISK), or not be stored at all, in which case they are recomputed from the contour and the original frame of the video.

Returns
List

List of integers indicating the linarized indices of the pixels that represent the blob.

property pixels_set

A set() version of blob.pixels, used in blob.overlaps_with()

Returns
Set

Set of integers indicating the linarized indices of the pixels that represent the blob.

propagate_identity(old_identity, new_identity, centroid)[source]

[Validation] Propagates the new identity to next and previous blobs.

This method called in the validation GUI when the used updates the identity of a given blob.

Parameters
old_identityint

Previous identity of the blob

new_identityint

New ientity of the blob

centroidtuple

[description]

removable_identity(identity_to_remove, blobs_in_frame)[source]

[Validation] Checks if the identity can be removed.

Parameters
identity_to_removeint

Identity to be removed

blobs_in_framelist

List of Blob objects in the frame where the identity is going to be removed

Returns
bool

True if the identity can be removed.

property resolution_reduction

Resolution reduction factor defined by the user

save_image_for_identification(identification_image_size, height, width, file_path)[source]

Saves in disk the image that will be used to train and evaluate the crossing detector CNN and the identification CNN.

This also updates the identification_image_index and the episode attributes. This helps to load the image from the correct file_path.

Parameters
identification_image_sizetuple

Tuple of integers (height, width, channels).

heightint

Video height considering the resolution reduction factor.

widthint

Video width considering the resolution reduction factor.

file_pathstr

Path to the hdf5 file where the images will be stored.

squared_distance_to(other)[source]

Returns the squared distance from the centroid of self to the centroid of other

Parameters
other<Blob object> or tuple

An instance of the class Blob or a tuple (x,y)

Returns
float

Squared distance between centroids

property summary

[Validation] Returns a summary string for some blob attributes.

Returns
str

Summary description of the blob

update_centroid(video, old_centroid, new_centroid, identity)[source]

[Validation] Updates the centroid of the blob.

Parameters
videoidtrackerai.video.Video

Instance of Video object

old_centroidtuple

Centroid to be updated

new_centroidtuple

Coordinates of the new centroid

identityint

Identity of the centroid to be updated

update_identity(old_identity, new_identity, centroid)[source]

[Validation] Updates the identity of the blob.

This method is used during the validation GUI. It populates the private attributes _user_generated_identities and _user_generated_centroids.

Parameters
new_identityint

new value for the identity of the blob

old_identityint

old value of the identity of the blob. It must be specified when the blob has multiple identities already assigned.

centroidtuple

centroid which identity must be updated.

property used_for_training

Flag indicating if the blob has been used to train the identification CNN

property user_generated_centroids

List of centroids generated by the user during the validation processes.

Returns
list or None

If the user has not generated any centroid for this blob, it returns None.

property user_generated_identities

List of identities of the blob some of which might have been give by a user during the validation process

property was_a_crossing

Flag indicating whether the blob was created after splitting a crossing blob during the crossings interpolation process

List of blobs

Fragment

List of fragments

Global fragment

List of global fragments