I am working with a small sample of the data saved with numpy.

sequence = np.load('data/MABe/example_sequence.npy')

For each frame (30fps video) the positions of keypoints on the mouse (nose, ears, neck, sides and tail) have been annotated. There is also a label for the behaviour happening at each time instant.

sequence[0] # Two mice, list of X and list of Y for each for one frame

array([[[298.99367828, 266.99367828, 248.99367828, 243.99367828,
         181.99367828, 144.99367828, 125.99367828],
        [140.75094149, 194.75094149, 138.75094149, 169.75094149,
         240.75094149, 167.75094149, 241.75094149]],

       [[191.68942719, 226.68942719, 167.68942719, 200.68942719,
         235.68942719, 161.68942719, 203.68942719],
        [283.87298664, 323.87298664, 329.87298664, 342.87298664,
         429.87298664, 433.87298664, 487.87298664]]])

imw, imh = 1024, 570 # The image dimensions in which the mice move about

With some basic cairo drawing we can repeatedly draw the mice and animate them to watch the motion:

def drawmouse(ctx, keypoints, w, h):
    x, y = keypoints[0]*w/imw, keypoints[1]*h/imh
    
    # Head 
    ctx.move_to(x[3], y[3])
    ctx.line_to(x[1], y[1])
    ctx.line_to(x[0], y[0])
    ctx.line_to(x[2], y[2])
    ctx.line_to(x[3], y[3])
    ctx.close_path()
    ctx.set_source_rgb(1, 0.5, 0)
    ctx.fill_preserve()
    ctx.set_source_rgb(1, 1, 0)
    ctx.set_line_width(3)
    ctx.stroke()
    
    # Body 
    ctx.line_to(x[4], y[4])
    ctx.line_to(x[6], y[6])
    ctx.line_to(x[6], y[5])
    ctx.line_to(x[3], y[3])
    ctx.close_path()
    ctx.set_source_rgb(1, 0.5, 0)
    ctx.fill_preserve()
    ctx.set_source_rgb(1, 1, 0)
    ctx.set_line_width(3)
    ctx.stroke()

import ipywidgets as widgets

# Set up surface
surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, 512, 300) # Create the surface
ctx = cairo.Context(surface)

widget = widgets.Image(
    value=surface.write_to_png(),
    format='png',
    width=512,
    height=512,
)
display(widget)

for ks in sequence[:200]:
    drawmouse(ctx, ks[0], 512, 300)
    drawmouse(ctx, ks[1], 512, 300)
    widget.value = surface.write_to_png()
    
display_surface(surface)

We want a static frame, so we modify the above with some colour to show both mice and 'trails' of where they have been and are going.

surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, 512, 512) # Create the surface
ctx = cairo.Context(surface)
draw_instant(ctx, sequence, 512, 512, 200, n=80, step=5)
surface.write_to_png('outputs/mabe_movement_viz.png') # Saving
display_surface(surface)

This needs to be quite fast - let's see how long it takes to make 100 images:

%%time
for u in range(100):
    surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, 256, 256) # Create the surface
    ctx = cairo.Context(surface)
    draw_instant(ctx, sequence, 256, 256, u, n=80, step=5)
display_surface(surface)

I fed a few thousand of these images to a resnet and trained it for a bit, but couldn't get more than ~66% accuracy so I gave up. The competition took the bulk of my coding time today so this is a short notebook. See you tomorrow.

Meeting Mice

`draw_instant`[source]

Meeting Mice

draw_instant[source]

`draw_instant`[source]