Overview

This assignment is designed to excercise your understanding of David Bolme's work on Correlation Filters. The goal is less about a general implementation than it is about a working demonstration. So, for example, in this assignment you will be implementing ASEF rather than MOSSE. Therefore, while you may want to experiment with tracking (see below), the primary reason for doing this assignment is to build code that will construct and display the correlation filters that develop while tracking a known object given ground-truth knowledge of the object position.

Holding onto an Object in Video

Given any arbitrary video there is a startup issue and also a ground-truth issue that must be addressed when implementing a correlation tracker. Namely, where is the object of interest in frame 1 and then in subsequent frames. In the YouTube videos of David Bolme's tracker startup was based upon a rectangle drawn by a person and then updates in the first few frames were based upon a best match using a still reltaively untrained correlation filter. That means there were details involved that we've not fully discussed to get the tracker off to a solid start.

So, this assignment is not primarily about building at tracker. Instead, you are to use a video sequence where tracking is easily accomplished using a simple algorithm. Namely, the frigate bird video from assignment 1. What you are to do in this assignment is implement the ASEF algorithm and show vidually the developing correlation filter as the frigate bird video progresses.

Specifics - What to Show

When your program is run four disctinct windows will be displayed and these will update for each successive frame.

  • Video: This window will show a 512 by 512 edge magnitude image drawn from the larger 1280 by 720 video. Your 512 x 512 image will not move over the course of the video. However, it will simplify everything else in this assignment to avoid using the full frame size of the video.
  • Goal: This window will show the desired correlation surface that will result from convolving the video frame with an exact filter. Keep in mind this is a grayscale image in the spatial domain with a Gaussian centered on the center of the frigate bird.
  • Exact: This window will show the exact filter that, when convolved with the video frame, generates the goal correlation surface.
  • ASEF: This window will show an evolving running average of the exact filters. Note on frame 1 it will match the display in the window Exact. After that it should start to become more readily understood as the sillouette of the frigate bird. The exact approach to weighting exact filters is up to you to choose. In other words, you are being asked to experiement with some different options and draw conclusions about strengths and weaknesses of different approaches.

A Few Details

The hardest part of this assignment will be implementing the quotient operation in the frequency domain. When in doubt seek help from the instructor and others in the class to make sure that your code is a faithful rendition of the algebra presented in lecture.

Along the way, you will probably want to display and inspect the FFT of the current frame, the target correlation surface, and the result of dividing the one by the other.

Here is one incredibly minor by helpful detail, the moveWindow command in openCV is useful for laying out your four windows in such a way that each is readily seen as the video plays.

One More Thing

Find a single additional video where you can also demonstrate the development of ASEF filters. By default, find some other object easily tracked using a simple feature so that you can generate the goal image for each frame. That is sufficient for this assignment. However, if you choose, you may want to try a video where after some number of frames (16, 32, etc.) your system starts using the correlation filter from frame k to first identifiy the most likely position of the object in fram k+1 and build the next goal image accordinly. In other words, you may wish to build a tracker.

Teams and Grading

You are expected to work in pairs - teams - for this project. Unless worked out between yourselves and approved by me in advance, I strongly suggest you keep your partner from Assignment 2. The grading will be done in the same fashion as for Assignment 2, in face-to-face interviews with the instructor.