{ "cells": [ { "cell_type": "markdown", "id": "c3109b7e-0092-47a3-aca2-288e745731ba", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "## Lecture 5 - Image Features and Feature Detection" ] }, { "cell_type": "markdown", "id": "c0357b50-2962-423e-85f1-80217ec3d134", "metadata": {}, "source": [ "#### Announcements\n", "\n", "* \n", "\n" ] }, { "cell_type": "markdown", "id": "38efd636-c09e-4d1f-a828-33e0fcc75686", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "#### Goals\n", "* Know what is meant by **local image features**.\n", "* Understand the motivation for **detecting**, **describing**, and **matching** local image **features**.\n", "* Understand why **uniqueness** and **invariance** are desirable properties of features and their descriptors.\n", "* Know how to detect corner features using the **Harris corner detector**.\n", "* Analyze the invariances provided (and not provided) by the Harris corner detector\n" ] }, { "cell_type": "code", "execution_count": 2, "id": "4a295613-e6e3-4f35-b603-27b158385acc", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The autoreload extension is already loaded. To reload it, use:\n", " %reload_ext autoreload\n" ] } ], "source": [ "# boilerplate setup\n", "%load_ext autoreload\n", "%autoreload 2\n", "\n", "%matplotlib inline\n", "\n", "import os\n", "import sys\n", "\n", "src_path = os.path.abspath(\"../src\")\n", "if (src_path not in sys.path):\n", " sys.path.insert(0, src_path)\n", "\n", "# Library imports\n", "import numpy as np\n", "import imageio.v3 as imageio\n", "import matplotlib.pyplot as plt\n", "import skimage as skim\n", "import cv2\n", "\n", "# codebase imports\n", "import util\n", "import filtering" ] }, { "cell_type": "markdown", "id": "c6ed4ad7-132d-4906-8a67-fae476b90b39", "metadata": {}, "source": [ "#### Plan\n", "* Brief slides - (recap) setup for feature matching\n", "* Which points should we try to match, and how should we describe them?\n", "* Uniqueness and invariance\n", "* Arrive at: corner features, local uniqueness; address invariance with the descriptor\n", "* Corner finding: whiteboard discussion; error function visualization demo; whiteboard --> eigenvalues\n", "* Decode the structure tensor: how do we compute it? Convolution!\n", "* Implement features.harris_score\n", "* Play with that on some images\n", "* Implement non-max suppression via maximum filter" ] }, { "cell_type": "markdown", "id": "b72febe7-2780-4f03-955e-e6253d9b69c7", "metadata": {}, "source": [ "What makes good features?\n", "\n", "* Uniqueness: features **shouldn't** match if they're from different points in the scene. ", " ", "\n", "* Invariance: features **should** match if they do come from the same point in the scene. " ] }, { "cell_type": "markdown", "id": "2dedf759-300a-45e8-949d-96eaff32c6ca", "metadata": {}, "source": [ "##### Homework Problem 1\n", "\n", "In terms of uniqueness and invariance, discuss why arbitrary single pixels, described using their RGB values, would not make good features for image matching." ] }, { "cell_type": "markdown", "id": "eec79b9e-4cf1-42d3-b353-fb23e7e2bb5b", "metadata": {}, "source": [ "What might be a *little* bit better?" ] }, { "cell_type": "code", "execution_count": null, "id": "ae5e4498-9dde-4ea0-aea4-8571dce73a28", "metadata": {}, "outputs": [], "source": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "848fc61b-03f3-41d4-8cf7-a4e54b2816b6", "metadata": {}, "source": [ "Idea: patches. Is it perfect?\n", "\n", "\n", "\n", "No! But it's hard to imageine something that is. Let's settle for this and talk about how to find patches that will be **locally** unique." ] }, { "cell_type": "markdown", "id": "79a1e607-89f6-4e7d-abd6-80eea8f81144", "metadata": {}, "source": [ "#### Whiteboard: Harris Corner Detection\n", "\n", "\n", "\n", "\n", "Here are the rough notes I used for the whiteboard presentation.\n", "\n", "Hey how much something changes in a direction... this sounds like a gradient magnitude!\n", "\n", "* Right angle example - this seems like it's going to work!\n", "* Diagonal line example - darn. This isn't going to work.\n", "\n", "Throw out the gradient idea and brute force it: compare a patch to all its neighbors. What do I mean by all?\n", "\n", "* Shift to every position (within a window) and compute a distance.\n", " * $E(u, v) = \\sum_{x, y \\in W} \\left[ I(x+u, y+v) - I(x, y) \\right]^2$\n", " * For each pixel (x, y) in the window, compare the pixel at (u,v) offset from (x, y) to the un-offset (x, y)\n", "* **Q**: Suppose we did that, and got a window's worth of distance measurements. How would you turn that into a single cornerness score?\n", " * **A**: The **minimum** distance would work - a large minimum distance means the most similar thing is not very similar." ] }, { "cell_type": "markdown", "id": "d5f31922-21d4-4102-b2be-1ccb6959b914", "metadata": {}, "source": [ "Okay but this is really expensive. Let's linearize!\n", "\n", "Taylor expansion reminder:\n", "$$\n", "f(x) = f(a) + f'(a)(x-a) + f''(a)(x-a)^2 + \\ldots\n", "$$\n", "\n", "For a first-order (linear) Taylor approximation,\n", "$$\n", "f(x) \\approx f(a) + f'(a)(x-a)\n", "$$\n", "\n", "\n", "In our bivariate context:\n", "$$\n", "I(x+u, y+v) \\approx I(x, y) + \n", "\\frac{\\partial{I}}{\\partial x} u + \n", "\\frac{\\partial{I}}{\\partial y} v\n", "$$\n", "Look, the derivatives came back!\n", "\n", "Plug this into our \"error\" function:\n", "$$\n", "\\begin{align}\n", "E(u, v) &= \\sum_{x, y \\in W} \\left[ I(x+u, y+v) - I(x, y) \\right]^2\\\\\n", "&\\approx \\sum_{x, y \\in W} \\left[\n", "\tI(x, y) + \\frac{\\partial{I}}{\\partial x} u\n", " \t\t\t\t\t\t\t+ \\frac{\\partial{I}}{\\partial y} v - I(x, y))\\right]^2\\\\\n", "&\\approx \\sum_{x, y \\in W} \\left[\n", "\t\t\t\t\t\t\t\t \\frac{\\partial{I}}{\\partial x} u\n", " + \\frac{\\partial{I}}{\\partial y} v\\right]^2\n", "\\end{align}\n", "$$" ] }, { "cell_type": "markdown", "id": "b43a75b5-b038-44b7-9534-9d73faf1564f", "metadata": {}, "source": [ "**Pause** At this point, it's fair to ask what the heck we're looking at.\n" ] }, { "cell_type": "code", "execution_count": 3, "id": "0baad59d-e0f4-450b-a108-a545fd5a218b", "metadata": {}, "outputs": [], "source": [ "from features import visualize_harris\n", "\n", "%matplotlib widget\n", "\n", "h = imageio.imread(\"../data/harris_crop.jpg\")\n", "h = skim.color.rgb2gray(h.astype(np.float32) / 255)\n", "\n", "hfull = imageio.imread(\"../data/harris.jpg\")\n", "hfull = filtering.down_2x(skim.color.rgb2gray(hfull.astype(np.float32) / 255))" ] }, { "cell_type": "markdown", "id": "a1aa6531-b21f-492b-ab24-d4b90c0e83fc", "metadata": {}, "source": [ "Here's a visualization of the error function $E(u, v)$ as we shift a patch around in a real image.\n", "* Top left: the original image with the patch highlighted\n", "* Top right: the actual error computed by shifting that patch to each position in a (by defualt 13x13) windo.\n", "* Bottom left: the approximate error function computed by substituting the linearized approximation based on the derivatives.\n", "* Bottom right: a contour plot of the same data as the bottom left.\n", "\n", "Notice tha the top right and bottom left are 3D plots, so you can drag them to change views." ] }, { "cell_type": "code", "execution_count": 4, "id": "8a6306f6-5bb1-4320-9470-ecac155d0c6a", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "e16082d4fb16427aa9ab11d72d7178f6", "version_major": 2, "version_minor": 0 }, "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyAAAAMgCAYAAADbcAZoAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjgsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvwVt1zgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAEIVJREFUeJzt2bENgAAQAzFg/52fLa6I7AVSn/Le3T0AAACBrxgBAAAQIAAAQMoDAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAIAAAQAA9nhAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAQIAAAAB7PCAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAIAAAQAA9nhAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAAAAAQIAAOzxgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAIAAAQAA9nhAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAAAAAQIAAOzxgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAAE/lB0rXCjwUMYpZAAAAAElFTkSuQmCC", "text/html": [ "\n", "