{ "cells": [ { "cell_type": "markdown", "id": "c3109b7e-0092-47a3-aca2-288e745731ba", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "## Lecture 10" ] }, { "cell_type": "markdown", "id": "c0357b50-2962-423e-85f1-80217ec3d134", "metadata": {}, "source": [ "#### Announcements\n", "\n", "* Project 1 is in! Artifacts due tonight.\n", " * I'm aiming to grade them by EOD Friday\n", " * Artifact showcase tomorrow\n", "* Problems for today: P10\n", "* Today's tea: Yunnan Mao Feng, a green tea from China" ] }, { "cell_type": "markdown", "id": "38efd636-c09e-4d1f-a828-33e0fcc75686", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "#### Goals\n", "* Know how to resample images using forward and inverse warping (and why the latter is usually preferred)\n", "* Know how to implement bilinear interpolation\n", "* Be prepared to implement the full panorama stitching pipeline, including:\n", " * Warping images into a common coordinate system\n", " * Blending them with a simple linear falloff near the edges" ] }, { "cell_type": "code", "execution_count": null, "id": "4a295613-e6e3-4f35-b603-27b158385acc", "metadata": {}, "outputs": [], "source": [ "# boilerplate setup\n", "%load_ext autoreload\n", "%autoreload 2\n", "\n", "%matplotlib inline\n", "\n", "import os\n", "import sys\n", "\n", "src_path = os.path.abspath(\"../src\")\n", "if (src_path not in sys.path):\n", " sys.path.insert(0, src_path)\n", "\n", "# Library imports\n", "import numpy as np\n", "import imageio.v3 as imageio\n", "import matplotlib.pyplot as plt\n", "import skimage as skim\n", "import cv2\n", "\n", "# codebase imports\n", "import util\n", "import filtering\n", "import features\n", "import geometry" ] }, { "cell_type": "markdown", "id": "a0d17035-d7ad-4271-affa-5d4a4fc61fd7", "metadata": {}, "source": [ "### Context: Panorama Stitching Overview\n", "\n", "* [x] Detect features - Harris corners\n", "* [x] Describe features - MOPS descriptor\n", "* [x] Match features - SSD + ratio test\n", "* Estimate motion model from correspondences\n", " * [x] Translation\n", " * [x] Affine\n", " * [x] Projective\n", " * [x] Robustness to outliers - RANSAC\n", "* Warp image(s) into common coordinate system and blend\n", " * [ ] Inverse warping\n", " * [ ] Blending\n", " * [ ] 360 panoramas" ] }, { "cell_type": "markdown", "id": "71425a03-00aa-43ed-a1ab-3260ae88ba1e", "metadata": {}, "source": [ "### Warping: Forward and Inverse\n", "\n", "See whiteboard notes.\n", "\n", "Forward warping:\n", "```\n", "for x, y in src:\n", " x', y' = T(x, y)\n", " dst[x', y'] = src[x, y]\n", "```\n", "\n", "Inverse warping:\n", "```\n", "Tinv = np.linalg.inv(T)\n", "for (x', y') in dst:\n", " x, y = Tinv(x', y')\n", " dst[x', y'] = src[x, y]\n", "```" ] }, { "attachments": {}, "cell_type": "markdown", "id": "d58f7218-bcfd-453a-aa4f-daa1d310dd2d", "metadata": {}, "source": [ "#### Bilinear Interpolation\n", "\n", "A reasonable way to sample a floating-point pixel value from the four surrounding known pixel values.\n", "\n", "\n", "##### First: linear interpolation.\n", "\n", "\n", "\n", "##### Next: bilinear interpolation\n", "\n", "Three equivalent interpretations:\n", "* Weight using a tent filter centered at $(x, y)$\n", "* Weight each corner by the area of the rectangle opposite it.\n", "* Linearly interpolate along 2 parallel sides, then linearly interpolate again between thsoe points.\n", "\n", "\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "a51a6e6c-2f15-499c-a3bb-a0fc073018cd", "metadata": {}, "source": [ "#### Other interpolation schemes\n", "\n", "![](https://upload.wikimedia.org/wikipedia/commons/thumb/9/90/Comparison_of_1D_and_2D_interpolation.svg/640px-Comparison_of_1D_and_2D_interpolation.svg.png)" ] }, { "cell_type": "markdown", "id": "f58df44e-6bca-4ab2-984f-ff20d2828d79", "metadata": {}, "source": [ "##### Homework Problem 1\n", "\n", "1. Complete the following function with Python-esque pseudocode (or working code in the lecture codebase), which performs inverse warping with nearest-neighbor sampling in the source image.\n", "\n", "```python\n", "def warp(img, tx, dsize=None)\n", " \"\"\" Warp img using tx, a matrix representing a geometric transformation.\n", " Pre: tx is 3x3 (or some upper-left slice of a 3x3 matrix). img is grayscale.\n", " Returns: an output image of shape dsize with the warped img\"\"\"\n", " H, W = img.shape[:2]\n", "\n", " # turn a 2x2 or 2x3 tx into a full 3x3 matrix\n", " txH, txW = tx.shape\n", " M = np.eye(3)\n", " M[:txH,:txW] = tx\n", "\n", " # set the output size to the input size if not specified\n", " if dsize is None:\n", " DH, DW = (H, W)\n", " else:\n", " DH, DW = dsize[::-1]\n", " out = np.zeros((DH, DW))\n", "\n", " # your code here\n", " \n", " return out\n", "\n", "```\n" ] }, { "cell_type": "markdown", "id": "f7d588f2-024b-451e-b141-66d1407c7c94", "metadata": {}, "source": [ "**Start with nearest-neighbor interpolation**, and test with the progressively more interesting test cases below. Once you have that working, you can try implementing bilinear interpolation if time allows." ] }, { "cell_type": "markdown", "id": "ddb601ed-c5ab-46ef-97f2-797bdded1657", "metadata": {}, "source": [ "##### Test case:" ] }, { "cell_type": "code", "execution_count": null, "id": "7370db80-e40a-41df-b278-a4f8bd125a0e", "metadata": {}, "outputs": [], "source": [ "y1 = imageio.imread(\"../data/yos1.jpg\").astype(np.float32) / 255\n", "y1 = skim.color.rgb2gray(y1)" ] }, { "cell_type": "code", "execution_count": null, "id": "91945844-fa92-4764-8e4e-1152c862c4f5", "metadata": {}, "outputs": [], "source": [ "h, w = y1.shape\n", "\n", "tx = np.eye(3)\n", "tx[:2,2] = [10, 20] # translation only\n", "#tx[0,1] = 0.1 # add a shear\n", "#tx[2,0] = 0.0005 # add a perspective warp\n", "util.imshow_gray(geometry.warp(y1, tx))" ] }, { "cell_type": "markdown", "id": "657ec2bb-c511-4c29-b329-ea44d290be92", "metadata": {}, "source": [ "### Context: Panorama Stitching Overview\n", "\n", "* [x] Detect features - Harris corners\n", "* [x] Describe features - MOPS descriptor\n", "* [x] Match features - SSD + ratio test\n", "* Estimate motion model from correspondences\n", " * [x] Translation\n", " * [x] Affine\n", " * [x] Projective\n", " * [x] Robustness to outliers - RANSAC\n", "* Warp image(s) into common coordinate system and blend\n", " * [x] Inverse warping\n", " * [ ] Blending\n", " * [ ] 360 panoramas" ] }, { "cell_type": "markdown", "id": "5409233a-857d-452c-bec6-2d5d962f7c7c", "metadata": {}, "source": [ "### Panorama Stitching Pipeline - Overview\n", "with some blending details thrown in\n", "\n", "(see whiteboard notes)\n", "\n", "\n", "Somewhere in there:\n", "\n", "##### Homework Problem 2\n", "\n", "Suppose you are stitching a panorama with three images $I_1, I_2, I_3$ and you've fitted transformations $T_{12}$ and $T_{23}$ that map coordinates from image 1 to 2 and from 2 to 3, respectively. Give the transformation that maps points from image 3's coordinates to image 1's coordinates.\n", "\n", "##### Homework Problem 3\n", "\n", "Give a strategy (describe, or write pseudocode) for finding the corners of the bounding box of a given image `img` after it has been warped using a homography `T`." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.12" } }, "nbformat": 4, "nbformat_minor": 5 }