CS 1053 Project 4: Neural Radiance Fields

In this assignment, you’ll implement a (slightly simplified) version of the technique described in the influential 2020 paper NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis by Mildenhall et al. Given a collection of images taken from multiple perspectives, along with pose information for those cameras, NeRF is an elegant technique that encodes a 3D volumetric representation of the scene in a neural network.

Dates

Assigned: Monday, January 26th, 2026

Code Due: Thursday, January 29th, 2026 at 10:00pm (via Github)

Overview

Purpose

This project gives you practice applying both 3D geometry concepts (in determining the color of a ray through the volume) as well as machine learning concepts, helping you to demonstrate aspects of the following course outcomes:

• Basic understanding single-view projective geometry and image formation under the pinhole camera model
• Basic understanding of the applications of modern machine learning techniques to high-level computer vision problems such as image recognition and image generation.

Setup

Skeleton Code

Accept the Project 4 assignment on Github Classroom and clone your repository. This repository contains the assignment’s single Jupyter notebook, which includes detailed instructions as well as skeleton code and TODOs for you to complete.

Hardware and Software

This project requires the use of a GPU to train NeRF models using the PyTorch framework. The recommended approach for this is to use Google Colab, which provides a Jupyter-like environment for working with notebooks, and allows you to use a GPU-equipped runtime for free.

Free Colab usage has dynamic limits based on availability, and students have had issues with the service timing out or restricting GPU usage after a while. As of early December 2025, Google offered their Colab Pro subscription for free for a year to students and educators; as of December 30th 2025, I don’t see this option, but it might come back. Check the subscription page and see if there’s a “Free for students and educators” button or link. The Colab Pro subscription gives you higher limits, access to faster GPUs, and so on; you should be fine without it, but it may be nicer to have it.

Head over to https://colab.research.google.com/ and upload the Project4.ipynb notebook found in your Github repo. From there, using Colab is very similar to using Jupyter Lab. To make sure you are using a runtime with a GPU, go to Runtime > Change runtime type and ensure that you have some kind of GPU selected under Hardware Accelerator (e.g., T4 GPU).

Running Locally

If you have a computer with a cuda-enabled GPU and wish to run the notebook locally, you can do this using Jupyter Lab. The skeleton code inclues the Python dependencies needed, so you should be able to uv run jupyter lab from the repo’s root directory and work with the notebook as usual. Make sure that when the first cell is executed, the value of device when printed is cuda. If it’s cpu, you can still work on many parts of the notebook, but the actual training and inference may be debilitatingly slow.

If you’re on a mac, you may be able to use Apple GPU acceleration using device = torch.device("mps") instead of "cuda".

Data

The input and test data for this project will be downloaded (and cached) by the notebook as needed. Do not commit the data files to your Github repository.

Tasks - Overview

There are several pieces we ask you to implement in this assignment:

1. TODO 1: Fit a single 2D image using an MLP with and without positional encoding;
2. TODO 2: Compute the origin and direction of rays through all pixels of an image
3. TODO 3: Sample the 3D points on the camera rays
4. Implemented for you: Compute compositing weight of samples on camera ray
5. TODO 4: Render an image with NeRF

The instructions about individual TODOs are present in detail in the notebook.

Testing

To help you verify the correctness of your solutions, we provide tests at the end of each TODO block, as well as qualitative and quantitative evaluation at the end of the notebook.

Expected Outputs

2D Image Fitting

With no positional encoding:

With 3 frequencies:

With 6 frequencies:

NeRF Training:

The optimization for this scene takes around 1000 to 3000 iterations to converge on a single GPU (in 10 to 30 minutes). We use peak signal-to-noise ratio (PSNR) to measure structual similarity between the prediction and target image, and we expect a converged model to reach a PSNR score higher than 20, and produce a reasonable depth map. Here’s an output after 1000 iterations of training with default settings:

Here’s the resulting 360 video:

Submission

1. Notebook Execute your completed notebook in Jupyter and commit the resulting version of your .ipynb file to your github repository; make sure the cell outputs are contained in the notebook.

2. Sample Output Also commit the 360 video output (lego_spiral_001000_rgb.mp4) to your repository.

3. Hours On the first line of hours.txt, include a single integer estimate of the number of hours you spent working on this assignment. Below that, you may optionally include a reflection on how it went, anything you found particularly confusing or helpful, and/or suggestions for improving the assignment.

Rubric

Points are awarded for correctness and efficiency, and deducted for issues with clarity or submission mechanics:

• 10 points for each correctly passed test case in the notebook (for TODOs 1-4)
• 5 points for your answer to TODO 1(c)
• 5 points for efficiency: training on a 360 scene with default parameters takes under 10 minutes on a lab machine with an RTX 3060

Clarity Deductions for poor coding style may be made. Please see the syllabus for general coding guidelines. Points may be deducted for any of the following:

• Methods should be written as concisely and clearly as possible
• Methods should not be too long - use helper methods to break code into sensible subroutines
• Code should not be cryptic and terse - explain nontrivial blocks with comments
• Methods you introduce should be accompanied by a precise specification
• Variable and function names should be informative but not overly verbose

Acknowledgements

This assignment is based on an assignment from Noah Snavely; thanks to Noah and numerous underappreciated TAs.