Overview

The goal of this project is to implement a simple and efficient setup for generating custom backgrounds for synthesized views from Neural Radiance Field (NeRF).

The idea is to utilize diffusion models to generate a spherical panorama from an input text prompt, and then render views for the input view directions for the object (with NeRF) and for the background (with spherical projection) simultaneously. Finally, by modifying the volumetric rendering function of NeRF, the rendered background is injected into the final rendered view. Additionally, distortion removal on the rendered background image is an important step to greatly improve the output.

Using the above presented setup which works with any NeRF model, the users can simply provide a text prompt and view direction to synthesize view of the object with a background that fits both the prompt literally and the object geometrically. The user can also specify more than one view directions for the same prompt, which would synthesize a view for each view direction with consistent background.

Results

Prompt: "snowy mountain"

Prompt: "forest"

Prompt: "on the moon with earth in view"

Prompt: "futuristic spaceship"

Prompt: "spacious room with toys on the floor"

The above results use the same set of view directions that generally produce the best results. However, the setup can work for any view directions which covers the entire hemisphere of possible view directions. The results at various different view directions are shown below.

Methodologies

In this section, an overview of the main components of the setup is presented. More technical details can be found in the report paper.

Generating Views from Spherical Panorama

Distortion Removal

NeRF Background Injection

Limitations & Future Directions

Limitation #1: Lighting & Shadows

The current setup does not account for lighting and shadows which will make the synthesized views more realistic.

Limitation #2: Lack of Generalized Setup for All View Directions

The current setup requires manual tuning of some parameters for different vertical view directions (camera height above ground).