Reservoir Splatting for Temporal Path Resampling and Motion Blur

Early real-time ray tracing researchers lacked the benefits of hardware acceleration [Kilgariff et al. 2018], often making do with significantly less than one ray per pixel. Frameless rendering [Bishop et al. 1994] recomputes a subset of pixels each frame, reusing others until they get updated. Adding prior-frame reprojection [Adelson and Hodges 1995; Corso et al. 2017] repositions reused pixels, and a render cache [Nehab et al. 2007; Scherzer et al. 2007; Walter et al. 1999] reuses computation over multiple frames. Adaptive frameless rendering [Dayal et al. 2005] carefully places new samples and runs (cached) samples through spatiotemporal reconstruction. Yang et al. [2011] reprojected bidirectionally, i.e., from both forward and backward frames, to help reconstruct intermediate frames.

Outside of ray tracing, games widely apply temporal antialiasing (TAA) [Yang et al. 2020] to reduce undersampling using prior-frame data. Modern TAA methods also upsample their output [Yang et al. 2009], but can introduce ghosting, blur, and shimmer. To reduce these artifacts, many apply fast neural networks [Liu 2020; Xiao et al. 2020] that extend and accelerate offline superresolution networks (e.g., Dong et al. [2015]).

Real-time denoisers (e.g., Schied et al. [2017; 2018]) often ingest low-sample inputs, so they employ temporal reprojection to increase effective sample counts [NVIDIA 2020b]. As in temporal superresolution, many denoisers also apply neural networks [Bako et al. 2017; Chaitanya et al. 2017; Işık et al. 2021], sometimes to uncover adaptive sampling opportunities [Hasselgren et al. 2020; Kuznetsov et al. 2018].

Our method is the first to demonstrate how prior-frame samples can be used via forward-reprojection without introducing bias.

Path reuse [Bekaert et al. 2002] amortizes costs by reusing paths or segments within pixel blocks, though such reuse has challenges; Xu and Sbert [2007] explore ways to reduce tile correlations. Gradient-domain rendering builds correlated paths for close-by pixels with shift mappings [Kettunen et al. 2015; Lehtinen et al. 2013] to evaluate finite differences. Bauszat et al. [2017] use these shift mappings to better reuse specular paths within pixel blocks.

Recently, resampling algorithms [Bitterli et al. 2020] have significantly improved path reuse, demonstrating real-time performance for many light transport problems; our work fits into this category of spatiotemporal reservoir resampling (or ReSTIR) algorithms.

Resampled importance sampling (RIS) [Talbot et al. 2005] aggregates M samples, which are then resampled into N samples, approximately distributed according to a normalized target function. Bitterli et al.’s [2020] ReSTIR DI combines RIS (using N = 1) with weighted reservoir sampling [Chao 1982], with each per-pixel reservoir storing a sample whose distribution is refined by resampling from spatial and temporal neighbors. Ouyang et al. [2021] treat longer paths as virtual point lights [Keller 1997] to handle indirect light. Lin et al. [2022] shift full paths between neighbors to properly handle glossy surfaces. Their generalized RIS (GRIS) provides a mathematical foundation for ReSTIR. Zhang et al.’s [2024] Area ReSTIR improves robustness for depth of field and subpixel details by reusing lens and subpixel coordinates with fractional motion vectors. We build on Area ReSTIR, but redesign its temporal reuse.

Motion blur arises as cameras have non-zero exposure times. Perceptually, it conveys relative motion within a frame. But temporal integration is expensive, so real-time methods use various approximations, e.g., accumulating multiple frames [Haeberli and Akeley 1990], blurring per-pixel [Rosado 2007], extruding geometry [Gribel et al. 2010; Tatarchuk et al. 2003], or screen-space velocity maps [McGuire et al. 2012]. See Navarro et al. [2011] for a survey. Such approximations typically favor performance over accuracy.

Ray tracers can stochastically sample time to render motion blur; this is generally more expensive but produces better results. Multidimensional adaptive sampling [Hachisuka et al. 2008; Meister and Hachisuka 2022] concentrates samples in motion-blurred regions. Frequency analysis [Egan et al. 2009] enables sparser sampling driven by a sheared space-time filter. Light field reconstruction [Lehtinen et al. 2011; 2012; Munkberg et al. 2014] can efficiently render motion blur. Covariance tracing [Belcour et al. 2013] computes a required per-pixel sample count, and reconstructs in image-space. Manzi et al. [2016] accelerate motion blur by evaluating temporal differences with cross-frame shift maps. Oberberger et al. [2022] further account for motion blur in the denoising process.

To our knowledge, we achieve the first motion blur via resampling, potentially enabling real-time motion blur in low-sample renderers.

Since the result of RIS does not have a tractable PDF, Lin et al. [2022] abandon the traditional f/p estimatorsin favor of the more general f(X)W_X estimators withwhere random variable W_X is an unbiased contribution weight (UCW) for X. A random variable W_X is a UCW for X if and only if Equation (2) is true for all integrable f; this is equivalent to $\mathbb{E}\left[W_X|X\right]=1/p_X(X)$ [Lin et al. 2022].

While RIS resampling generally results in intractable PDFs, it allows unbiased integration and iterative resampling with simple UCWs via this f(X)W_X integration framework.

Reusing a path between pixels requires modifying some vertices to change which pixel it contributes to. Shift mappings T move paths between domains, e.g., pixels. Formally, a shift map from Ω₁ to Ω₂ is a bijective function from a subset of Ω₁ to a subset of Ω₂.

The reconnection shift [Lehtinen et al. 2013] is a commonly used shift mapping. Given two camera positions x₀ and y₀ and primary hits x₁ and y₁, this shift maps base path $\overline{x}=[x_0{x}_1{x}_2\dots x_n]$ into offset path $T(\overline{x})=[y_0{y}_1{x}_2\dots x_n]$, reconnecting to $\overline{x}$ immediately after primary hit y₁. This works well on rough surfaces, but fails the goal of good shifts (that $f(\overline{x})\approx f(T(\overline{x}))$) if any of x₁, x₂, or y₁ are nearly specular. Kettunen et al.’s [2015] half-vector shift and Lin et al.’s [2022] hybrid shift postpone reconnection until rough vertices are found, more effectively handling specular materials.

Forward and backward shifts T and T^{− 1} need not be defined for all paths, but when defined, bijectivity is essential for unbiased path reuse¹. Shift mappings also change path and probability densities, so formulas using shifts need Jacobian determinants |T′(x)| or |∂T/∂x|.

Given candidate samples X₁, …, X_N, their corresponding domains Ω₁, …, Ω_N, a target domain Ω for resampling, and a non-negative target function $\hat{p}$ defined in Ω, GRIS [Lin et al. 2022] aggregates X₁, …, X_N into a result Y ∈ Ω, approximately distributed proportional to $\hat{p}$. First, each X_i ∈ Ω_i is shifted to a similar sample Y_i ∈ Ω via shift map T_i:Then, a resampling weight w_i is computed for each Y_i:where m_i is a resampling MIS weight, $W_{X_i}$ is sample X_i’s UCW in its original domain Ω_i, and |∂T_i/∂X_i| is the Jacobian of shift T_i. Finally, output Y is resampled from the Y_i proportional to weights w_i. The new contribution weight for Y isIf the supports of candidates Y_i together cover the support of $\hat{p}$, W_Y is unbiased, as per section 3.1. Taking one canonical sample guarantees this coverage; a canonical X_c is sampled from Ω_c = Ω, with an identity shift map T_c, so that X_c alone covers $\hat{p}$’s support.

Lin et al. [2022] introduce the generalized balance heuristic to compute resampling MIS weights:which uses the “$\hat{p}$ from” functionas an (unnormalized) proxy for the PDF of y = T_j(z_j), originating in domain Ω_j as $z_j=T_j^{-1}(y)$. The Jacobian modifies the proxy the way probability densities transform in shift mappings. The confidence weights c_j control the relative weight of candidate samples.

ReSTIR repeatedly applies GRIS to share samples spatiotemporally. Each pixel i stores a reservoir containing a sample X_i, its UCW $W_{X_i}$, and confidence weight c_i. See Wyman et al.’s [2023] course notes for details, but at a high level, the process typically goes as follows:

Initial sampling. Each pixel i gets some number N_init initial candidates, e.g., newly-traced independent paths. One is selected via RIS, producing a canonical initial sample $X_i^{\ast }$ with UCW $W_{X_i^{\ast }}$.

Temporal resampling. Each pixel i backprojects along its backward motion vector to choose prior-frame reservoir j. The results of GRIS between Y_j = T_j(X_j) and $X_i^{\ast }$, using confidence weights c_j and N_init, replaces X_i, a new $W_{X_i}$ is computed, and c_i is set to min (c_cap, c_j + N_init), where c_cap is a fixed confidence cap.

Spatial resampling. Each pixel i chooses N_spat spatial neighbors $j_1,\dots ,j_{N_{\text{spat}}}$ randomly from a box around i, and performs GRIS between X_i and each $X_{j_k}$, using the corresponding confidence weights. The result overwrites X_i; $W_{X_i}$ is updated with Equation (5), and c_i sums the samples’ confidence weights, again clamped to c_cap.

Shading. The final color for pixel i is evaluated as $f(X_i)W_{X_i}$, and the same process repeats in the next frame.

Prior to Area ReSTIR [Zhang et al. 2024], screen-space algorithms such as Lin et al. [2022] applied ReSTIR to path space starting at secondary hits, x₂, finding primary hits x₁ by ray tracing at predetermined subpixel locations (e.g., pixel centers).

Zhang et al. [2024] note this adds instabilities near high-frequency details: even if a reused path suffix [x₂x₃⋅⋅⋅x_n] remains valid, it may be incompatible with a new pixel’s implicitly defined prefix [x₀x₁]. To cure this, they add subpixel location (and lens position) as additional dimensions to reservoirs, essentially storing full paths [x₀x₁⋅⋅⋅x_n]. Target functions no longer vary with each pixel’s implicit prefix, as the shift maps propose complete, reusable paths. This significantly boosts reuse quality.

To improve subpixel precision for temporal path reuse, Zhang et al. [2024] backproject the hit point at each pixel i’s center to the prior

frame, via an image-space shift with motion vector δ_i, building a 1 × 1 pixel fractional reservoir around the result. This off-grid fractional reservoir is populated with samples from the overlapping 2 × 2 block of pixels (via RIS), and the chosen sample is shifted back to the current frame with motion vector − δ_i, approximately retaining the primary hit.

As no samples in the 2 × 2 block of prior-frame pixels may fall in the fractional reservoir, Zhang et al.’s “fast” variant can leave this reservoir empty, producing excess noise. Prior to RIS, their “robust” variant first spatially shifts all samples in the 2 × 2 block to the fractional reservoir. This solves the issue but is more expensive.

Mathematically, Area ReSTIR integrates the measurement contribution function for each pixel i:where pixel i’s path space Ω_i ⊂ Ω is the set of paths $\overline{x}$ for which the pixel filter $h_i(\overline{x})>0$, and $f(\overline{x})$ is the path contribution. We use ${\hat{p}}_i(\overline{x})$ to denote the pixel-dependent target function for resampling, which we assume is defined asoften with $\hat{p}=f$, but cheaper approximations can also be used.

Note that Area ReSTIR only approximately retains primary hits; this may degrade temporal reuse near subpixel details. Our reservoir splatting exactly retains primary hits, improving temporal reuse.

Zhang et al.’s Area ReSTIR integrates the measurement integral at a fixed time. However, rendering motion blur requires integrating over a time interval [t₀, t₁), where Δt = t₁ − t₀ is the exposure time. Intensity of pixel i is thenwhere $h_i(\overline{x},t)$ is the camera’s pixel filter at time t ∈ [t₀, t₁).

We experimented by naïvely extending Area ReSTIR for motion blur by adding sample time t to the reservoirs. But backprojecting each pixel’s motion δ_i at a single time causes suboptimal reuse, as apparent motion depends on both sample time and subpixel location, which vary independently. Our splatting uses samples’ real motion between frames, exactly contributing to the relevant pixels.

Lin et al.’s [2022] GRIS theory requires defining input domains without looking at the samples. Prior work worked around this by reusing based on pixel-center motion vectors. But exactly preserving prior-frame samples’ primary hits requires examining which samples contribute to which pixel, which is not allowed. To reconcile this, we conceptually define all prior samples as contributing to every pixel, but ensure zero weight for those not splatting into current pixel j.² This allows applying GRIS for temporal resampling assuming:

For efficient computation, we first initialize a GRIS reservoir with a canonical sample for each current-frame pixel. We then shift all prior paths X_i to the current frame as Y_i = T(X_i), identifying the pixels each contributes to, and stream Y_i to these pixels’ reservoirs (while avoiding race conditions).

While a balance heuristic (section 4.1.2) for all-to-all reuse for all N pixels requires O(N³) total shifts, the splat operation implicitly zero-weights most prior-frame samples for a given pixel. This reduces to two shifts per pixel (forward-splatting the prior reservoir and reverse-splatting the initial sample) totaling O(N) shifts.

Splatting projects to the current frame using the last frame’s forward motion vectors; this leaves holes where no prior-frame pixel contributes. We can optionally fill holes with backup samples, e.g., backprojecting to the prior frame to find relevant samples.

To obtain a backup, we follow the rounded motion vector δ at each pixel center back to a prior-frame pixel b, whose sample X^b we give as an additional input to our temporal reuse. We directly use Zhang et al.’s [2024] proposed temporal shift T_b.

This simple method improves robustness, albeit at increased cost similar to Zhang et al.’s [2024] robust variant. Like any shift moving the shading point, this works best for low frequency geometry and lighting. In the supplementary material, we extend the MIS weights from Equations 11 and 13 to correctly account for backup samples.

Earlier methods set the confidence weight c_j to the capped sum of the inputs (section 3.4). However, we conceptually have H × W inputs, of which only a few contribute. Setting the confidence weight based only on contributing reservoirs produces bias. Instead, we model confidence after Zhang et al. [2024], backprojecting the current pixel-center into the prior frame and bilinearly interpolating the confidence weights of the 2 × 2 overlapped pixels:Here, ${\beta }_{k^{\prime }}$ is the bilinear weight corresponding to how much pixel k′ overlaps with the backprojected pixel j. With the backup, we also add its confidence before clamping.

No prior spatiotemporal resampling algorithms handle motion blur, which requires integrating over a shutter time [t₀, t₁) and associating a specific time t with each sample.

Zhang et al. [2024] achieved real-time depth of field by reusing the subpixel location u. Shifts could then be defined by either reusing the lens position s (the lens vertex copy shift), or the primary hit x₁ (the primary hit reconnection shift). Primary hit reconnection improves reuse of bokeh samples for large apertures, but often leads to shift failures for small apertures. Neither shift produces acceptable results alone, so they must be combined via heuristic-based MIS weights.

In contrast, splatting explicitly preserves the primary hit x₁ and also allows reuse of a fixed lens sample s, while the image-space location u is simply defined by splatted sample location. This achieves high-quality bokeh using our single shift map. This saves compute time and simplifies the code. See Figure 4 for an illustration.