ReSTIR BDPT: Bidirectional ReSTIR Path Tracing with Caustics

ReSTIR PT [Lin et al. 2022] improves real-time rendering quality by orders of magnitude by reusing path samples between pixels and frames. However, ReSTIR PT inherits weaknesses from unidirectional path tracing (PT) for scenes with caustics or hard-to-reach lights (Figure 1). Our work aims to significantly improve ReSTIR PT in these scenes by incorporating BDPT.

Naïvely applying generalized resampled importance sampling (GRIS), the main engine behind ReSTIR PT, to real-time bidirectional PT is not feasible. When reusing across pixels or frames, GRIS does not directly account for which technique generates a path sample; and in BDPT, many sampling techniques can form each path. This makes parameterizing a path with one random seed infeasible, calling for expensive path storage. Additionally, light tracing poses additional challenges for screen-space reuse as it produces samples contributing to all pixels.

To address these challenges, we apply GRIS theory in an extended path space consisting of path–technique pairs $(\overline{x},\tau )$ joined by technique-specific multiple importance sampling (MIS) weights. These extended paths allow principled reasoning about the sampling technique at path reuse without introducing bias. Our method enables a seemingly infeasible choice, using [Lin et al. 2022] hybrid shift for camera sub-paths and random replay for light sub-paths.

We achieve interactive caustic rendering by implementing a bidirectional hybrid shift, as well as storing an additional caustic reservoir in each pixel. The caustic reservoir enables unbiased temporal accumulation of caustics over multiple frames in dynamic scenes. Scene changes may move caustics between pixels, but they often remain nearly converged. Accumulation is supported by merging new caustic paths into caustic reservoirs each frame. Our caustic reservoirs automatically aggregate and stratify the caustics into the pixels, allowing interactive updates and retaining high caustic quality as the scene evolves.

We further accelerate our ReSTIR BDPT by deriving a novel variant of recursive MIS weights [van Antwerpen 2011] for fast MIS weight evaluation with shift mappings. We also propose two faster alternatives, a biased variant which copies the unshifted MIS weight for cases where full unbiasedness is not required, and an unbiased lightweight (LW) variant which omits vertex connection techniques.

Our method runs interactively, taking around 50ms per frame in most of our test scenes (Figure 11). On difficult scenes at one sample per pixel, we achieve much lower error compared to ReSTIR PT in equal time due to the improved efficiency of bidirectional sampling.

We briefly review resampled importance sampling (RIS) [Talbot et al. 2005], its generalization [Lin et al. 2022], and bidirectional PT [Veach and Guibas 1995a]. We refer readers to [Wyman et al. 2023] course notes for a more comprehensive ReSTIR review [Bitterli et al. 2020; Lin et al. 2022], and physically-based rendering textbooks [Pharr et al. 2016] and Veach’s thesis [1997] for an introduction to path space and bidirectional PT.

Our notation and key symbols are summarized in Table 1.

We aim to produce paths with BDPT and apply GRIS to reuse these paths spatiotemporally. BDPT often produces a much better sample distribution, enabling rendering of intricate light paths such as caustics, and improving the sampling quality at low sample counts.

Lin et al. [2022] apply GRIS to unidirectional PT, which corresponds to techniques $s=0$ (camera subpath hits an emissive surface) and $s=1$ (next event estimation). A main challenge in applying GRIS to BDPT is handling the large set of sampling techniques. In particular, we often want to apply random replay as part of Lin et al.’s hybrid reconnection shift. However, we cannot use random replay for bidirectional paths unless we know which technique generated the path: a path combining two subpaths can only be reproduced by replaying the random number states for the camera and light subpaths up until the connection vertices.

Another challenge comes from light tracing techniques, i.e., $t\le 1$, as such paths can contribute to any pixel. We must correctly weight our samples to account for this fact. Furthermore, we need to design a reuse scheme that works well for light tracing and caustics.

Finally, computing the BDPT MIS weight ${\omega }_{s,t}$ (Equation (6)) during spatiotemporal reuse can be quite involved. We cannot directly apply [van Antwerpen 2011] efficient, recursive MIS formulation, since shift mappings may change parts of the paths.

Below, we address the aforementioned challenges:

Finally, in Section 7, we discuss our GPU implementation of a ReSTIR-based bidirectional path tracer.

In this section, we derive our bidirectional ReSTIR estimator by applying GRIS in an extended path space. We want to apply GRIS to resample over all bidirectional sampling techniques, using correct MIS weights ${\omega }_{\tau }$ to allow unbiased spatiotemporal reuse.

Previous ReSTIR methods directly apply GRIS to the path integral (Equation (1)) by sampling paths and merging them into reservoirs. As mentioned, using random replay shift mappings complicates this case: existing formulations do not allow shift mapping to depend on a path’s sampling technique. Thus, when shifting via random replay, we do not know the camera and light subpath lengths.

Therefore, we apply GRIS in an extended path space. We pair path samples $X$ with the technique indices $\tau$ used to generate them in extended paths $\hat{X}=(X,\tau )$. All extended paths of a given technique $\tau$ define the technique’s path space ${\mathrm{\Omega }}_{\tau }$ and our extended path space $\hat{\mathrm{\Omega }}$ is the union of all sample-technique pairs, $\hat{\mathrm{\Omega }}=\bigcup _{\tau }{\mathrm{\Omega }}_{\tau }$. Our extended path space integral is then:where ${\omega }_{\tau }(\overline{x})$ is the technique MIS weight of path $\overline{x}$ under technique $\tau$. We weight the measurement contribution $f$ with these MIS weights to avoid counting a path’s contribution multiple times.

Now, our goal is to apply GRIS on the extended path space integral (Equation (18)). To apply GRIS, we first need two items: the target function $\hat{p}$ and shift mapping $T$. We set the target function to be always weighted by the technique MIS weight ${\omega }_{\tau }$:where $\hat{x}=(\overline{x},\tau )$ and $\hat{q}$ is a user-defined target function. We always set $\hat{q}$ to be the luminance of the measurement contribution $f$. We then assume technique-specific shift mappings, i.e., the output path depends on the technique, but the technique is not changed in the shift:where $T_{\tau }$ is the shift mapping defined for technique $\tau$. We will specify $T_{\tau }$ in the next section. The Jacobian is simplyCombining these, we can directly apply GRIS to estimate the extended path space integral (Equation (18)). For the resampling output $\hat{Y}=(Y,\tau )$ sampled from the shifted inputs $T_i({\hat{X}}_i)$ proportionally to resampling weights $w_i$, we getThe UCW for the chosen sample at GRIS resampling iswhere $\hat{p}$ is given by Equation (19), and the candidates’ resampling weights are, denoting ${\hat{Y}}_i=T_i({\hat{X}}_i)$,Here, $m_i$ is the generalized balance heuristic (Equation (17)), with a substitution of our $\hat{p}$ (Equation (19)) and Jacobian (Equation (21)) into the definition of ${\hat{p}}_{\leftarrow i}$ in Equation (15).

Like Lin et al. [2022], we use the generalized balance heuristic for temporal reuse and pairwise MIS [Lin et al. 2022; Wyman et al. 2023] for spatial reuse. We use pairwise MIS for spatial reuse to avoid the quadratic cost of the balance heuristic. For temporal reuse, we use the balance heuristic since temporal reuse always has just two candidates (the current and prior frame’s samples).

In initial sampling, the UCW $W_{{\hat{X}}_i}$ for a bidirectional sample-technique pair ${\hat{X}}_i=(X_i,{\tau }_i)$ from technique ${\tau }_i=(s,t)$ is the reciprocal of the subpath sampling PDFs:where $Z_i$ and $Y_i$ are the camera and light subpaths, respectively, and $p_L(Y_i)$ is the probability of selecting $Y_i$ for subpath connection.

Initial sampling generates a path for each technique within a pixel, and we use GRIS to select one. The resampling MIS weights for resampling one path-technique pair $\hat{x}=(\overline{x},\tau )$ from the initial candidates are, with $\tau =(s,t)$,where $N_L$ is the number of light subpaths. The number of samples a pixel can receive varies by technique: Light tracing techniques $t\le 1$ can contribute to any pixel, so each pixel receives $N_L$ light tracing samples, requiring the divide by $N_L$ to normalize. Other techniques $t\ge 2$ generate samples only for the camera subpath’s pixel. With one camera subpath per pixel and each subpath vertex connected to one light subpath, all the other path-technique spaces receive one sample each per pixel.

Now, we define our technique-specific bidirectional hybrid shift mapping $T_{\tau }$ (Equation (20)) for $\tau =(s,t)$. We separately discuss three cases: camera tracing techniques ($t\ge 2$), light tracing techniques ($t\le 1$) where the path is non-caustic, and light tracing techniques ($t\le 1$) where the path is caustic. A path is caustic if the third vertex from the camera, $x_2$, is classified as non-rough. A key caustic path property is reconnections from the primary hit are not possible as $x_2$ is non-rough, leading us to use random replay instead.

Path tracing, next event estimation, and light subpath connections ($t\ge 2$). When techniques have a camera subpath, we shift the light subpath (if $s>0$) with random replay and the camera subpath (if $t>1$) with the hybrid shift mapping of ReSTIR PT [Lin et al. 2022]. Case $s=0$ corresponds to PT from the camera and hitting an emitter; $s=1$ corresponds to next event estimation, and $s\ge 2$ connects to a non-degenerate light subpath.

For techniques with subpath connections ($t>1$ and $s>0$), we always reconnect (at latest) to the first light subpath vertex. As Manzi et al. [2015], we only allow subpath connections on vertices classified as rough at sampling time. Thus, for BDPT to connect a camera and light subpath, the connected vertices must be classified rough, so a reconnection shift will also succeed. We do not impose a distance constraint for reconnection between the light and camera subpaths, as technique MIS weights $w_{\tau }$ naturally lower weights for short connections. Additionally, if a shift changes the classification of either vertex (e.g., a rough vertex becomes non-rough) we then force a shift failure to maintain bijectivity.

Figure 3 illustrates a temporal shift for a path with $t=6$ and $s=4$. We first trace a new camera subpath from the target pixel’s primary hit $x_1^{\prime }$. If we sample two consecutive rough vertices, we can reconnect to the second. Otherwise, we reuse the random numbers that generated the original path’s next direction (i.e., apply a random replay shift), and keep tracing the shifted path.

Light tracing ($t\le 1$), non-caustic. For $t=1$ techniques, light subpaths are connected directly to the camera. As we only perform bidirectional connections between rough vertices, this implies the primary hit $x_1$ is rough. If the secondary hit $x_2$ is also rough, then the path is non-caustic and a spatial shift is possible. We apply a reverse hybrid shift which shifts the light subpath using random replay until the secondary hit $x_2^{\prime }$, then reconnects to the new primary hit $x_1^{\prime }$ (see Figure 4, top). If the new primary hit $x_1^{\prime }$ is not rough, reconnection is not possible and the shift fails. Just like the forward hybrid shift used for camera subpaths, our reverse hybrid always reconnects the first two consecutively-rough vertices from the camera, which are always the primary and secondary hits $x_1\leftrightarrow x_2$ as we only use this shift for non-caustic paths.

Despite replacing the last BSDF direction sampling with a reconnection to the primary hit, the technique index does not change: The shift mapping shifts the extended path ($\overline{x},\tau$) into another extended path ($T_{\tau }(\overline{x}),\tau$) inside the same layer of the extended path space. Our shift mappings never change the technique index.

Light tracing ($t\le 1$), caustic. For caustic paths, the secondary hit $x_2$ is non-rough, making reconnection impossible. Prior work defined shifts on such paths (e.g., [Lehtinen et al. 2013] manifold shift), but they are poor fits for real-time, requiring expensive optimization procedures and per-vertex storage. We instead shift caustics temporally via random replay only (see Figure 4, bottom).

Unlike our shifts for other techniques, this changes the primary hit $x_1$ and thus the path’s pixel filter contribution ($W_{\text{e}}$ in Equation (2)). To best preserve the sample’s contribution, the sample should be shifted to the pixel where the pixel filter contribution is largest. Our shift mapping accomplishes this by projecting the shifted primary hit $x_1^{\prime }$ onto the screen; the shifted sample is stored at this new pixel. This may shift multiple caustic samples to the same pixel, and GRIS automatically selects one of them. This essentially forward-projects caustic paths to the next frame, using random replay instead of motion vectors, and GRIS combines the samples. We justify this mathematically in Appendix A.

After shifting a path, we must recompute its MIS weight ${\omega }_{\tau }$ in the new domain. This is prohibitively expensive, requiring PDF evaluations at every path vertex. We could simply copy the candidate path’s MIS weight, but this introduces bias (see Figure 5). Instead, we provide an efficient recursive formulation for recomputing ${\omega }_{\tau }$ during reconnection. We also provide bounds on the bias associated with copying the MIS weight in Section 6.4.

We build on prior work [van Antwerpen 2011; Georgiev 2012] which optimizes the formulas for ${\omega }_{\tau }$ allowing computation using only information at the connected subpath vertices $y_{s-1}$ and $z_{t-1}$. Specifically, we store extra partial MIS quantities per vertex, and compute ${\omega }_{\tau }$ from these quantities at $y_{s-1}$ and $z_{t-1}$. Partials are computed incrementally, i.e., values at some vertex $z_i$ derive from quantities at $z_{i-1}$, allowing efficient computation during sampling.

We aim to compute partial MIS quantities at the end of the camera subpath $z_{t-1}$ only using information available during reconnection, without visiting the rest of the camera subpath. We store additional information with the subpath and use that to recover the partial MIS quantities needed to compute ${\omega }_{\tau }$ during reconnection. Importantly, the size of the additional information is fixed and independent of the subpath length, which is key for performance.

If the scene changes or animates, these quantities must be updated. For temporal reuse, we already retrace the whole path, so we can compute the MIS weight as usual. During spatial reuse, we use our recursive MIS algorithm to recover only the information needed to compute the MIS weight, without traversing the entire path.

We implemented BDPT in the Falcor rendering framework [Kallweit et al. 2022] using the Slang shading language. At a high-level, our initial sampling resembles SmallVCM [Georgiev et al. 2012], but without vertex merging. We first sample a fixed number of light subpaths via Algorithm 1, parallelizing over subpaths. Initial vertices $y_0$ are selected according to each light’s emissive power. After sampling light subpaths, we sample camera subpaths, parallelizing over pixels, connecting them to light subpaths in Algorithm 2. Finally, pixels reuse full paths from spatiotemporal neighbors, using Algorithm 3 to shift paths between pixel domains.

In Figure 11, we show several scenes [Bitterli 2016] that are challenging with unidirectional PT. All results use an RTX 4090 GPU, a Ryzen 7 3700x CPU, at a resolution of 1,920$\times$1,080. We compare numerical errors using MAPE.³

We sample $N_L=1920\cdot 1080\approx 2M$ light subpaths, unless otherwise noted. All results use a maximum path length of 20 bounces, or 8 diffuse bounces (whichever is reached first), where a “diffuse bounce” refers to sampling a BSDF lobe with roughness above the connectability threshold. We classify vertices with roughness greater than 0.08 as connectable, which is the minimum roughness our material system supports. See Figure 7 for comparison with other thresholds. We apply Russian roulette by randomly terminating paths inside TraceNewVertex (Algorithm 2, line 5) according to the probabilitywhere $\text{rgh}(x_i)$ returns roughness of the material (not just the sampled BSDF lobe) at vertex $x_i$.⁴ We limit confidence weights $c_i$ to a maximum of 20 in all experiments.

Our strongest results are on scenes lit primarily by caustics, e.g., the Bathroom and Sponza that only have emissive geometry inside glass light fixtures. The glass fixtures in Sponza and Bathroom cause next event estimation ($s=1$) to fail, which makes BSDF sampling ($s=0$) the only viable technique for ReSTIR PT. The glass also causes failed reconnections, forcing ReSTIR PT to use (unidirectional) random replay for most scene lighting; this introduces obvious blotchy correlation artifacts. In contrast, our work efficiently samples light paths through the glass, greatly improving sampling of LSD*E paths and better resolving scene caustics.

In the VeachBidir scene, the caustics on the wall are challenging without bidirectional techniques. Even at high sample counts, unidirectional PT fails to resolve them. Additionally, when the light is animated (Figure 9), our temporal caustic shift enables efficient resampling between frames, greatly improving the result.

For diffuse lighting, as in the WhiteRoom, our improvement over ReSTIR PT is reduced. Reconnection works well on these scenes, enabling spatial reuse to share many light paths between pixels. However, BDPT still improves the candidate distribution, so our method still shows benefits, especially at low sample counts where candidate path distribution especially matters.

In Figure 8, we show a cut-down version of our algorithm without camera connections ($t\le 1$ techniques), similar to prior methods [Liu and Gan 2023; Nabata et al. 2020]. Importantly, $t\le 1$ techniques excel at sampling caustic paths, making it the dominant technique on these scenes. Even with only the $t=1$ technique, most illumination in the scene is well-sampled, except for regions where the primary hits are not rough, which makes connecting to the camera impossible.

Our algorithm costs roughly twice as much as ReSTIR PT at the same sample count. This is mostly because BDPT traces roughly twice as many rays as regular PT, due to light tracing and subpath connections. Our algorithm also employs additional data structures (the LVC and caustic reservoirs) and a larger path reservoir to store light subpath and recursive reconnection MIS information, increasing register and memory pressures.

Our per-pixel reservoirs have size 244B if using our recursive reconnection MIS, and 160B otherwise. The size of LVC vertices is 112B. We give the full contents of our reservoir and light vertex data structures in the supplementary document. For 1,920$\times$1,080 renders, the total storage cost of our full unbiased algorithm is 8.2 GB. We did not attempt to optimize storage.

Figure 12 shows error (MAPE) over time for an offline variant of our algorithm, compared to ReSTIR PT. As Lin et al. [2022] note, temporal resampling is less effective for offline rendering as temporal correlations hurt offline convergence (when averaging multiple frames). Instead, our offline variant uses spatial resampling only. As BDPT has a higher-quality initial candidate distribution, our error is lower than Lin et al. [2022].

Spatial reuse from pixels with very different target functions can sometimes reduce efficiency [Pan et al. 2024; Tokuyoshi 2023]. While we observe better offline results than standard BDPT in all of our scenes when measured per-sample, in some scenes the reduced efficiency no longer compensates for the added computational cost of resampling, leading to standard BDPT converging faster. This suggests a careful study on how to best benefit from ReSTIR algorithms in the offline context.

We presented a method to combine BDPT with ReSTIR. By combining spatiotemporal reuse and path-space sampling, we enable interactive rendering of scenes with caustics and complex occlusion; this was previously limited to offline renderers.

We thank Aaron Lefohn for the discussions and support, and the anonymous reviewers for their constructive feedback.

The pixel a caustic path $(t\le 1)$ hits is determined by its primary hit vertex $x_1$. We shift caustic paths temporally by random replay, i.e., re-tracing the path in the new frame with the same random numbers. This means its pixel may change. This initially seems incompatible with GRIS, which requires pre-selecting the reuse domains, and the selection cannot depend on their samples. To resolve this, we reformulate the reuse process as follows.

First, the domain of current and previous pixels covers the whole image, but their support covers only the current pixel $i$. The target function $\hat{q}$ is zero outside of the pixel, e.g., by a pixel filter $h$,where $\hat{p}$ is the target function, $\overline{x}$ is the path of the path-technique pair $\hat{x}$, and $h_i(\overline{x})$ evaluates the pixel filter at the subpixel location of $\overline{x}$’s primary hit.

Second, in candidate sampling, all $N_L$ $t\le 1$ paths are conceptually treated as candidates for each pixel $i$, leading to resampling weightswhere pixel filter $h_i$ automatically discards samples not in pixel $i$, and $\hat{p}$ discards non-caustic paths. The initial sampling’s output $\hat{X}$, now a caustic path in pixel $i$ or a zero-contribution null sample $\mathrm{\varnothing }$ [Wyman et al. 2023], is then chosen proportionally to the $w_i$.

Third, in temporal reuse, each current pixel $i$ conceptually reuses from all prior frame pixels $j$ (and pixel $i$’s initial sampling). Most pixels do not contribute, as the temporal random replay shift to $i$ fails. Substituting $\hat{q}$ (Equation (48)) to the generalized balance heuristic function (Equation (17)) with confidence weights $M_i$ yieldswhere $T^{-1}$ is the random replay shift to the previous frame, shared with all pixels, $i$ runs over all previous-frame pixels and the current pixel’s initial sampling result, and $j$ runs over all previous-frame pixels. $M_{\text{is}}$ is the confidence weight given to the initial sample. We immediately see that the pixel a caustic path $\hat{x}$ shifts to with random replay is unique, and hence the denominator contains only two active terms, assuming the box filter, and simple sum over multiple pixels for wider filters (the shift mapping, its Jacobian, and $\hat{p}$ do not depend on the pixel).