NDSplat
arXiv

Feed-Forward Adaptive Gaussian Splatting for Interactive Medical Volume Rendering

United Imaging Intelligence, Boston MA, USA
zhongpai.gao@uii-ai.com *Corresponding author
+2.8 / +2.4 dB
PSNR over Render-FM · CT / MR
4–6 ms
Per 1600² view (A100)
1–2 s
Per new scan, one forward pass

One Model, Two Modalities

A whole-body CT and a whole-body MR scan from the test set, each turned into Gaussians by one forward pass of the same network. Every frame, transfer-function switch and clipping-plane move in the video is rendered from those Gaussians, with no fitting to the scan.

Render-FM with a fixed 3-mm grid versus RenderFM-Next with learned 3 to 1.5-mm refinement, on clipped CT and MR clinical presets

Adaptive refinement for feed-forward medical Gaussians. (a) Render-FM uses a fixed 3-mm grid, which blurs organ boundaries and cut faces. (b) RenderFM-Next, one CT+MR model, refines to 1.5-mm children (orange) where a learned score expects the most benefit, under a user-set budget. Right: clipped clinical presets; PSNR in dB.

Abstract

Path-traced rendering shows CT and MR volumes with realistic lighting, but it restarts after every change of view, clipping plane or transfer function (TF), the map from voxel intensity to colour and opacity, which limits the frame rate. Feed-forward Gaussian splatting models such as Render-FM predict the Gaussians directly from the volume instead of fitting them to each scan, and are fast. Render-FM, however, handles only CT, runs its network again after every TF edit, has no clipping operator, and places its Gaussians on a uniform 3-mm grid that blurs organ boundaries and cut faces.

We present RenderFM-Next, a feed-forward model that addresses these limits. One network handles both CT and MR. Its Gaussians are predicted once per volume: a new TF needs no pass of this network, and a clipping plane cuts them exactly, using operators from prior work. The model predicts coarse 3-mm Gaussians for the whole volume and refines them to 1.5 mm only where refinement improves the image. A small score head ranks the coarse Gaussians by how much refining them would reduce the rendering error; at render time the user chooses how many are refined, and those next to a clipping plane are always refined, which sharpens cut faces.

We build a benchmark of about 2,000 CT and MR scans with path-traced references under clinical presets, TF edits and clipping. RenderFM-Next improves over Render-FM retrained on this benchmark by 2.8 dB on CT and 2.4 dB on MR and renders a 1600×1600 view in 4–6 ms. Refinement adds 0.30 / 0.69 dB over the same model trained without it (0.51 / 1.17 dB on cut faces), and refining a learned 10% of the coarse Gaussians matches a random 50% with 2.4× fewer Gaussians.

How It Works

Once per volume, one U-Net for CT and MR predicts 3-mm parent Gaussians, up to eight 1.5-mm children per parent, and a score that ranks the parents. At render time the network is not run again: the top-ranked parents under the budget, and those next to the clipping plane, are replaced by their children.

RenderFM-Next overview: once-per-volume prediction of parents, children and scores; render-time refinement, transfer-function head and exact clipping
Predict once · CT and MR

One network, one pass

The same weights serve both modalities. The transfer function and the clipping plane are not network inputs, so editing either never re-runs the U-Net.

Refine · learned score

Detail where it pays off

A small score head ranks coarse Gaussians by how much refining them would reduce the rendering error. The user sets the budget at render time.

Interact · TF and clip

Edit and cut in real time

A light TF head recolours the same Gaussians, and an exact clipping operator cuts them. Gaussians next to the plane are always refined, which sharpens cut faces.

Quantitative Results

Test set: 67 CT and 115 MR patients, with path-traced references under the base TF, clinical presets, TF edits and clipping. PSNR in dB, per-patient mean.

Method All reported families PSNR↑ by family #G (M)FPS↑
PSNR↑SSIM↑LPIPS↓ Base uncutBase clippedClinical presetsPreset + edit
CTRender-FM (retrained) 26.870.9360.08526.9127.2525.7226.320.3599
Ours (parents only) 29.390.9480.06829.9629.6927.3228.980.35225
Ours (joint) 29.690.9510.05730.3130.0527.4228.950.89162
Ours (CT specialist) 29.800.9520.05630.4130.2027.5328.850.90163
MRRender-FM (retrained) 26.250.9210.14826.7425.5628.4026.780.17159
Ours (parents only) 27.950.9320.12628.7127.2728.7927.500.17350
Ours (joint) 28.630.9390.10729.2328.1129.3128.130.42229
Ours (MR specialist) 28.930.9400.10729.5528.3829.5528.440.41224

Render-FM is retrained on this benchmark, one model per modality. Ours = RenderFM-Next. Parents only: trained without children (same number of Gaussians as Render-FM). #G: Gaussians per view in millions. FPS at 1600² on one A100. Bold = best single model; grey rows are one model per modality, shown for reference.

Render-FM and RenderFM-Next side by side

Capabilities on top, quality and cost below (CT / MR). All times on one A100 GPU at 1600².

Render-FMRenderFM-Next (Ours)
Parents onlyChildren (20% refined)
CT and MR in one network–✓✓
New TF without a U-Net pass–✓✓
Exact clipping-plane cut–✓✓
Adjustable refinement––✓
Refined cut faces––✓
PSNR (dB)↑26.87 / 26.2529.39 / 27.9529.69 / 28.63
Frame rate (FPS)↑99 / 159225 / 350162 / 229
New TF (ms)↓71 / 3421 / 1551 / 35
New volume (s)↓1.99 / 1.560.82 / 0.541.73 / 1.06
Peak memory (GB)↓3.3 / 1.73.0 / 1.57.7 / 4.9

Moving a clipping plane takes 17 / 12 ms, because only the band next to the cut changes.

Qualitative Comparison

Clipped clinical presets on held-out scans; full-frame PSNR in dB. Children sharpen cut faces and thin structures that the parents-only model and Render-FM (retrained) blur.

Qualitative comparison on clipped CT and MR clinical presets: reference, Render-FM, parents only, RenderFM-Next

BibTeX

@article{gao2026renderfmnext,
  title={Feed-Forward Adaptive Gaussian Splatting for Interactive Medical Volume Rendering},
  author={Gao, Zhongpai and Planche, Benjamin and Zheng, Meng and Choudhuri, Anwesa
          and Chen, Terrence and Wu, Ziyan},
  journal={arXiv preprint},
  year={2026}
}