ReRound
CALIBRATION-FREE LLM QUANTIZATION

ReRound

Reconstructive Rounding to Resolve Midpoint Ambiguity
in Calibration-Free LLM Quantization

He-Yen Hsieh·H. T. Kung
Harvard University
Paper Code Walkthrough BibTeX
AFTER QUANTIZATION SCALING

The scaled weight is 1.48. Now it must be rounded to 1 or 2.

Quantization scaling has mapped the original weight into quantized coordinates. The remaining choice is between the two adjacent integer levels.

1 = floor · round down 2 = ceiling · round up
1 floorround down 2 ceilinground up midpoint 1.5 scaled weight 1.48 reconstruction 1.76
RTN

RTN picks the closer integer.

1.48 is 0.48 away from 1 and 0.52 away from 2, so round-to-nearest (RTN) chooses 1.

ReRound

ReRound adds guidance from a diffusion prior learned from the pretrained LLM's own weights.

The reconstruction is 1.76, on the other side of the midpoint. When it remains close to the scaled weight, it can support changing the rounding choice to 2.

TL;DR

ReRound is designed for weights near a rounding midpoint.

When a scaled weight is clearly closer to one quantization level, round-to-nearest (RTN) already gives a clear choice. Near a midpoint, the distances to the floor and ceiling become very similar. ReRound uses a diffusion prior of the pretrained LLM's own weights as an additional signal for these uncertain cases.

No calibration data is needed. Reconstructed weights only guide rounding; the deployed model remains low-bit quantized.

01 · INTUITION

Why look beyond the nearest choice?

When only 0.48 km and 0.52 km are shown, the shorter distance points to the left path. Once the full routes are visible, that small difference may no longer be enough to choose the route. ReRound uses the same idea when a scaled weight lies near the midpoint between two quantization levels.

Two paths whose first segments are 0.48 km and 0.52 km
STEP 1 OF 3The shorter distance looks preferable.

With only 0.48 km and 0.52 km shown, the left path looks like the natural choice.

CHOOSING A PATH 0.48 km vs. 0.52 km The first segment gives only a local distance.
→
ROUNDING A WEIGHT distance to 1 vs. 2 A learned weight prior adds another signal when the scaled weight is near the midpoint.
02 · WALKTHROUGH

From RTN to ReRound.

This short walkthrough starts with nearest rounding, shows why weights near a midpoint are uncertain, and then shows how ReRound uses reconstructed weights to revisit those choices.

Playback tip: To adjust the animation speed, click ⋮ in the video player and select ⏱ Playback speed.

03 · METHOD

Where does ReRound get the extra signal?

Three steps are enough to understand the idea. The paper gives the full formulation.

Conditional diffusion model used to reconstruct weight patches
STEP 1

Learn a prior from the LLM's own weights.

ReRound trains a conditional diffusion model on small patches from the pretrained LLM's weight matrices. The model learns to recover a continuous full-precision patch from a low-bit observation.

Later, RTN-quantized patches are fed to the diffusion model to reconstruct a continuous weight matrix. The reconstructed matrix guides rounding; it is not the final deployed weight matrix.

The prior comes from the pretrained model's own weight matrices.
Reconstruction-guided rounding with a position-dependent tolerance near the midpoint
STEP 2

Start from RTN, then reconsider uncertain weights.

After scaling, each weight still has to be assigned to an integer level. ReRound starts from the RTN choice. If the reconstructed weight points to the opposite adjacent level, ReRound considers changing that assignment.

The blue tolerance curve allows more reconstruction guidance near the midpoint and less near the integer levels. If the reconstruction is too far from the scaled weight, ReRound keeps the RTN assignment.

Near the midpoint, the reconstruction can provide useful guidance for the rounding direction.
Candidate selection by comparing singular values with the original full-precision weight matrix
STEP 3

Choose the candidate that best preserves the weight matrix.

Different tolerance settings produce a small set of candidate quantized matrices. ReRound dequantizes each candidate only for comparison and checks how closely its leading singular values match those of the original full-precision matrix.

The candidate with the smallest spectral gap is kept. This selection uses model weights only—no calibration text or task examples.

The final candidate is the one with the smallest spectral discrepancy from the original matrix.
04 · RESULTS

Results across eight small LLMs.

Across eight small LLMs, ReRound improves matched group-wise RTN at both W4 and W3.

W4
+1.3 pts

Largest W4 gain over matched group-wise RTN among the eight small LLMs.

W3
+1.6 pts

Largest W3 gain over matched group-wise RTN among the eight small LLMs.

Accuracy gain of ReRound over matched group-wise RTN across eight small LLMs at W4 and W3
Gain over matched group-wise RTN across eight small LLMs. Positive values indicate improvement over the RTN baseline.
05 · CITATION

BibTeX