Ozaki II supplement: INT8 range conditions and paper performance results

Back to the article · 中文版

This supplement collects range derivations, bit budgets, and the complete paper performance tables and stage breakdowns. The main article concentrates on preprocessing costs on H20.

1 The Ozaki II pipeline and the conditions for INT8 computation

Let $A\in\mathbb R^{p\times q}$, $B\in\mathbb R^{q\times r}$, and $C=AB$. Each output is a dot product of length $q$. In the square experiments, $p=q=r=n$. We use $s$ for the number of moduli and $k$ for a bit budget after scaling.

Ozaki II scales floating point inputs into integer values, represents each integer through $s$ small residues, computes the residue matrix products with $s$ INT8 GEMMs, and combines the results before restoring the original scale. Before discussing performance, we need to establish what each INT8 computation can reliably do. Paper, Section 3

1.1 From FP64 inputs to multiple INT8 GEMMs

Choose pairwise coprime moduli $m_1,\ldots,m_s$, with product $M=\prod_t m_t$. Choose diagonal powers of two, $D\in\mathbb R^{p\times p}$ and $E\in\mathbb R^{r\times r}$, to scale rows of A and columns of B.

CRT, the Chinese Remainder Theorem, combines the residue channels. Scaling must enforce $2c_{\max}<M$, where $c_{\max}$ bounds the absolute integer product $A'B'$. With the range and arithmetic requirements satisfied, the reconstruction target is $X=A'B'$.

Two checks are needed: whether the input residues fit in INT8, and whether a length-$q$ accumulation fits in INT32. They establish the reliability of the low precision core. See Section 2 of the main article for how fast and accu choose scaling.

1.2 Why the modulus limit is 256

INT8 represents 256 values in $[-128,127]$. Conventional nonnegative residues lie in $[0,m_t-1]$; for moduli larger than 128, some of them would not fit in signed INT8. Ozaki II uses centered residues:

Subtracting a modulus preserves the represented congruence. For $2\le m_t\le256$, this entire interval fits in INT8. At the boundary $m_t=256$, it is exactly $[-128,127]$, with the nonnegative residue 128 represented by the congruent value −128.

What is preserved here is the input's information modulo the selected modulus. The original integer $a'$ can be far larger than the INT8 range; each channel stores only one of its residues. Multiple moduli jointly represent the required information, so the complete integer $a'$ does not have to fit into eight bits.

The moduli must also be pairwise coprime. Subject to this constraint, a larger modulus contributes more range bits, $\log_2m_t$, helping reach the required total budget with fewer GEMMs. Paper, Section 3.1

1.3 The dot product length determines whether INT32 accumulation is safe

After fitting the inputs, we must prevent overflow during multiplication and accumulation. For the $t$-th modulus, one output is

The largest absolute INT8 value is 128, so an individual product has magnitude at most $128^2=2^{14}$. Signed INT32 represents $[-2^{31},2^{31}-1]$. Summing absolute values gives a conservative bound independent of cancellation:

The condition

therefore prevents overflow in the final result. Every intermediate partial sum is bounded by the same total of absolute values, so the guarantee does not rely on eventual cancellation and also covers different accumulation orders.

When integer products and sums stay within range, they are exact; they do not suffer from floating point significand rounding. This is the reason for using INT8 × INT8 → INT32: narrow inputs reduce hardware cost, while a wider accumulator preserves the dot product.

The strict inequality matters. At $q=2^{17}$, the worst-case bound reaches $2^{31}$, exceeding the largest positive INT32 value. This is a sufficient condition. Longer dot products do not always overflow, but the general guarantee no longer holds.

For larger $q$, the inner dimension can be partitioned into blocks with safe accumulations. Reduce each block result modulo the selected modulus, then combine those residues within a safe range. Overflow must be prevented before it occurs; applying a remainder operation after an INT32 overflow does not generally recover the correct residue. Paper, Section 3.1

1.4 How many GEMMs are required

Each modulus contributes a pair of residue matrices and one main GEMM, so $s$ moduli require $s$ main GEMMs. Each channel uses its own INT32 accumulation; increasing $s$ does not change the preceding dot product length into $sq$.

The choice of $s$ depends on the required total range budget:

A modulus close to 256 contributes almost eight bits. The paper's 16-modulus configuration supplies about 125 range bits in total, shared by the two inputs, accumulation headroom, and the signed range. With mild magnitude variation and suitable scaling and reconstruction precision, 16 main GEMMs can therefore plausibly support FP64-like accuracy. It is not a fixed count guaranteed to work for arbitrary inputs.

The two limits serve different purposes:

The suffix in fast-$s$ and accu-$s$ denotes the number of moduli used for the main products. Accu also performs a bound-estimation GEMM, which belongs to preprocessing.

2 Why constrain scaled entries by $2^k$

Bounding inputs provides a sufficient bound on their product. If $|a'|\le2^{k_A}$ and $|b'|\le2^{k_B}$, then

This lets us bound every output before computing the product. We also want scaling to be as large as possible: the integer grid corresponds to a step of $1/D_{ii}$ or $1/E_{jj}$ in the original scale. A larger scale generally loses less information during truncation.

For equal budgets on both inputs, Equation (18) in the paper is

Expanding the product of moduli gives

Ignoring the floor and the small final correction explains the trend:

The moduli supply range bits. The dot product length consumes accumulation headroom, the signed interval costs one bit, and the two inputs share the remaining budget. Adding a modulus close to eight bits contributes roughly four bits per input; multiplying $q$ by four costs roughly one bit per input.

Paper Figure 3. Source: Section 3.1. This is a fixed point range budget, not a guarantee that every element retains the same number of floating point significant bits.

This formula connects the modulus count to the input budget, but shared scaling remains sensitive to magnitude variation. Large and small entries in the same row can cause the small ones to be truncated away. A bound that better reflects the actual product can reduce unnecessarily conservative scaling.

3 Performance gains and where execution time goes

3.1 How much faster can INT8 emulation be than native FP64

The dense peak specifications in paper Table 1 provide useful context. Floating point columns use TFLOPS, and the integer column uses TOPS. These are specifications, not measured rates.

Source: paper Table 1 and Section 4.1. We do not apply GH200 specifications to H20.

Native DGEMM is direct double precision matrix multiplication. The following performance values are reproduced from the paper, in effective TFLOPS: the final product's work, $2n^3$, divided by elapsed time. They do not measure the total internal throughput of all $s$ integer products.

GH200 throughput across sizes and modulus counts (paper Table 3).

RTX 4090 throughput across sizes and modulus counts (paper Table 4).

Source: paper Tables 3 and 4. The suffixes 14–18 are modulus counts.

GH200 has strong native FP64 performance. For small matrices, OS II's extra work outweighs the benefit. At size 16384, fast-14 reaches $80.2/60.9\approx1.32\times$. RTX 4090 has much weaker native FP64 throughput; at size 8192, the corresponding ratio is $9.81/0.62\approx15.8\times$. The hardware's throughput ratio between low and high precision matters considerably.

For every matched size and modulus count in these tables, fast is faster than accu. Increasing the modulus count lowers throughput for both. However, choosing a method requires comparing time to satisfy the same error target. If accu reaches that target with fewer moduli, the ranking at equal $s$ is insufficient. The throughput tables alone do not establish accuracy thresholds for each input; a practical configuration still needs validation on the target data.

3.2 Where one Ozaki II call spends its time

Paper Figure 6, GH200, $n=1024,16384$, accu and fast.

Paper Figure 7, RTX 4090, $n=2048,8192$, accu and fast.

Original paper figures. Source: Figures 6 and 7. Each bar is normalized to 100%; the horizontal axis is the number of moduli.

In Figure 6, moving from size 1024 to 16384 on GH200 increases the share of main GEMMs, especially for fast. Preprocessing and reconstruction are more easily amortized at the larger size. Accu retains a heavier green stage because it includes an additional bound product.

Figure 7 shows a similar increase in the yellow share from size 2048 to 8192 on RTX 4090, while green conversion and red reconstruction remain substantial. This is consistent with Table 4's large speedup: the weak native FP64 baseline allows emulation to win despite its overhead. The two figures use different sizes, so bars at matching positions are not direct GPU comparisons at equal dimensions.

These plots show that work outside the main GEMMs matters. Percentages locate time, but do not explain its cause. A smaller green fraction can coexist with longer absolute preprocessing time. A GEMM inside green also means that complexity cannot be inferred from color alone.

For the operations and execution resources inside green, continue to Part II of the main article.