badge icon

This article was automatically translated from the original Turkish version.

Blog
Blog
Avatar
AuthorElif İnceNovember 6, 2026 at 4:00 AM

Quantization (1/3): Memory, Bits, and Breakpoint

T3 AI+2 More
Quote
T3 AI Developer
T3 AI Developer

Key findings: In this article, six dense models from the Qwen3 family were quantized using six different quantization levels and evaluated on Turkish benchmark tests: first, the reliability of the measurement pipeline (E0), followed by the impact of model size and bit width (E1). The findings are summarized as follows:

  • 8-bit: Models quantized to 8-bit precision exhibited loss levels close to noise compared to the BF16 (bfloat16) version; the only 8-bit condition where the accuracy difference (Δ) from BF16 had a 95% confidence interval (CI) excluding zero was Qwen3-8B FP8 (−0.87 points).
  • 4-bit: Results depended on the method. GPTQ-W4 emerged as the first breakpoint for the 8B and smaller models (losses between −2.25 and −3.07 points; breakpoint defined as at least a 2-point loss with CI excluding zero); GGUF Q4_K_M remained within −0.67 to +0.11 points across models from 1.7B to 32B (−1.79 points at 0.6B).
  • 3-bit: All six models exceeded the breakpoint threshold at approximately 3-bit (Q2_K).
  • Language: On Belebele, Turkish loss averaged 0.7 points higher than English; the difference was statistically significant in only five conditions (1.7B, 4B, and 8B Q2_K; 8B FP8; and 1.7B GPTQ-W4 in reverse direction) and not significant at 14B and 32B.


Limitations: The findings are limited to six dense models from the Qwen3 family, evaluated on Turkish multiple-choice benchmarks and specific software versions.


This article is the first in a three-part series titled "Quantization." Throughout the series, the Qwen3 family is evaluated using Turkish data and Turkish benchmarks: the second article examines method selection, calibration language, memory budget, and speed; the third article addresses mixed precision, KV cache, and language model head (lm_head) quantization.

What Is Quantization?

Aynı istem, iki cevap (BF16 ve GGUF kuant)

Same Prompt, Two Responses (BF16 and GGUF Quant) (Illustration: Elif İnce)

One of the two answers shown in the image comes from a model whose weights have been compressed to approximately one-third of their original size. Which one originates from the quantized model will be explained at the end of the section "What Does Quantization Do to a Weight?"


A common challenge in AI applications is maximizing performance from available hardware. For model providers, the goal is to serve as many users as possible on the same hardware. In setups where models are hosted on private servers to prevent customer data from leaving the organization, each additional GPU increases monthly costs. In mobile applications, models may need to run locally without internet connectivity. When running an open-source coding model on a personal laptop, GPU memory may range between 8 and 24 GB.


Acquiring more GPUs increases cost, while selecting a smaller model introduces quality loss.


Quantization addresses this issue by modifying the model itself rather than the hardware: it stores weights and, if needed, intermediate values during inference using fewer bits. The method is now widely used in AI applications. For instance, some of the largest open-weight models, such as Kimi K3, publish their weights directly in 4-bit MXFP4 format;【1】 DeepSeek-V3 performed the majority of its training computations using FP8 mixed precision.【2】 The rationale behind publishing a model from the start in 4-bit will be revisited at the end of the final article in this series.


In this article, we first calculate why a large model cannot fit into memory and where the memory is consumed. We then examine step by step how quantization affects weights and the challenges involved in the process. Finally, we compress six Qwen3 models ranging from 0.6 billion to 32 billion parameters using six different quantization levels and measure how model size and bit width affect Turkish accuracy.

Why Doesn’t a Giant Model Fit in Memory?

The memory required for a model’s weights can be estimated with a simple multiplication: number of parameters × bytes per parameter. Today, most models are released in BF16 (bfloat16) format, where each parameter occupies 2 bytes. This same multiplication explains why the largest models cannot fit on conventional hardware.


Table 1: Weight Memory Requirements and Number of H100 GPUs Needed for Specific Large Language Models in BF16, FP8, and Approximately 4-bit Precision (Weights Only)


Çeşitli modellerde ağırlık belleği için gereken H100 sayısı

Number of H100s Required for Weight Memory in Various Models (Diagram: Elif İnce)


The first four models in the table use a Mixture-of-Experts (MoE) architecture: each token activates only a few experts. Memory requirements are determined by the total number of parameters, while computational load per token is determined by the number of active parameters. Although Kimi K3 has 2.8 trillion parameters, only about 104 billion are activated per token; however, all 2.8 trillion must remain in memory.


The table lists only the weights and excludes KV cache and activation memory. The ~4-bit column assumes an average of 4.25 bits for MXFP4, which uses an 8-bit scale for 32 values. The first four models require between 19 and 70 H100 GPUs in BF16 format; reducing precision to 4-bit lowers this number to roughly one-quarter. Even with four H100 GPUs (320 GB total), none of these four models can be loaded in 4-bit precision. Therefore, experiments were conducted using the Qwen3 family, which fits on a single GPU.

Where Does Memory Go?

Bellek kalemleri ve büyüyen KV cache payı (temsili)

Memory Pens and Growing KV Cache Share Representation (Illustration: Elif İnce)

A running model’s memory is divided into three main components. Weights occupy memory when the model is loaded and remain constant. Key-Value (KV) cache consists of key and value vectors stored at each layer for every token the model has processed so far. It grows linearly as context length increases and the number of concurrent users rises. Activations are temporary values generated during the computation of a layer and are freed once the computation is complete. Additional small fixed allocations such as frame buffers (CUDA context) and communication buffers in multi-GPU setups are also included.


The weight memory requirement has been shown above. The size of the KV cache for Qwen3-32B can be calculated as follows: 64 layers, 8 KV heads per layer, 128 dimensions per head, and BF16 storage.【8】 In this model, each token occupies 0.25 MiB. A single conversation of 32,000 tokens requires 8 GiB; 16 concurrent conversations of the same length require 128 GiB, approaching twice the size of the model’s BF16 weights. In long-context services, the KV cache can dominate memory usage over the weights themselves. This situation brings the compression of non-weight structures into focus.

Why Does Quantization Accelerate the Model?

During decoding, when the model generates each new token, it reads all weights from memory once. For a single user, this read operation takes longer than the computation itself; thus, the theoretical upper bound of generation speed is approximately memory bandwidth ÷ model size. An H100 has a bandwidth of approximately 3.35 TB/s. For a 66 GB BF16 Qwen3-32B model, this yields an upper limit of about 51 tokens per second; for its 4-bit quantized version of 17.4 GB, the limit rises to approximately 192 tokens per second. These figures represent only theoretical upper bounds. Quantization therefore not only reduces the model’s memory footprint but also raises the theoretical upper bound of decoding speed. In MoE models, the calculation uses active parameters rather than total parameters.

Deep Dive: KV Cache and Bandwidth Formulas

Since the KV cache stores one key and one value tensor per layer, the formula begins with a factor of 2. In models using grouped-query attention, the number of KV heads is smaller than the number of attention heads, reducing the cache proportionally; in Qwen3-32B, there are 8 KV heads for 64 attention heads.



This limit applies to a single user. When multiple requests are processed in batches, weights are read once and reused across many tokens, shifting the bottleneck from memory bandwidth to computational capacity.


Keep in mind: Weight memory is the product of the number of parameters and the number of bits per parameter. Reducing the number of bits per parameter decreases both the required number of GPUs and the amount of data read from memory per token.

What Does Quantization Do to a Weight?

A number format determines which numbers can be represented using a limited number of bits. Floating point numbers consist of three parts: a sign bit, an exponent that determines the magnitude, and a mantissa that determines precision within that magnitude. The number of bits allocated to the exponent defines the representable range, while the bits allocated to the mantissa define the granularity within that range. Integer formats, by contrast, represent values on an equally spaced grid.


Table 2: Bit Structure and Typical Usage of Number Formats

Sayı formatlarında bit paylaşımı

Bit Sharing in Number Formats (Drawing: Elif İnce)


As the number of bits decreases, either the representable range or the granularity within the range is lost. Low-bit formats compensate for this loss with a scale: numbers are first divided by a common factor and then mapped onto a smaller grid.

How Does a Weight Grid Get Mapped?

The simplest method is the absmax mapping, which is based on the absolute maximum value. The scale is determined by dividing the largest absolute value in the row by the largest integer representable in the target format: 127 for INT8 and 7 for INT4. Each weight is divided by this scale and rounded to the nearest integer. During model inference, these integers are multiplied back by the scale; this process is called dequantization. The difference between the returned value and the original weight is the rounding error.


Throughout this section, we will use the same eight-weight row: [0.21, −0.47, 0.08, 0.33, −0.15, 0.62, −0.29, 3.10]. The final value is a large outlier, typical of those found in real models. This row is first quantized to INT8 and then to INT4.


Table 3: Mapping of the Example Row to INT8 and INT4 (Symmetric Absmax)

The table is illustrative; values are calculated using symmetric absolute maximum and mean squared error (MSE). In INT8, each weight returns with a maximum error of 0.012. In INT4, three out of eight weights are rounded to zero, and the remaining small weights converge to approximately ±0.443, increasing the error by about 330 times.

How Does a Single Outlier Corrupt an Entire Row?

Izgaraya yuvarlama: INT8 ve INT4

Quantization to Grid: INT8 and INT4 Methods (Illustration: Elif İnce)

The primary cause of the INT4 result is the value 3.10. Because this single value determines the scale, the grid step increases to 0.443, and the rest of the row is rounded using this step. If the first seven weights of the same row were quantized to INT4 without the outlier, the scale would drop to one-fifth (0.089), reducing the error of these seven weights by a factor of 37.


The same phenomenon occurs in real models. In large language models, systematically very large values appear in certain channels; these outliers are particularly prominent in activations and become more frequent as the model grows.【12】 Even though they are rarer in weights, a single large value reduces the precision of all values sharing the same scale.

How Does Dividing the Scale into Smaller Groups Limit the Outlier Effect?

One solution is to divide the scale into smaller segments. The fineness of this partitioning is called granularity. In per-tensor quantization, a single scale is used for the entire tensor. In per-channel quantization, each channel has its own scale. In per-group quantization, a channel is divided into groups—for example, groups of 128 weights—and each group has its own scale stored separately. MXFP4 applies the same idea at the hardware level using blocks of 32 values.【13】 In the example, the row is divided into two groups of four values each.


Table 4: INT4 Quantization of the Example Row in Groups of Four


Granülerlik düzeylerinde ölçek paylaşımı

Scale Sharing at Granularity Levels (Drawing: Elif İnce)

The first group returns nearly error-free, and the outlier's impact is limited to the second group. The row's OKH value decreases from 0.0177 to 0.0097. The small weights in the second group again disappear.


Granularity has a cost: one scale is stored for each group. With 128-element groups and 16-bit scales, the average bits per weight amount to 4 + 16/128 = 4.125 bits, whereas MXFP4's 32-element blocks consume 4.25 bits. As group size decreases, error reduces but the number of stored scales increases.


The answer to the problem posed at the beginning is as follows. Compressed model A produces answer A: Qwen3-32B's GGUF Q4_K_M quantization (llama.cpp's format that retains most weights around 4 bits), reducing weights from 65.5 GB to 19.8 GB. Answer B comes from the BF16 model.


The two answers diverge noticeably at the word level (approximately 51% of words differ); however, the first sentence is identical, their lengths are nearly equal (34 and 36 words; 87 and 89 tokens), and both convey the same two ideas: accumulated unnecessary files and programs alongside insufficient hardware. The difference lies in word choice and the direction of the final sentence: A states that read-write speed drops when the disk fills up; B asserts that memory and processor capacity remain limited. Almost all weights have changed due to rounding errors of the type seen in this section; yet the answer is preserved. This is merely a single example and not proof; there exists a threshold of robustness that will be sought through experimentation.


Dive Deeper: Symmetric and Asymmetric Quantization

The quantization in this section is symmetric: the grid distributes evenly around zero. In tensors with values skewed in one direction (e.g., activations after ReLU), half the grid goes unused. Asymmetric quantization adds a zero-point to shift the grid to align with the actual range of the data.【14】



Asymmetric quantization uses the range more efficiently; however, it requires storing a zero-point alongside each scale and makes multiplication slightly more expensive. Since weights are typically distributed around zero, symmetric quantization is common in weight quantization.


Key Takeaway: Quantization maps each weight to the nearest point on a scaled grid. Because the grid step is determined by the largest value in the row, a single outlier can distort the entire row. Dividing the scale into smaller groups confines this damage within each group.

Experiments

Deneylerin bağlantısı

Connection of Experiments (Illustration: Elif İnce)

The concepts described in previous sections were evaluated on six dense models of the Qwen3 family using Turkish benchmark tests. The primary metric is task accuracy; English was measured only for comparison. The central questions addressed are:

  • E0 – Measurement Pipeline Reliability: What is the noise floor of the measurement pipeline on Qwen3-8B, are the two KL paths consistent, and does the ranking of quantization types by accuracy and answer distribution (KL divergence) follow the expected order?
  • E1 – Model Size and Bit Width: How does accuracy in Turkish degrade as bit width decreases, and are larger models more resilient to quantization than smaller ones?

For E1, all quantizations of the six models were generated and measured. Prior to this, E0 verified the reliability of the measurement pipeline.


In the diagram, the orange EN (English) badge indicates experiments involving English measurements; gray boxes denote experiments covered in other parts of the series. All runs were performed on a single H100 (80 GB), totaling 10.7 GPU-hours.

Common Setup

All experiments share the same models, quantizations, and evaluation metrics.


Table 5: Common Setup: Models, Quantizations, Calibration, and Metrics


Quantizations: The quantizations in the table come from three families. GGUF is the format of llama.cpp;【24】 the number in the name indicates the approximate bit count (Q8_0 is 8 bits, Q4_K_M is approximately 4 bits). Types with a K in their name do not store all tensors at the same bit level; they preserve some of the most sensitive tensors at higher bit precision, which is why the measured average bit rate of Q4_K_M exceeds 4. imatrix derives from calibration data—sample texts passed through the model during quantization—which tensors are most important and directs rounding accordingly. GPTQ weights are rounded sequentially, and at each step the error introduced is compensated by adjusting weights that have not yet been rounded.【25】 FP8 is an 8-bit floating-point format that compresses both weights and activations. The notations W8A8 and W4A16 specify the bit depth of weights (W) and activations (A): in W4A16, weights are stored in 4 bits while computation is performed in 16 bits.


Measurement: The primary metric is the difference in accuracy between the quantized model and its BF16 counterpart. Since each quantized model produces the same answers, this difference is reported as a 95% confidence interval using a resampling (bootstrap) method with paired questions. Answer flip rate measures how often correct answers from the BF16 model become incorrect and incorrect answers become correct in the quantized version; since the mean accuracy may remain unchanged while a significant portion of answers change, this metric is interpreted alongside accuracy.【26】 KL divergence measures how far the probability distribution of each token in the quantized model deviates from that of the BF16 model; it is more sensitive because it does not require correct answers and is reported as an additional metric. The breakpoint of a model is the first quantization level, ordered by measured average bits, at which a loss of at least 2 points is observed in the average of Belebele-tr and ARC-tr, with a confidence interval excluding zero.

Deep Dive: Setup Details

The Turkish calibration dataset is randomly sampled from the OzenliDerlem subset of BellaTurca, which consists of carefully cleaned general web texts.【27】 The number of calibration samples follows the original GPTQ study’s use of 128 examples.【28】 KL measurement is performed on a separate dataset of approximately 100,000 tokens from Havadis news texts.【29】

The Turkish and English datasets of Belebele consist of identical questions based on the same paragraphs; the paragraphs are drawn from FLORES-200.【30】 This ensures comparable content across both languages. ARC-tr is the human-verified Turkish translation of the ARC-Challenge test set. In GPQA-tr, since the correct answer appears in the same column on every row, the options are shuffled using a fixed random seed. GPQA-tr is not measured for 0.6B and 1.7B models, as even in BF16 form their performance is near chance level; it is measured only for the 4B model, but the BF16 score is at chance level, so that column is displayed in gray.


Benchmarks are run using log-likelihood-based multiple-choice scoring; since the model does not generate text, Qwen3’s reasoning mode has no effect on results. GPTQ and FP8 quantizations run with their own native kernels in vLLM.【31】 GGUF quantization weights are loaded back into BF16 and evaluated on the same engine, ensuring all quantizations pass through the same measurement pipeline. This approach was verified on E0 to produce identical results to llama.cpp itself. The KL divergence of GGUF quantizations is computed directly using llama.cpp’s perplexity tool. The KL divergence of FP8 and GPTQ quantizations is calculated from vLLM’s token probabilities on the same Havadis dataset.

Pre-Control: Is the Measurement Tool Reliable?

Before presenting other results, the measurement pipeline itself was tested on the Qwen3-8B model. Three conditions were verified. First, when the BF16 model is run twice with different kernel settings, the flip rate must be close to zero; this rate forms the baseline noise level for all graphs, and if it is small, the measurement pipeline is reproducible, and any differences smaller than this band are considered noise. Second, the KL divergence computed from GGUF weights reloaded into BF16 must match the KL divergence computed by llama.cpp directly; a large discrepancy would indicate that GGUF results should be interpreted as “reloaded model” outputs. Third, accuracy loss and KL divergence must increase monotonically in the sequence Q8_0 → Q4_K_M → Q3_K_M → Q2_K; any deviation from this order indicates an error in measurement or quantization and is explicitly noted alongside the results.


Findings: When the Qwen3-8B BF16 model was measured twice using default and enforce_eager kernel settings, the average of Belebele-tr and ARC-tr changed by +0.13 points (95% CI [−0.21, +0.47]); the flip rate was 0.7%. This band (±0.47 points; upper bound of CI) is drawn as a gray band in subsequent graphs. In path consistency, the two KL measurement paths for all four GGUF quantizations were closely aligned: for Q8_0, 0.0019 and 0.0019 (difference 0.5%); for Q4_K_M, llama.cpp KL was 0.0370 and reloaded weights gave 0.0359 (difference 3%); for Q3_K_M, 0.1305 and 0.1287 (difference 1.4%); for Q2_K, 0.4077 and 0.4048 (difference 0.7%). Loss and KL increased monotonically along the sequence Q8_0 → Q4_K_M → Q3_K_M → Q2_K (Δ −0.06, −0.67, −1.48, −7.57 points; KL values as above). All three conditions were satisfied; the pre-control passed.


Table 6: Measurement Baseline (E0, Qwen3-8B)


The Flip rate was calculated jointly over Belebele-tr and ARC-tr. The "—" symbol is used in the reopened KL column for BF16 and FP8 because they are not GGUF formats; FP8 is outside the order check. The largest KL difference (3%) is in Q4_K_M and may stem from llama.cpp rounding activations to 8 bits on CUDA, though this has not been tested.


Result E0: The measurement chain is reliable: baseline noise is ±0.47 points, the two KL paths are consistent, and loss and KL are in correct order. Subsequent differences should be interpreted within this band.

Does Model Size Increase Resistance to Quantization?

Research Question: How does Turkish accuracy change as the number of bits decreases across six Qwen3 models, and is this change dependent on model size?


Setup: Six models × six quantizations and BF16 references in the common setup. For each, average bits, disk size, Belebele-tr, ARC-tr, and GPQA-tr accuracy, flip rate, and KL divergence were measured. Belebele-English was also measured; this allows comparison of loss in both languages across the same questions and model sizes.


Expectation: FP8 and 8-bit quantizations are expected to show minimal loss near the E0 baseline, with quality declining slowly from 8 to 4 bits, and loss accelerating sharply as bit count drops toward 3. 【32】 Smaller models are expected to suffer greater loss under the same quantization, with the breakdown point shifting to higher bit levels as model size decreases. Languages are not equally affected by quantization; 【33】 Belebele-tr is expected to show greater loss than Belebele-en.

How to Read the Results?

  • If loss curves approach zero in a regular pattern as model size increases, larger models are more resistant to quantization, confirming the warning against aggressive compression of small models in this family.
  • If curves overlap regardless of model size, resistance depends primarily on bit count, not size; the breakdown point occurs at the same bit level across all sizes.
  • If accuracy remains unchanged while flip rate increases, the model achieves the same score by correctly answering different questions; KL divergence further quantifies this deviation.
  • In GPQA-tr, if the BF16 score remains near chance level, the benchmark cannot measure quantization loss; such cells are shown in gray in the table.
  • If the loss gap between Turkish and English in Belebele widens in small models and narrows in large ones, language fragility is tied to model capacity; if no gap emerges, this negative result is also reported.


Model başarısı, koşul ve ortalama ölçülen bit

Model Performance, Condition, and Average Measured Bits (Drawing: Elif İnce)

Findings: For each of the six models, BF16 reference was compared against six quantizations; the metric is the difference between the average of Belebele-tr and ARC-tr and the BF16 baseline. Loss at 8 bits is at noise level: Q8_0 ranges from −0.18 to +0.51 points across all six models, with no GA excluding zero; FP8 ranges from −0.87 to +0.17, with only one 8-bit cell excluding zero: Qwen3-8B FP8 (−0.87 points, GA [−1.66; −0.11]).


The breakdown point in the comparative performance visualization is defined according to the Measurement definition in the Setup section.


Results at 4 bits and below diverged by method. GGUF Q4_K_M remained between −0.67 and +0.11 points across models from 1.7B to 32B; only at 0.6B did it exclude zero (−1.79 points, GA [−3.33; −0.24]), but it did not reach the −2 point breakdown threshold. Q3_K_M remained lossless (0.00) at 14B, −0.40 at 32B, −1.29 at 4B, and −1.48 at 8B; however, it dropped to −3.03 at 1.7B and −5.63 at 0.6B. Q2_K exceeded the breakdown threshold in all models: −2.33 at 32B, −2.74 at 14B, −7.57 at 8B, −8.40 at 0.6B, −9.21 at 4B, and −11.58 at 1.7B.


Kuant × model × benchmark ısı haritası

Quant × Model × Benchmark Heat Map (Drawing: Elif İnce)

Breakdown Point: According to the measured average bit order (primary criterion), the first breakdown occurred at GPTQ-W4 for 0.6B, 1.7B, 4B, and 8B models (−2.67, −2.83, −3.07, and −2.25 points respectively; all GAs excluded zero). At 14B and 32B, GPTQ-W4 did not break down (−0.35 and −0.07), and the first breakdown occurred with Q2_K; the nominal bit order yielded the same result. Thus, the breakdown point shifted to lower bit levels as model size increased. However, bit count alone is not explanatory: although GPTQ-W4 has a measured average bit range of 5.3–5.9 and Q4_K_M of 4.2–4.95, GPTQ-W4 produced 0.9 to 2.6 points more loss than Q4_K_M in models of 8B and below.


Size Effect: Loss decreased with increasing model size under the same quantization: Spearman correlation between size and Δ was +0.77 for GPTQ-W4, +0.77 for Q4_K_M, +0.89 for Q3_K_M, and +0.83 for Q2_K (positive: larger models lost less). However, the reduction was not uniform: Q2_K at 1.7B (−11.58) lost more than at 0.6B (−8.40), and GPTQ-W4 at 4B (−3.07) lost more than at 0.6B (−2.67). No clear relationship with size was observed for 8-bit quantizations.


Flip Rate and KL. Answers changed while accuracy remained constant: in 15 of 36 cells, |Δ| was below the noise band (0.47 points), yet flip rate exceeded the noise baseline (0.7%). For example, at 14B Q3_K_M, Δ = 0.00 but flip rate was 7.9%; at 1.7B Q4_K_M, Δ = −0.23 but flip rate was 9.7%. Flip rate ranged from 0.7% to 2.7% for Q8_0, 3.8% to 11.5% for Q4_K_M, and 12.5% to 28.8% for Q2_K. KL divergence followed the same pattern and decreased with model size: for Q4_K_M, 0.122 at 0.6B, 0.037 at 8B, and 0.024 at 32B; for Q2_K, 1.32, 0.41, and 0.31. KL divergence for FP8 was 0.0016–0.0072 (slightly above Q8_0), and for GPTQ-W4, 0.031–0.162. Despite its higher average bit count, GPTQ-W4 produced 1.3 to 1.6 times higher KL divergence than Q4_K_M in the same model size, consistent with the accuracy gap.


Belebele-en was measured only on these six models; GAs were calculated by resampling the same rows across both languages.

Belebele veri setinde kuantizasyonun Türkçe ve İngilizce üzerindeki etkisi

The Effect of Quantization on Turkish and English in the Belebele Dataset (Illustration: Elif İnce)


Two Languages: Across the six models, in 27 of 36 model-quantization conditions, Turkish lost more than English in Belebele; the average difference was +0.7 points. The GA excluded zero for only five conditions: 4B Q2_K (+6.56 points), 8B Q2_K (+5.00), 1.7B Q2_K (+4.5, GA [+2.3; +6.7]), 8B FP8 (+1.44; here English accuracy slightly exceeded BF16), and inversely 1.7B GPTQ-W4 (−3.9, GA [−6.2; −1.6]; English lost more). At 0.6B, the difference was small (+0.2 to +1.6 points; Turkish accuracy was near chance, possibly influenced by baseline effects). At 14B, the difference ranged from −0.1 to +0.8 points; at 32B, from −0.2 to +0.5 points. Turkish’s additional loss was most pronounced at the lowest bit quantizations (Q2_K) and small-to-medium models (1.7B–8B); it was not evident at 14B and above.


GPQA-tr: In the GPQA-tr dataset with 546 questions, confidence intervals range from 4 to 7 points in width (for example, 32B GPTQ-W4 −3.66, GA [−7.33; 0.00]) and differences are small. The Qwen3-4B BF16 score of 27.1% is close to chance level, so this column is shown in gray; GPQA-tr was not measured for 0.6B and 1.7B.


Table 7: Qwen3-8B and Qwen3-32B Results

All six models are included in the appendix.


Accuracies in % (Bel: Belebele). Bold Δ: excludes 95% GA zero. Bit: measured average bit. KL: Havadis; GGUF: llama.cpp, FP8; GPTQ-W4: vLLM logprobs.


Result E1: The loss at 8 bits is close to the noise floor (Qwen3-8B FP8 is the only 8-bit condition excluding GA zero, −0.87 points). The breakpoint occurs at GPTQ-W4 for models ≤8B (approximately 5.3–5.9 bits), and at Q2_K for 14B and 32B models (approximately 3 bits). Turkish additional loss is only significant in five conditions (details: Two languages).


Table 8: Summary of Experiments

Discussion and Limitations

Bit limit: Post-training quantization introduces rounding noise not encountered during training. In this study, noise remained at the level of measurement error at 8 bits, varied by method at 4 bits, and exceeded the breakpoint threshold for all six models at approximately 3 bits. Findings in the literature support this trend: GPTQ shows that loss remains small at 4 bits but increases significantly at 3 bits, especially for smaller models;【34】 a large-scale study examining the 3–8 bit range found 4 bits to be the optimal point in most cases, based on total model bit count and zero-shot accuracy.【35】 Calibration and mixed precision can reduce this loss only up to a certain point; as weights are learned without quantization constraints, this limit is expected to be shiftable by methods but not eliminable.


Scope: Findings apply only to the six dense models of the Qwen3 family, Turkish (English only on Belebele), multiple-choice benchmarks, and the software versions used. Other formats such as AWQ, NF4, and EXL3, MoE models, and quantizations below 2 bits are excluded; the same ranking should not be assumed in these cases.


Measurement limitations: The quality of open-ended text generation and human evaluation were not measured. Multiple-choice scoring does not capture Turkish affixes or alignment-related formal errors; furthermore, automatic metrics may underestimate the impact of quantization on users compared to human evaluation.【36】 ARC-tr and GPQA-tr are translation datasets; translation errors add noise independent of the model.

Frequently Asked Questions

If Accuracy Does Not Change, Does the Quantized Model Produce the Same Answers as BF16?

It does not always produce the same answers. While average accuracy remains similar, some correct answers may become incorrect and some incorrect answers may become correct.【37】 In E1, for 15 of the 36 conditions, the accuracy difference remained within the noise band while the flip rate exceeded the noise floor; for example, in 14B Q3_K_M, accuracy remained unchanged but 7.9% of answers flipped. Therefore, accuracy must be evaluated alongside flip rate and KL divergence.

Why Is FP8 Often Considered Safe?

FP8’s floating-point structure preserves a wide dynamic range; additionally, hardware support for FP8 is available on Ada Lovelace and Hopper architectures and later GPUs (not present on Ampere or earlier); prior evaluations showed that FP8 W8A8 preserves quality almost entirely in large models.【38】 In E1, FP8 did not exceed a 0.87 point loss in Turkish for any of the six models (GA was the only exception, at zero for 8B only).

Conclusion

Example Input: One of the two answers in the input came from a model whose weights were compressed to approximately one third; in this single example, the two answers were content-wise similar. The mechanisms described in this paper also operated in this example: weights were rounded to a grid, outlier effects were confined within their own group, and sensitive tensors were preserved with additional bits.


Different Quantizations: 8-bit quantizations exhibited noise-level losses compared to BF16; the only condition where GA deviated from zero was Qwen3-8B FP8 (−0.87 points). Results at 4-bit depend on the method. GGUF Q4_K_M remained below the breaking threshold for all six models (largest loss: −1.79 points at 0.6B); GPTQ-W4 was the first to break the threshold among 8B and smaller models. At approximately 3-bit (Q2_K), all six models exceeded the breaking threshold (smallest loss: −2.33 points at 32B); for 14B and 32B, this was the first quantization to break the threshold. Loss decreased as model size increased, but this reduction was not consistent.


Language Difference: In Belebele, Turkish loss averaged 0.7 points higher than English; the difference was significant in only five conditions and not noticeable at 14B and 32B.


Measurement Reliability: Results were obtained using the measurement pipeline tested in E0 (noise band ±0.47 points, two KL paths aligned, loss and KL in expected order); all results for the six models are provided in the Appendix.


Limitations: Findings are limited to six dense models from the Qwen3 family, Turkish multiple-choice benchmarks, and specific software versions; AWQ, NF4, EXL3, MoE models, and quantizations below 2-bit are outside the scope.


The second article in this series will focus on how these quantizations are generated and how to select among them: it will examine how calibration-based methods work, how the language of calibration data affects Turkish performance, and which model-quantization combinations perform best under 8, 16, and 24 GB memory budgets; the speedup achieved by quantization will also be measured.

Appendix: All Results

The BF16 reference and six quantizations for the six Qwen3 models are listed below.


Table 9: BF16 Reference and Six Quantizations for the Six Qwen3 Models (All Results)


Accuracies % (Bel: Belebele). Thick Δ: excludes 95% GA zero; gray cell: not measured (GPQA-tr 0.6B and 1.7B) or BF16 score at chance level (GPQA-tr 4B). Bit: measured average bit. KL: from Havadis; GGUF: from llama.cpp; FP8 and GPTQ-W4: from vLLM logprobs.

Bibliographies

Bandarkar, L., D. Liang, B. Muller, et al. “The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants.” *arXiv* 2024. Accessed October 14, 2026.https://arxiv.org/abs/2308.16884

Clark, P., I. Cowhey, O. Etzioni et al. “Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge.” *arXiv* 2018. Accessed October 9, 2026.https://arxiv.org/abs/1803.05457

DeepSeek-AI, A. Liu, B. Feng, et al. “DeepSeek-V3 Technical Report.” *arXiv* 2024. Accessed October 14, 2026.https://arxiv.org/abs/2412.19437

DeepSeek-AI. "DeepSeek-V4 Pro." Hugging Face. Accessed October 1, 2026.https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro

Dettmers, T. and L. Zettlemoyer. “The Case for 4-bit Precision: k-bit Inference Scaling Laws.” *arXiv* 2022. Accessed October 4, 2026.https://arxiv.org/abs/2212.09720

Dettmers, T., A. Pagnoni, A. Holtzman, and L. Zettlemoyer. “QLoRA: Efficient Finetuning of Quantized LLMs.” *arXiv* 2023. Accessed October 6, 2026.https://arxiv.org/abs/2305.14314

Dettmers, T., M. Lewis, Y. Belkada, and L. Zettlemoyer. “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale.” *arXiv* 2022. Accessed September 30, 2026.https://arxiv.org/abs/2208.07339

Dutta, A., S. Krishnan, N. Kwatra, and R. Ramjee. “Accuracy is Not All You Need.” *arXiv* 2024. Accessed October 11, 2026.https://arxiv.org/abs/2407.09141

EleutherAI. “Language Model Evaluation Harness.” GitHub. Accessed October 13, 2026.https://github.com/EleutherAI/lm-evaluation-harness

Frantar, E., S. Ashkboos, T. Hoefler, and D. Alistarh. “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.” *arXiv* 2023. Accessed October 11, 2026.https://arxiv.org/abs/2210.17323

Jacob, B., S. Kligys, B. Chen et al. "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference." *arXiv* 2017. Accessed October 10, 2026.https://arxiv.org/abs/1712.05877

Kurtic, E., A. Marques, S. Pandit et al. “‘Give Me BF16 or Give Me Death’? Accuracy-Performance Trade-Offs in LLM Quantization.” *arXiv* 2025. Accessed October 8, 2026.https://arxiv.org/abs/2411.02355

Kwon, W., Z. Li, S. Zhuang, et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention." *arXiv* 2023. Accessed October 8, 2026.https://arxiv.org/abs/2309.06180

Marchisio, K., S. Dash, H. Chen, et al. “How Does Quantization Affect Multilingual LLMs?” arXiv, 2024. Accessed October 12, 2026.https://arxiv.org/abs/2407.03211

Micikevicius, P., D. Stosic, N. Burgess, et al. “FP8 Formats for Deep Learning.” *arXiv* 2022. Accessed October 8, 2026.https://arxiv.org/abs/2209.05433

Moonshot AI. “Kimi K3.” GitHub, 2026. Accessed October 12, 2026.https://github.com/MoonshotAI/Kimi-K3

Nagel, M., M. Fournarakis, R. A. Amjad, et al. “A White Paper on Neural Network Quantization." *arXiv* 2021. Accessed September 30, 2026.https://arxiv.org/abs/2106.08295

Open Language Data Initiative. “FLORES+.” Hugging Face. Accessed October 3, 2026.https://huggingface.co/datasets/openlanguagedata/flores_plus

Rein, D., B. L. Hou, A. C. Stickland, et al. “GPQA: A Graduate-Level Google-Proof Q&A Benchmark.” *arXiv* 2023. Accessed October 2, 2026.https://arxiv.org/abs/2311.12022

Rouhani, B. D., R. Zhao, A. More et al. “Microscaling Data Formats for Deep Learning.” *arXiv* 2023. Accessed October 15, 2026.https://arxiv.org/abs/2310.10537

Turkish NLP Suite. "BellaTurca." Hugging Face. Accessed October 5, 2026.https://huggingface.co/datasets/turkish-nlp-suite/BellaTurca

Turkish NLP Suite. "Havadis." Hugging Face. Accessed October 1, 2026.https://huggingface.co/datasets/turkish-nlp-suite/Havadis

Xiaomi MiMo Team. “MiMo-V2.6-Pro-RL.” Hugging Face. Accessed October 5, 2026.https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL

Yang, A., A. Li, B. Yang et al. “Qwen3 Technical Report.” *arXiv* 2025. Accessed October 14, 2026.https://arxiv.org/abs/2505.09388

Yıldız Teknik Üniversitesi Bilgisayar Mühendisliği Bölümü COSMOS Araştırma Grubu. “gpqa-extended_tr.” Hugging Face. Accessed October 13, 2026.https://huggingface.co/datasets/ytu-ce-cosmos/gpqa-extended_tr

Z.ai. “GLM-5.3.” Hugging Face. Accessed October 15, 2026.https://huggingface.co/zai-org/GLM-5.3

ggml-org. "llama.cpp." GitHub. Accessed October 6, 2026.https://github.com/ggml-org/llama.cpp

malhajar. "arc-tr-v0.2." Hugging Face. Accessed October 8, 2026.https://huggingface.co/datasets/malhajar/arc-tr-v0.2

vLLM Project. "LLM Compressor." GitHub. Accessed October 3, 2026.https://github.com/vllm-project/llm-compressor

Citations

  • [1]

    Moonshot AI. “Kimi K3.” GitHub. Accessed October 12, 2026.

  • [2]

    DeepSeek-AI, A. Liu, B. Feng, et al. “DeepSeek-V3 Technical Report.” arXiv, 2024, pp. 2-3. Accessed October 14, 2026.

  • [3]

    Moonshot AI, “Kimi K3,” GitHub, 2026. Accessed October 12, 2026.

  • [4]

    DeepSeek-AI, “DeepSeek-V4 Pro,” Hugging Face, Accessed October 1, 2026.

  • [5]

    Xiaomi MiMo Team, “MiMo-V2.6-Pro-RL,” Hugging Face, Accessed October 5, 2026.

  • [6]

    Z.ai, “GLM-5.3,” Hugging Face, Accessed October 15, 2026.

  • [7]

    A. Yang, A. Li, B. Yang et al. "Qwen3 Technical Report," *arXiv* 2025, 1. Accessed October 14, 2026.

  • [8]

    A. Yang, A. Li, B. Yang, et al. "Qwen3 Technical Report," 7.

  • [9]

    P. Micikevicius, D. Stosic, N. Burgess, et al. “FP8 Formats for Deep Learning.” *arXiv* 2022, 4. Accessed October 8, 2026.

  • [10]

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA: Efficient Finetuning of Quantized LLMs,” *arXiv* 2023, 7–9, Accessed October 6, 2026.

  • [11]

    B. D. Rouhani, R. Zhao, A. More et al. "Microscaling Data Formats for Deep Learning." *arXiv* 2023, 4-5. Accessed October 15, 2026.

  • [12]

    T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale.” *arXiv* 2022, no. 2. Accessed September 30, 2026.

  • [13]

    B. D. Rouhani, R. Zhao, A. More et al. “Microscaling Data Formats for Deep Learning,” 10.

  • [14]

    B. Jacob, S. Kligys, B. Chen et al. “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference,” *arXiv* 2017, 8. Accessed October 10, 2026.https://arxiv.org/abs/1712.05877M. Nagel, M. Fournarakis, R. A. Amjad et al. “A White Paper on Neural Network Quantization,” *arXiv* 2021, 5, 7–8. Accessed September 30, 2026.https://arxiv.org/abs/2106.08295

  • [15]

    Yang, A., A. Li, B. Yang, et al. "Qwen3 Technical Report," 15.

  • [16]

    vLLM Project, “LLM Compressor,” GitHub, Accessed October 3, 2026.

  • [17]

    ggml-org. "llama.cpp." GitHub, Accessed October 6, 2026.

  • [18]

    Turkish NLP Suite, “BellaTurca,” Hugging Face, Accessed October 5, 2026.

  • [19]

    L. Bandarkar, D. Liang, B. Muller, et al. “The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants,” *arXiv* 2024, 3-4, 7-8. Accessed October 14, 2026.

  • [20]

    malhajar “arc-tr-v0.2,” Hugging Face, Accessed October 8, 2026; P. Clark, I. Cowhey, O. Etzioni et al., “Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge,” arXiv 2018, Accessed October 9, 2026.

  • [21]

    Yıldız Teknik Üniversitesi Bilgisayar Mühendisliği Bölümü COSMOS Araştırma Grubu, “gpqa-extended_tr,” Hugging Face, Accessed October 13, 2026. ; D. Rein, B. L. Hou, A. C. Stickland, et al., “GPQA: A Graduate-Level Google-Proof Q&A Benchmark,” arXiv 2023, 1, Accessed October 2, 2026.

  • [22]

    EleutherAI, “Language Model Evaluation Harness,” GitHub, Accessed October 13, 2026.

  • [23]

    Turkish NLP Suite. "Havadis." Hugging Face, Accessed October 1, 2026.

  • [24]

    ggml-org. "llama.cpp."

  • [25]

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers,” *arXiv* 2023, 5-6, Accessed October 11, 2026.

  • [26]

    A. Dutta, S. Krishnan, N. Kwatra, and R. Ramjee, “Accuracy is Not All You Need,” *arXiv* 2024, 3, 6, 8-9, Accessed October 11, 2026.

  • [27]

    Turkish NLP Suite, “BellaTurca.”

  • [28]

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers,” 6.

  • [29]

    Turkish NLP Suite, “Havadis.”

  • [30]

    Open Language Data Initiative, “FLORES+,” Hugging Face, Accessed October 3, 2026.

  • [31]

    W. Kwon, Z. Li, S. Zhuang, et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention," *arXiv* 2023, 1-2. Accessed October 8, 2026.

  • [32]

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers,” 3; T. Dettmers and L. Zettlemoyer, “The Case for 4-bit Precision: k-bit Inference Scaling Laws,” *arXiv* 2022, 3-4, Accessed October 4, 2026.

  • [33]

    K. Marchisio, S. Dash, H. Chen, et al. "How Does Quantization Affect Multilingual LLMs?" *arXiv* 2024, 1, 6. Accessed October 12, 2026.

  • [34]

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers,” 9.

  • [35]

    T. Dettmers and L. Zettlemoyer, “The Case for 4-bit Precision: k-bit Inference Scaling Laws,” 9-10.

  • [36]

    K. Marchisio, S. Dash, H. Chen, et al. “How Does Quantization Affect Multilingual LLMs?” 8.

  • [37]

    A. Dutta, S. Krishnan, N. Kwatra, and R. Ramjee, “Accuracy is Not All You Need,” 3.

  • [38]

    E. Kurtic, A. Marques, S. Pandit, et al. “‘Give Me BF16 or Give Me Death’? Accuracy-Performance Trade-Offs in LLM Quantization,” *arXiv* 2025, 1-2, 5. Accessed October 8, 2026.

T3 AI
Türkiye's Large Language Model,is growing together with you.

Contents

  • What Is Quantization?

  • Why Doesn’t a Giant Model Fit in Memory?

    • Where Does Memory Go?

    • Why Does Quantization Accelerate the Model?

      • Deep Dive: KV Cache and Bandwidth Formulas

    • What Does Quantization Do to a Weight?

      • How Does a Weight Grid Get Mapped?

      • How Does a Single Outlier Corrupt an Entire Row?

      • How Does Dividing the Scale into Smaller Groups Limit the Outlier Effect?

      • Dive Deeper: Symmetric and Asymmetric Quantization

  • Experiments

    • Common Setup

      • Deep Dive: Setup Details

    • Pre-Control: Is the Measurement Tool Reliable?

    • Does Model Size Increase Resistance to Quantization?

      • How to Read the Results?

    • Discussion and Limitations

    • Frequently Asked Questions

      • If Accuracy Does Not Change, Does the Quantized Model Produce the Same Answers as BF16?

      • Why Is FP8 Often Considered Safe?

  • Conclusion

  • Appendix: All Results