badge icon

This article was automatically translated from the original Turkish version.

Blog
Blog
Avatar
AuthorElif İnceNovember 16, 2026 at 4:00 AM

Quantization (3/3): Mixed Precision, KV Cache, and LM Head

T3 AI+2 More
Quote
T3 AI Developer
T3 AI Developer

Key findings: In this article, two experiments were conducted on the Qwen3 family using Turkish benchmark tests: a mixed-precision quantization recipe (E1) and quantization of the KV cache and lm_head (E2). The results are summarized as follows:

  • Mixed recipe (E1): A mixed recipe built from a tensor sensitivity map derived from Turkish text for Qwen3-4B achieved +0.9 points higher accuracy than the uniform Q3_K_M quantization (%95 GA [+0.1, +1.7]) and a 13% improvement in KL divergence on Havadis; it was indistinguishable in accuracy from Unsloth Dynamic and distinct from randomly distributed control recipes.
  • Transfer (E1): When the same rule was applied to Qwen3-8B, a similar trend was observed (+0.93; GA lower bound +0.0; single sample).
  • KV cache (E2): In llama.cpp, an 8-bit KV cache reduced memory usage by 47% and a 4-bit KV cache by 72%; the cost is approximately a 25% reduction in decode speed at 32K context. On 8B, recall remained practically preserved with q8_0 (−1.3 points) but dropped by a total of −4.0 points with q4_0 (−8.0 points at 32K); on 1.7B, the drop was larger (−4.0 and −12.0 points).
  • lm_head (E2): Reducing lm_head from Q8_0 to Q4_K reduced file size by 18% at 0.6B and 6% at 8B with no measurable accuracy loss, but increased KL divergence by 29–32%.


Limitations: The findings are limited to the dense models of the Qwen3 family, Turkish data, llama.cpp, and mostly single-seed runs.


This article is the third and final installment of the three-part "Quantization" series. Throughout the series, the Qwen3 family is evaluated using Turkish data and Turkish benchmarks: in the first article the memory challenge of large models, the impact of quantization on weights, and the effect of model size and bit count; in the second article method selection, calibration language, memory budget, and speed were addressed. This article builds upon the memory accounting from the first article, incorporating the concepts of scale, outliers, and grouping, as well as the common experimental setup; it is recommended to read the first article and the second article on calibration-based methods before proceeding.

Weights Are Not the Only Memory Component: Bit Budget and KV Cache

One of the two answers at the beginning of the first article in this series was generated by the Qwen3-32B Q4_K_M quantization. Although the name includes the number 4, not all tensors are stored in 4 bits—some are stored in 6 bits.【1】 Certain bits are allocated to tensors that suffer the most from quantization.


Weights are not the only component of memory. The model stores a key and a value vector for each token it has processed at every layer; this Key-Value (KV) cache for Qwen3-32B, with 32K context and 16 concurrent conversations in Brain Floating Point 16 (BF16) format, requires 128 GiB of memory—approximately twice the size of the model’s weights. In long-context services, the KV cache can account for a larger share of memory usage than the weights themselves.


This article first examines how the bit budget can be distributed across tensors, then explores how activations, the KV cache, the embedding table, and lm_head can be compressed. Two experiments are then conducted with the Qwen3 family: a tensor sensitivity map is generated from Turkish text to construct a mixed recipe of equivalent size, and the memory, speed, and quality costs of compressing the KV cache and lm_head are measured.

Should Every Layer Be Quantized the Same Way?

Not all tensors in a model are affected equally by quantization. Embedding tables, the language modeling head (lm_head), the first and last several layers, and some projections in the attention mechanism are typically more sensitive. Assigning different bit counts to different tensors is called mixed-precision. llama.cpp’s K-quant family is a common example: in the Q4_K_M quantization, some tensors are stored in the 6-bit Q6_K format.【2】 Unsloth’s “Dynamic” quantizations apply this idea at the layer level by preserving sensitive layers at higher bit precision.【3】


Mixture of Experts (MoE) models rely on mixed-precision for a different reason. Expert weights constitute a large portion of total parameters, but each token uses only a few experts; attention and shared layers, however, operate on every token. Therefore, in quantized versions of large MoE models such as Kimi K3 and DeepSeek V4 Pro, experts can be stored at lower bit precision while shared components are kept at higher bit precision.


The eight-weight example row from the first article is reconsidered: [0.21, −0.47, 0.08, 0.33, −0.15, 0.62, −0.29, 3.10]. When the row is quantized to INT4 using a single absmax scale, the 3.10 outlier inflates the scale, causing some smaller weights to round to zero. Dividing the row into two groups of four reduced the error, but could not prevent the smaller weights in the group containing the outlier from rounding to zero. Error was measured using mean squared error (MSE); instead of distributing bits equally across the entire row, the bit count was increased only for the group containing the outlier.


Table 1: Average Bits and Mean Squared Error (MSE) for the Example Row Under Different Schemes


The scheme that promoted only the group containing the outlier to INT8 achieved slightly lower error than the all-INT8 row, despite using only an average of 6 bits. The core principle of mixed-precision is to allocate the bit budget where error is most concentrated. Leaving the outlier alone in BF16 reduces error by approximately 37 times compared to all-INT4 (0.0177 → 0.00048); however, this is still about 10 times higher than promoting the entire group to INT8. This approach is a simplified analog of LLM.int8()’s idea of isolating outlier channels.【4】

How Is the Sensitive Layer Identified?

Katman ve tensör türüne göre şematik kuantizasyon sapması (Çizim: Elif İnce)

Schematic Quantization Error According to Layer and Tensor Type (Drawing: Elif İnce)

One direct method is sensitivity analysis: the model is kept at high precision, only one tensor at a time is reduced to low bit-width, and the deviation of the output from the original model is measured. The process is repeated for each tensor, and the deviations are visualized as a heatmap. The bit budget is then allocated starting with the most sensitive tensors. In E1, this analysis will be performed on Qwen3-4B using Turkish text to establish a heatmap-based recipe.

Dive Deeper: The Two Meanings of “Dynamic”

In quantization literature, the term “dynamic” is used for two different concepts. In PyTorch’s definition, dynamic quantization refers to pre-quantizing weights to INT8 and computing activation scales on-the-fly for each input; in static quantization, activation scales are fixed in advance via calibration.【5】 The “FP8 dynamic” quantizations in vLLM also follow this meaning.


Unsloth’s “Dynamic” quantizations, however, represent a mixed-precision recipe: the bit-width for each layer is determined on a per-model basis.【6】 These two concepts are independent; a model can be both mixed-precision and dynamically scaled in its activations.

In this article, the term “dynamic” is used exclusively in the first sense; the second meaning is referred to as “mixed precision.”


Keep in mind: Since precision varies across tensors, concentrating the bit budget on the most sensitive tensors results in lower error for the same overall size.

What Else Can Be Compressed Beyond Weights?

All examples so far have involved weight-only quantization. In this scheme, weights are stored at 4 or 8 bits, expanded to 16 bits during computation, and multiplied with activations at 16 bits; this setup is briefly denoted as W4A16. The gains are seen in memory usage and single-user generation (decode: the stage where tokens are produced one at a time) speed, because in this phase all weights are read from memory for each new token, and the bottleneck is typically this memory access.

Why Are Activations Harder to Compress Than Weights?

In the prefill stage, where the entire input is processed at once, and in multi-user services, the bottleneck shifts from memory to computation. To gain speed here, the matrix multiplication itself is performed at low bit-width: both weights and activations are quantized together (W8A8, using 8-bit floating-point format FP8). The challenge lies in the fact that outliers, which were already observed in weights in the first article, can be even more pronounced in activations.【7】 SmoothQuant addresses this challenge by transferring the outlier burden from activations to weights.【8】 Activation scales can be fixed in advance via calibration (static) or computed on-the-fly for each token (dynamic).


The same row is multiplied by an activation vector: x = [1.0, 0.5, −1.2, 0.8, 2.0, −0.4, 1.5, 0.1]. The output with original weights is −0.530.


Table 2: Output Deviation When Multiplying the Same Row with the Activation Vector


The error at the row level does not perfectly align with the output error: although the full INT8 recipe has a row error of 0.000054, which is smaller than the outlier BF16 recipe’s 0.00048, its output deviates more in this input (+0.037 and −0.028). This is why methods using calibration focus on output errors generated by actual inputs rather than weight errors.

How Does the KV Cache Expand Memory in Long Contexts?

The size of the KV cache is calculated as 2 × layers × kv_head × head_size × context × concurrent_users × bytes; the initial factor of 2 arises because separate tensors are stored for the key and value. In Qwen3-32B (64 layers, 8 KV heads, 128 dimensions per head), the KV cache for a 32,000-token conversation with 16 concurrent users occupies 128 GiB in BF16.【9】 When the same cache is stored in FP8, it reduces to 64 GiB, and to approximately 32 GiB at 4 bits (excluding scale information; llama.cpp’s q8_0 and q4_0 types, which include scales, measure ratios of approximately 53% and 28% respectively, see E2). Since the distributions of key and value tensors differ, methods like KIVI quantize them separately: KIVI applies channel-wise scaling to keys and token-wise scaling to values.【10】 The risk of KV cache quantization becomes more pronounced in long contexts: details related to a single piece of information in a long context may be lost.

Embedding and lm_head

The embedding table and lm_head each contain one row per token in the vocabulary. With Qwen3’s vocabulary of approximately 152,000 tokens, Qwen3-32B’s lm_head amounts to 778 million parameters, representing 2.4% of the model’s total parameters. In Qwen3-0.6B, however, the embedding table alone accounts for roughly one-quarter of the model’s parameters.【11】 Since lm_head directly produces logit values for all tokens at each step, its error affects the probability of every token; hence, some recipes retain it at higher precision. In smaller models, the memory cost of this choice becomes more significant. In Qwen3’s 0.6B, 1.7B, and 4B models, the embedding table and lm_head share the same weights; in these models, they cannot be compressed separately and must be updated together.


Table 3: Memory, Speed, and Risk Trends by Quantized Component

In this paper, only the KV cache and lm_head were measured.


Bir transformer katmanında kuantize edilebilen kalemler (Çizim: Elif İnce)

Quantizable Pens in a Transformer Layer (Drawing: Elif İnce)

Table 3 summarizes which component alleviates which bottleneck: weight quantization dominates in single-user local deployments, while activation and KV cache quantization are more prominent in high-load services.

Dive Deep: What Does SmoothQuant Impose?

The output of a layer is given by Y = X W. Some channels of the activation matrix X have very large values, while the weight matrix W has a more homogeneous distribution. SmoothQuant selects a scale s for each channel, divides X by this scale, and multiplies W by the same scale:【12】


The product remains unchanged; extreme channels in the activation are reduced, while the weights increase slightly. The parameter α determines how the difficulty is shared between the two sides. Since the scales are precomputed and embedded into the weights of the previous layer, they introduce no additional runtime cost.


Key Takeaway: Weight quantization targets memory usage and single-user speed, while activation and KV cache quantization targets computation in dense services and memory for long contexts.

Hybrid Recipe, KV Cache, and lm_head: Two Experiments

Building on the previous sections, this section evaluates hybrid precision with KV cache and lm_head quantization on the dense models of the Qwen3 family, using Turkish text and Turkish benchmarking tests; activation quantization was not experimentally tested. The questions addressed are:

  • E1 – Layer sensitivity and hybrid recipe: Which tensors are sensitive in Turkish; is a recipe built from this mapping better than uniform quantization of the same size?
  • E2 – KV cache and lm_head: How much memory does KV cache quantization save in long Turkish contexts, and does it impair recall; does compressing lm_head yield the same effect in small and large models?


Deneylerin birbirine bağlantısı: E1 ve E2 (Çizim: Elif İnce)

Connection Between Experiments: E1 and E2 (Diagram: Elif İnce)

The two experiments are independent; both use the uniform GGUF quantizations, common setup, and pre-control generated in the first paper as their baseline for comparison.


In the diagram, gray boxes represent experiments covered in earlier papers of the series; dashed boxes indicate the independent experiments of this paper. All runs in the series were performed on a single H100 graphics processing unit (GPU; 80 GB), with a total runtime of 10.7 GPU-hours.

Models, Quantizations, and Metrics Used in the Experiments

The common setup from the first paper was used in these experiments; relevant details for this paper are summarized below.


Table 4: Common Setup: Models, Quantizations, Calibration, and Metrics (Relevant to this paper)


Measurement: The primary metric is the difference in accuracy between the quantized model and its BF16 counterpart; results are reported as 95% confidence intervals (CI) using paired bootstrap resampling. The answer flip rate measures how often correct answers from the BF16 model are changed to incorrect ones, and incorrect answers are changed to correct ones. KL divergence measures how far the probability distribution of each token in the quantized model deviates from that of the BF16 model; since it does not require correct answers, it can be more sensitive than accuracy and was used as the primary metric in single-tensor analysis. In the preliminary control of the first paper, the noise floor of the measurement chain was measured at ±0.47 points; differences smaller than this band were considered noise.【21】

Dive Deep: Calibration Details

The Turkish calibration set was randomly sampled from the OzenliDerlem subset, which consists of carefully cleaned general web texts from BellaTurca.【22】 The number of calibration samples followed the original GPTQ study with 128 examples.【23】 The KL measurement was performed on a separate dataset of approximately 100,000 tokens drawn from Havadis news texts.【24】


Belebele-tr is a multiple-choice reading comprehension test built on FLORES-200 paragraphs.【25】 ARC-tr is a human-reviewed Turkish translation of the ARC-Challenge test section.


Benchmark tests were run using log-likelihood based multiple-choice scoring; since the model does not generate text, the reasoning mode results of Qwen3 were deemed irrelevant. The weights of GGUF quantizations were converted back to BF16 and evaluated on the same engine, ensuring all quantizations passed through the same measurement pipeline. This approach was verified in the initial control test to produce identical results to llama.cpp itself. The KL divergence of GGUF quantizations was computed directly using llama.cpp’s perplexity tool.

Which Tensors Are Sensitive and What Does the Map-Based Recipe Gain?

Question: Does the hybrid recipe derived from the tensor sensitivity map of Qwen3-4B in Turkish text outperform a uniform quantization of the same size?


Setup: In the first phase, Qwen3-4B was kept at BF16 precision; each of the 36 layers had its seven tensor types (q, k, v, o, gate, up, down) individually reduced to 2-bit RTN (no calibration, rounding to nearest value; group size 128), and KL divergence was measured on a subset of approximately 20,000 tokens (20,480) from the Havadis dataset, resulting in a total of 252 measurements. Due to the high cost of running benchmark tests for this number of measurements, the map was derived from KL, and recipes were compared using accuracy.


In the second phase, using the map, bit allocation was distributed such that the total size was approximately equal to Q3_K_M (at most 1% difference): the most sensitive tensors were assigned Q6_K, moderately sensitive ones Q4_K, and the rest Q3_K or Q2_K; the recipe was generated as GGUF using llama.cpp’s tensor-type-specific configuration option. For comparison, a uniform Q3_K_M, the closest Unsloth Dynamic quantization【26】, and two control recipes with the same bits randomly distributed across layers within each tensor type (two random seeds) were used; this randomization preserved the total size exactly.


In the third phase, the same recipe rule was transferred to Qwen3-8B: no separate map was derived for the 8B model; instead, the layer-tensor pattern from the 4B map (first and last layers, gate/up/down) was applied proportionally, and the size was kept equal to uniform Q3_K_M.


Table 5: E1 Comparison Conditions (Uniform, Random, Unsloth Dynamic, and Map-Based Recipes)


Expectation and Interpretation: The observation that tensors such as first and last layers and down_proj are highly sensitive suggests that the map-based recipe is expected to significantly outperform uniform quantization and the random control, and to approach the performance of the prebuilt hybrid quantization. Since the behavior of the prebuilt hybrid quantization may be influenced by the content of its calibration data, it is also expected that the map-based recipe could match or surpass it in Turkish. If the map-based recipe outperforms both uniform quantization and the random control, the gain is consistent with sensitivity information; if the random recipe also outperforms uniform quantization, part of the gain may be attributed to irregular bit distribution; if the recipe transferred from 4B to 8B yields improvement, the sensitivity map appears transferable within the model family; if it does not, separate analysis per model may be required.


Qwen3-4B hassasiyet haritası: KL, logaritmik ölçek (Çizim: Elif İnce)

Qwen3-4B Sensitivity Map: KL, Logarithmic Scale (Diagram: Elif İnce)

Findings (Map): When reducing 252 tensors individually to 2 bits, the median KL divergence was 0.013; however, the distribution has a long tail: the most sensitive cell was the gate_proj of layer 2 (0-based index; Figure 4) with KL 4.32, followed by layer 1 (1.36), layer 3 (0.625), and layer 4 (0.196); none of the other cells exceeded 0.065. Thirteen of the 20 most sensitive cells were located in the first four or last four layers; 12 were gate/up projections and 8 were down projections. Mean values by type: gate 0.200 (median 0.022), down 0.026, up 0.024, attention tensors 0.008–0.010.


Benzer boyutta reçeteler: Δacc ve KL (Çizim: Elif İnce)

Similar-sized prescriptions: Δacc and KL (Drawing: Elif İnce)

Recipes: From the map, 44 tensors were assigned as Q6_K, 88 as Q4_K, 101 as Q3_K, and 19 as Q2_K; the map recipe is approximately 2.1 GB (measured average 4.15 bits), compared to uniform Q3_K_M at 4.12 bits and Unsloth Dynamic (Q3_K_XL) at 4.23 bits, making the map recipe 1.9% larger than the map recipe and 2.7% larger than uniform Q3_K_M. In accuracy, the map recipe yielded the smallest loss: relative to BF16, the map recipe showed Δ −0.40 points (GA [−1.7, +0.9]), uniform Q3_K_M showed −1.29, Unsloth showed −1.37, and random recipes showed −2.99 and −2.11. The difference between map recipe and uniform was +0.89 points (GA [+0.1, +1.7]), map recipe versus Unsloth +0.97 [−0.3, +2.3], map recipe versus random +2.6 [+1.0, +4.2] (s0) and +1.7 [+0.2, +3.2] (s1). The difference relative to uniform lies at the lower bound of GA (+0.1) and exceeds zero relative to random controls; it is indistinguishable from Unsloth (point estimates are similar). On Belebele-tr, random recipes lost −4.4 and −4.2 points; the map recipe lost −0.7 (GA [−2.4, +1.0]). The answer flip rate was lowest in the map recipe at 9.0%. KL ranking: map recipe 0.187, Unsloth 0.210, uniform 0.215, random 0.255 and 0.247; the map recipe achieved KL values 13% lower than uniform and approximately 25% lower than the average of random controls.


Transfer: When the same recipe rule was applied to Qwen3-8B (4.14 GB, 4.05 bits), the loss relative to BF16 was Δ −0.55 points (GA [−1.6, +0.5]); for uniform Q3_K_M, it was −1.48. The difference is +0.93 points (GA [+0.0, +1.8]; lower bound at zero). The answer flip rate decreased from 9.4% to 7.6%, and the KL divergence in the Havadis text outside the mapped portion dropped from 0.131 to 0.114 (13% reduction).


Table 6: Similar-Sized Recipes (Qwen3-4B and Qwen3-8B)


The uniform Q3_K_M accuracies were derived from the measurements in the first paper; KL was measured excluding the first 20,480 tokens. The Qwen3-8B recipe is a proportional application of the rule derived from the 4B map (no separate map was extracted for 8B). Random recipes consist of rows generated using two distinct random seeds; all conditions were obtained in a single run.


Result E1: The most sensitive tensors are concentrated in the gate_proj of the first layers (layers 1–4, zero-based) and in the down/up projections at the first and last layers. The rule-based recipe built from the map outperforms the uniform quantization of the same size by +0.9 points in benchmark accuracy and by 13% in KL, and is approximately +2 points better than random distribution; it is indistinguishable from Unsloth Dynamic in accuracy (single run). When the rule was transferred to 8B, a similar trend was observed (+0.9 points, GA lower bound +0.0; KL 13%); both models represent a single sample.

How Much Can KV Cache and lm_head Be Compressed?

Question: How much memory does KV cache quantization save in long Turkish contexts, and does it impair recall; does compressing lm_head affect model size differently?


Setup: In the first part, the Q8_0 copy of Qwen3-8B (memory, speed, and recall) and the Q8_0 copy of the smaller Qwen3-1.7B model (recall only) were fixed; KV cache was stored in llama.cpp as F16, q8_0, and q4_0. Context lengths were set to 4K, 16K, and 32K; peak video memory (VRAM) and decode speed were measured under each condition. For quality evaluation, a needle-in-a-haystack test 【27】 was constructed: the haystack comprised long Turkish texts from the AcademicCorpus subset of the BellaTurca dataset 【28】 (with multiple texts concatenated end-to-end if necessary), and the needle consisted of Turkish sentences written for this study, each with a numerical answer. The needle was placed at five different depths within the context, and five distinct needles were tested for each context length and depth. In the second part, only the lm_head type was changed in the Q4_K_M copies of Qwen3-0.6B and Qwen3-8B (to Q8_0 and Q4_K); Belebele-tr, ARC-tr, KL divergence, and disk size were measured. In Qwen3-0.6B, the embedding table shares weights with lm_head; 【29】 in this model, both change together.


Expectations and Interpretation: KV cache is expected to grow linearly with context length, and q8_0 is expected to reduce this growth by approximately half, preserving recall, while q4_0 is expected to begin degrading at the longest context lengths. Compressing lm_head to 4 bits is expected to yield a significant memory gain and significant loss in the 0.6B model, but only a small gain and small loss in the 8B model. If q8_0 KV preserves recall at all lengths, 8-bit KV could become the default choice for long-context local usage; if degradation concentrates on needles placed in the middle of the context, KV quantization may be exacerbating an already weak region of the model; if recall drops significantly more in small models than in large ones, the cost of KV quantization is model-size dependent.


Findings (KV Cache): The KV cache of Qwen3-8B in Q8_0 grew linearly with context length, and the measured values matched the formula exactly: F16 showed 576, 2304, and 4608 MiB for context lengths of 4K, 16K, and 32K respectively. q8_0 KV reduced this to 306, 1224, and 2448 MiB (53% of F16), while q4_0 reduced it further to 162, 648, and 1296 MiB (28% of F16). Peak VRAM decreased from 12.7 GiB at 32K to 10.7 GiB (q8_0) and 9.5 GiB (q4_0); since weights (approximately 8 GiB) remained constant, the total memory savings were 16% and 25% respectively.


Speed Cost: Quantized KV cache resulted in slowdown, not speedup: at 32K context, decode speed was 141.7 tokens/s in F16, 105.6 tokens/s in q8_0, and 103.7 tokens/s in q4_0 (75% and 73% of F16); at 4K, the ratios were 90–91%. These results were measured using llama.cpp’s own kernels; results may differ with other engines.


Recall: In the needle-in-a-haystack test (3 context lengths × 5 depths × 5 needles = 75 trials), recall varied by KV type and model size. In Qwen3-8B, F16 KV achieved 74/75 correct answers, q8_0 achieved 73/75, and q4_0 achieved 71/75: one missing needle in q8_0 (−1.3 points), three missing in q4_0 (−4.0 points). Loss was not observed at 4K; at 16K, it occurred only at 50% depth; at 32K, it accumulated at 25% and 50% depth in q4_0. In Qwen3-1.7B, the same pattern was more pronounced: F16 achieved 67/75, q8_0 achieved 64/75, and q4_0 achieved 58/75 (−4.0 and −12.0 points); loss began at 4K in q4_0 (24/25) and at 16K in q8_0; losses were concentrated at medium depths in 32K contexts.


Table 7: Recall in the Needle-in-a-Haystack Test


KV önbelleği kuantizasyonu: VRAM, hız ve geri çağırma (Çizim: Elif İnce)

KV Cache Quantization: VRAM, Speed, and Recall (Illustration: Elif İnce)

Each cell contains 5 depth × 5 heads (25 trials); the Qwen3-8B and Qwen3-1.7B models were run with q8_0 weights using llama.cpp. KV memory usage matched the formula exactly (F16: 576, 2304, 4608 MiB); quantized KV reduced decode speed at 32K to approximately 75% of F16 performance.


lm_head: In Qwen3-0.6B (where lm_head shares the embedding table and thus changes together), the proportion of lm_head in total size was 32.7% (127.6 MB) with default Q4_K_M, 38.6% (165.3 MB) with Q8_0, and 25.0% (87.5 MB) with Q4_K; in Qwen3-8B these values were 10.2% (510.5 MB), 12.8% (661.2 MB), and 7.2% (350.1 MB) respectively. Switching from Q8_0 to Q4_K reduced file size by 18% (428 → 351 MB) for 0.6B and by 6% (5.17 → 4.86 GB) for 8B. No significant difference in accuracy was observed: the Q4_K − Q8_0 difference was +0.40 points for 0.6B (GA [−0.75, +1.55]) and −0.03 points for 8B ([−0.64, +0.56]). KL divergence increased: from 0.120 to 0.154 (+29%) for 0.6B and from 0.036 to 0.048 (+32%) for 8B.


Table 8: Size Accuracy and KL by lm_head Type and Model Size


The base rows are the default Q4_K_M copies from the first paper. Each cell represents a single run with a single seed; the 95% confidence interval for the accuracy difference between Q4_K and Q8_0 (average of Belebele-tr and ARC-tr) includes zero (0.6B: +0.40 [−0.75; +1.55]; 8B: −0.03 [−0.64; +0.56]).


Result E2: An 8-bit KV cache reduced memory usage by 47%, while a 4-bit KV cache reduced it by 72%; the cost was a ≈25% decrease in decode speed at 32K context (≈10% at 4K, ≈19% at 16K). Recall was largely preserved at 8B with q8_0 (−1.3 points), but dropped by a total of −4.0 points with q4_0 (−8.0 points at 32K); in the 1.7B model, the drop was larger (−4.0 and −12.0 points) and concentrated in medium-depth layers. Reducing lm_head from Q8_0 to Q4_K reduced file size by 18% in the small model without significant change in benchmark accuracy, while KL increased by 29–32%; in the large model, the gain was minor (6%).


Table 9: Summary of Experiments

Discussion and Limitations

Portability: The sensitivity map was derived from a single model (Qwen3-4B), a single language (Turkish), and a single bit budget (2-bit RTN); the recipe was only transferred to another model in the same family (Qwen3-8B). It has not been measured how well the map generalizes to other model families, languages, or bit budgets; the recipe must be validated separately under these conditions.


Scope: Findings apply to dense models in the Qwen3 family, Turkish language, multiple-choice benchmark tasks, llama.cpp, a single H100 GPU, and specific software versions; the recipe and lm_head configurations were tested with a single random seed. The needle-in-a-haystack test measured recall of a single number only within contexts up to 32K tokens; the cost of KV quantization may differ in longer contexts or tasks requiring integration of multiple pieces of information. Activation quantization has not been experimentally evaluated; MoE models and training with quantization are outside the scope.


Precision: Quantization is an approximate transformation: each quantized value introduces a measurable deviation from the original model. Whether this deviation is significant for practical use can only be determined by measuring performance on the target task and target language.

Frequently Asked Questions

How Can One Tell If a Pre-Quantized Model Is Mixed-Precision?

GGUF files store the type of each tensor within the file itself; llama.cpp tools or Hugging Face's file viewer indicate which tensors are Q6_K or Q4_K.【30】 llama.cpp's K-quant types are already mixed-precision recipes; Unsloth's Dynamic quantizations determine the recipe on a per-model basis.【31】 The behavior of these recipes in Turkish may depend on the content of the calibration data used during their creation; the recipe built from the Turkish map in E1 outperformed the uniform quantization of the same size in both accuracy and KL (single model, single run).

When Should the KV Cache Be Compressed?

Memory bottlenecks arise more from context length or the number of concurrent users than from weights. In E2, 8-bit KV cache reduced memory usage by approximately half while largely preserving recall performance (−4.0 points at 1.7B); 4-bit KV cache significantly reduced recall, especially with long contexts and smaller models. In llama.cpp, both types slow down decode speed by approximately 25% at a 32k token context.

Consequences of Allocating Bit Budget and Compressing the KV Cache

In the measurements of this article, when the bit budget was allocated not equally across all tensors but rather to those most affected by quantization, lower variance was achieved at the same size; Q4_K_M quantization, one of the two answers at the beginning of the first article, is a common implementation of this idea. For Qwen3-4B, the recipe derived from the Turkish sensitivity map achieved 13% lower KL divergence and +0.9 points higher accuracy than uniform quantization of the same size (GA [+0.1, +1.7]); it diverged from randomly distributed control recipes (approximately +2 points, ~25% in KL) and could not be distinguished from Unsloth Dynamic (+0.97, GA [−0.3, +2.3]). When applied to Qwen3-8B, a similar trend was observed (+0.93, GA [+0.0, +1.8]); however, these two models represent only a single example.


KV Cache and lm_head: In E2, 8-bit KV cache reduced memory usage by 47%, and 4-bit KV cache by 72%; recall was practically preserved at q8_0 (−1.3 points) but dropped by a total of −4.0 points at q4_0 (−8.0 points at 32K), with a larger drop at 1.7B (−4.0 and −12.0 points); the cost is a decode speed reduced to approximately 75% of F16 at 32K context (90–91% at 4K). Reducing lm_head from Q8_0 to Q4_K reduced file size by 18% at 0.6B and 6% at 8B, with no measurable accuracy loss (within GA), but increased KL divergence by 29–32%.


General Trend: As measured in the first and second articles, the loss in bit count due to quantization varies according to method, calibration, model size, and metric: at 8 bits, loss is generally close to noise levels (except Qwen3-8B FP8); at 4 bits, it depends on method; at approximately 3 bits (Q2_K), the breaking point was exceeded across all six models.


Limitations: The sensitivity map was derived from a single model (Qwen3-4B), a single language (Turkish), and a single bit level (2-bit RTN); the recipe rule was only transferred to Qwen3-8B, and its applicability to other model families, languages, or bit budgets has not been measured. Findings apply only to Qwen3’s dense models, Turkish language, multiple-choice benchmarking tests, llama.cpp, and a single H100 GPU; the recipe and lm_head configurations are based on a single seed; the needle-in-a-haystack test measures only one number at a maximum of 32k tokens; activation quantization was not tested, and MoE models or training with quantization were excluded (see Discussion and Limitations).


Open Questions: Two questions remain open: how transferable are sensitivity maps across languages (the 4B → 8B transfer within the family yielded a similar trend; single example), and can the limit below 3 bits be overcome with post-training methods? This second question relates to a topic mentioned in the introduction of the first article in this series. Some K3 models distribute weights directly as MXFP4;【32】 previously, some K2 Thinking’s MoE components obtained low-bit INT4 weights during post-training, and gpt-oss had already reduced MoE weights to MXFP4 after final training.【33】 In these models, quantization was not applied only after training; low-bit sensitivity was accounted for during the training process itself.【34】 This approach is called quantization-aware training (quantization-aware training, QAT). To what extent can accounting for low-bit sensitivity during training surpass the limits of post-training quantization? This question will be addressed in a separate article comparing post-training quantization (post-training quantization, PTQ) with quantization-aware training.

Bibliographies

Turkish NLP Suite. “BellaTurca.” Hugging Face. Accessed October 5, 2026.https://huggingface.co/datasets/turkish-nlp-suite/BellaTurca

Turkish NLP Suite. “Havadis.” Hugging Face. Accessed October 1, 2026.https://huggingface.co/datasets/turkish-nlp-suite/Havadis

Unsloth. “Unsloth Dynamic 3.0 GGUFs.” Unsloth Dokümantasyonu. Accessed October 14, 2026.https://unsloth.ai/docs/basics/dynamic-3.0-ggufs

ggml-org. “llama.cpp: LLM Inference in C/C++.” GitHub. Accessed October 6, 2026.https://github.com/ggml-org/llama.cpp

Bandarkar, L., D. Liang, B. Muller, M. Artexte, S. N. Shukla, et al. “The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants.” arXiv 2024. Accessed October 14, 2026.https://arxiv.org/abs/2308.16884

Clark, P., I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, et al. “Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge.” arXiv 2018. Accessed October 9, 2026.https://arxiv.org/abs/1803.05457

Dettmers, T., M. Lewis, Y. Belkada, and L. Zettlemoyer. “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale.” arXiv 2022. Accessed September 30, 2026.https://arxiv.org/abs/2208.07339

Dutta, A., S. Krishnan, N. Kwatra, and R. Ramjee. “Accuracy is Not All You Need.” arXiv 2024. Accessed October 11, 2026.https://arxiv.org/abs/2407.09141

EleutherAI. “Language Model Evaluation Harness.” GitHub. Accessed October 13, 2026.https://github.com/EleutherAI/lm-evaluation-harness

Frantar, E., S. Ashkboos, T. Hoefler, and D. Alistarh. “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.” arXiv 2023. Accessed October 11, 2026.https://arxiv.org/abs/2210.17323

Kamradt, Greg. “Needle in a Haystack.” GitHub. Accessed October 4, 2026.https://github.com/gkamradt/needle-in-a-haystack

Liu, Z., J. Yuan, H. Jin, S. Zhong, Z. Xu, et al. “KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache.” arXiv 2024. Accessed October 4, 2026.https://arxiv.org/abs/2402.02750

Moonshot AI. “Kimi K2 Thinking.” Hugging Face. Accessed October 10, 2026.https://huggingface.co/moonshotai/Kimi-K2-Thinking

Moonshot AI. “Kimi K3.” GitHub. Accessed October 12, 2026.https://github.com/MoonshotAI/Kimi-K3

Open Language Data Initiative. “FLORES+.” Hugging Face. Accessed October 3, 2026.https://huggingface.co/datasets/openlanguagedata/flores_plus

OpenAI, S. Agarwal, L. Ahmad, J. Ai, S. Altman, et al. “gpt-oss-120b & gpt-oss-20b Model Card.” arXiv 2025. Accessed October 13, 2026.https://arxiv.org/abs/2508.10925

PyTorch. “Quantization.” PyTorch 2.5 Dokümantasyonu. Accessed October 10, 2026.https://docs.pytorch.org/docs/2.5/quantization.html

Xiao, G., J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han. “SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models.” arXiv 2024. Accessed October 1, 2026.https://arxiv.org/abs/2211.10438

Yang, A., A. Li, B. Yang, B. Zhang, B. Hui, et al. “Qwen3 Technical Report.” arXiv 2025. Accessed October 14, 2026.https://arxiv.org/abs/2505.09388

malhajar. "arc-tr-v0.2." Hugging Face. Accessed October 8, 2026.https://huggingface.co/datasets/malhajar/arc-tr-v0.2

Citations

  • [1]

    ggml-org, “llama.cpp: LLM Inference in C/C++,” GitHub, Accessed October 6, 2026.https://github.com/ggml-org/llama.cpp

  • [2]

    ggml-org, “llama.cpp: LLM Inference in C/C++.”

  • [3]

    Unsloth, “Unsloth Dynamic 3.0 GGUFs,” Unsloth Dokümantasyonu, Accessed October 14, 2026.https://unsloth.ai/docs/basics/dynamic-3.0-ggufs

  • [4]

    T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale,” arXiv 2022, 9. Accessed September 30, 2026.https://arxiv.org/abs/2208.07339

  • [5]

    PyTorch, “Quantization,” PyTorch 2.5 Documentation, Accessed October 10, 2026.https://docs.pytorch.org/docs/2.5/quantization.html

  • [6]

    Unsloth, “Unsloth Dynamic 3.0 GGUFs.”

  • [7]

    T. Dettmers, et al. "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale," 9.

  • [8]

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models,” arXiv 2024, 2-4, 8-9. Accessed October 1, 2026.https://arxiv.org/abs/2211.10438

  • [9]

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, et al., “Qwen3 Technical Report,” arXiv 2025, 5-8, Accessed October 14, 2026.https://arxiv.org/abs/2505.09388

  • [10]

    Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, et al. “KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache,” arXiv 2024, 2, 6. Accessed October 4, 2026.https://arxiv.org/abs/2402.02750

  • [11]

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, et al. “Qwen3 Technical Report,” 5-8.

  • [12]

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models,” arXiv 2024, 2-4, 8-9. Accessed October 1, 2026.https://arxiv.org/abs/2211.10438

  • [13]

    A. Yang et al. “Qwen3 Technical Report,” 5–8.

  • [14]

    ggml-org, “llama.cpp: LLM Inference in C/C++.”

  • [15]

    Turkish NLP Suite, “BellaTurca,” Hugging Face, Accessed October 5, 2026.https://huggingface.co/datasets/turkish-nlp-suite/BellaTurca

  • [16]

    L. Bandarkar, D. Liang, B. Muller, M. Artexte, S. N. Shukla, et al., “The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants,” arXiv 2024, 7. Accessed October 14, 2026.https://arxiv.org/abs/2308.16884

  • [17]

    malhajar, “arc-tr-v0.2,” Hugging Face, Accessed October 8, 2026.https://huggingface.co/datasets/malhajar/arc-tr-v0.2; P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, et al., “Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge,” arXiv 2018, 2, 7, Accessed October 9, 2026.https://arxiv.org/abs/1803.05457

  • [18]

    G. Kamradt, “Needle in a Haystack,” GitHub, Accessed October 4, 2026.https://github.com/gkamradt/needle-in-a-haystack

  • [19]

    EleutherAI, “Language Model Evaluation Harness,” GitHub, Accessed October 13, 2026.https://github.com/EleutherAI/lm-evaluation-harness

  • [20]

    Turkish NLP Suite, “Havadis,” Hugging Face, Accessed October 1, 2026.https://huggingface.co/datasets/turkish-nlp-suite/Havadis

  • [21]

    A. Dutta, S. Krishnan, N. Kwatra, and R. Ramjee. “Accuracy is Not All You Need,” arXiv 2024, 7-9. Accessed October 11, 2026.https://arxiv.org/abs/2407.09141

  • [22]

    Turkish NLP Suite, “BellaTurca.”

  • [23]

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers,” arXiv 2023, 4, 6-8, Accessed October 11, 2026.https://arxiv.org/abs/2210.17323

  • [24]

    Turkish NLP Suite, “Havadis.”

  • [25]

    L. Bandarkar, et al., “The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants,” 7; Open Language Data Initiative, “FLORES+,” Hugging Face, Accessed October 3, 2026.https://huggingface.co/datasets/openlanguagedata/flores_plus

  • [26]

    Unsloth, “Unsloth Dynamic 3.0 GGUFs.”

  • [27]

    Greg Kamradt, “Needle in a Haystack.”

  • [28]

    Turkish NLP Suite, “BellaTurca.”

  • [29]

    A. Yang, et al., “Qwen3 Technical Report,” 5–8.

  • [30]

    ggml-org, “llama.cpp: LLM Inference in C/C++.”

  • [31]

    Unsloth, “Unsloth Dynamic 3.0 GGUFs.”

  • [32]

    Moonshot AI, “Kimi K3,” GitHub, Accessed October 12, 2026.https://github.com/MoonshotAI/Kimi-K3

  • [33]

    Moonshot AI, “Kimi K2 Thinking,” Hugging Face, Accessed October 10, 2026.https://huggingface.co/moonshotai/Kimi-K2-Thinking; OpenAI, S. Agarwal, L. Ahmad, J. Ai, S. Altman, et al., “gpt-oss-120b & gpt-oss-20b Model Card,” arXiv 2025, 3, Accessed October 13, 2026.https://arxiv.org/abs/2508.10925

  • [34]

    Moonshot AI, “Kimi K3"; Moonshot AI, “Kimi K2 Thinking”; OpenAI et al., "gpt-oss-120b & gpt-oss-20b Model Card," 3.

T3 AI
Türkiye's Large Language Model,is growing together with you.

Contents

  • Weights Are Not the Only Memory Component: Bit Budget and KV Cache

  • Should Every Layer Be Quantized the Same Way?

    • How Is the Sensitive Layer Identified?

      • Dive Deeper: The Two Meanings of “Dynamic”

    • What Else Can Be Compressed Beyond Weights?

    • Why Are Activations Harder to Compress Than Weights?

    • How Does the KV Cache Expand Memory in Long Contexts?

      • Embedding and lm_head

      • Dive Deep: What Does SmoothQuant Impose?

    • Hybrid Recipe, KV Cache, and lm_head: Two Experiments

    • Models, Quantizations, and Metrics Used in the Experiments

      • Dive Deep: Calibration Details

    • Which Tensors Are Sensitive and What Does the Map-Based Recipe Gain?

    • How Much Can KV Cache and lm_head Be Compressed?

  • Discussion and Limitations

  • Frequently Asked Questions

    • How Can One Tell If a Pre-Quantized Model Is Mixed-Precision?

    • When Should the KV Cache Be Compressed?

  • Consequences of Allocating Bit Budget and Compressing the KV Cache