Opt.Gear-QAT: Preserving On-Device Performance with QAT

🤗 HUGGING FACE 🏠 OptAI ✉️ CONTACT

Opt.Gear is a foundation model developed in-house for on-device environments. The family includes the general-purpose language models Opt.Gear-1B and Opt.Gear-270M, and an ultra-lightweight model with approximately 1M parameters for microcontrollers.

Making a small model is not enough for deployment on real devices. In addition to parameter count, weight memory, memory bandwidth, compute precision, and the kernels supported by the accelerator determine inference performance.

The question is how much of the original performance can be retained when the model is reduced to 4 bits .

For Opt.Gear-1B, we applied Quantization-Aware Training (QAT) so that the trained model adapts directly to the INT4 grid used for deployment. These weights also minimize performance loss when subsequently used with other INT4 PTQ algorithms.

Quantization parameters of Opt.Gear-1B

Opt.Gear-1B QAT uses the following final deployment conditions.

Group size 32 uses an independent scale for each group of 32 weights. This increases scale metadata compared with per-tensor quantization, but allows a suitable range to be assigned to each region of weights with a different distribution.

Symmetric quantization fixes the zero-point at 0 and represents positive and negative values around the scale. This simplifies quantization operations and deployment kernels, but without a suitable scale, the limited INT4 grid may be used inefficiently.

Opt.Gear-1B therefore uses layer output distortion measured with calibration activations instead of simple max-based scale initialization.

Opt.Gear-1B QAT Pipeline

1. Quantization scale is important

Uniform weight quantization projects real-valued weights onto a limited integer grid:

Here, Δ is the quantization scale, or step size. Since INT4 has very few representable integer values, choosing the scale is critical:

• If the scale is too large, clipping decreases but quantization intervals become coarse.
• If the scale is too small, small weights can be represented precisely but large weights are clipped.
• Expanding the range to accommodate a few outliers concentrates most weights into a small number of quantization bins.

2. Hessian-based scale initialize

Simple PTQ methods such as RTN generally determine scales from the maximum absolute weight or weight range. This is straightforward to implement, but assumes that every weight perturbation has the same effect on the model output . In practice, even weight errors of equal magnitude have a greater impact on layer outputs when they occur along directions frequently traversed by input activations.

For Opt.Gear-1B, we estimated the sensitivity of each layer using calibration inputs and determined the initial INT4 scale for each group with the following output reconstruction objective:

This approach starts from GPTQ and its second-order output reconstruction perspective, but does not apply the full GPTQ algorithm unchanged. Opt.Gear uses this objective to find the initial group-wise scales for QAT, then jointly optimizes weights and scales against the overall training objective.

3. LSQ-based QAT

Hessian-based initialization effectively reduces initial output distortion in each layer. However, a layer-wise reconstruction objective does not directly optimize the performance of the entire language model. Small errors can accumulate across layers; some barely affect token prediction, while others substantially change the final logits.

After finding a good starting point with Hessian initialization, QAT with the full language modeling objective is therefore necessary. Opt.Gear-1B initializes scales from the Hessian-based values, then uses Learned Step Size Quantization(LSQ) to make the scales learnable parameters.

The forward pass simulates the same symmetric INT4 grid used for deployment:

round cannot be differentiated directly, so the backward pass approximates gradients with a Straight-Through Estimator (STE). LSQ propagates task-loss gradients to the step size Δ as well as the weights. Scale gradients are normalized by the number of elements in each group and the number of positive quantization levels. This prevents scale updates from overwhelming weight updates while allowing group-wise step sizes to be optimized jointly with model parameters.

Two adaptations take place together during QAT:

• Weights move toward distributions that are easier to represent on the INT4 grid.
• Group-wise scales adjust to clipping ranges and resolutions suited to the task loss.

Hessian-based initialization and LSQ play different roles:

Performance of QAT

We compared the Base, QAT, and PTQ results of Opt.Gear-1B and Gemma3-1B on six shared English and Korean benchmarks. RTN, AWQ, and GPTQ results come from requantizing the master weights of the QAT-trained checkpoints with each PTQ method. The QAT rows evaluate the final QAT quantizer, including the learned scales.

The six-benchmark average for Opt.Gear-1B decreased from Base 49.7 to QAT 48.6, a reduction of approximately 1.0 point . The average decline was approximately 1.3 points across the three English metrics and 0.8 points across the three Korean metrics. Crucially, the actual symmetric W4A16 deployment conditions with group size 32 were applied unchanged , yet just 1,000-step QAT preserved both English and Korean performance.

Under the same conditions, Gemma3-1B decreased from a Base average of 47.3 to a QAT average of 42.8, a reduction of approximately 4.6 points . Differences in architecture, pretraining, and QAT recipes mean that this gap cannot be attributed to a single factor. The results nevertheless suggest that Hessian initialization and LSQ gave Opt.Gear-1B a stable starting point and convergence path even with short QAT training.

Quantization stability

Opt.Gear-1B performs best with its learned scales, but the QAT-trained weights themselves also remain relatively stable under other INT4 quantizers.

Gemma3-1B also underwent PTQ from its QAT checkpoint, but switching to RTN reduced its average by 3.6 points. Under the same RTN requantization conditions, Opt.Gear-1B lost only 1.3 points on average, demonstrating greater stability.

Conclusion

Opt.Gear-1B QAT was designed around three principles:

1. Rather than looking only at weight ranges, measure the impact on outputs using calibration activations .
2. Hessian-based output reconstruction provides a good group-wise INT4 scale from which to start training.
3. Learned Step Size Quantization jointly optimizes weights and scales for the final task objective.

Using the same symmetric W4A16 grid with group size 32 throughout initialization, QAT, and final export prevents the learned quantization structure from being lost during deployment conversion. After just 1,000-step QAT, Opt.Gear-1B differed from Base by only about 1.0 point in the unweighted average across six English and Korean benchmarks.

The next article introduces Opt.Gear-1M, which extends this design philosophy to microcontroller scale.

Citation

@misc{optgear2026,
    title  = {{Opt-Gear} Technical Report},
    author = {{OptAI Gear Team}},
    year   = {2026},
    url    = {https://huggingface.co/OptGear}
}

Correspondence: contact@opt-ai.kr · Hugging Face: huggingface.co/OptAI