Opt.Gear-1M: A Generative Language Model Running on an MCU

🤗 HUGGING FACE 🏠 OptAI ✉️ CONTACT

The goal of Opt.Gear-1M is not simply to shrink Opt.Gear-270M and 1B. It is to run an auto-regressive language model on a microcontroller , where resources are far more constrained than on a mobile NPU or server GPU.

To this end, we deployed Opt.Gear-1M directly on the ARM Cortex-M7 core of the STM32H747I-DISCO. With W4A32, using 4-bit weights and FP32 activations, the measured generation speed is:

These results were measured using an embedded C runtime running on a real MCU , not a host simulator.

Optimization Is a Different Problem on an MCU

On server GPUs, computational workload and GPU utilization are the main optimization targets. On mobile NPUs, operator support, tensor layout, quantization formats, and accelerator memory are important.

Microcontrollers impose even tighter constraints:

• Flash limits the size of the weights and tokenizer that can be stored.
• SRAM limits the size of activations, the KV cache, and intermediate buffers.
• Without a large matrix accelerator, operations that execute simply on the CPU are important.
• Dynamic allocation and complex runtime dependencies must be minimized.
• More than long context, a predictable memory footprint and stable latency matter.

A small parameter count alone is therefore insufficient for an MCU language model. A small vocabulary, limited context, small KV cache, fixed-size state, simple activations, and static memory allocation must be designed together. Opt.Gear-1M starts from these MCU constraints.

Configuration of Opt.Gear-1M

Opt.Gear comes in three sizes: 1B, 270M, and 1M. Almost every configuration in Opt.Gear-1M was scaled down again for MCU execution.

Opt.Gear-1M does not aim to cover broad knowledge benchmarks. Its priority is to reliably handle short commands, local generation in constrained domains, embedded-control prompts, and offline human-machine interfaces within a small memory budget.

Architecture: The Power of Fixed-Size State

All Opt.Gear models use a hybrid sequence mixing architecture that combines attention with the ConvKV-Gated Mixer.

Attention can directly model relationships between distant tokens, but decoding requires storing the keys and values of previous tokens in a KV cache. The cache grows with sequence length. The ConvKV-Gated Mixer instead applies causal depthwise convolution to keys and values, maintaining a fixed-size state determined by the convolution kernel rather than the entire context.

Opt.Gear-1B and 270M use kernel size 3, while Opt.Gear-1M uses kernel size 4. This slightly widens the local receptive field when the hidden dimension is very small. The convolution layer is causal: it does not reference future tokens when processing the current token. Auto-regressive decoding therefore only needs to retain the small preceding convolution state.

The purpose of this architecture is not to remove attention entirely:

• GQA attention handles selective interactions between tokens.
• ConvKV-Gated Mixer handles short, repetitive local patterns with a small, fixed state.

This balances model expressiveness against the MCU memory footprint.

Simple Operations Mean Performance: ReLU6GLU

The FFNs in Opt.Gear-1B and 270M use GeGLU-based gated dense MLPs. Opt.Gear-1M uses a smaller FFN with hidden dimension 128 and intermediate dimension 272, with a ReLU6-based gated configuration, ReLU6GLU:

Because the output is bounded between 0 and 6, the activation range is predictable. It can be implemented with relatively simple operations, without complex nonlinear functions such as sigmoid or GELU.

On a Cortex-M7 without a large accelerator, implementation complexity, temporary buffers, and memory access matter alongside theoretical operation counts. Opt.Gear-1M therefore uses a smaller, simpler FFN rather than a large gated FFN.

Results of STM32H747I-DISCO

What 20 tokens/s Means

20 tokens/s means approximately 50ms to generate one token. Generating up to 40 tokens takes about 2 seconds for decoding alone. End-to-end response time also includes prompt prefill and application processing.

This is not server-class throughput for generating long documents, but for short command responses or local status generation, it is fast enough to observe tokens appearing in real time . Gear-1M does more than fit in MCU Flash: it performs auto-regressive generation at a speed the user can observe.

Measuring Footprint from the ELF

Analyzing the final ELF with arm-none-eabi-size shows that .text and .rodata total 1,596,400 bytes (approximately 1.5MB). This is not just the size of the model weights:

• .text contains the executable code of the embedded runtime.
• .rodata contains quantized weights and read-only constants.
• The 16-bit embedding and LM head are also included in the read-only data.

The theoretical size of W4 weights therefore differs from the final ELF section sizes. A device build uses Flash for inference code, lookup tables, scales, and other constants as well as model weights. For MCU models, this makes footprint measurement based on the final ELF more important than parameter count alone.

In the final build, the .bss section is 444,404 bytes. Including .data and reserved heap/stack regions, the static RAM footprint is 449,032 bytes (approximately 449KB). RAM is primarily used for:

• Layer activation
• GQA KV cache
• ConvKV fixed-size state
• Intermediate tensor
• Logit and sampling buffers
• Embedded runtime workspace
• Heap and stack

What can we do

Opt.Gear-1M is not intended to run a general-purpose chatbot on an MCU. Its model size and 2,048 vocabulary have clear limits. The question we wanted to explore was whether a generative language model can run at a practical speed on a microcontroller CPU itself, without a server GPU or mobile NPU .

The 20 tokens/s measured on the STM32H747I-DISCO suggests the following possibilities:

• Short natural-language summaries of sensor data
• Local command responses for small robots
• Offline interfaces for industrial controllers
• Generation of structured status messages
• Simple question answering on devices with limited connectivity
• Short embedded assistants tailored to specific domains

Conclusion

We deployed Opt.Gear-1M, a language model with approximately 1M parameters, directly on the ARM Cortex-M7 of the STM32H747I-DISCO. The key design choices are:

• 128-dimensional hidden state and 5 decoder layers
• GQA with 2 query heads and 1 KV head
• ConvKV-Gated Mixer maintaining a fixed-size state
• Custom BPE tokenizer with a 2,048 vocabulary
• 512-token maximum context tailored to the MCU
• 4-bit weight, 16-bit embedding·LM head, FP32 activation
• Embedded C runtime without ONNX or X-CUBE-AI dependencies

The deployed build uses approximately 1.5MB of .text/.rodata and approximately 449KB of static RAM. Measured generation speed on the 400MHz Cortex-M7 is 20 tokens/s, or approximately 50ms/token.

This result means more than simply fitting a 1M model onto an MCU.

Even with severely limited Flash, SRAM, and compute, designing the architecture and runtime together enables auto-regressive generation at an observable speed.

Opt.Gear-1M is a first step toward extending the Opt.Gear family beyond mobile and edge accelerators to deeply embedded systems such as sensors, controllers, small robots, and offline human-machine interfaces.

Citation

@misc{optgear2026,
    title  = {{Opt-Gear} Technical Report},
    author = {{OptAI Gear Team}},
    year   = {2026},
    url    = {https://huggingface.co/OptGear}
}

Correspondence: contact@opt-ai.kr · Hugging Face: huggingface.co/OptAI