Hello, I'm Junyoung Park from OptAI. I will be attending ACL 2026 in San Diego, USA, this July. We look forward to your interest.

This technical review builds on the core ideas of a paper our team recently submitted to EMNLP. Across parameter-efficient research from LoRA to model quantization, rank collapse is an important challenge affecting practical model optimization quality.

FORA overview diagram
FORA overview
1. Introduction: Structural Limitations of LoRA and Rank Collapse

LoRA freezes pretrained weights W0 and learns a low-rank matrix ΔW = BA to maximize parameter efficiency (PEFT). Recent research, however, shows that trained LoRA adapters undergo spectral collapse: their effective rank falls substantially below the nominal rank r.

2. Core Mechanism 1: Layer Selection with Diagonal Fisher Scores

Instead of placing adapters in every transformer layer, FORA selectively allocates them only to the top K layers carrying the most information for a given task.

Fl=1N∑n=1N∑θ∈θlbase∥∇θL(xn,yn)∥2F_l=\frac{1}{N}\sum_{n=1}^{N}\sum_{\theta\in\theta_l^{base}}\lVert\nabla_{\theta}\mathcal{L}(x_n,y_n)\rVert^2

Here, θ_l^{base} denotes the base-model parameters of layer l, and N is the number of mini-batches. This diagonal trace serves as an approximation to positive-semidefinite curvature invariant under layer-wise reparameterization.

< 1%Layer scan
Top-KAdapter layers
FullRank usage
StiefelConstraint
3. Core Mechanism 2: Stiefel Manifold Constraints and the Cayley Transform

Concentrating task demands on a few selected layers requires their adapters to use the full r-dimensional rank space. FORA constrains the down-projection matrix B_l to the Stiefel manifold of column-orthogonal matrices.

St(dout,r)={B∈Rdout×r:B⊤B=Ir}St(d_{out},r)=\{B\in\mathbb{R}^{d_{out}\times r}:B^{\top}B=I_r\}

When B lies on the Stiefel manifold, BA has the same singular values as A. This structurally prevents rank loss.

Q=(I−α2W)−1(I+α2W),B(t+1)=QB(t)Q=(I-\frac{\alpha}{2}W)^{-1}(I+\frac{\alpha}{2}W),\quad B^{(t+1)}=QB^{(t)}
4. Summary of Cross-Architecture Analysis
  • LLaMA family: Fisher mass concentrated in the early stack close to the input.
  • Qwen3 family: selected layers shifted toward the middle of the stack.
  • Gemma-2 family: a polarized pattern concentrated at both the top and bottom of the stack.
5. Conclusion: The Emergence of Super-Additivity

Layer selection and Stiefel constraints combine orthogonally in FORA, producing super-additive synergy. Beyond simply reducing parameter count, this presents an efficiency frontier that makes fuller use of adapter capacity within a limited parameter budget.

References
  1. Hu, E. J., et al. (2022). LoRA: Low-rank adaptation of large language models. ICLR.
  2. Park, J. Y., et al. (2025). Riemannian optimization for LoRA on the Stiefel manifold. Findings of EMNLP.
  3. Wen, Z., & Yin, W. (2013). A feasible method for optimization with orthogonality constraints. Mathematical Programming.
  4. Zhang, Q., et al. (2023). AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. ICLR.