Can metaheuristic coordinate search improve Fourier fine-tuning efficiency?

Task feedback replaces fixed frequency sampling rules

FourierFT builds weight updates from a small set of Fourier components, but its default uniform sampling and optional Gaussian band-pass sampling pick those coordinates through fixed, task-agnostic rules. The selected frequency coordinates therefore cannot adapt to the downstream task. HeuFouFT changes the selection step: it searches for coordinates using downstream performance rather than a preset sampling pattern.

The motivation is spectral coverage. The paper's Figure 1 shows a learned LoRA update on GPT-2-Medium layer-1 Q projection (rank 8, E2E) whose spectral energy is localized, and it argues that uniform or radially constrained sampling covers that energy poorly. The authors note this spectrum is not an oracle for FourierFT, but it motivates selecting coordinates from task feedback.

Block probes, a Random Forest filter, then metaheuristic search

HeuFouFT first builds a coarse intensity map from lightweight block-level probes. The 1024×1024 frequency-domain matrix is split into 100 non-overlapping 10×10 blocks, and each block is lightly fine-tuned on 10% of the E2E training data (4,200 samples, batch size 4, max sequence length 128, 1 epoch) to compute heuristic intensity from weighted NLG metrics. Each block takes about 45 seconds, giving roughly 75 minutes and 8.2×10^17 FLOPs for the map.

That map initializes three metaheuristic optimizers: Genetic Algorithm with Simulated Annealing (GA-SA), Particle Swarm Optimization (PSO), and Cuckoo Search (CS). During search, a Random Forest surrogate filters each population so only the top 30% of candidates proceed to proxy fine-tuning. The surrogate is trained on 5,000 randomly generated sampling schemes, each labeled by a lightweight fine-tuning run taking 30 seconds, for about 6.94 hours and 2.05×10^19 FLOPs. The paper reports the surrogate cuts overall computational cost by 68% by filtering out 70% of low-potential candidates, though data collection dominates the RF-related cost at 81%.

All three variants beat fixed sampling and LoRA on five metrics

The evaluation uses GPT-2-Medium on E2E with 42,000 training examples and 4,600 examples each for validation and testing. Baselines are Full FT, LoRA, LoCA, and FourierFT with uniform or Gaussian band-pass sampling. Metrics are BLEU, NIST, METEOR, ROUGE-L, and CIDEr. All PEFT methods adapt the Q and V projections, while Full FT updates the complete model.

Across these five metrics, all three HeuFouFT variants outperform random-uniform FourierFT, Gaussian band-pass FourierFT, and LoRA. PSO goes further and outperforms LoCA, the best-performing baseline, on four metrics while using 37.6% fewer trainable spectral coefficients. The abstract and discussion both state the 37.6% reduction relative to the FourierFT and LoCA configurations.

Different optimizers reach similar performance from different spectra

A notable internal comparison is that the three searches reach similar performance from distinct frequency distributions. The authors read this as evidence that the benefit stems from task-guided selection itself rather than from any particular optimizer. In other words, GA-SA, PSO, and CS are alternative ways to explore the same task-guided objective, and the shared ingredient is downstream feedback rather than the search algorithm's identity.

This matters for readers comparing methods: the paper's claim is about the selection principle, not about crowning one metaheuristic. PSO's edge over LoCA on four metrics is a result for that variant, while the broader claim covers all three variants beating fixed sampling and LoRA.

Once coordinates are chosen, fine-tuning needs 15–18% of Full FT FLOPs

The efficiency claim has two parts that should not be conflated. The 37.6% figure concerns trainable spectral coefficients relative to the FourierFT and LoCA configurations. The 15–18% figure concerns FLOPs of Full FT once coordinates are selected. These are different denominators: one counts parameters, the other counts compute during final fine-tuning.

The paper also reports that the FLOP and memory reduction metrics in the main text are end-to-end, covering heuristic intensity map construction, surrogate labeling and training, metaheuristic sampling search, and final LLM fine-tuning. That end-to-end accounting is what makes the comparison with baselines such as LoRA and vanilla FourierFT, which only include final training, a comprehensive one.

The evidence is one model, one dataset, one hardware setup

The reported results come from GPT-2-Medium on E2E, with all experiments on a single NVIDIA A100 80GB GPU in FP16 precision using PyTorch 2.1 and CUDA 12.2. The paper's own cost discussion frames the approach as scalable to larger models such as Llama 3-7B, but that is a stated expectation, not a measured result in this evidence.

The search pipeline also carries a one-time overhead: the heuristic intensity map takes about 75 minutes, and labeling 5,000 schemes for the surrogate takes about 6.94 hours. The paper notes data collection dominates RF-related cost at 81% and is a one-time overhead for multiple metaheuristic iterations. Readers weighing adoption should treat these costs as part of the method, not as incidental setup.

The priority is task-guided selection, not the optimizer

The paper's most decision-relevant claim is that task-guided coordinate search allocates a limited spectral adaptation budget more effectively than fixed sampling. The supporting pattern is that all three variants beat fixed sampling and LoRA across five metrics, PSO beats LoCA on four metrics with 37.6% fewer trainable spectral coefficients, and final fine-tuning needs only 15–18% of Full FT FLOPs once coordinates are selected.

Because the three searches reach similar performance from different frequency distributions, the variable worth prioritizing is whether coordinate selection is driven by downstream performance at all. If a reader is comparing Fourier fine-tuning strategies, the threshold this paper sets is not which metaheuristic to pick but whether the selection step responds to the task; the optimizer choice appears secondary within the tested setup.

Sources used in this answer