Can second-order optimization train variable projection SVMs for road abnormality detection?

Why the kernel makes training hard

A variable projection support vector machine (VP-SVM) pairs an SVM classifier with an adaptive feature transformation. The transformation maps each input signal into a low-dimensional space, and its parameters carry physical meaning; in this application they describe adaptive Hermite functions fitted to tire-sensor waveforms. Those transformation parameters are optimized together with the SVM weights, so the model keeps interpretable parameters rather than the tens of millions typical of deep networks.

The cost is numerical. Kernel methods build a kernel matrix whose size scales with the number of training signals, and in a VP-SVM the kernel also depends on the learnable transformation parameters. The paper notes that an earlier VP-SVM formulation handled this by optimizing a reduced objective with stochastic subgradient methods, which the authors say can yield suboptimal classification and weaker interpretability. The question the paper poses is therefore narrow: can second-order information make this training both cheaper and better?

Analytic derivatives replace finite differences

The paper's technical contribution is a set of analytic formulas for the derivatives needed by second-order optimizers. It derives second-order derivatives of Moore-Penrose pseudo-inverses and of variable projection operators, plus Hessians of the primal quadratic VP-SVM objective. It also gives first- and second-order partial derivatives of the objective terms with respect to the transformation parameters, assuming the kernel is twice differentiable, and a mixed derivative with respect to both the SVM weights and the transformation parameters.

This matters because a trust-region method needs curvature information to choose its steps. Supplying that information in closed form, rather than approximating it, is what lets the solver exploit the structure of the variable projection problem. The authors state that although the results are written for VP-SVM, they extend to any kernel-based model whose kernel depends on learnable parameters.

A 5,171-signal tire-sensor benchmark

The evaluation uses a dataset of q = 5,171 one-dimensional signals recorded on the public road network of Budapest. Each signal is a voltage change from a force sensor implanted in a tire, so it reflects tire deformation; the signals were segmented into single tire revolutions and zero-padded into equal-length, compactly supported waveforms. The task is to recognize road surface abnormalities such as potholes or bumps.

The model represents each signal with only five Hermite-Fourier coefficients, built from adaptive Hermite functions. The paper notes that a prior VP-based neural network used eleven adaptive Hermite functions and reached an average of 97% accuracy on this dataset. The comparison here is not against that network but between two training schemes for the simpler VP-SVM, both evaluated with 5-fold cross-validation.

Trust region wins on accuracy and iteration count

Table 1 reports mean accuracy, minimum and maximum accuracy across folds, mean gradient norm, iteration count and wall-clock time. The trust-region method reached 95.5% mean accuracy (range 92.2% to 98.1%) in 15 iterations and 1.73 seconds, with a mean gradient norm of 9.16e-04. Gradient descent needed 1,000 iterations and 13.63 seconds to reach 92.1% mean accuracy (range 90.3% to 96.2%), with a mean gradient norm of 8.24e-04.

The intermediate gradient-descent rows show the trade-off clearly: 25 iterations gave 72.3% mean accuracy in 0.33 seconds, 250 iterations gave 88.2% in 3.39 seconds, and 500 iterations gave 91.3% in 6.78 seconds. So gradient descent is cheaper at very small iteration budgets, but it only approaches the trust-region result after roughly two orders of magnitude more iterations and about eight times the training time. The paper also reports a second trust-region run at 25 iterations with 96.3% mean accuracy and a 2.24e-04 gradient norm in 5.39 seconds, so the advantage is not confined to a single budget.

A simpler model lands near the prior 97%

The authors frame the accuracy result as competitive rather than superior. Their VP-SVM uses five Hermite-Fourier coefficients, while the earlier VP-based neural network used eleven adaptive Hermite functions and averaged 97% accuracy on the same dataset. The trust-region-trained VP-SVM's 95.5% mean is described by the authors as close to that state of the art, achieved with a much simpler model.

This is a comparison across different model classes and feature counts, not a controlled head-to-head test, so it should be read as context for the training result rather than as evidence that the VP-SVM is the better classifier. The paper's own controlled comparison is the solver comparison within the same VP-SVM setup.

What the evidence does and does not cover

All experiments ran on a single workstation with an Intel Xeon W-2123 CPU at 3.60 GHz, 64 GB RAM, Python 3.10.12 and PyTorch 2.4.0 with automatic differentiation, and all models were trained on the CPU. The reported times are therefore specific to that hardware and implementation, and the paper does not report a multi-hardware or large-scale scaling study. The dataset is one collection of tire-sensor signals from Budapest roads, so the accuracy figures describe that benchmark rather than road abnormality detection in general.

The paper also does not report a statistical significance test between solvers; the comparison rests on cross-validation means and ranges across five folds. The extension to other parameter-dependent kernels is stated as a property of the derivation, not demonstrated experimentally here.

Choose trust region when iterations are the bottleneck

The decision-relevant split in this paper is between iteration budget and per-iteration cost. If the constraint is the number of optimizer steps, the trust-region method is the clear choice: 15 iterations produced 95.5% mean accuracy, while gradient descent at 25 iterations produced 72.3% and needed 1,000 iterations to reach 92.1%. If the constraint is strictly wall-clock time at a very small budget, gradient descent's 0.33 seconds for 25 iterations is cheaper than any trust-region run reported.

For a VP-SVM on this tire-sensor task, the practical priority is to pay for curvature information up front rather than to run many cheap first-order steps. The paper's own framing supports that reading: the authors set out to show that training time can be greatly reduced with a properly chosen second-order scheme while generalization improves, and the reported numbers show both a lower iteration count and a higher mean accuracy than the gradient-descent baseline.

Sources used in this answer