How Can Optimal Control Be Learned from Data in Reproducing Kernel Hilbert Spaces?

What the method removes from the control problem

The target is a control-affine stochastic diffusion: the state drifts according to an unknown function plus an unknown input matrix times the control, and is perturbed by a Wiener process scaled by a diffusion parameter. The paper's setting is deliberately sparse. Both the drift and input functions and the stage cost are unknown; only a control penalty function and the admissible control set are given. Learning therefore has to start from snapshot data of the form state, input, next state, and stage cost, collected at a small sampling time.

This matters because the usual route to optimal control needs a model. The paper positions itself against local methods that require accurate initialization and can settle in suboptimal local solutions, and against dictionary-based Koopman approaches that first fix a finite set of observables. The proposed alternative is nonparametric: the approximation space comes from the kernel and the sampled data, so the effective basis adapts to the data rather than being chosen in advance.

Probability densities become the object being learned

Instead of tracking individual trajectories, the method tracks how the probability density of the state evolves. For a fixed feedback law, the transition operator maps the initial density to the density at a later time, and its infinitesimal version is the Fokker-Planck-Kolmogorov operator. The paper embeds these densities into a reproducing kernel Hilbert space, a function space with an inner product defined by a kernel, so that operator regression techniques can be applied.

The practical payoff is that the dynamics enter as a linear operator acting on densities even though the underlying system is nonlinear. The paper estimates finite-rank approximations of the Markov transition operators of the controlled diffusion by kernel regression, and reports that the sample and time complexity of this identification step scales linearly with the number of states and controls. That linear scaling is the claim that makes the approach relevant to higher-dimensional problems, where solving Hamilton-Jacobi-Bellman equations directly becomes intractable.

A convex recursion replaces policy iteration

The learned operators plug into a dynamic programming recursion derived from a convex reformulation of the stochastic control problem. By changing variables so that the control appears multiplied by the density, the problem becomes a convex optimization over densities, and the optimal feedback law is recovered as the ratio of the optimized control variable to the optimized density. Discretizing time turns this into a backward recursion over value functions, where each step applies the adjoint of the estimated transition operators and minimizes the control penalty plus a term involving the next value function.

Two consequences follow. First, the optimal policy is extracted directly from the value function, so no initial stabilizing policy is needed, unlike policy-iterative schemes or methods that blend local pre-stabilizing controllers with global ones. Second, the recursion accommodates general non-quadratic stage costs, in contrast to kernel-based linear quadratic regulator analogs, and it avoids the synthesis bottlenecks of sum-of-squares approaches. The paper also notes that, unlike an earlier method relying on specific control excitations to learn infinitesimal generators, this approach imposes no such excitation condition, which is what makes it applicable to real-system data.

One-dimensional benchmarks show uneven accuracy

The first numerical evidence comes from four one-dimensional systems, each with a known optimal control law used as ground truth. Identification used 1000 training points for systems S1, S2, and S3, and 400 points for S4, with the recursion run for 5000 steps at a sampling time of 10^-3 s for some settings and 500 steps at 10^-2 s for others to ensure value-function convergence. Accuracy is reported as root mean squared error between the estimated and optimal control laws over 100 test points.

The errors differ substantially across systems. S1 reached a mean RMSE of 1.29×10^-2 with a standard deviation of 1.59×10^-3, S3 reached 2.17×10^-2 with 8.13×10^-3, and S4 reached 1.91×10^-2 with 2.94×10^-4. S2 was the outlier at 1.31×10^-1 with a standard deviation of 1.53×10^-1, an order of magnitude worse and far more variable. The paper does not attribute that gap to a specific cause in the extracted text, so the honest reading is that the method's accuracy is system-dependent rather than uniformly tight.

A learned law stabilizes an uncontrollable equilibrium

The two-dimensional test is a Van der Pol oscillator with a linearly uncontrollable equilibrium at the origin and an unstable limit cycle. The system admits an explicit Hamilton-Jacobi-Bellman solution with globally optimal feedback law equal to minus the product of the two states. The method used 2500 training points sampled from a grid with both states bounded between -3 and 3, with a radial basis function scale of 20.

The learned feedback law closely matched the known optimum, with an RMSE of 7.24×10^-2 on 900 test points, and the closed-loop system under the learned law exhibited an asymptotically stable origin. This is a stronger test than the one-dimensional cases because the target behavior is not merely tracking a reference but removing an unstable limit cycle while the linearization at the origin provides no control authority. The result supports the claim that the approach captures nonlinear structure that a linear design would miss, though it remains a simulated benchmark with a known analytical answer.

The AUV test compares cost, not model knowledge

The headline application is depth control of an autonomous underwater vehicle moving at a constant forward velocity of 2 m/s in a vertical plane. The state has four components: heave velocity, pitch rate, depth, and pitch angle; the control is the fin deflection angle. Training used 8000 independent and identically distributed samples drawn within specified bounds on the four states with random inputs, collected at a sample rate of 0.5 s, with a radial basis function bandwidth of 35, a diffusion parameter of 0.001, and a regularization value of 10^-8. The recursion ran for 1000 steps.

The comparison is against a linear quadratic regulator using the same cost matrices, which benefits from exact knowledge of the system parameters. The learned law had no access to the model or its parameters. Over 50 simulation runs of 50 s each, tracking two depth references that switch at 25 s, the learned law achieved a mean accumulated-cost reduction of 5.09% and a standard deviation reduction of 30.11% relative to that baseline. The paper explains the result by noting that the linearized model is exact at the operating point reached after the transient, because depth is translation-invariant in the nonlinear dynamics, while the learned law exploits the full nonlinear dynamics.

The fin deflection was constrained to a magnitude of at most 0.4363 rad, about plus or minus 25 degrees, and the constrained minimizer was obtained by projecting the unconstrained one onto that interval. That detail matters for interpreting the comparison: the constraint is handled by projection in this example, so the reported gains apply to a setting with a box constraint on a single input, not to general state-safety constraints.

Where the linear-scaling claim holds and stops

The paper's central structural claim is that both identification and control have polynomial runtime complexity because the approach scales linearly with state dimension. That claim is supported by the complexity summary and by the four-dimensional AUV example, which is the largest system demonstrated. It is not a demonstration on genuinely high-dimensional systems, and the paper lists large-scale regression via data-efficient low-rank estimators and feature spaces for further complexity reduction as future directions rather than completed results.

A second open question is error propagation. The paper explicitly names quantifying how operator-estimation errors propagate through the dynamic-programming recursion and translate into policy suboptimality as future work. Until that is established, the reported RMSE values and the AUV cost reductions should be read as empirical outcomes of specific benchmarks, not as guarantees that follow from the method's structure. The S2 result, with its much larger and more variable error, is the clearest internal signal that per-system behavior can differ markedly.

The priority is model-free nonlinearity, not universal superiority

The most decision-relevant comparison in this paper is not learned control versus a naive baseline but learned control versus a well-informed linear design. The LQR controller knew the exact parameters and still incurred higher mean accumulated cost and roughly 30% more variability in that cost across runs, while the learned law used only sampled data. The paper's own explanation is that the linearization is exact at the post-transient operating point, so the learned law's advantage comes from exploiting nonlinear dynamics outside that point rather than from compensating for missing model knowledge.

That framing sets the practical threshold for a reader. The method is most attractive when the dynamics and stage cost are genuinely unknown, the stage cost is not restricted to a quadratic form, and the state dimension is large enough that direct dynamic programming is intractable but still within the range where kernel regression with the reported sample sizes is feasible. The AUV result, at four states and 8000 training samples, marks the demonstrated scale; the linear-scaling argument and the unresolved error-propagation question mark where the evidence ends.

What the benchmarks leave unsettled

Several boundaries sit close to the results they qualify. The one-dimensional and Van der Pol tests compare against analytically known optimal laws, so they measure approximation accuracy rather than closed-loop robustness under model mismatch. The AUV test does evaluate closed-loop cost over 50 stochastic runs, but the comparison baseline is a linear controller on a model chosen to permit linear design, and the constraint is a single box constraint handled by projection. The paper states that systems requiring stricter state-safety specifications could be addressed by assigning large penalties to unsafe regions in the stage cost, and flags this as future work rather than a demonstrated capability.

The assumptions also bound the setting. The analysis assumes continuously differentiable drift and input functions, a stage cost that is continuously differentiable, bounded below, and radially unbounded, a closed convex control set with nonempty interior, and a strongly convex control penalty, plus a growth condition as the state grows. The paper describes the first five as standard and typically not restrictive, and the growth condition as often readily met because one constant can be large. These are conditions on the problem class, not on the data, and they are what the convergence and optimality statements rest on.

Reading the result as a trade of assumptions

The paper's contribution is best understood as moving where knowledge is required. It gives up model equations and the stage cost, and in exchange requires sampled transitions, a kernel choice with a bandwidth selected per system, and a control penalty plus constraint set supplied by the designer. The reported numbers are conditional on those choices: the radial basis function scales differ across benchmarks, ranging from 1 to 35, and the paper reports them as part of each experiment rather than deriving them.

So the fair summary is narrower than full model-free optimal control. The method delivers a globally optimal policy in the convex reformulation without an initial stabilizing policy, without an explicit dictionary, and without restrictive cost structure, and it does so at a cost that scales linearly with state dimension in the paper's accounting. Whether that combination beats a parameter-aware linear design depends on how much nonlinearity the operating regime exercises, which is exactly what the AUV comparison illustrates and what the unresolved error-propagation question would need to quantify.

The paper-specific takeaway

For a reader deciding whether this approach applies, the decisive split is between problems where a linear design is already exact at the relevant operating point and problems where the operating regime crosses nonlinear structure. In the AUV case the linearization was exact after the transient, and the learned law still won on mean cost and on cost variability, which suggests the gain came from the transient and from the switching between depth references rather than from steady-state behavior. That is the specific scenario the evidence supports: unknown nonlinear dynamics, a non-quadratic or general stage cost, a single box input constraint, and a state dimension small enough for kernel regression at the reported sample sizes.

The paper's own priority for future work points to the same gap. Quantifying how operator-estimation errors propagate into policy suboptimality is the missing link between the benchmark RMSE values and any claim of near-optimality in a new setting. Until that link exists, the strongest defensible statement is that a fully data-driven kernel method reproduced known optimal laws to within reported errors on small benchmarks and reduced mean and variability of accumulated cost against a parameter-aware LQR on a four-state underwater vehicle task.

How the pieces fit together

The method's logic runs in one direction: densities are embedded in a kernel space, transition operators are estimated from snapshot data by kernel regression, and those estimates drive a convex backward recursion that returns a value function and, from it, a feedback law. Each stage removes a requirement that earlier approaches imposed. Kernel embedding removes the need for an explicit dictionary of observables. The convex reformulation removes the need for an initial stabilizing policy. The general stage-cost handling removes the quadratic-cost restriction of kernel-based linear quadratic analogs.

What remains is a set of empirical questions the paper opens rather than closes: how estimation error propagates, how the approach behaves at larger state dimensions than four, and how state-safety constraints beyond a box input constraint would be handled. The paper's contribution is a coherent pipeline with linear scaling in its accounting and competitive benchmark numbers, not a settled answer to those questions.

The comparison that matters most

Among all the numbers in the paper, the AUV comparison carries the most weight because it is the only one where a data-driven law faces a baseline with privileged information. The 5.09% mean cost reduction and 30.11% standard deviation reduction are modest in the first case and substantial in the second, and both are measured over 50 runs of a 50 s tracking task with two reference switches. The paper's explanation, that the linear controller is exact at the post-transient operating point while the learned law captures the full nonlinear dynamics, tells the reader where to expect similar gains: in regimes where the trajectory leaves the neighborhood of a single operating point.

The one-dimensional benchmarks and the Van der Pol test serve a different purpose. They show that the learned law can approximate a known optimal law, with errors ranging from about 1.3×10^-2 to 1.3×10^-1 in mean RMSE across the four one-dimensional systems and 7.24×10^-2 for the Van der Pol case. Those are accuracy measurements against analytical ground truth, not closed-loop cost comparisons, and they should not be read as evidence about robustness or about performance under model mismatch. Keeping the two kinds of evidence separate is what makes the paper's claim legible: accurate approximation on small problems with known answers, and a favorable cost comparison on one four-state problem against a parameter-aware linear design.

What a practitioner should check first

Before adopting the approach, a practitioner should check four things the paper makes explicit. First, whether the available data are snapshot transitions of state, input, next state, and stage cost at a small sampling time, since that is the input the identification step consumes. Second, whether the control penalty is strongly convex and the admissible set is closed, convex, and has nonempty interior, since those are among the assumptions the analysis rests on. Third, whether the stage cost can be evaluated on sampled states, because the method learns it from data rather than being told its form. Fourth, whether the state dimension and sample size are in the range the paper demonstrates, which is up to four states with 8000 samples in the AUV case.

The kernel bandwidth is a further practical choice the paper treats per system, reporting values from 1 to 35 across benchmarks without a derivation rule. A reader planning to reuse the method should treat bandwidth selection as an open engineering decision rather than a settled part of the recipe, and should expect per-system accuracy to vary, as the S2 result demonstrates.

What remains open after the benchmarks

The paper is explicit about two open directions that bound its claims. Error propagation from operator estimation into policy suboptimality is not quantified, so the relationship between the reported RMSE values and the optimality gap of the resulting policy is unknown. And state-safety constraints are not demonstrated; the paper suggests penalizing unsafe regions in the stage cost as a possible route and leaves it for future work. The AUV example handles only a box constraint on the fin deflection, obtained by projecting the unconstrained minimizer onto the interval.

These are not peripheral caveats. They determine whether the method's favorable comparison against a parameter-aware linear controller extends to settings where safety matters and where estimation error is larger than in the reported benchmarks. The paper's contribution stands on what it demonstrates: a fully data-driven pipeline that estimates Markov transition operators in a kernel space, solves a convex recursion with linear scaling in state dimension, and produces competitive control on benchmarks including a four-state underwater vehicle.

The final judgment on this paper

The paper's most valuable specific claim is that a kernel-embedded, operator-theoretic recursion can produce a feedback law from snapshot data alone that beats a parameter-aware linear quadratic regulator on both mean accumulated cost and cost variability in a four-state depth-tracking task. The 5.09% mean reduction and 30.11% standard deviation reduction over 50 runs are the numbers to remember, together with the paper's explanation that the advantage comes from exploiting nonlinear dynamics away from the post-transient operating point where the linearization is exact.

Everything else in the paper supports or bounds that claim. The one-dimensional and Van der Pol benchmarks show approximation accuracy against known optimal laws, with S2 standing out as a case of much larger and more variable error. The linear-scaling argument explains why the approach is tractable, and the unresolved error-propagation question marks the limit of what the benchmarks establish. A reader should take away the scenario split rather than a general verdict: expect the largest benefit where the operating regime exercises nonlinear structure that a linear design cannot represent, and expect the evidence to thin out as state dimension grows beyond four or as safety constraints become stricter than a single box constraint.

How to read the reported errors

The error numbers in the paper measure different things and should not be pooled. The one-dimensional RMSE values compare the estimated control law to the known optimal control law over 100 test points, with means from 1.29×10^-2 to 1.31×10^-1 and standard deviations from 2.94×10^-4 to 1.53×10^-1. The Van der Pol RMSE of 7.24×10^-2 compares the learned feedback law to the analytical optimum over 900 test points. Neither is a closed-loop cost, and neither says anything about behavior under a different diffusion parameter or a different sampling time.

The AUV numbers are different in kind: they are accumulated costs over 50 closed-loop simulation runs of 50 s each, compared against an LQR baseline, and reported as percentage reductions in mean and standard deviation. A 5.09% mean reduction and a 30.11% standard deviation reduction answer a question about closed-loop cost under stochastic simulation, not about how closely the learned law matches an optimal law, which is not available for that system in the paper. Treating the RMSE values and the cost reductions as interchangeable would misstate what the evidence shows.

Why the kernel choice carries the method

The reproducing kernel Hilbert space is not a cosmetic detail. It is what allows the transition operator to be estimated without fixing a finite dictionary: the approximation space is induced by the kernel and the sampled data, so the effective basis adapts. The paper contrasts this with extended dynamic mode decomposition and related Koopman approaches, which first choose a dictionary of observables and then identify a finite-dimensional operator on the span of that dictionary. By using suitable infinite-dimensional kernel spaces, the paper argues, arbitrary operators can be approximated to arbitrary accuracy.

That flexibility comes with a per-system tuning burden. The reported radial basis function scales are 1.2, 1.8, 2, and 1 for the four one-dimensional systems, 20 for the Van der Pol oscillator, and 35 for the AUV, with no derivation rule given in the extracted text. The S2 case, with a mean RMSE of 1.31×10^-1 and a standard deviation of 1.53×10^-1, is the clearest reminder that the kernel and its bandwidth materially affect the result and that the paper reports them as experimental choices rather than as outcomes of a selection procedure.

What the linear-scaling claim covers

The paper states that the approach scales linearly with state dimension, enabling polynomial runtime complexity for both identification and control, and summarizes this in a complexity table. The claim is about how computational cost grows with dimension, and it is the reason the authors present the method as a route to high-dimensional and uncertain optimal control problems. It is not a claim that accuracy is preserved as dimension grows, and the largest demonstrated system has four states.

The distinction matters because a reader could conflate tractability with reliability. The paper's own future-work list includes large-scale regression via data-efficient low-rank estimators and feature spaces for further complexity reduction, which indicates that the current implementation is not presented as the final word on scale. The demonstrated evidence covers one to four states, with sample sizes from 400 to 8000 training points, and the linear-scaling argument is what the authors offer beyond that range.

The assumptions that enable the guarantee

The convex reformulation and the existence of a unique minimizer rest on a set of conditions the paper states up front. The drift and input functions must be continuously differentiable, the diffusion parameter must be positive, the stage cost must be continuously differentiable, bounded below, and radially unbounded, the initial state must have a density in the appropriate space, the control set must be closed, convex, and have nonempty interior, and the control penalty must be strongly convex. A further growth condition bounds the drift and cost as the state grows.

The paper describes the first five as standard and typically not restrictive, and the growth condition as often readily met because one of its constants can be large. These conditions are what allow the problem to be recast as a convex optimization over densities with a unique minimizer and an optimal feedback law recovered as a ratio. A reader evaluating whether the method applies to a new system should check these conditions against the system rather than assuming the benchmarks generalize.

What the AUV experiment actually shows

The AUV experiment is a depth-tracking task with two references, 5 m and 2 m, switching at 0 s and 25 s, evaluated over 50 simulation runs with a diffusion parameter of 0.001. The learned law was trained on 8000 samples with random inputs and ran for 1000 recursion steps. The comparison baseline was an LQR controller using the same cost matrices and exact system parameters. The reported outcome is a 5.09% reduction in mean accumulated cost and a 30.11% reduction in the standard deviation of that cost.

The paper's interpretation is that the LQR controller performs well because the vertical-plane model is built to permit linear design and the linearization is exact at the operating point reached after the transient, owing to the translation invariance of depth. The learned law, without model access, achieved lower cost and lower variability by exploiting the full nonlinear dynamics. That interpretation is the authors' explanation, and it points to where the advantage is expected to appear: in the transient and in the switching between references, not in steady-state regulation at a single operating point.

The bottom line for readers

The paper answers its title question with a concrete pipeline and a concrete comparison. A feedback law can be learned from snapshot data by embedding state densities in a reproducing kernel Hilbert space, estimating Markov transition operators by kernel regression, and solving a convex Hamilton-Jacobi-Bellman recursion that scales linearly with state dimension. The evidence is a set of small benchmarks with known optimal laws, where mean RMSE ranged from 1.29×10^-2 to 1.31×10^-1 across four one-dimensional systems and reached 7.24×10^-2 on a Van der Pol oscillator, plus a four-state AUV task where the learned law reduced mean accumulated cost by 5.09% and its standard deviation by 30.11% against a parameter-aware LQR.

The judgment a reader should carry away is conditional. The method's advantage is demonstrated where nonlinear structure matters and where a linear design, even with exact parameters, cannot represent it. The evidence does not yet establish how estimation error propagates into suboptimality, how the approach behaves beyond four states, or how it handles safety constraints stricter than a box constraint on the input. Those are the questions that separate the paper's demonstrated result from a general claim about data-driven optimal control.

Sources used in this answer