Improved analysis‐error covariance matrix for high‐dimensional variational inversions: application to source estimation using a 3D atmospheric transport model
Bibliographic record
Abstract
Variational methods are widely used to solve geophysical inverse problems. Although gradient‐based minimization algorithms are available for high‐dimensional problems (dimension >10 6 ), they do not provide an estimate of the errors in the optimal solution. In this study, we assess the performance of several numerical methods to approximate the analysis‐error covariance matrix, assuming reasonably linear models. The evaluation is performed for a CO 2 flux estimation problem using synthetic remote‐sensing observations of CO 2 columns. A low‐dimensional experiment is considered in order to compare the analysis error approximations to a full‐rank finite‐difference inverse Hessian estimate, followed by a realistic high‐dimensional application. Two stochastic approaches, a Monte‐Carlo simulation and a method based on random gradients of the cost function, produced analysis error variances with a relative error <10 % . The long‐distance error correlations due to sampling noise are significantly less pronounced for the gradient‐based randomization, which is also particularly attractive when implemented in parallel. Deterministic evaluations of the inverse Hessian using the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm are also tested. While existing BFGS preconditioning techniques yield poor approximations of the error variances (relative error >120 % ), a new preconditioner that efficiently accumulates information on the diagonal of the inverse Hessian dramatically improves the results (relative error <50 % ). Furthermore, performing several cycles of the BFGS algorithm using the same gradient and vector pairs enhances its performance (relative error <30 % ) and is necessary to obtain convergence. Leveraging those findings, we proposed a BFGS hybrid approach which combines the new preconditioner with several BFGS cycles using information from a few (3–5) Monte‐Carlo simulations. Its performance is comparable to the stochastic approximations for the low‐dimensional case, while good scalability is obtained for the high‐dimensional experiment. Potential applications of these new BFGS methods range from characterizing the information content of high‐dimensional inverse problems to improving the convergence rate of current minimization algorithms.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.001 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".