Inverse Problems & Regularization

Definition

Inverse Problem: given a forward model $d=Ax+ϵ$ (known observation/physics operator $A$, data $d$, unknown $x$, noise $ϵ$), recover $x$.

Ill-posed (Hadamard 1902): fails one of existence, uniqueness, or stability (small data perturbations yield small solution perturbations).

Discrete Inverse Problem: the discretization $Ax=d$ often has a very large condition number — small data noise amplifies into large solution perturbations.

Regularization trades strict data fit for stability via a regularizer $R(x)$ and parameter $\lambda$: ${\min }_x\Vert Ax-d{\Vert }^2+\lambda R(x)$ Common choices: Tikhonov $\Vert x{\Vert }^2$ or $\Vert Lx{\Vert }^2$, sparsity $\Vert x{\Vert }_1$ (compressive sensing).

Core Arguments: choosing $\lambda$

$\lambda$ too small → noise-dominated; too large → regularizer-dominated. Zhenyu compared four methods:

1. L-Curve: log-log plot of $\Vert x_{\lambda }\Vert$ vs $\Vert A{x}_{\lambda }-d\Vert$; the “corner” marks best $\lambda$. Geometric intuition, needs no noise estimate; fragile when no clear corner exists.
2. L-Curve Curvature: numerical curvature $\kappa (\lambda )$ automates corner detection; sensitive to noise in the differentiation.
3. Generalized Cross Validation (GCV) [Golub, Heath, Wahba 1979]: $\text{GCV}(\lambda )=\Vert A{x}_{\lambda }-d{\Vert }^2/[m-\text{tr}(I-H(\lambda )){]}^2$; closed-form approximation to leave-one-out CV. Statistically optimal for prediction MSE; may be flat; sometimes underregularizes.
4. Morozov Discrepancy Principle: pick $\lambda$ s.t. $\Vert A{x}_{\lambda }-d\Vert =\delta$ (noise level). Principled; requires known $\delta$.

Different Perspectives

• Hansen (2010): L-Curve preferred when noise is unknown — strongest geometric intuition
• Golub (1979): GCV statistically optimal under Gaussian noise
• Zhenyu’s DSCOVR observation: L-Curve ≈ Curvature; GCV tends to underregularize on real data — no silver bullet, use multiple + check physical plausibility

Core Tools

• SVD + Tikhonov closed form: $x_{\lambda }={\sum }_i\frac{{\sigma }_i}{{\sigma }_i^2+\lambda }{u}_i^{T}d\cdot v_i$ — the filter-factor view reveals which singular directions are kept vs suppressed
• Condition number analysis: $\text{cond}(A)={\sigma }_1/{\sigma }_n$
• Picard condition: check $|u_i^{T}d|$ decays faster than ${\sigma }_i$ down to noise — otherwise recovery is fundamentally hopeless
• Filter factors: Tikhonov → smooth transition; TSVD → step (keep top $k$ singular components)

Open Questions

• Nonlinear inverse problems: most real retrievals are $d=f(x)+ϵ$ with nonlinear $f$ — how do the four methods extend?
• Bayesian unification: $\lambda$ ↔ prior precision; choosing $\lambda$ ↔ Type-II MLE/MAP — the tradeoff vs full MCMC over hyperparameters (see Gaussian Process Bayesian Inversion)?
• Deep-learning “regularization” (dropout, weight decay, augmentation) — formal link to classical Tikhonov?

Applications

Used/planned: DSCOVR retrieval, MODIS×CALIPSO joint aerosol, nuclear winter soot profile retrieval, exoplanet spin-orbit tomography. Transferable to: medical imaging (CT, MRI), geophysical prospection, atmospheric trace gas retrieval (CH4, CO2), astronomical image reconstruction.

Sources

• — code + comparison experiments + self-study notes
• Undergraduate Research @ Peking University & Caltech/UCR — project background
• Textbook (raw): Hansen Discrete Inverse Problems 2010
• Original paper (raw): Golub, Heath, Wahba 1979 (GCV)
• Lecture notes (raw): Morozov discrepancy principle
• [Pending] Further .docx study notes from Zhenyu’s research-angle LLM Wiki, to be merged in later

Related Pages

• Gaussian Process Bayesian Inversion — Bayesian dual of regularization
• MCMC & Bayesian Inversion — skill page
• Satellite Remote Sensing & Data Processing — application vehicle