Least Angle when ATAATA is singular

计算科学 regression least-squares constrained-optimization
2021-12-26 06:16:48

I'm teaching myself this regression stuff, so forgive me if this is a basic question. I can't seem to find a discussion of my particular problem.

So I'm least-squares-ing this overdetermined system Ax=b, and I'm interested in minimizing ||x||1.

Problem is, my ATA is singular, so I can't use Least Angle.

I don't know how many of the columns of A are redundant without computing rank explicitly, and I don't really care anyway. I'd rather use that redundancy to get the best 1-norm I can.

Suggestions?

EDIT:

I'd settle for a minimized ||x||2.

1个回答

Minimizing the 2-norm of x among all least squares solutions is relatively easy to do- this is the pseudoinverse solution. It can be computed using either a rank revealing version of the QR factorization of A (there are specialized versions of this that work well on large and sparse A matrices) or by using the Singular Value Decomposition. Since you're really more interested in minimizing x1, I'll move on to answering that question.

The problem

minx1

subject to

Axb2δ

is equivalent (with a suitable choice of λ) to the basis pursuit denoising problem (BPDN)

minAxb22+λx1

Minimizing x1 has (in many circumstances) the effect of finding a sparse solution to the problem of approximating Ax=b. These sparse approximation problems have been extensively studied because of their application to problems of compressive sensing.

As a practical matter, you probably don't want to limit your x to being an actual least squares solution but rather to set some tolerance on Axb2. If for some reason you do need to have an x that is actually a least squares solution, you can start by finding any least squares solution x^, then let δ=Ax^b2.

Algorithms for solving these sparse norm approximation problems are specialized enough that you're probably best off making use of existing software rather than trying to implement the algorithms yourself. I would recommend the list of software packages (as well as the many references to books and papers) at

http://dsp.rice.edu/cs