Robust Statistics

For the last couple of days I got reacquainted with an old friend - robust statistics. Many of us in engineering learned quite a bit of statistics and probability theory. Most of us apply normal statistics (that is, statistics built on the assumption that the distribution is normal) in our day-to-day work. For example, the mean is a consequence of normal statistics. But, in the "real world" the data doesn't have to follow a normal distribution, and measurements are often corrupted with non-normal (a.k.a. non-gaussian) deviations. This can wreak havoc with normal statistics.

Enter robust statistics. Robust statistics is much more tolerant of the errant data that exists in the real world. Machine learning has flocked to robust statistics like a moth to a candle, so you see more of it now than you did 10 years ago (at least it seems that way). But I don't do machine learning. Rather, I was focused on a problem of measurement.

What I wanted was a way to separate form from deviation. But a typical filter would be "mislead" by the potentially massive deviation that I was trying to filter out. In my searching I stumbled upon ISO 16610-71: "Robust areal filters: Gaussian regression filters" and I saw my old friend - the biweight function. A splendid M-Estimator that I used to solve a different problem in 2004. I hadn't really "forgotten" about it, but it certainly had drifted from my mind over the years. The math was instantly familiar and I had to try it out on my problem. A few hours of some hack-and-slash Matlab later, I had a working prototype and some very nice results.

So what's the point? (1) Don't forget what you spent years learning in school. It wasn't useless, and it takes constant effort to retain those skills. (2) Research, search, and search again. Someone else has also faced your problem, and they probably have a good idea you can leverage. (3) Deep understanding lets you solve a breadth of problems. Spending some time building technical depth and foundational knowledge beyond what is needed day-to-day gives you the tools to tackle the truly hard problems that come across your desk every now and then.