Convex optimization: some thoughts and after thoughts

Last semester I took a course on convex optimization. Unless you are a math major in a graduate school, you may not have heard about this subject before (I certainly had not). I knew it was going to be very heavy on math, but I still decided to take the course. If for nothing else but to convince the older (but not necessarily wiser) me that it could still do what the (much) younger (but not necessarily more foolish) me could do.

Convex optimization deals with minimizing (or maximizing) multi variable functions given several (typically) multi variable constraints. Maximizing the returns from a stock market portfolio given constraints on risks and cap on investment, estimating the best parameters for a machine learning model to minimize prediction errors, device sizing in electronic circuits by minimizing power consumption are all examples of real world convex optimization problems.

The course turned out to be more mathematical than I thought (which meant all my late weeknights were spent doing homework) but it was a fun ride and in process I was pleasantly surprised to gain a realization that I did not see coming: solving math problems is not very different from how you would approach solving a hard problem in day to day life. Indulge me a bit here.

The first thing I learnt about convex optimization was to ask a few questions before even beginning to solve a problem, does a solution even exist? If it does exist can we ever find it? Can we find something that is not the true solution but close enough to be acceptable? If not posed in a mathematical context, these questions would fit in perfectly in the context of any real non-mathematical problem as well (peace in the Middle East, averting a government shutdown, your favorite hard to solve problem).

Many problems (certainly the non-trivial ones) do not have a perfect analytical solution, but they do have a numerical solution. In other words, there is no single formula, no one system of equations to solve which would give the solution. Instead, what we have is an iterative approach where we constantly keep on trying to minimize the error and keep getting closer and closer to the true solution until we decide, this is good enough, let’s stop. Sounds familiar?

Finally, and we don’t often think in these terms, but, constraints (or limitations to use a different term) are good. They bind the search space. They give us a defined place where to look for a solution. Think of how many times at work the constraints of time and resources has forced a solution, may not have been the best solution but certainly better than having a completely open ended problem.

Challenging as it was, I highly recommend a course on optimization as it helps you develop a keenly mathematical perspective. Especially for data scientists this is an absolute must have to understand how machine learning algorithms work under the hood.