Calculus of Variations, 1/2

I've decided to post on this topic for two reasons. First, I want to solidify my own comprehension; I was introduced to variational calculus as a supplemental portion of a Finite Element Analysis course that I took during the third year of my BME. When my FEA professor lectured on it, I wasn't particularly comfortable with the method; I got the gist of what was being done but I wasn't able to reconstruct it on my own when I sat down to a piece of paper. Second, since that lecture, I have recognized it in a handful of applications in the realm of optimization, mechanics, control theory, and FEA; Those topics are pretty damned important for someone interested in robotics. For those reasons, although plenty of people have produced better summaries and  course materials on the topic, I feel compelled to make a modest attempt for my own sense of completion.

To start, it is sometimes useful to learn about the history of a subject in order to understand the context in which it was developed. As far as mathematical methods go, there is usually some famous problem for which it was developed. In the case of variational calculus, that problem was finding the Brachistochrone Curve. The problem was posed by Johann Bernoulli as part of a correspondence between friends and contemporary mathematicians of his time. He asked his colleagues to determine the optimal curve that a bead would follow through a uniform gravitational field between two points in the shortest amount of time. As one might expect, the name is derived from the latin brachis meaning "shortest" and chron meaning "time".

The Brachistochrone Problem

My Introduction to Machine Learning

Open classroom exercises

A couple of months ago, I worked through some of the exercises for Andrew Ng's OpenClassroom Machine Learning Class. I think the site is now defunct (or maybe morphed into coursera) but the assignments were still available at the time of this post. The experience left me with a pretty good basis for starting to learn about using artificial neural networks (in addition to other ML algorithms) to solve optimization and classification problems. This is a very exciting field for me to peer into as it involves some promising prospects for the creation of intelligent machines. Deep learning, one ML research area that seems to be prevalent, is essentially just the clever utilization of very large neural network models to learn complex relationships. One of my current goals is to comprehend some of the strengths, weaknesses, and challenges involved with deep learning networks and their implementation.

Childhood dream

My first exposure to neural networks was Jeff Hawkins' book, On Intelligence, where he proposed ways that neural networks might be organized to mimic the neocortex of the human brain and achieve a powerful self-teaching mechanism. At the time, this book had me very excited - the idea seemed simple enough to implement (a 13 year old kid could loosely comprehend it) and the notion of building an actual learning machine is very appealing. Mary Shelley's Frankenstein had people excited when it was written and all Dr. Frankenstein did was assemble limbs - he had an intact brain and didn't have to put one together from scratch! However, I wasn't scrutinizing enough to consider questioning the claims in the book and I didn't know enough about neural networks to evaluate them. Since then, I've heard some critical things from people in the machine learning and artificial intelligence communities but I still haven't been exposed to the topic sufficiently to make a judgement on the value of that book nor the validity of the claims therein. It looks like Mr. Hawkins' company, Numenta, has been pretty closed about their work but recently announced that they would be open sourcing their Cortical Learning Algorithm. I'm interested in learning more about the progress he has made -  in any case, his book made a lasting impression on me what with inspiration and so forth.

My current conception

While neural networks may provide powerful and flexible algorithms, there is still a great deal to be discovered about their behavior and, historically, a kind of stigma surrounding their use. It seems that there was a little bit of hype followed by disenchantment in their history. Machines weren't powerful enough to implement them at the scale required for useful or impressive results and they were seen as flexible but inefficient toys for solving problems that weren't necessarily novel. Another limitation was the lack of an appropriate back-propagation algorithm that could allow for many-layer ANNs. Today, neural networks can be seen in a variety of useful applications, and the niche seems to be growing.

One strong application of ANNs is the classification of datasets. Classification problems involve training a network with samples of a known class such that a decision boundary is defined between two regions in the feature space. Instances that lie on one side of that boundary will be classified in one way while instances that lie on the other will be classified in the opposite way. The process of 'learning' in this case can be reduced to manipulating the boundary between those two classifications. If the data used to 'train' a neural network is representative of the task as a whole, then one can expect relatively accurate classifications of new data.