In this lesson we are going to walk through the exciting topic of big data analysis. At a high level, big data analysis is the process that helps to build algorithms that can analyze vast amounts of data and be able to generate behavior-based decisions from that data.

For big data analysis in this section, we'll be using the Decision Tree gem. It uses the ID3 algorithm and is very efficient in taking data and making decisions based on it.

I'd like to show you a practical big data project I built for a client about a year ago. This was a large company that has more than a thousand trucks and they wanted an application that showed when each truck should be retired based on historical data. For years the company had been forced to make arbitrary decisions based on gut decisions. An application like this would allow for more informed, data driven, decisions.

This application took in all of company's data such as model, make, year, mileage, maintenance and other pertinent factors and used an algorithm that generated a recommendation for when a particular truck should be retired.

These are some of the factors that the algorithm took into consideration to decide if a particular truck should retire or not.

In this section we will walk through a number of ways that we can build big data applications using Ruby.