The most common approach of the stochastic gradient boosting is known by many other names as well. In the past years it has come up to be known as the most sought after methods used for data mining. a few implementations that we see of this technique also make it capable enough to be use not just for regression but also other classification techniques. It is seen getting used with continuous as well as the categorical variables. These boosting methods further got evolved into boosting trees with their application into regression trees. The fundamental concept is to calculate a series of some simple trees. In this case, the successive tree is made on the prediction residual of the previous tree. This method would be further used to build binary trees. However the problem with these trees is of over fitting. As it is a machine learning algorithm, it becomes difficult to know when to stop. To explain it further, it is a challenge to stop or avoid the learning algorithm to fit into the esoteric aspects of the training data and it would not improve the predictive validity of the model in the study. This problem of over fitting crops up in most of the machine learning algorithms that are found to be used in the predictive data mining. This problem can be dealt with, to an extent, by the general solution to evaluate the quality of the fitted model by predicting those observations that haven’t been to put to test earlier. It helps basically to gauge the predictive accuracy of the solution.
The procedure that is used for applying this method to the classification problems requires calls for a separate sequencing to build for each category. The computation effort mostly multiplies and becomes much larger than what effort would go into solving a basic prediction problem. It would not be a wise decision to analyse the category dependent variable with more than 100 classes as it would require a huge investment of time and effort as a great number of individual trees would be yielded in the process.
