Decision Tree Regression

As a marketing manager, you want a set of customers who are most likely to purchase your product. This is how you can save your marketing budget by finding your audience. As a loan manager, you need to identify risky loan applications to achieve a lower loan default rate. This process of classifying customers into a group of potential and non-potential customers or safe or risky loan applications is known as a classification problem. Classification is a two-step process, learning step and prediction step. In the learning step, the model is developed based on given training data. In the prediction step, the model is used to predict the response for given data. Decision Tree is one of the easiest and popular classification algorithms to understand and interpret. It can be utilized for both classification and regression kind of problem.

In this tutorial, you are going to cover the following topics:

A decision tree is a flowchart-like tree structure where an internal node represents feature(or attribute), the branch represents a decision rule, and each leaf node represents the outcome. The topmost node in a decision tree is known as the root node. It learns to partition on the basis of the attribute value. It partitions the tree in recursively manner call recursive partitioning. This flowchart-like structure helps you in decision making. It’s visualization like a flowchart diagram which easily mimics the human level thinking. That is why decision trees are easy to understand and interpret.

Decision Tree is a white box type of ML algorithm. It shares internal decision-making logic, which is not available in the black box type of algorithms such as Neural Network. Its training time is faster compared to the neural network algorithm. The time complexity of decision trees is a function of the number of records and number of attributes in the given data. The decision tree is a distribution-free or non-parametric method, which does not depend upon probability distribution assumptions. Decision trees can handle high dimensional data with good accuracy.

The basic idea behind any decision tree algorithm is as follows:

Shannon invented the concept of entropy, which measures the impurity of the input set. In physics and mathematics, entropy referred as the randomness or the impurity in the system. In information theory, it refers to the impurity in a group of examples. Information gain is the decrease in entropy. Information gain computes the difference between entropy before split and average entropy after split of the dataset based on given attribute values. ID3 (Iterative Dichotomiser) decision tree algorithm uses information gain.

Where, Pi is the probability that an arbitrary tuple in D belongs to class Ci.

Where,

The attribute A with the highest information gain, Gain(A), is chosen as the splitting attribute at node N().

C4.5, an improvement of ID3, uses an extension to information gain known as the gain ratio. Gain ratio handles the issue of bias by normalizing the information gain using Split Info. Java implementation of the C4.5 algorithm is known as J48, which is available in WEKA data mining tool.

Where,

The gain ratio can be defined as

Another decision tree algorithm CART (Classification and Regression Tree) uses the Gini method to create split points.

Where, pi is the probability that a tuple in D belongs to class Ci.

The Gini Index considers a binary split for each attribute. You can compute a weighted sum of the impurity of each partition. If a binary split on attribute A partitions data D into D1 and D2, the Gini index of D is:

In case of a discrete-valued attribute, the subset that gives the minimum gini index for that chosen is selected as a splitting attribute. In the case of continuous-valued attributes, the strategy is to select each pair of adjacent values as a possible split-point and point with smaller gini index chosen as the splitting point.

The attribute with minimum Gini index is chosen as the splitting attribute.

Let’s first load the required libraries.

pregnantglucosebpskininsulinbmipedigreeagelabel061487235033.60.62750111856629026.60.35131028183640023.30.672321318966239428.10.16721040137403516843.12.288331

Here, you need to divide given columns into two types of variables dependent(or target variable) and independent variable(or feature variables).

To understand model performance, dividing the dataset into a training set and a test set is a good strategy.

Let’s create a Decision Tree Model using Scikit-learn.

Let’s estimate, how accurately the classifier or model can predict the type of cultivars.

Accuracy can be computed by comparing actual test set values and predicted values.

Well, you got a classification rate of 67.53%, considered as good accuracy. You can improve this accuracy by tuning the parameters in the Decision Tree Algorithm.

In the decision tree chart, each internal node has a decision rule that splits the data. Gini referred as Gini ratio, which measures the impurity of the node. You can say a node is pure when all of its records belong to the same class, such nodes known as the leaf node.

Here, the resultant tree is unpruned. This unpruned tree is unexplainable and not easy to understand. In the next section, let’s optimize it by pruning.

Well, the classification rate increased to 77.05%, which is better accuracy than the previous model.

This pruned model is less complex, explainable, and easy to understand than the previous decision tree model plot.

Related posts:

Every two weeks I chat with great hosts to get insights on how they run their shows. It’s a series called Meet the Podcaster. Today there’s Brett Gajda and Nick Jaworski, respectively host and…

The largest Content Management System in the world today is none other than “WordPress” and at the same time the demand for WordPress designer and developers is at the top. The IT organizations and…

We all know the drill now. During the elections, politicians promise to do everything they can to help the environment. Business leaders state that they will lend their support to various…