Logistic regression - Software comparison

Logistic regression is a popular supervised learning method.

There are several reasons for this. The theoretical foundation of the method is attractive. It is in line with the generalized regression. Thus the logistic regression is a well identified variant which one can implement according the kind of the dependent variable (class attribute). Their performance in prediction is comparable to the other approaches. Furthermore, it puts forward some indicators for the interpretation of the results. Among them, the famous odds-ratio enables to identify precisely the contribution of each predictor.

Logistic regression is available in many free tools. In this tutorial, we compare the implementation of this technique with Tanagra 1.4.27, R 2.7.2 (GLM command), Orange 1.0b2, Weka 3.5.6, and the package RWeka 0.3-13 for R. Beyond the comparison, this tutorial is also an opportunity to show how to achieve the succession of operations with these tools: importing an ARFF file (Weka file format); split the data into learning and test set; computing the predictive model on the learning set; testing the model on the test set; selecting the relevant variable using criterion in agreement with the logistic regression; evaluating again the performance of the simplified model.

Keywords: logistic regression, supervised learning, software comparison
Components: BINARY LOGISTIC REGRESSION, SUPERVISED LEARNING, TEST, DISCRETE SELECT EXAMPLES
Tutorial: en_Tanagra_Perfs_Reg_Logistique.pdf
Dataset: wave_2_classes_with_irrelevant_attributes.zip
References:
D. Garson, "Logistic Regression"
Wikipedia, "Logistic Regression"

Association rule mining - Software comparison

This document extends a previous tutorial dedicated to the comparison of various implementations of association rules mining (http://data-mining-tutorials.blogspot.com/2008/10/association-rule-learning.html). We had analyzed Tanagra, Orange and Weka. We extend here the comparison to R, RapidMiner and Knime.

We handle an attribute-value dataset. It is not the usual data format for the association rule mining where the "native" format is rather the transactional database. We see in this tutorial than some of tools can automatically recode the data. Others require an explicit transformation. Thus, we must find the right components and the correct sequence of treatments to produce the transactional data format. The process is not always easy according to the software.

The tools studied in this tutorial are: Tanagra 1.4.28, R 2.7.2 (arules package 0.6-6), Orange 1.0b2, RapidMiner Community Edition, Knime 1.3.5 and Weka 3.5.6. These programs load the data and perform the calculations in memory. When the size of the database increases, the real bottleneck is the memory available on our personal computer.

Keywords: association rule, frequent itemset
Components: A PRIORI, A PRIORI PT
Tutorial: en_Tanagra_Assoc_Rules_Comparison.pdf
Dataset: credit-german.zip
References:
R. Rakotomalala, « Règles d’association »
Wikipedia, "Association rule learning"

K-Means - Classification of a new instance

The deployment is an important step of the Data Mining framework. In the case of a clustering, after the construction of clusters with a learning algorithm, we want to determine to which particular cluster (group) a new unlabelled instance belongs.

In this tutorial, we use the K-Means algorithm. We assign each new instance to the group which is closest using the distance to the center of groups. The method is fair because the technique used to assign a group in the deployment phase is consistent with the learning algorithm. It is not true if we use another learning algorithm e.g. HAC (hierarchical agglomerative clustering) with de single linkage aggregation rule. The distance to the center of groups is inadequate in this context. Thus, the classification strategy must be consistent with the learning strategy.

All the descriptors are discrete in our dataset. The K-Means algorithm does not handle directly this kind of data. We must transform them. We use a multiple correspondence analysis algorithm.

In this tutorial, we compare the results of Tanagra 1.4.28 and R 2.7.2.

Keywords: data clustering, k-means, multiple correspondence analysis, factorial analysis, clusters interpretation, data exportation
Components: MULTIPLE CORRESPONDENCE ANALYSIS, K-MEANS, GROUP CHARACTERIZATION, CONTINGENCY CHI-SQUARE, EXPORT DATASET
Tutorial: en_Tanagra_KMeans_Deploiement.pdf
Dataset: banque_classif_deploiement.zip
References:
Wikipedia (en), « K-Means algorithm ».
F. Husson, S. Lê, J. Josse, J. Mazet, « FactoMineR – A package dedicated to Factor Analysis and Data Mining with R ».

Decision tree and large dataset

Dealing with large dataset is on of the most important challenge of the Data Mining. In this context, it is interesting to analyze and to compare the performances of various free implementations of the learning methods, especially the computation time and the memory occupation. Most of the programs download all the dataset into memory. The main bottleneck is the available memory.

In this tutorial, we compare the performance of several implementations of the C4.5 algorithm (Quinlan, 1993) when processing a file containing 500,000 observations and 22 variables. The programs used are: Knime 1.3.5; Orange 1.0b2; R (rpart package) 2.6.0; RapidMiner Community Edition; Sipina Research; Tanagra 1.4.27; Weka 3.5.6.

Our data file is well-known artificial dataset described in the CART book (Breiman et al., 1984). We have generated a dataset with 500.000 observations. The class attribute has 3 values, there are 21 continuous predictors.

Keywords: c4.5, decision tree, classification tree, large dataset, knime, orange, r, rapidminer, sipina, tanagra, weka
Components: SUPERVISED LEARNING, C4.5
Tutorial: en_Tanagra_Perfs_Comp_Decision_Tree.pdf
Dataset: wave500k.zip
Reference: R. Quinlan, « C4.5 : Programs for Machine Learning », Morgan Kaufman, 1993.

Decision tree and cross validation (continued)

In a previous tutorial, we compare the implementation of the decision tree induction and cross validation evaluation performances with three programs: TANAGRA, ORANGE and WEKA.

In this paper, we extent the same framework for the comparison of three new programs: R 2.7.2, KNIME 1.3.51 and RAPIDMINER Community Edition.

Keywords: supervised learning, decision tree, classification tree, classifier assessment
Components: Supervised learning, C-RT, Cross validation
Tutorial: en_Tanagra_Validation_Croisee_Suite.pdf
Dataset: heart.zip

Interactive induction of decision tree

Interactive induction of decision trees with SIPINA.

Various functionalities of SIPINA are not documented. In this tutorial, we show how to explore nodes of a decision tree, in order to obtain a better understanding of the characteristics of the subpopulation on a node. This is an important task, for instance when we want to validate the rules with an expert domain.

Keywords: decision tree, classification tree, interactive analysis
Tutorial: en_sipina_interactive.pdf
Dataset: blood_pressure_levels.xls

Decision tree and contextual descriptive statistics

SIPINA proposes some descriptive statistics functionalities. In itself, the information is not really exceptional; there is a large number of freeware which do that.

It becomes more interesting when we combine these tools with the decision tree. The exploratory phase is improved. Indeed, every node of the tree corresponds to a subpopulation. The variables which do not appear in the tree are not necessarily irrelevant. Perhaps, some of them were hided during the tree learning which selects the “best” variables. By computing contextual descriptive statistics, in connection with the each node, we better understand the prediction rules highlighted during the induction process.

Keywords: descriptive statistics, decision tree, interactive exploration
Tutorial: en_sipina_descriptive_statistics.pdf
Dataset: heart_disease_male.xls

Cost-sensitive Decision Tree

Error rate evaluation is a key point of the induction process. A usual approach is to partition the dataset in a learning set, which is used for the induction of the classification model, and in a test set, which is used for the performance evaluation.

The first subject of this tutorial is to show how to make a partition of the dataset with SIPINA. Then, we build the tree on the first part of the dataset. Later, we classify the examples of the second part of the dataset. We compare the predicted value and the true value. We obtain honest error rate estimation.

The second main subject of this document is to show how to take into account the misclassification costs during the learning process and the evaluation process. We use a slightly modified version of C4.5 (Quinlan, 1993).

Keywords: decision trees, C4.5, classifier evaluation, cost-sensitive learning, F-Measure, spams detection
Tutorial: en_sipina_cost_sensitive.pdf
Dataset: spam.xls

Semi-partial correlation

The semi-partial correlation measures the additional information of an independent variable (X), compared with one or several control variables (Z1,..., Zp), that we can used for the explanation of a dependent variable (Y).

We can compute the semi-partial correlation in various ways. The square of the semi-partial correlation can be obtained with the difference between the square of the multiple correlation coefficient of regression Y / X, Z1...,Zp (including X) and the same quantity for the regression Y / Z,...,Zp (without X).

We can also obtain the semi-partial correlation by computing the residuals of the regression X/Z1,...,Zp; then, we compute the correlation between Y and these residuals. In other words, we seek to quantify the relationship between X and Y, by removing the effect of Z on the latter. The semi-partial correlation is an asymmetrical measure.

In this tutorial, we show the different ways for computing the semi-partial correlation.

Keywords: correlation, Pearson's correlation, semi-partial correlation, multiple linear regression
Components: LINEAR CORRELATION, MULTIPLE LINEAR REGRESSION, SEMI-PARTIAL CORRELATION
Tutorial: en_Tanagra_Semi_Partial_Correlation.pdf
Dataset: cars_semi_partial_correlation.xls
Reference: M. Brannick, « Partial and Semipartial Correlation », University of South Florida.

Partial correlation

Partial correlation measures the degree of association between two random variables, with the effect of a set of controlling variables removed.

In this tutorial, we show how to use the PARTIAL CORREALTION component of Tanagra. We reproduce the example described online (see Reference). Thus, in addition to the presentation of the theoretical method, we can trace the detail of all the calculations that we will achieve.

Keywords: correlation, Pearson's correlation, rank correlation, Spearman's rho, partial correlation
Components: LINEAR CORRELATION, SPEARMAN’S RHO, PARTIAL CORRELATION
Tutorial: en_Tanagra_Partial_Correlation.pdf
Dataset: wechsler_adult_intelligence_scale.xls
Reference: S. Rathbun, A. Wiesner, « STAT 505 – Applied Multivariate Statistical Analysis », The Pennsylvania State University, Lesson 7 : Partial Correlations

Friedman Anova by Ranks

In this tutorial, we show how to use the FRIEDMAN’S ANOVA BY RANKS component. We use this test when we want to check the null hypothesis that K related (matched) samples come from the same population. That matching can be achieved by studying the same group of individuals under each of the K conditions (repeated measure with various conditions).

Keywords: comparison of population, matched samples, analysis of variance, ranks, nonparametric, ANOVA
Components: Friedman’s ANOVA by Rank, One-way ANOVA, Kruskal-Wallis 1-way ANOVA
Tutorial: en_Tanagra_Friedman_Anova.pdf
Dataset: howell_book_friedman_anova_dataset.zip
Reference: Wikipedia, « Friedman test »

Manova - Multivariate analysis of variance

In this tutorial, we show how to use the ONE WAY MANOVA component (Multivariate Analysis of Variance): unlike classical ANOVA, there is more than one dependent variables.

We will see that a multivariate test and a combination of univariate tests give a different conclusion.

Keywords: multivariate analysis of variance, variance covariance matrix
Components: One-way ANOVA, One-Way MANOVA
Tutorial: en_Tanagra_Manova.pdf
Dataset: tomassone_p_29.xls
Reference:
S. Rathburn, A. Wiesner, "STAT 505 - Applied Multivariate Statistical Analysis", Penn State University, Departement of Statistics.

Normality test

A goodness-of-fit test is used to decide if a sample comes from a population with specific distribution. TANAGRA has a new component, which uses several tests in order to check the normality assumption.

We use artificial dataset in this tutorial, we have generated the dataset from 3 distributions: uniform, normal and log-normal.

Keywords: Shapiro-Wilk's test, Lilliefors' test, Anderson-Darling's test, d’Agostino's test
Components: More Univariate cont stat, Normality Test
Tutorial: en_Tanagra_Normality_Test.pdf
Dataset: normality_test_simulation.xls
Reference:
Wikipedia - "Normality test"

Two-sample t-test

In this tutorial, we show how to use TANAGRA to determine if two populations means are equal. The conditional variance may be assumed as equal or unequal i.e. we use pooled or separate variance estimation.

Keywords: test for mean comparison, Student's t-test
Components: T-Test, T-Test Unequal Variance
Tutorial: en_Tanagra_Two_Sample_T_Test_For_Equal_Means.pdf
Dataset: auto83b.xls
Reference: NIST/SEMATECH, « e-Handbook of Statistical Methods », Section 7.3.1 « Do two processes have the same mean ? ».

ANOVA and test for equality of variances

In this tutorial, we show how to use TANAGRA in an analysis of variance problem. We test also homogeneity of variances assumption on the same dataset.

We use the GEAR dataset (NIST/SEMATECH e-Handbook of Statistical Methods, http://www.itl.nist.gov/div898/handbook/). We consider that we have 10 machine tools, which produce gears. We have 10 batches of 10 observations. We want to test various assumptions: (1) the average diameter of the gears is the same one for the whole of the machines? (2) The variability of the gear diameter is the same for the whole of the machines?

Keywords: analysis of variance, test for equality of variances, bartlett's test, levene's test, brown-forsythe's test
Components: One-way ANOVA, Bartlett’s test, Levene’s test, Brown-Forsythe test
Tutorial: Anova and Tests for Equality of Variances
Dataset: gear_data_from_nist.xls
Reference: NIST/SEMATECH, « e-Handbook of Statistical Methods », Chapitre 7 « Product and Process Comparisons ».