Johnny Deng's Column

Autoregressive Moving-Average Process

2008-05-15T17:49:00.000+01:00

An n-dimensional autoregressive moving-average process of orders p and q, ARMA(p,q), is a stochastic process of the form

[1]

where a is an n-dimensional vector, the and are nn matrices, and W is n-dimensional white noise (see the notation conventions documentation). As the name suggests, this combines an AR(p) model with an MA(q) model of the same dimension n. In applications, ARMA(1,1) processes are common.

Exhibit 1 indicates a realization of the univariate ARMA(1,1) process

[2]

where W is variance 1 Gaussian white noise.

ARMA Process
Exhibit 1

A realization of the ARMA(1,1) process [2].

Exercises

Below are indicated a realization of 50 consecutive terms of a variance 1 Gaussian white noise.

0.293	0.317	0.047	-0.286	-1.237
-0.554	0.535	-1.640	-0.899	-0.704
-1.886	0.271	0.418	1.651	0.078
0.528	1.013	2.296	0.086	1.471
-0.580	-1.776	-2.217	0.502	-1.104
-1.211	0.205	0.110	0.011	0.778
-1.036	1.195	-1.169	-0.162	-0.504
-0.679	-1.366	0.885	-0.476	1.644
-1.665	0.129	2.882	0.978	0.054
-0.396	0.685	1.403	-0.009	0.918
Realization of 50 consecutive terms of a variance 1 Gaussian white noise.

Use this to generate a corresponding realization of the ARMA(1,1) process

[e1]

where ^tW is a variance 1 Gaussian white noise. Initialize the realization with term ⁰x = 0. [spreadsheet solution]

White Noise

2008-05-15T17:38:00.000+01:00

A white noise is a simple type of stochastic process. Precise definitions vary. One simple definition is that a white noise is a (univariate or multivariate) discrete-time stochastic process whose terms are independent and identically distributed (IID), all with zero mean. While this definition captures the spirit of what constitutes a white noise, the IID requirement is often too restrictive for applications. Typically the IID requirement is replaced with a requirement that terms have constant second moments, zero autocorrelations and zero means. Let's formalize this.

If you have not already done so, see the notation conventions documentation. A one-dimensional stochastic process

..., ^t–2W, ^t–1W, ^tW, ^t+1W, ...

[1]

is said to be white noise if unconditional means, standard deviations and autocorrelations satisfy

E(^tW) = 0

[2]

std(^tW) =

[3]

cor(^tW, ^t+nW) = 0

[4]

for some constant and any integer n. To distinguish this definition of white noise from that which requires IID terms, we call the latter an independent white noise or strong white noise. Note that the definition of white noise is more restrictive than that of independent white noise in just one respect. With a white noise, means, standard deviations and autocorrelations must exist. For independent white noise, they need not.

While the definition of independent white noise is otherwise more restrictive than that of white noise, it is also simpler. An independent white noise is necessarily a very simple process. Conditions [2] through [4], which define a white noise, can accommodate more complicated processes. For example, conditions [2] and [3] apply only to unconditional moments. There is nothing to stop a white noise from being conditionally heteroskedastic. That is impossible with an independent white noise.

An independent white noise whose terms are all normally distributed is called a Gaussian white noise. A realization of a univariate Gaussian white noise with variance 1 is graphed in Exhibit 1.

Univariate Gaussian White Noise
Exhibit 1

A realization of a univariate Gaussian white noise with variance 1.

All these concepts generalize to multivariate processes. An n-dimensional stochastic process

[5]

is said to be white noise if unconditional expectations satisfy

	[6]
	[7]

for some constant covariance matrix . Condition [7] does not require that the be independent. If we make this stronger assumption, the process is called independent white noise. If we further assume the are joint normal, it is called Gaussian white noise.

White noises are important in time series analysis because more complicated stochastic processes are generally defined in terms of white noises.

ARCH and GARCH Processes

2008-05-15T17:31:00.001+01:00

Autoregressive conditional heteroskedastic (ARCH) processes are a form of stochastic process that are widely used in finance and economics for modeling conditional heteroskedasticity and volatility clustering. First proposed by Engle (1982), ARCH processes are univariate conditionally heteroskedastic white noises. An ARCH(q) process X is defined by two interrelated formulas (see the notation conventions documentation):

	[1]
	[2]

where W is a standard normal Gaussian white noise. This means that the time t distribution of X, conditional on information available at time t–1, is normal, with constant mean 0 and a conditional variance that changes with time. Our notation indicates that is a variance at time t, but conditional on information available at time t–1. Formula [2] defines as a function of preceding values of X. Together, formulas [1] and [2] ensure that, if X takes on large positive or negative values at some point in time, its conditional variance will be elevated for subsequent points in time, thereby making it likely that X will also take on large positive or negative values at those times too. In this manner, an ARCH process models volatility clustering—periods of high or low volatility.

Bollerslev (1986) extended the model by allowing to also depend on its own past values. His generalized ARCH, or GARCH(p,q), process has form

	[3]
	[4]

See Hamilton (1994) for stationarity conditions. In applications, GARCH(1,1) processes are common. Exhibit 1 indicates a realization of the GARCH(1,1) process

	[5]
	[6]

GARCH(1,1) Process
Exhibit 1

A realization of the GARCH(1,1) process defined by [5] and [6].

There have been many attempts to generalize GARCH models to multiple dimensions. Attempts include:

the vech and BEKK models of Engle and Kroner (1995),

the CCC-GARCH of Bollerslev (1990),

the orthogonal GARCH of Ding (1994), Alexander and Chibumba (1997), and Klaassen (2000), and

the DCC-GARCH of Engle (2000), and Engle and Sheppard (2001).

With some of these approaches, the number of parameters that must be specified becomes unmanageable as dimensionality n increases. With some, estimation requires considerable user intervention or entails other challenges. Some require assumptions that are difficult to reconcile with phenomena to be modeled. This is an area of ongoing research.

Greek alphabet, again

2007-12-10T13:15:00.001Z

Dummy Variables

2007-11-29T00:27:00.001Z

Why use dummies?

Regression analysis is used with numerical variables. Results only have a valid interpretation if it makes sense to assume that having a value of 2 on some variable is does indeed mean having twice as much of something as a 1, and having a 50 means 50 times as much as 1.

However, social scientists often need to work with categorical variables in which the different values have no real numerical relationship with each other. Examples include variables for race, political affiliation, or marital status. If you have a variable for political affiliation with possible responses including Democrat, Independent, and Republican, it obviously doesn't make sense to assign values of 1 - 3 and interpret that as meaning that a Republican is somehow three times as politically affiliated as a Democrat.

The solution is to use dummy variables - variables with only two values, zero and one. It does make sense to create a variable called "Republican" and interpret it as meaning that someone assigned a 1 on this varible is Republican and someone with an 0 is not.

Nominal variables with multiple levels

If you have a nominal variable that has more than two levels, you need to create multiple dummy variables to "take the place of" the original nominal variable. For example, imagine that you wanted to predict depression from year in school: freshman, sophomore, junior, or senior. Obviously, "year in school" has more than two levels.

What you need to do is to recode "year in school" into a set of dummy variables, each of which has two levels. The first step in this process is to decide the number of dummy variables. This is easy; it's simply k-1, where k is the number of levels of the original variable.

You could also create dummy variables for all levels in the original variable, and simply drop one from each analysis.

In this instance, we would need to create 4-1=3 dummy variables. In order to create these variables, we are going to take 3 of the levels of "year of school", and create a variable corresponding to each level, which will have the value of yes or no (i.e., 1 or 0). In this instance, we can create a variable called "sophomore," "junior," and "senior." Each instance of "year of school" would then be recoded into a value for "sophomore," "junior," and "senior." If a person were a junior, then "sophomore" would be equal to 0, "junior" would be equal to 1, and "senior" would be equal to 0.

Interpreting results

The decision as to which level is not coded is often arbitrary. The level which is not coded is the category to which all other categories will be compared. As such, often the biggest group will be the not- coded category. For example, often "Caucasian" will be the not-coded group if that is the race of the majority of participants in the sample. In that case, if you have a variable called "Asian", the coefficient on the "Asian" variable in your regression will show the effect being Asian rather than Caucasian has on your dependant variable.

In our example, "freshman" was not coded so that we could determine if being a sophomore, junior, or senior predicts a different depressive level than being a freshman. Consequently, if the variable, "junior" was significant in our regression, with a positive beta coefficient, this would mean that juniors are significantly more depressed than freshman. Alternatively, we could have decided to not code "senior," if we thought that being a senior is qualitatively different from being of another year.

Levels of Measurement

2007-11-29T00:07:00.000Z

The level of measurement refers to the relationship among the values that are assigned to the attributes for a variable. What does that mean? Begin with the idea of the variable, in this example "party affiliation." That variable has a number of attributes. Let's assume that in this particular election context the only relevant attributes are "republican", "democrat", and "independent". For purposes of analyzing the results of this variable, we arbitrarily assign the values 1, 2 and 3 to the three attributes. The level of measurement describes the relationship among these three values. In this case, we simply are using the numbers as shorter placeholders for the lengthier text terms. We don't assume that higher values mean "more" of something and lower numbers signify "less". We don't assume the the value of 2 means that democrats are twice something that republicans are. We don't assume that republicans are in first place or have the highest priority just because they have the value of 1. In this case, we only use the values as a shorter name for the attribute. Here, we would describe the level of measurement as "nominal".

Why is Level of Measurement Important?

First, knowing the level of measurement helps you decide how to interpret the data from that variable. When you know that a measure is nominal (like the one just described), then you know that the numerical values are just short codes for the longer names. Second, knowing the level of measurement helps you decide what statistical analysis is appropriate on the values that were assigned. If a measure is nominal, then you know that you would never average the data values or do a t-test on the data.

There are typically four levels of measurement that are defined:

Nominal
Ordinal
Interval
Ratio

In nominal measurement the numerical values just "name" the attribute uniquely. No ordering of the cases is implied. For example, jersey numbers in basketball are measures at the nominal level. A player with number 30 is not more of anything than a player with number 15, and is certainly not twice whatever number 15 is.

In ordinal measurement the attributes can be rank-ordered. Here, distances between attributes do not have any meaning. For example, on a survey you might code Educational Attainment as 0=less than H.S.; 1=some H.S.; 2=H.S. degree; 3=some college; 4=college degree; 5=post college. In this measure, higher numbers mean more education. But is distance from 0 to 1 same as 3 to 4? Of course not. The interval between values is not interpretable in an ordinal measure.

In interval measurement the distance between attributes does have meaning. For example, when we measure temperature (in Fahrenheit), the distance from 30-40 is same as distance from 70-80. The interval between values is interpretable. Because of this, it makes sense to compute an average of an interval variable, where it doesn't make sense to do so for ordinal scales. But note that in interval measurement ratios don't make any sense - 80 degrees is not twice as hot as 40 degrees (although the attribute value is twice as large).

Finally, in ratio measurement there is always an absolute zero that is meaningful. This means that you can construct a meaningful fraction (or ratio) with a ratio variable. Weight is a ratio variable. In applied social research most "count" variables are ratio, for example, the number of clients in past six months. Why? Because you can have zero clients and because it is meaningful to say that "...we had twice as many clients in the past six months as we did in the previous six months."

It's important to recognize that there is a hierarchy implied in the level of measurement idea. At lower levels of measurement, assumptions tend to be less restrictive and data analyses tend to be less sensitive. At each level up the hierarchy, the current level includes all of the qualities of the one below it and adds something new. In general, it is desirable to have a higher level of measurement (e.g., interval or ratio) rather than a lower one (nominal or ordinal).

AUC: Z distribution

2007-11-28T22:39:00.001Z

What is overfitting and how can I avoid it?

2007-11-28T19:27:00.000Z

The critical issue in developing a neural network is generalization: how
well will the network make predictions for cases that are not in the
training set? NNs, like other flexible nonlinear estimation methods such as
kernel regression and smoothing splines, can suffer from either underfitting
or overfitting. A network that is not sufficiently complex can fail to
detect fully the signal in a complicated data set, leading to underfitting.
A network that is too complex may fit the noise, not just the signal,
leading to overfitting. Overfitting is especially dangerous because it can
easily lead to predictions that are far beyond the range of the training
data with many of the common types of NNs. Overfitting can also produce wild
predictions in multilayer perceptrons even with noise-free data.

For an elementary discussion of overfitting, see Smith (1996). For a more
rigorous approach, see the article by Geman, Bienenstock, and Doursat (1992)
on the bias/variance trade-off (it's not really a dilemma). We are talking
about statistical bias here: the difference between the average value of an
estimator and the correct value. Underfitting produces excessive bias in the
outputs, whereas overfitting produces excessive variance. There are
graphical examples of overfitting and underfitting in Sarle (1995, 1999).

The best way to avoid overfitting is to use lots of training data. If you
have at least 30 times as many training cases as there are weights in the
network, you are unlikely to suffer from much overfitting, although you may
get some slight overfitting no matter how large the training set is. For
noise-free data, 5 times as many training cases as weights may be
sufficient. But you can't arbitrarily reduce the number of weights for fear
of underfitting.

Given a fixed amount of training data, there are at least six approaches to
avoiding underfitting and overfitting, and hence getting good
generalization:

o Model selection
o Jittering
o Early stopping
o Weight decay
o Bayesian learning
o Combining networks

The first five approaches are based on well-understood theory. Methods for
combining networks do not have such a sound theoretical basis but are the
subject of current research. These six approaches are discussed in more
detail under subsequent questions.

The complexity of a network is related to both the number of weights and the
size of the weights. Model selection is concerned with the number of
weights, and hence the number of hidden units and layers. The more weights
there are, relative to the number of training cases, the more overfitting
amplifies noise in the targets (Moody 1992). The other approaches listed
above are concerned, directly or indirectly, with the size of the weights.
Reducing the size of the weights reduces the "effective" number of
weights--see Moody (1992) regarding weight decay and Weigend (1994)
regarding early stopping. Bartlett (1997) obtained learning-theory results
in which generalization error is related to the L_1 norm of the weights
instead of the VC dimension.

Overfitting is not confined to NNs with hidden units. Overfitting can occur
in generalized linear models (networks with no hidden units) if either or
both of the following conditions hold:

1. The number of input variables (and hence the number of weights) is large
 with respect to the number of training cases. Typically you would want at
 least 10 times as many training cases as input variables, but with
 noise-free targets, twice as many training cases as input variables would
 be more than adequate. These requirements are smaller than those stated
 above for networks with hidden layers, because hidden layers are prone to
 creating ill-conditioning and other pathologies.

2. The input variables are highly correlated with each other. This condition
 is called "multicollinearity" in the statistical literature.
 Multicollinearity can cause the weights to become extremely large because
 of numerical ill-conditioning--see "How does ill-conditioning affect NN
 training?"

Methods for dealing with these problems in the statistical literature
include ridge regression (similar to weight decay), partial least squares
(similar to Early stopping), and various methods with even stranger names,
such as the lasso and garotte (van Houwelingen and le Cessie, ????).

References:

 Bartlett, P.L. (1997), "For valid generalization, the size of the weights
 is more important than the size of the network," in Mozer, M.C., Jordan,
 M.I., and Petsche, T., (eds.) Advances in Neural Information Processing
 Systems 9, Cambrideg, MA: The MIT Press, pp. 134-140.

 Geman, S., Bienenstock, E. and Doursat, R. (1992), "Neural Networks and
 the Bias/Variance Dilemma", Neural Computation, 4, 1-58.

 Moody, J.E. (1992), "The Effective Number of Parameters: An Analysis of
 Generalization and Regularization in Nonlinear Learning Systems", in
 Moody, J.E., Hanson, S.J., and Lippmann, R.P., Advances in Neural
 Information Processing Systems 4, 847-854.

 Sarle, W.S. (1995), "Stopped Training and Other Remedies for
 Overfitting," Proceedings of the 27th Symposium on the Interface of
 Computing Science and Statistics, 352-360,
 ftp://ftp.sas.com/pub/neural/inter95.ps.Z (this is a very large
 compressed postscript file, 747K, 10 pages)

 Sarle, W.S. (1999), "Donoho-Johnstone Benchmarks: Neural Net Results,"
 ftp://ftp.sas.com/pub/neural/dojo/dojo.html

 Smith, M. (1996). Neural Networks for Statistical Modeling, Boston:
 International Thomson Computer Press, ISBN 1-850-32842-0.

 van Houwelingen,H.C., and le Cessie, S. (????), "Shrinkage and penalized
 likelihood as methods to improve predictive accuracy,"
 http://www.medstat.medfac.leidenuniv.nl/ms/HH/Files/shrinkage.pdf and
 http://www.medstat.medfac.leidenuniv.nl/ms/HH/Files/figures.pdf

 Weigend, A. (1994), "On overfitting and the effective number of hidden
 units," Proceedings of the 1993 Connectionist Models Summer School,
 335-342.

Copyright 1997, 1998, 1999, 2000, 2001, 2002 by Warren S. Sarle, Cary, NC,
USA. Answers provided by other authors as cited below are copyrighted by
those authors, who by submitting the answers for the FAQ give permission for
the answer to be reproduced as part of the FAQ in any of the ways specified
in part 1 of the FAQ.

Overfitting and Underfitting

2007-11-28T14:24:00.000Z

Overfitting and Underfitting

When fitting a model to noisy data (ALWAYS), we make the assumption that the data have been generated from some “TRUE” model by making predictions at given values of the inputs, then adding some amount of noise to each point, where the noise is drawn from a normal distribution with an unknown variance.

Our task is to discover both this model and the width of the noise distribution. In doing so, we aim for a compromise between bias, where our model does not follow the right trend in the data (and so does not match well with the underlying truth), and variance, where our model fits the data points too closely, fitting the noise rather than trying to capture the true distribution. These two extremes are known as underfitting and overfitting.

IMPORTANT! : the number of parameters in a model; the higher, the more complexly the model can fit the data. If the number of parameters in our model is larger than that the “true one”, then we risk overfitting, and if our model contains fewer parameters than the truth, we could underfit.

The illustration shows how increasing the number of parameters in the model can result in overfitting. The 9 data points are generated from a cubic polynomial which contains 4 parameters (the true model) and adding noise. We can see that by selecting candidate models containing more parameters than the truth, we can reduce, and even eliminate, any mismatch between the data points and our model. This occurs when the number of parameters is the same as the number of data points (an 8th order polynomial has 9 parameters).

Data Mining Techniques

2007-11-24T16:11:00.000Z

Data Mining Techniques

referenced from http://www.statsoft.com/textbook/stdatmin.html

Data Mining

Data Mining is an analytic process designed to explore data (usually large amounts of data - typically business or market related) in search of consistent patterns and/or systematic relationships between variables, and then to validate the findings by applying the detected patterns to new subsets of data. The ultimate goal of data mining is prediction - and predictive data mining is the most common type of data mining and one that has the most direct business applications. The process of data mining consists of three stages: (1) the initial exploration, (2) model building or pattern identification with validation/verification, and (3) deployment (i.e., the application of the model to new data in order to generate predictions).

Stage 1: Exploration. This stage usually starts with data preparation which may involve cleaning data, data transformations, selecting subsets of records and - in case of data sets with large numbers of variables ("fields") - performing some preliminary feature selection operations to bring the number of variables to a manageable range (depending on the statistical methods which are being considered). Then, depending on the nature of the analytic problem, this first stage of the process of data mining may involve anywhere between a simple choice of straightforward predictors for a regression model, to elaborate exploratory analyses using a wide variety of graphical and statistical methods (see Exploratory Data Analysis (EDA)) in order to identify the most relevant variables and determine the complexity and/or the general nature of models that can be taken into account in the next stage.

Stage 2: Model building and validation. This stage involves considering various models and choosing the best one based on their predictive performance (i.e., explaining the variability in question and producing stable results across samples). This may sound like a simple operation, but in fact, it sometimes involves a very elaborate process. There are a variety of techniques developed to achieve that goal - many of which are based on so-called "competitive evaluation of models," that is, applying different models to the same data set and then comparing their performance to choose the best. These techniques - which are often considered the core of predictive data mining - include: Bagging (Voting, Averaging), Boosting, Stacking (Stacked Generalizations), and Meta-Learning.

Stage 3: Deployment. That final stage involves using the model selected as best in the previous stage and applying it to new data in order to generate predictions or estimates of the expected outcome.

The concept of Data Mining is becoming increasingly popular as a business information management tool where it is expected to reveal knowledge structures that can guide decisions in conditions of limited certainty. Recently, there has been increased interest in developing new analytic techniques specifically designed to address the issues relevant to business Data Mining (e.g., Classification Trees), but Data Mining is still based on the conceptual principles of statistics including the traditional Exploratory Data Analysis (EDA) and modeling and it shares with them both some components of its general approaches and specific techniques.

However, an important general difference in the focus and purpose between Data Mining and the traditional Exploratory Data Analysis (EDA) is that Data Mining is more oriented towards applications than the basic nature of the underlying phenomena. In other words, Data Mining is relatively less concerned with identifying the specific relations between the involved variables. For example, uncovering the nature of the underlying functions or the specific types of interactive, multivariate dependencies between variables are not the main goal of Data Mining. Instead, the focus is on producing a solution that can generate useful predictions. Therefore, Data Mining accepts among others a "black box" approach to data exploration or knowledge discovery and uses not only the traditional Exploratory Data Analysis (EDA) techniques, but also such techniques as Neural Networks which can generate valid predictions but are not capable of identifying the specific nature of the interrelations between the variables on which the predictions are based.

Data Mining is often considered to be "a blend of statistics, AI [artificial intelligence], and data base research" (Pregibon, 1997, p. 8), which until very recently was not commonly recognized as a field of interest for statisticians, and was even considered by some "a dirty word in Statistics" (Pregibon, 1997, p. 8). Due to its applied importance, however, the field emerges as a rapidly growing and major area (also in statistics) where important theoretical advances are being made (see, for example, the recent annual International Conferences on Knowledge Discovery and Data Mining, co-hosted by the American Statistical Association).

For information on Data Mining techniques, please review the summary topics included below in this chapter of the Electronic Statistics Textbook. There are numerous books that review the theory and practice of data mining; the following books offer a representative sample of recent general books on data mining, representing a variety of approaches and perspectives:

Berry, M., J., A., & Linoff, G., S., (2000). Mastering data mining. New York: Wiley.

Edelstein, H., A. (1999). Introduction to data mining and knowledge discovery (3rd ed). Potomac, MD: Two Crows Corp.

Fayyad, U. M., Piatetsky-Shapiro, G., Smyth, P., & Uthurusamy, R. (1996). Advances in knowledge discovery & data mining. Cambridge, MA: MIT Press.

Han, J., Kamber, M. (2000). Data mining: Concepts and Techniques. New York: Morgan-Kaufman.

Hastie, T., Tibshirani, R., & Friedman, J. H. (2001). The elements of statistical learning : Data mining, inference, and prediction. New York: Springer.

Pregibon, D. (1997). Data Mining. Statistical Computing and Graphics, 7, 8.

Weiss, S. M., & Indurkhya, N. (1997). Predictive data mining: A practical guide. New York: Morgan-Kaufman.

Westphal, C., Blaxton, T. (1998). Data mining solutions. New York: Wiley.

Witten, I. H., & Frank, E. (2000). Data mining. New York: Morgan-Kaufmann.

Crucial Concepts in Data Mining

Bagging (Voting, Averaging)
The concept of bagging (voting for classification, averaging for regression-type problems with continuous dependent variables of interest) applies to the area of predictive data mining, to combine the predicted classifications (prediction) from multiple models, or from the same type of model for different learning data. It is also used to address the inherent instability of results when applying complex models to relatively small data sets. Suppose your data mining task is to build a model for predictive classification, and the dataset from which to train the model (learning data set, which contains observed classifications) is relatively small. You could repeatedly sub-sample (with replacement) from the dataset, and apply, for example, a tree classifier (e.g., C&RT and CHAID) to the successive samples. In practice, very different trees will often be grown for the different samples, illustrating the instability of models often evident with small datasets. One method of deriving a single prediction (for new observations) is to use all trees found in the different samples, and to apply some simple voting: The final classification is the one most often predicted by the different trees. Note that some weighted combination of predictions (weighted vote, weighted average) is also possible, and commonly used. A sophisticated (machine learning) algorithm for generating weights for weighted prediction or voting is the Boosting procedure.

Boosting
The concept of boosting applies to the area of predictive data mining, to generate multiple models or classifiers (for prediction or classification), and to derive weights to combine the predictions from those models into a single prediction or predicted classification (see also Bagging).

A simple algorithm for boosting works like this: Start by applying some method (e.g., a tree classifier such as C&RT or CHAID) to the learning data, where each observation is assigned an equal weight. Compute the predicted classifications, and apply weights to the observations in the learning sample that are inversely proportional to the accuracy of the classification. In other words, assign greater weight to those observations that were difficult to classify (where the misclassification rate was high), and lower weights to those that were easy to classify (where the misclassification rate was low). In the context of C&RT for example, different misclassification costs (for the different classes) can be applied, inversely proportional to the accuracy of prediction in each class. Then apply the classifier again to the weighted data (or with different misclassification costs), and continue with the next iteration (application of the analysis method for classification to the re-weighted data).

Boosting will generate a sequence of classifiers, where each consecutive classifier in the sequence is an "expert" in classifying observations that were not well classified by those preceding it. During deployment (for prediction or classification of new cases), the predictions from the different classifiers can then be combined (e.g., via voting, or some weighted voting procedure) to derive a single best prediction or classification.

Note that boosting can also be applied to learning methods that do not explicitly support weights or misclassification costs. In that case, random sub-sampling can be applied to the learning data in the successive steps of the iterative boosting procedure, where the probability for selection of an observation into the subsample is inversely proportional to the accuracy of the prediction for that observation in the previous iteration (in the sequence of iterations of the boosting procedure).

CRISP
See Models for Data Mining.

Data Preparation (in Data Mining)
Data preparation and cleaning is an often neglected but extremely important step in the data mining process. The old saying "garbage-in-garbage-out" is particularly applicable to the typical data mining projects where large data sets collected via some automatic methods (e.g., via the Web) serve as the input into the analyses. Often, the method by which the data where gathered was not tightly controlled, and so the data may contain out-of-range values (e.g., Income: -100), impossible data combinations (e.g., Gender: Male, Pregnant: Yes), and the like. Analyzing data that has not been carefully screened for such problems can produce highly misleading results, in particular in predictive data mining.

Data Reduction (for Data Mining)
The term Data Reduction in the context of data mining is usually applied to projects where the goal is to aggregate or amalgamate the information contained in large datasets into manageable (smaller) information nuggets. Data reduction methods can include simple tabulation, aggregation (computing descriptive statistics) or more sophisticated techniques like clustering, principal components analysis, etc.

Deployment
The concept of deployment in predictive data mining refers to the application of a model for prediction or classification to new data. After a satisfactory model or set of models has been identified (trained) for a particular application, one usually wants to deploy those models so that predictions or predicted classifications can quickly be obtained for new data. For example, a credit card company may want to deploy a trained model or set of models (e.g., neural networks, meta-learner) to quickly identify transactions which have a high probability of being fraudulent.

Drill-Down Analysis
The concept of drill-down analysis applies to the area of data mining, to denote the interactive exploration of data, in particular of large databases. The process of drill-down analyses begins by considering some simple break-downs of the data by a few variables of interest (e.g., Gender, geographic region, etc.). Various statistics, tables, histograms, and other graphical summaries can be computed for each group. Next one may want to "drill-down" to expose and further analyze the data "underneath" one of the categorizations, for example, one might want to further review the data for males from the mid-west. Again, various statistical and graphical summaries can be computed for those cases only, which might suggest further break-downs by other variables (e.g., income, age, etc.). At the lowest ("bottom") level are the raw data: For example, you may want to review the addresses of male customers from one region, for a certain income group, etc., and to offer to those customers some particular services of particular utility to that group.

Feature Selection
One of the preliminary stage in predictive data mining, when the data set includes more variables than could be included (or would be efficient to include) in the actual model building phase (or even in initial exploratory operations), is to select predictors from a large list of candidates. For example, when data are collected via automated (computerized) methods, it is not uncommon that measurements are recorded for thousands or hundreds of thousands (or more) of predictors. The standard analytic methods for predictive data mining, such as neural network analyses, classification and regression trees, generalized linear models, or general linear models become impractical when the number of predictors exceed more than a few hundred variables.

Feature selection selects a subset of predictors from a large list of candidate predictors without assuming that the relationships between the predictors and the dependent or outcome variables of interest are linear, or even monotone. Therefore, this is used as a pre-processor for predictive data mining, to select manageable sets of predictors that are likely related to the dependent (outcome) variables of interest, for further analyses with any of the other methods for regression and classification.

Machine Learning
Machine learning, computational learning theory, and similar terms are often used in the context of Data Mining, to denote the application of generic model-fitting or classification algorithms for predictive data mining. Unlike traditional statistical data analysis, which is usually concerned with the estimation of population parameters by statistical inference, the emphasis in data mining (and machine learning) is usually on the accuracy of prediction (predicted classification), regardless of whether or not the "models" or techniques that are used to generate the prediction is interpretable or open to simple explanation. Good examples of this type of technique often applied to predictive data mining are neural networks or meta-learning techniques such as boosting, etc. These methods usually involve the fitting of very complex "generic" models, that are not related to any reasoning or theoretical understanding of underlying causal processes; instead, these techniques can be shown to generate accurate predictions or classification in crossvalidation samples.

Meta-Learning
The concept of meta-learning applies to the area of predictive data mining, to combine the predictions from multiple models. It is particularly useful when the types of models included in the project are very different. In this context, this procedure is also referred to as Stacking (Stacked Generalization).

Suppose your data mining project includes tree classifiers, such as C&RT and CHAID, linear discriminant analysis (e.g., see GDA), and Neural Networks. Each computes predicted classifications for a crossvalidation sample, from which overall goodness-of-fit statistics (e.g., misclassification rates) can be computed. Experience has shown that combining the predictions from multiple methods often yields more accurate predictions than can be derived from any one method (e.g., see Witten and Frank, 2000). The predictions from different classifiers can be used as input into a meta-learner, which will attempt to combine the predictions to create a final best predicted classification. So, for example, the predicted classifications from the tree classifiers, linear model, and the neural network classifier(s) can be used as input variables into a neural network meta-classifier, which will attempt to "learn" from the data how to combine the predictions from the different models to yield maximum classification accuracy.

One can apply meta-learners to the results from different meta-learners to create "meta-meta"-learners, and so on; however, in practice such exponential increase in the amount of data processing, in order to derive an accurate prediction, will yield less and less marginal utility.

Models for Data Mining
In the business environment, complex data mining projects may require the coordinate efforts of various experts, stakeholders, or departments throughout an entire organization. In the data mining literature, various "general frameworks" have been proposed to serve as blueprints for how to organize the process of gathering data, analyzing data, disseminating results, implementing results, and monitoring improvements.

One such model, CRISP (Cross-Industry Standard Process for data mining) was proposed in the mid-1990s by a European consortium of companies to serve as a non-proprietary standard process model for data mining. This general approach postulates the following (perhaps not particularly controversial) general sequence of steps for data mining projects:

Another approach - the Six Sigma methodology - is a well-structured, data-driven methodology for eliminating defects, waste, or quality control problems of all kinds in manufacturing, service delivery, management, and other business activities. This model has recently become very popular (due to its successful implementations) in various American industries, and it appears to gain favor worldwide. It postulated a sequence of, so-called, DMAIC steps -

- that grew up from the manufacturing, quality improvement, and process control traditions and is particularly well suited to production environments (including "production of services," i.e., service industries).

Another framework of this kind (actually somewhat similar to Six Sigma) is the approach proposed by SAS Institute called SEMMA -

- which is focusing more on the technical activities typically involved in a data mining project.

All of these models are concerned with the process of how to integrate data mining methodology into an organization, how to "convert data into information," how to involve important stake-holders, and how to disseminate the information in a form that can easily be converted by stake-holders into resources for strategic decision making.

Some software tools for data mining are specifically designed and documented to fit into one of these specific frameworks.

The general underlying philosophy of StatSoft's STATISTICA Data Miner is to provide a flexible data mining workbench that can be integrated into any organization, industry, or organizational culture, regardless of the general data mining process-model that the organization chooses to adopt. For example, STATISTICA Data Miner can include the complete set of (specific) necessary tools for ongoing company wide Six Sigma quality control efforts, and users can take advantage of its (still optional) DMAIC-centric user interface for industrial data mining tools. It can equally well be integrated into ongoing marketing research, CRM (Customer Relationship Management) projects, etc. that follow either the CRISP or SEMMA approach - it fits both of them perfectly well without favoring either one. Also, STATISTICA Data Miner offers all the advantages of a general data mining oriented "development kit" that includes easy to use tools for incorporating into your projects not only such components as custom database gateway solutions, prompted interactive queries, or proprietary algorithms, but also systems of access privileges, workgroup management, and other collaborative work tools that allow you to design large scale, enterprise-wide systems (e.g., following the CRISP, SEMMA, or a combination of both models) that involve your entire organization.

Predictive Data Mining
The term Predictive Data Mining is usually applied to identify data mining projects with the goal to identify a statistical or neural network model or set of models that can be used to predict some response of interest. For example, a credit card company may want to engage in predictive data mining, to derive a (trained) model or set of models (e.g., neural networks, meta-learner) that can quickly identify transactions which have a high probability of being fraudulent. Other types of data mining projects may be more exploratory in nature (e.g., to identify cluster or segments of customers), in which case drill-down descriptive and exploratory methods would be applied. Data reduction is another possible objective for data mining (e.g., to aggregate or amalgamate the information in very large data sets into useful and manageable chunks).

SEMMA
See Models for Data Mining.

Stacked Generalization
See Stacking.

Stacking (Stacked Generalization)
The concept of stacking (short for Stacked Generalization) applies to the area of predictive data mining, to combine the predictions from multiple models. It is particularly useful when the types of models included in the project are very different.

Suppose your data mining project includes tree classifiers, such as C&RT or CHAID, linear discriminant analysis (e.g., see GDA), and Neural Networks. Each computes predicted classifications for a crossvalidation sample, from which overall goodness-of-fit statistics (e.g., misclassification rates) can be computed. Experience has shown that combining the predictions from multiple methods often yields more accurate predictions than can be derived from any one method (e.g., see Witten and Frank, 2000). In stacking, the predictions from different classifiers are used as input into a meta-learner, which attempts to combine the predictions to create a final best predicted classification. So, for example, the predicted classifications from the tree classifiers, linear model, and the neural network classifier(s) can be used as input variables into a neural network meta-classifier, which will attempt to "learn" from the data how to combine the predictions from the different models to yield maximum classification accuracy.

Other methods for combining the prediction from multiple models or methods (e.g., from multiple datasets used for learning) are Boosting and Bagging (Voting).

Text Mining
While Data Mining is typically concerned with the detection of patterns in numeric data, very often important (e.g., critical to business) information is stored in the form of text. Unlike numeric data, text is often amorphous, and difficult to deal with. Text mining generally consists of the analysis of (multiple) text documents by extracting key phrases, concepts, etc. and the preparation of the text processed in that manner for further analyses with numeric data mining techniques (e.g., to determine co-occurrences of concepts, key phrases, names, addresses, product names, etc.).

Voting
See Bagging.

To index

Data Warehousing

StatSoft defines data warehousing as a process of organizing the storage of large, multivariate data sets in a way that facilitates the retrieval of information for analytic purposes.

The most efficient data warehousing architecture will be capable of incorporating or at least referencing all data available in the relevant enterprise-wide information management systems, using designated technology suitable for corporate data base management (e.g., Oracle, Sybase, MS SQL Server. Also, a flexible, high-performance (see the IDP technology), open architecture approach to data warehousing - that flexibly integrates with the existing corporate systems and allows the users to organize and efficiently reference for analytic purposes enterprise repositories of data of practically any complexity - is offered in StatSoft enterprise systems such as SEDAS (STATISTICA Enterprise-wide Data Analysis System) and SEWSS (STATISTICA Enterprise-wide SPC System), which can also work in conjunction with STATISTICA Data Miner and WebSTATISTICA Server Applications.

To index

On-Line Analytic Processing (OLAP)

The term On-Line Analytic Processing - OLAP (or Fast Analysis of Shared Multidimensional Information - FASMI) refers to technology that allows users of multidimensional databases to generate on-line descriptive or comparative summaries ("views") of data and other analytic queries. Note that despite its name, analyses referred to as OLAP do not need to be performed truly "on-line" (or in real-time); the term applies to analyses of multidimensional databases (that may, obviously, contain dynamically updated information) through efficient "multidimensional" queries that reference various types of data. OLAP facilities can be integrated into corporate (enterprise-wide) database systems and they allow analysts and managers to monitor the performance of the business (e.g., such as various aspects of the manufacturing process or numbers and types of completed transactions at different locations) or the market. The final result of OLAP techniques can be very simple (e.g., frequency tables, descriptive statistics, simple cross-tabulations) or more complex (e.g., they may involve seasonal adjustments, removal of outliers, and other forms of cleaning the data). Although Data Mining techniques can operate on any kind of unprocessed or even unstructured information, they can also be applied to the data views and summaries generated by OLAP to provide more in-depth and often more multidimensional knowledge. In this sense, Data Mining techniques could be considered to represent either a different analytic approach (serving different purposes than OLAP) or as an analytic extension of OLAP.

To index

Exploratory Data Analysis (EDA)

EDA vs. Hypothesis Testing

As opposed to traditional hypothesis testing designed to verify a priori hypotheses about relations between variables (e.g., "There is a positive correlation between the AGE of a person and his/her RISK TAKING disposition"), exploratory data analysis (EDA) is used to identify systematic relations between variables when there are no (or not complete) a priori expectations as to the nature of those relations. In a typical exploratory data analysis process, many variables are taken into account and compared, using a variety of techniques in the search for systematic patterns.

Computational EDA techniques

Computational exploratory data analysis methods include both simple basic statistics and more advanced, designated multivariate exploratory techniques designed to identify patterns in multivariate data sets.

Basic statistical exploratory methods. The basic statistical exploratory methods include such techniques as examining distributions of variables (e.g., to identify highly skewed or non-normal, such as bi-modal patterns), reviewing large correlation matrices for coefficients that meet certain thresholds (see example above), or examining multi-way frequency tables (e.g., "slice by slice" systematically reviewing combinations of levels of control variables).

Multivariate exploratory techniques. Multivariate exploratory techniques designed specifically to identify patterns in multivariate (or univariate, such as sequences of measurements) data sets include: Cluster Analysis, Factor Analysis, Discriminant Function Analysis, Multidimensional Scaling, Log-linear Analysis, Canonical Correlation, Stepwise Linear and Nonlinear (e.g., Logit) Regression, Correspondence Analysis, Time Series Analysis, and Classification Trees.

Neural Networks. Neural Networks are analytic techniques modeled after the (hypothesized) processes of learning in the cognitive system and the neurological functions of the brain and capable of predicting new observations (on specific variables) from other observations (on the same or other variables) after executing a process of so-called learning from existing data.

For more information, see Neural Networks; see also STATISTICA Neural Networks.

Graphical (data visualization) EDA techniques

A large selection of powerful exploratory data analytic techniques is also offered by graphical data visualization methods that can identify relations, trends, and biases "hidden" in unstructured data sets.

Brushing. Perhaps the most common and historically first widely used technique explicitly identified as graphical exploratory data analysis is brushing, an interactive method allowing one to select on-screen specific data points or subsets of data and identify their (e.g., common) characteristics, or to examine their effects on relations between relevant variables. Those relations between variables can be visualized by fitted functions (e.g., 2D lines or 3D surfaces) and their confidence intervals, thus, for example, one can examine changes in those functions by interactively (temporarily) removing or adding specific subsets of data. For example, one of many applications of the brushing technique is to select (i.e., highlight) in a matrix scatterplot all data points that belong to a certain category (e.g., a "medium" income level, see the highlighted subset in the fourth component graph of the first row in the illustration left) in order to examine how those specific observations contribute to relations between other variables in the same data set (e.g, the correlation between the "debt" and "assets" in the current example). If the brushing facility supports features like "animated brushing" or "automatic function re-fitting", one can define a dynamic brush that would move over the consecutive ranges of a criterion variable (e.g., "income" measured on a continuous scale or a discrete [3-level] scale as on the illustration above) and examine the dynamics of the contribution of the criterion variable to the relations between other relevant variables in the same data set.

Other graphical EDA techniques. Other graphical exploratory analytic techniques include function fitting and plotting, data smoothing, overlaying and merging of multiple displays, categorizing data, splitting/merging subsets of data in graphs, aggregating data in graphs, identifying and marking subsets of data that meet specific conditions, icon plots,

shading, plotting confidence intervals and confidence areas (e.g., ellipses),

generating tessellations, spectral planes,

integrated layered compressions,

and projected contours, data image reduction techniques, interactive (and continuous) rotation

with animated stratification (cross-sections) of 3D displays, and selective highlighting of specific series and blocks of data.

Verification of results of EDA

The exploration of data can only serve as the first stage of data analysis and its results can be treated as tentative at best as long as they are not confirmed, e.g., crossvalidated, using a different data set (or and independent subset). If the result of the exploratory stage suggests a particular model, then its validity can be verified by applying it to a new data set and testing its fit (e.g., testing its predictive validity). Case selection conditions can be used to quickly define subsets of data (e.g., for estimation and verification), and for testing the robustness of results.

To index

Neural Networks
(see also Neural Networks chapter)

Neural Networks are analytic techniques modeled after the (hypothesized) processes of learning in the cognitive system and the neurological functions of the brain and capable of predicting new observations (on specific variables) from other observations (on the same or other variables) after executing a process of so-called learning from existing data. Neural Networks is one of the Data Mining techniques.

The first step is to design a specific network architecture (that includes a specific number of "layers" each consisting of a certain number of "neurons"). The size and structure of the network needs to match the nature (e.g., the formal complexity) of the investigated phenomenon. Because the latter is obviously not known very well at this early stage, this task is not easy and often involves multiple "trials and errors." (Now, there is, however, neural network software that applies artificial intelligence techniques to aid in that tedious task and finds "the best" network architecture.)

The new network is then subjected to the process of "training." In that phase, neurons apply an iterative process to the number of inputs (variables) to adjust the weights of the network in order to optimally predict (in traditional terms one could say, find a "fit" to) the sample data on which the "training" is performed. After the phase of learning from an existing data set, the new network is ready and it can then be used to generate predictions.

The resulting "network" developed in the process of "learning" represents a pattern detected in the data. Thus, in this approach, the "network" is the functional equivalent of a model of relations between variables in the traditional model building approach. However, unlike in the traditional models, in the "network," those relations cannot be articulated in the usual terms used in statistics or methodology to describe relations between variables (such as, for example, "A is positively correlated with B but only for observations where the value of C is low and D is high"). Some neural networks can produce highly accurate predictions; they represent, however, a typical a-theoretical (one can say, "a black box") research approach. That approach is concerned only with practical considerations, that is, with the predictive validity of the solution and its applied relevance and not with the nature of the underlying mechanism or its relevance for any "theory" of the underlying phenomena.

However, it should be mentioned that Neural Network techniques can also be used as a component of analyses designed to build explanatory models because Neural Networks can help explore data sets in search for relevant variables or groups of variables; the results of such explorations can then facilitate the process of model building. Moreover, now there is neural network software that uses sophisticated algorithms to search for the most relevant input variables, thus potentially contributing directly to the model building process.

One of the major advantages of neural networks is that, theoretically, they are capable of approximating any continuous function, and thus the researcher does not need to have any hypotheses about the underlying model, or even to some extent, which variables matter. An important disadvantage, however, is that the final solution depends on the initial conditions of the network, and, as stated before, it is virtually impossible to "interpret" the solution in traditional, analytic terms, such as those used to build theories that explain phenomena.

Some authors stress the fact that neural networks use, or one should say, are expected to use, massively parallel computation models. For example Haykin (1994) defines neural network as:

"a massively parallel distributed processor that has a natural propensity for storing experiential knowledge and making it available for use. It resembles the brain in two respects: (1) Knowledge is acquired by the network through a learning process, and (2) Interneuron connection strengths known as synaptic weights are used to store the knowledge." (p. 2).

However, as Ripley (1996) points out, the vast majority of contemporary neural network applications run on single-processor computers and he argues that a large speed-up can be achieved not only by developing software that will take advantage of multiprocessor hardware by also by designing better (more efficient) learning algorithms.

Neural networks is one of the methods used in Data Mining; see also Exploratory Data Analysis. For more information on neural networks, see Haykin (1994), Masters (1995), Ripley (1996), and Welstead (1994). For a discussion of neural networks as statistical tools, see Warner and Misra (1996). See also, STATISTICA Neural Networks.

Supervised learning

2007-11-23T16:02:00.000Z

Supervised learning

From Wikipedia, the free encyclopedia

Supervised learning is a machine learning technique for creating a function from training data. The training data consist of pairs of input objects (typically vectors), and desired outputs. The output of the function can be a continuous value (called regression), or can predict a class label of the input object (called classification). The task of the supervised learner is to predict the value of the function for any valid input object after having seen a number of training examples (i.e. pairs of input and target output). To achieve this, the learner has to generalize from the presented data to unseen situations in a "reasonable" way (see inductive bias). (Compare with unsupervised learning.) The parallel task in human and animal psychology is often referred to as concept learning.

Overfitting

2007-11-18T16:25:00.000Z

we will look at some techniques for preventing our model becoming too powerful (overfitting). In the next, we address the related question of selecting an appropriate architecture with just the right amount of trainable parameters.

Bias-Variance trade-off

Consider the two fitted functions below. The data points (circles) have all been generated from a smooth function, h(x), with some added noise. Obviously, we want to end up with a model which approximates h(x), given a specific set of data y(x) generated as:

(1)

In the left hand panel we try to fit the points using a function g(x) which has too few parameters: a straight line. The model has the virtue of being simple; there are only two free parameters. However, it does not do a good job of fitting the data, and would not do well in predicting new data points. We say that the simpler model has a high bias.

The right hand panel shows a model which has been fitted using too many free parameters. It does an excellent job of fitting the data points, as the error at the data points is close to zero. However it would not do a good job of predicting h(x) for new values of x. We say that the model has a high variance. The model does not reflect the structure which we expect to be present in any data set generated by equation (1) above.

Clearly what we want is something in between: a model which is powerful enough to represent the underlying structure of the data (h(x)), but not so powerful that it faithfully models the noise associated with this particular data sample.

The bias-variance trade-off is most likely to become a problem if we have relatively few data points. In the opposite case, where we have essentially an infinite number of data points (as in continuous online learning), we are not usually in danger of overfitting the data, as the noise associated with any single data point plays a vanishingly small role in our overall fit. The following techniques therefore apply to situations in which we have a finite data set, and, typically, where we wish to train in batch mode.

Preventing overfitting

Early stopping

One of the simplest and most widely used means of avoiding overfitting is to divide the data into two sets: a training set and a validation set. We train using only the training data. Every now and then, however, we stop training, and test network performance on the independent validation set. No weight updates are made during this test! As the validation data is independent of the training data, network performance is a good measure of generalization, and as long as the network is learning the underlying structure of the data (h(x) above), performance on the validation set will improve with training. Once the network stops learning things which are expected to be true of any data sample and learns things which are true only of this sample (epsilon in Eqn 1 above), performance on the validation set will stop improving, and will typically get worse. Schematic learning curves showing error on the training and validation sets are shown below. To avoid overfitting, we simply stop training at time t, where performance on the validation set is optimal.

One detail of note when using early stopping: if we wish to test the trained network on a set of independent data to measure its ability to generalize, we need a third, independent, test set. This is because we used the validation set to decide when to stop training, and thus our trained network is no longer entirely independent of the validation set. The requirements of independent training, validation and test sets means that early stopping can only be used in a data-rich situation.

Weight decay

The over-fitted function above shows a high degree of curvature, while the linear function is maximally smooth. Regularization refers to a set of techniques which help to ensure that the function computed by the network is no more curved than necessary. This is achieved by adding a penalty to the error function, giving:

(2)

One possible form of the regularizer comes from the informal observation that an over-fitted mapping with regions of large curvature requires large weights. We thus penalize large weights by choosing

(3)

Using this modified error function, the weights are now updated as

(4)

where the right hand term causes the weight to decrease as a function of its own size. In the absence of any input, all weights will tend to decrease exponentially, hence the term "weight decay".

Training with noise

A final method which can often help to reduce the importance of the specific noise characteristics associated with a particular data sample is to add an extra small amount of noise (a small random value with mean value of zero) to each input. Each time a specific input pattern x is presented, we add a different random number, and use instead.

At first, this may seem a rather odd thing to do: to deliberately corrupt ones own data. However, perhaps you can see that it will now be difficult for the network to approximate any specific data point too closely. In practice, training with added noise has indeed been shown to reduce overfitting and thus improve generalization in some situations.

If we have a finite training set, another way of introducing noise into the training process is to use online training, that is, updating weights after every pattern presentation, and to randomly reorder the patterns at the end of each training epoch. In this manner, each weight update is based on a noisy estimate of the true gradient.

[Top] [Next: Growing and pruning networks] [Back to the first page]

Algebra - Imperial College

2007-10-23T16:16:00.000+01:00

M2P2 Algebra

Lecturer: Prof M W Liebeck

Recommended books

For group theory, there are many good introductory books. Here are a few suggestions:

J.B. Fraleigh, A First Course in Abstract Algebra, Addison Wesley

R.. Allenby, Rings, Fields and Groups, Arnold

I.N. Herstein, Topics in Algebra

For linear algebra, here are a few good ones:

J. Fraleigh and R. Beauregard, Linear Algebra, Addison Wesley

S. Lipschutz and M. Lipson, Linear Algebra, Schaum Outline Series, McGraw Hill

M2S1 PROBABILITY AND STATISTICS II - Imperial College

2007-10-23T16:00:00.000+01:00

Recommended Texts
G. R. Grimmett and D. R. Stirzaker, Probability and Random Processes (2nd Edition/3rd Edition).
[Very useful for probability material of the course].
W. Feller, An Introduction to Probability Theory and Its Applications. Vols 1 and 2. [A classical
reference text].
G. Casella and R.L. Berger, Statistical Inference. [A very useful text, which covers statistical ideas as
well as probability material].
There are many such introductory texts in the Mathematics library. Other books relating to specific
parts of the course will be recommended when relevant.
Also, there will be a course WWW page accessible from http://stats.ma.ic.ac.uk/ayoung. It will
contain links to course handouts, exercises and solutions.
Professor A. Young (room 529, email alastair.young@imperial.ac.uk)

Confusion Matrix

2007-10-19T22:09:00.001+01:00

A confusion matrix (Kohavi and Provost, 1998) contains information about actual and predicted classifications done by a classification system. Performance of such systems is commonly evaluated using the data in the matrix. The following table shows the confusion matrix for a two class classifier.

The entries in the confusion matrix have the following meaning in the context of our study:

a is the number of correct predictions that an instance is negative,
b is the number of incorrect predictions that an instance is positive,
c is the number of incorrect of predictions that an instance negative, and
d is the number of correct predictions that an instance is positive.

		Predicted
		Negative	Positive
Actual	Negative	a	b
Actual	Positive	c	d

Several standard terms have been defined for the 2 class matrix:

The accuracy (AC) is the proportion of the total number of predictions that were correct. It is determined using the equation:

[1]

The recall or hit rate or true positive rate (TP) is the proportion of positive cases that were correctly identified, as calculated using the equation:

[2]

The false alarm rate or false positive rate (FP) is the proportion of negatives cases that were incorrectly classified as positive, as calculated using the equation:

[3]

The true negative rate (TN) is defined as the proportion of negatives cases that were classified correctly, as calculated using the equation:

[4]

The false negative rate (FN) is the proportion of positives cases that were incorrectly classified as negative, as calculated using the equation:

[5]

Finally, precision (P) is the proportion of the predicted positive cases that were correct, as calculated using the equation:

[6]

The accuracy determined using equation 1 may not be an adequate performance measure when the number of negative cases is much greater than the number of positive cases (Kubat et al., 1998). Suppose there are 1000 cases, 995 of which are negative cases and 5 of which are positive cases. If the system classifies them all as negative, the accuracy would be 99.5%, even though the classifier missed all positive cases. Other performance measures account for this by including TP in a product: for example, geometric mean (g-mean) (Kubat et al., 1998), as defined in equations 7 and 8, and F-Measure (Lewis and Gale, 1994), as defined in equation 9.

[7]

[8]

[9]

In equation 9, b has a value from 0 to infinity and is used to control the weight assigned to TP and P. Any classifier evaluated using equations 7, 8 or 9 will have a measure value of 0, if all positive cases are classified incorrectly.

Another way to examine the performance of classifiers is to use a ROC graph, described on the next page.

Timetable II

2007-10-19T20:04:00.001+01:00

Autumn 2007

MSc Advanced Computing (Weeks 2 - 11)

Week 2 start date: Monday 8 October, 2007

Date Published: 19 October 2007

Time	Monday	Tuesday	Wednesday	Thursday	Friday
0900	Advanced Topics in Software Engineering LEC (2-10) / jnm (2-10),sue (2-10) / 311	Modal and Temporal Logic LEC (2-10) / imh (2-10),mjs (2-10) / 144	Commemoration Day (No Teaching Week 4 - 24.10.07) Wks (4-4) / None (4-4) / Intelligent Data and Probabilistic Inference LEC (2-10) / dfg (2-10) / 145	Advanced Issues in Object Oriented Programming LEC (2-10) / scd (2-10) / 145	Network Security LEC (2-10) / ecl1 (2-10),mrh (2-10) / 308
1000	Advanced Topics in Software Engineering TUT (2-10) / jnm (2-10),sue (2-10) / 311 Laboratory Prolog LAB (2-10) / nr600 (2-10) / 202,206 Intelligent Data and Probabilistic Inference LEC (11-11) / dfg (11-11) / 308	Modal and Temporal Logic TUT (2-10) / imh (2-10),mjs (2-10) / 144	Commemoration Day (No Teaching Week 4 - 24.10.07) Wks (4-4) / None (4-4) / Intelligent Data and Probabilistic Inference LEC (2-10) / dfg (2-10) / 308	Advanced Issues in Object Oriented Programming TUT (2-10) / scd (2-10) / 145	Lexus Preperation Wks (11-11) / nr600 (11-11) / 202,206 Network Security TUT (2-10) / ecl1 (2-10),mrh (2-10) / 308
1100	Laboratory Prolog LAB (2-10) / nr600 (2-10) / 202,206 Advanced Topics in Software Engineering LEC (2-10) / jnm (2-10),sue (2-10) / 311 Intelligent Data and Probabilistic Inference LEC (11-11) / dfg (11-11) / 308 Lexus Preperation Wks (5-5) / nr600 (5-5) / 202,206	Models of Concurrent Computation LEC (2-10) / dirk (2-10),pg (2-10) / 145	Commemoration Day (No Teaching Week 4 - 24.10.07) Wks (4-4) / None (4-4) / Laboratory Prolog LAB (2-10) / nr600 (2-10) / 202,206 Intelligent Data and Probabilistic Inference TUT (2-10) / dfg (2-10) / 344	Machine Learning LAB (2-10) / maja (2-10),shm (2-10) / 219 Computing for Optimal Decisions LEC (2-10) / br (2-10) / 145	Machine Learning LAB (2-10) / maja (2-10),shm (2-10) / 219 Lexis Prolog LAB (11-11) / nr600 (11-11) / 202,206 Modal and Temporal Logic LEC (2-10) / imh (2-10),mjs (2-10) / 144
1200	Laboratory Workshop (MAC & MSc CS in depth pathway) LEC (2-11) / nr600 (2-11) / 311 Lexis Prolog LAB (5-5) / nr600 (5-5) / 202,206 Intelligent Data and Probabilistic Inference TUT (11-11) / dfg (11-11) / 308	Prolog Support Lectures LEC (2-10) / cjh (2-10),klc (2-10) / 144	Commemoration Day (No Teaching Week 4 - 24.10.07) Wks (4-4) / None (4-4) / Laboratory Prolog LAB (2-10) / nr600 (2-10) / 202,206	Machine Learning Alternative Lab LAB (2-10) / maja (2-10),shm (2-10) / 219	Machine Learning Alternative Lab LAB (2-10) / maja (2-10),shm (2-10) / 219 Lexis Prolog LAB (11-11) / nr600 (11-11) / 202,206
1300	Lexis Prolog LAB (5-5) / nr600 (5-5) / 202,206
1400	Advanced Issues in Object Oriented Programming LEC (2-10) / scd (2-10) / 144	Network Security LEC (2-10) / ecl1 (2-10),mrh (2-10) / 144		Automated Reasoning LEC (2-10) / kb (2-10) / 145	Automated Reasoning LEC (2-10) / kb (2-10) / 144
1500		Models of Concurrent Computation TUT (2-10) / dirk (2-10),pg (2-10) / 145		Computer Vision LEC (2-10) / gzy (2-10) / 144	Automated Reasoning TUT (2-10) / kb (2-10) / 144
1600	Machine Learning LEC (2-10) / maja (2-10),shm (2-10) / 311	Models of Concurrent Computation LEC (2-10) / dirk (2-10),pg (2-10) / 145		Computer Vision TUT (2-10) / gzy (2-10) / 144	Computing for Optimal Decisions LEC (2-10) / br (2-10) / 145
1700	Machine Learning LEC (2-10) / maja (2-10),shm (2-10) / 311	Project Lecture LEC (8-8) / / 308		Computer Vision LEC (2-10) / gzy (2-10) / 144	Computing for Optimal Decisions TUT (2-10) / br (2-10) / 145

Timetable

2007-10-19T20:03:00.001+01:00

Autumn 2007

MSc Computing Science (Weeks 2 - 11)

Week 2 start date: Monday 8 October, 2007

Date Published: 19 October 2007

Time	Monday	Tuesday	Wednesday	Thursday	Friday
0900		Programming in C and C++ LEC (2-2) / wjk (2-2) / 311	Commemoration Day (No Teaching Week 4 - 24.10.07) Wks (4-4) / None (4-4) / Integrated Programming Laboratory LAB (2-10) / ap7 (2-10) / 202,206 Programming in C and C++ LEC (2-2) / wjk (2-2) / 144	Programming in C and C++ LEC (2-2) / wjk (2-2) / 308
1000	Computer Systems LEC (3-11) / dirk (3-11),jamm (3-11) / 145	Integrated Programming Laboratory LAB (2-2) / ap7 (2-2) / 202 Object Oriented Design & Programming. LEC (3-11) / scd (3-11),wjk (3-11) / 343	Integrated Programming Laboratory LAB (2-10) / ap7 (2-10) / 202 Commemoration Day (No Teaching Week 4 - 24.10.07) Wks (4-4) / None (4-4) /	Integrated Programming Laboratory LAB (2-2) / ap7 (2-2) / 206	Programming in C and C++ LEC (2-2) / wjk (2-2) / 144 Object Oriented Design & Programming. LEC (3-11) / scd (3-11),wjk (3-11) / 343
1100	Computer Systems TUT (3-11) / dirk (3-11),jamm (3-11) / 144	Integrated Programming Laboratory LAB (2-2) / ap7 (2-2) / 202 Logic and AI Programming LEC (6-8) / klc (6-8) / 343	Integrated Programming Laboratory LAB (2-10) / ap7 (2-10) / 202 Commemoration Day (No Teaching Week 4 - 24.10.07) Wks (4-4) / None (4-4) /	Computer Systems LEC (3-9) / ajd (3-9) / 144 Integrated Programming Laboratory LAB (2-2) / ap7 (2-2) / 202,206	Object Oriented Design & Programming. TUT (3-11) / scd (3-11),wjk (3-11) / 343 Integrated Programming Laboratory LAB (1-2) / ap7 (1-2) / 202,206
1200	Computer Systems LEC (3-11) / dirk (3-11),jamm (3-11) / 145	Programming in C and C++ LEC (2-2) / wjk (2-2) / 311 Logic and AI Programming TUT (6-8) / klc (6-8) / 343,202,206	Integrated Programming Laboratory LAB (2-10) / ap7 (2-10) / 202,206 Commemoration Day (No Teaching Week 4 - 24.10.07) Wks (4-4) / None (4-4) /	Programming in C and C++ LEC (2-2) / wjk (2-2) / 308 Computer Systems TUT (3-9) / ajd (3-9) / 144,202,206	Computer Systems LEC (3-9) / ajd (3-9) / 144 Programming in C and C++ TUT (2-2) / wjk (2-2) / 308 Integrated Programming Laboratory LAB (2-2) / ap7 (2-2) / 202,206
1300
1400	Logic and AI Programming LEC (3-11) / fs (3-11),klc (3-11) / 145 Programming in C and C++ LEC (2-2) / wjk (2-2) / 145	Logic and AI Programming LEC (3-11) / fs (3-11),klc (3-11) / 343 Programming in C and C++ TUT (2-2) / wjk (2-2) / 344		Programming in C and C++ LEC (2-2) / wjk (2-2) / 144 Computer Systems LEC (3-9) / ajd (3-9) / 144	Programming in C and C++ TUT (2-2) / wjk (2-2) / 344
1500	Programming in C and C++ LEC (2-2) / wjk (2-2) / 145 Logic and AI Programming LEC (6-8) / klc (6-8) / 145	Logic and AI Programming TUT (3-11) / fs (3-11),klc (3-11) / 343 Logic and AI Programming LAB (9-10) / klc (9-10) / 202,206 Integrated Programming Laboratory LAB (2-2) / ap7 (2-2) / 206		Integrated Programming Laboratory LAB (2-2) / ap7 (2-2) / 202	Integrated Programming Laboratory LAB (2-2) / ap7 (2-2) / 202
1600	Integrated Programming Laboratory LAB (2-2) / ap7 (2-2) / 202,206	Integrated Programming Laboratory LAB (2-2) / ap7 (2-2) / 202,206 Lab Workshop (MSc CS) LAB (3-10) / ap7 (3-10) / 202,206		Integrated Programming Laboratory LAB (7-11) / ap7 (7-11) / 202 Integrated Programming Laboratory LAB (2-4) / ap7 (2-4) / 202	MSc in Computing Group Project Presentations Wks (8-8) / ap7 (8-8) / 308 Integrated Programming Laboratory LAB (2-2) / ap7 (2-2) / 202
1700	Integrated Programming Laboratory LAB (2-2) / ap7 (2-2) / 202,206	Integrated Programming Laboratory LAB (2-2) / ap7 (2-2) / 202,206		Integrated Programming Laboratory LAB (2-4) / ap7 (2-4) / 202,206	MSc in Computing Group Project Presentations Wks (8-8) / ap7 (8-8) / 308

ROC Graph

2007-10-19T20:01:00.001+01:00

ROC graphs are another way besides confusion matrices to examine the performance of classifiers (Swets, 1988). A ROC graph is a plot with the false positive rate on the X axis and the true positive rate on the Y axis. The point (0,1) is the perfect classifier: it classifies all positive cases and negative cases correctly. It is (0,1) because the false positive rate is 0 (none), and the true positive rate is 1 (all). The point (0,0) represents a classifier that predicts all cases to be negative, while the point (1,1) corresponds to a classifier that predicts every case to be positive. Point (1,0) is the classifier that is incorrect for all classifications. In many cases, a classifier has a parameter that can be adjusted to increase TP at the cost of an increased FP or decrease FP at the cost of a decrease in TP. Each parameter setting provides a (FP, TP) pair and a series of such pairs can be used to plot an ROC curve. A non-parametric classifier is represented by a single ROC point, corresponding to its (FP,TP) pair.

The above figure shows an example of an ROC graph with two ROC curves labeled C1 and C2, and two ROC points labeled P1 and P2. Non-parametric algorithms produce a single ROC point for a particular data set.

Features of ROC Graphs

An ROC curve or point is independent of class distribution or error costs (Provost et al., 1998).
An ROC graph encapsulates all information contained in the confusion matrix, since FN is the complement of TP and TN is the complement of FP (Swets, 1988).
ROC curves provide a visual tool for examining the tradeoff between the ability of a classifier to correctly identify positive cases and the number of negative cases that are incorrectly classified.
The closer the curve follows the left-hand border and then the top border of the ROC space, the more accurate the test.The closer the curve comes to the 45-degree diagonal of the ROC space, the less accurate the test.

Area-based Accuracy Measure

It has been suggested that the area beneath an ROC curve can be used as a measure of accuracy in many applications (Swets, 1988). Provost and Fawcett (1997) argue that using classification accuracy to compare classifiers is not adequate unless cost and class distributions are completely unknown and a single classifier must be chosen to handle any situation. They propose a method of evaluating classifiers using a ROC graph and imprecise cost and class distribution information.

Euclidian Distance Comparison

Another way of comparing ROC points is by using an equation that equates accuracy with the Euclidian distance from the perfect classifier, point (0,1) on the graph. We include a weight factor that allows us to define relative misclassification costs, if such information is available. We define AC_d as a distance based performance measure for an ROC point and calculate it using the equation:

[22]

where W is a factor, ranging from 0 to 1, that is used to assign relative importance to false positives and false negatives. AC_d ranges from 0 for the perfect classifier to sqrt(2) for a classifier that classifies all cases incorrectly, point (1,0) on the ROC graph. AC_d differs from g-mean₁, g-mean₂ and F-measure in that it is equal to zero only if all cases are classified correctly. In other words, a classifier evaluated using AC_d gets some credit for correct classification of negative cases, regardless of its accuracy in correctly identifying positive cases.

Example

Consider two algorithms A and B that perform adequately against most data sets. However, assume both A and B misclassify all positive cases in a particular data set and A classifies 10 times the number of infrequent itemsets as potentially frequent compared to B. Algorithm B is the better algorithm in this case because there has been less wasted effort counting infrequent itemsets.

Cumulative Gains and Lift Charts

2007-10-19T19:59:00.000+01:00

Lift is a measure of the effectiveness of a predictive model calculated as the ratio between the results obtained with and without the predictive model.
Cumulative gains and lift charts are visual aids for measuring model performance
Both charts consist of a lift curve and a baseline
The greater the area between the lift curve and the baseline, the better the model

Example Problem 1

A company wants to do a mail marketing campaign. It costs the company $1 for each item mailed. They have information on 100,000 customers. Create a cumulative gains and a lift chart from the following data.

Overall Response Rate: If we assume we have no model other than the prediction of the overall response rate, then we can predict the number of positive responses as a fraction of the total customers contacted. Suppose the response rate is 20%. If all 100,000 customers are contacted we will receive around 20,000 positive responses.

Cost ($)	Total Customers Contacted	Positive Responses
100000	100000	20000

Prediction of Response Model: A response model predicts who will respond to a marketing campaign. If we have a response model, we can make more detailed predictions. For example, we use the response model to assign a score to all 100,000 customers and predict the results of contacting only the top 10,000 customers, the top 20,000 customers, etc.

Cost ($)	Total Customers Contacted	Positive Responses
10000	10000	6000
20000	20000	10000
30000	30000	13000
40000	40000	15800
50000	50000	17000
60000	60000	18000
70000	70000	18800
80000	80000	19400
90000	90000	19800
100000	100000	20000

Cumulative Gains Chart:

The y-axis shows the percentage of positive responses. This is a percentage of the total possible positive responses (20,000 as the overall response rate shows).
The x-axis shows the percentage of customers contacted, which is a fraction of the 100,000 total customers.
Baseline (overall response rate): If we contact X% of customers then we will receive X% of the total positive responses.
Lift Curve: Using the predictions of the response model, calculate the percentage of positive responses for the percent of customers contacted and map these points to create the lift curve.

Lift Chart:

Shows the actual lift.
To plot the chart: Calculate the points on the lift curve by determining the ratio between the result predicted by our model and the result using no model.
Example: For contacting 10% of customers, using no model we should get 10% of responders and using the given model we should get 30% of responders. The y-value of the lift curve at 10% is 30 / 10 = 3.

Analyzing the Charts: Cumulative gains and lift charts are a graphical representation of the advantage of using a predictive model to choose which customers to contact. The lift chart shows how much more likely we are to receive respondents than if we contact a random sample of customers. For example, by contacting only 10% of customers based on the predictive model we will reach 3 times as many respondents as if we use no model.

Evaluating a Predictive Model

We can assess the value of a predictive model by using the model to score a set of customers and then contacting them in this order. The actual response rates are recorded for each cutoff point, such as the first 10% contacted, the first 20% contacted, etc. We create cumulative gains and lift charts using the actual response rates to see how much the predictive model would have helped in this situation. The information can be used to determine whether we should use this model or one similar to it in the future.

Example Problem 2

Using the response model P(x)=100-AGE(x) for customer x and the data table shown below, construct the cumulative gains and lift charts. Ties in ranking should be arbitrarily broken by assigning a higher rank to who appears first in the table.

Customer Name	Height	Age	Actual Response
Alan	70	39	N
Bob	72	21	Y
Jessica	65	25	Y
Elizabeth	62	30	Y
Hilary	67	19	Y
Fred	69	48	N
Alex	65	12	Y
Margot	63	51	N
Sean	71	65	Y
Chris	73	42	N
Philip	75	20	Y
Catherine	70	23	N
Amy	69	13	N
Erin	68	35	Y
Trent	72	55	N
Preston	68	25	N
John	64	76	N
Nancy	64	24	Y
Kim	72	31	N
Laura	62	29	Y

1. Calculate P(x) for each person x

2. Order the people according to rank P(x)

Customer Name P(x) Actual Response

Alex 88 Y

Amy 87 N

Hilary 81 Y

Philip 80 Y

Bob 79 Y

Catherine 77 N

Nancy 76 Y

Jessica 75 Y

Preston 75 N

Laura 71 Y

Elizabeth 70 Y

Kim 69 N

Erin 65 Y

Alan 61 N

Chris 58 N

Fred 52 N

Margot 49 N

Trent 45 N

Sean 35 Y

John 24 N

3. Calculate the percentage of total responses for each cutoff point

Response Rate = Number of Responses / Total Number of Responses (10)

Total Customers Contacted Number of Responses Response Rate

2

1

10%

4

3

30%

6

4

40%

8

6

60%

10

7

70%

12

8

80%

14

9

90%

16

9

90%

18

9

90%

20

10

100%

4. Create the cumulative gains chart:

The lift curve and the baseline have the same values for 10%-20% and 90%-100%.

5. Create the lift chart:

Introduction to ROC Curves

2007-10-19T19:52:00.000+01:00

About ROC curves

A ROC curve provides a graphical representation of the relationship between the true-positive and false-positive prediction rate of a model. The y-axis corresponds to the sensitivity of the model, i.e. how well the model is able to predict true positives (real cleavages) from sites that are not cleaved, and the y-coordinates are calculated as:

The x-axis corresponds to the specificity (expressed on the curve as 1-specificity), i.e. the ability of the model to identify true negatives. An increase in specificity (i.e. a decrease along the X-axis) results in an increase in sensitivity. The x-coordinates are calculated as:

The greater the sensitivity at high specificity values (i.e. high y-axis values at low X-axis values) the better the model. A numerical measure of the accuracy of the model can be obtained from the area under the curve, where an area of 1.0 signifies near perfect accuracy, while an area of less than 0.5 indicates that the model is worse than just random. The quantitative-qualitative relationship between area and accuracy follows a fairly linear pattern, such that the following could be used as a guide:

0.9-1: Excellent
0.8-0.9: Very good
0.7-0.8: Good
0.6-0.7: Average
0.5-0.6: Poor

Introduction to ROC Curves

The sensitivity and specificity of a diagnostic test depends on more than just the "quality" of the test--they also depend on the definition of what constitutes an abnormal test. Look at the the idealized graph at right showing the number of patients with and without a disease arranged according to the value of a diagnostic test. This distributions overlap--the test (like most) does not distinguish normal from disease with 100% accuracy. The area of overlap indicates where the test cannot distinguish normal from disease. In practice, we choose a cutpoint (indicated by the vertical black line) above which we consider the test to be abnormal and below which we consider the test to be normal. The position of the cutpoint will determine the number of true positive, true negatives, false positives and false negatives. We may wish to use different cutpoints for different clinical situations if we wish to minimize one of the erroneous types of test results.

We can use the hypothyroidism data from the likelihood ratio section to illustrate how sensitivity and specificity change depending on the choice of T4 level that defines hypothyroidism. Recall the data on patients with suspected hypothyroidism reported by Goldstein and Mushlin (J Gen Intern Med 1987;2:20-24.). The data on T4 values in hypothyroid and euthyroid patients are shown graphically (below left) and in a simplified tabular form (below right).

T4 value	Hypothyroid	Euthyroid
5 or less	18	1
5.1 - 7	7	17
7.1 - 9	4	36
9 or more	3	39
Totals:	32	93

Suppose that patients with T4 values of 5 or less are considered to be hypothyroid. The data display then reduces to:

T4 value	Hypothyroid	Euthyroid
5 or less	18	1
> 5	14	92
Totals:	32	93

You should be able to verify that the sensivity is 0.56 and the specificity is 0.99.

Now, suppose we decide to make the definition of hypothyroidism less stringent and now consider patients with T4 values of 7 or less to be hypothyroid. The data display will now look like this:

T4 value	Hypothyroid	Euthyroid
7 or less	25	18
> 7	7	75
Totals:	32	93

You should be able to verify that the sensivity is 0.78 and the specificity is 0.81.

Lets move the cut point for hypothyroidism one more time:

T4 value	Hypothyroid	Euthyroid
<>	29	54
9 or more	3	39
Totals:	32	93

You should be able to verify that the sensivity is 0.91 and the specificity is 0.42.

Now, take the sensitivity and specificity values above and put them into a table:

Cutpoint	Sensitivity	Specificity
5	0.56	0.99
7	0.78	0.81
9	0.91	0.42

Notice that you can improve the sensitivity by moving to cutpoint to a higher T4 value--that is, you can make the criterion for a positive test less strict. You can improve the specificity by moving the cutpoint to a lower T4 value--that is, you can make the criterion for a positive test more strict. Thus, there is a tradeoff between sensitivity and specificity. You can change the definition of a positive test to improve one but the other will decline.

The next section covers how to use the numbers we just calculated to draw and interpret an ROC curve.
.

Plotting and Intrepretating an ROC Curve

This section continues the hypothyroidism example started in the the previous section. We showed that the table at left can be summarized by the operating characteristics at right:

T4 value	Hypothyroid	Euthyroid
5 or less	18	1
5.1 - 7	7	17
7.1 - 9	4	36
9 or more	3	39
Totals:	32	93

Cutpoint	Sensitivity	Specificity
5	0.56	0.99
7	0.78	0.81
9	0.91	0.42

The operating characteristics (above right) can be reformulated slightly and then presented graphically as shown below to the right:

Cutpoint	True Positives	False Positives
5	0.56	0.01
7	0.78	0.19
9	0.91	0.58

This type of graph is called a Receiver Operating Characteristic curve (or ROC curve.) It is a plot of the true positive rate against the false positive rate for the different possible cutpoints of a diagnostic test.

An ROC curve demonstrates several things:

It shows the tradeoff between sensitivity and specificity (any increase in sensitivity will be accompanied by a decrease in specificity).
The closer the curve follows the left-hand border and then the top border of the ROC space, the more accurate the test.
The closer the curve comes to the 45-degree diagonal of the ROC space, the less accurate the test.
The slope of the tangent line at a cutpoint gives the likelihood ratio (LR) for that value of the test. You can check this out on the graph above. Recall that the LR for T4 <> 9 is 0.2. This corresponds to the far right, nearly horizontal portion of the curve.
The area under the curve is a measure of text accuracy. This is discussed further in the next section.

The Area Under an ROC Curve

| Previous Section | Main Menu | Next Section |

The graph at right shows three ROC curves representing excellent, good, and worthless tests plotted on the same graph. The accuracy of the test depends on how well the test separates the group being tested into those with and without the disease in question. Accuracy is measured by the area under the ROC curve. An area of 1 represents a perfect test; an area of .5 represents a worthless test. A rough guide for classifying the accuracy of a diagnostic test is the traditional academic point system:

.90-1 = excellent (A)
.80-.90 = good (B)
.70-.80 = fair (C)
.60-.70 = poor (D)
.50-.60 = fail (F)

Recall the T4 data from the previous section. The area under the T4 ROC curve is .86. The T4 would be considered to be "good" at separating hypothyroid from euthyroid patients.

ROC curves can also be constructed from clinical prediction rules. The graphs at right come from a study of how clinical findings predict strep throat (Wigton RS, Connor JL, Centor RM. Transportability of a decision rule for the diagnosis of streptococcal pharyngitis. Arch Intern Med. 1986;146:81-83.) In that study, the presence of tonsillar exudate, fever, adenopathy and the absence of cough all predicted strep. The curves were constructed by computing the sensitivity and specificity of increasing numbers of clinical findings (from 0 to 4) in predicting strep. The study compared patients in Virginia and Nebraska and found that the rule performed more accurately in Virginia (area under the curve = .78) compared to Nebraska (area under the curve = .73). These differences turn out not to be statistically different, however.

At this point, you may be wondering what this area number really means and how it is computed. The area measures discrimination, that is, the ability of the test to correctly classify those with and without the disease. Consider the situation in which patients are already correctly classified into two groups. You randomly pick on from the disease group and one from the no-disease group and do the test on both. The patient with the more abnormal test result should be the one from the disease group. The area under the curve is the percentage of randomly drawn pairs for which this is true (that is, the test correctly classifies the two patients in the random pair).

Computing the area is more difficult to explain and beyond the scope of this introductory material. Two methods are commonly used: a non-parametric method based on constructing trapeziods under the curve as an approximation of area and a parametric method using a maximum likelihood estimator to fit a smooth curve to the data points. Both methods are available as computer programs and give an estimate of area and standard error that can be used to compare different tests or the same test in different patient populations. For more on quantitative ROC analysis, see Metz CE. Basic principles of ROC analysis. Sem Nuc Med. 1978;8:283-298.

A final note of historical interest
You may be wondering where the name "Reciever Operating Characteristic" came from. ROC analysis is part of a field called "Signal Dectection Therory" developed during World War II for the analysis of radar images. Radar operators had to decide whether a blip on the screen represented an enemy target, a friendly ship, or just noise. Signal detection theory measures the ability of radar receiver operators to make these important distinctions. Their ability to do so was called the Receiver Operating Characteristics. It was not until the 1970's that signal detection theory was recognized as useful for interpreting medical test results.

Logistic Regression

2007-10-19T19:50:00.000+01:00

Logistic Regression

What is the logistic curve? What is the base of the natural logarithm? Why do statisticians prefer logistic regression to ordinary linear regression when the DV is binary? How are probabilities, odds and logits related? What is an odds ratio? How can logistic regression be considered a linear regression? What is a loss function? What is a maximum likelihood estimate? How is the b weight in logistic regression for a categorical variable related to the odds ratio of its constituent categories?

This chapter is difficult because there are many new concepts in it. Studying this may bring back feelings that you had in the first third of the course, when there were many new concepts each week.

For this chapter only, we are going to deal with a dependent variable that is binary (a categorical variable that has two values such as "yes" and "no") rather than continuous.

[Technical note: Logistic regression can also be applied to ordered categories (ordinal data), that is, variables with more than two ordered categories, such as what you find in many surveys. However, we won't be dealing with that in this course and you probably will never be taught it. If our dependent variable has several unordered categories (e.g., suppose our DV was state of origin in the U.S.), then we can use something called discriminant analysis, which will be taught to you in a course on multivariate statistics.]

It is customary to code a binary DV either 0 or 1. For example, we might code a successfully kicked field goal as 1 and a missed field goal as 0 or we might code yes as 1 and no as 0 or admitted as 1 and rejected as 0 or Cherry Garcia flavor ice cream as 1 and all other flavors as zero. If we code like this, then the mean of the distribution is equal to the proportion of 1s in the distribution. For example if there are 100 people in the distribution and 30 of them are coded 1, then the mean of the distribution is .30, which is the proportion of 1s. The mean of the distribution is also the probability of drawing a person labeled as 1 at random from the distribution. That is, if we grab a person at random from our sample of 100 that I just described, the probability that the person will be a 1 is .30. Therefore, proportion and probability of 1 are the same in such cases. The mean of a binary distribution so coded is denoted as P, the proportion of 1s. The proportion of zeros is (1-P), which is sometimes denoted as Q. The variance of such a distribution is PQ, and the standard deviation is Sqrt(PQ). {Why can't all of stats be this easy?}

Suppose we want to predict whether someone is male or female (DV, M=1, F=0) using height in inches (IV). We could plot the relations between the two variables as we customarily do in regression. The plot might look something like this:

Points to notice about the graph (data are fictional):

The regression line is a rolling average, just as in linear regression. The Y-axis is P, which indicates the proportion of 1s at any given value of height. (review graph)
The regression line is nonlinear. (review graph)
None of the observations --the raw data points-- actually fall on the regression line. They all fall on zero or one. (review graph)

Why use logistic regression rather than ordinary linear regression?

When I was in graduate school, people didn't use logistic regression with a binary DV. They just used ordinary linear regression instead. Statisticians won the day, however, and now most psychologists use logistic regression with a binary DV for the following reasons:

If you use linear regression, the predicted values will become greater than one and less than zero if you move far enough on the X-axis. Such values are theoretically inadmissible.
One of the assumptions of regression is that the variance of Y is constant across values of X (homoscedasticity). This cannot be the case with a binary variable, because the variance is PQ. When 50 percent of the people are 1s, then the variance is .25, its maximum value. As we move to more extreme values, the variance decreases. When P=.10, the variance is .1*.9 = .09, so as P approaches 1 or zero, the variance approaches zero.
The significance testing of the b weights rest upon the assumption that errors of prediction (Y-Y') are normally distributed. Because Y only takes the values 0 and 1, this assumption is pretty hard to justify, even approximately. Therefore, the tests of the regression weights are suspect if you use linear regression with a binary DV.

The Logistic Curve

The logistic curve relates the independent variable, X, to the rolling mean of the DV, P (). The formula to do so may be written either

where P is the probability of a 1 (the proportion of 1s, the mean of Y), e is the base of the natural logarithm (about 2.718) and a and b are the parameters of the model. The value of a yields P when X is zero, and b adjusts how quickly the probability changes with changing X a single unit (we can have standardized and unstandardized b weights in logistic regression, just as in ordinary linear regression). Because the relation between X and P is nonlinear, b does not have a straightforward interpretation in this model as it does in ordinary linear regression.

Loss Function

A loss function is a measure of fit between a mathematical model of data and the actual data. We choose the parameters of our model to minimize the badness-of-fit or to maximize the goodness-of-fit of the model to the data. With least squares (the only loss function we have used thus far), we minimize SS_res, the sum of squares residual. This also happens to maximize SS_reg, the sum of squares due to regression. With linear or curvilinear models, there is a mathematical solution to the problem that will minimize the sum of squares, that is,

b = (X'X)^-1X'y

b = R^-1r

With some models, like the logistic curve, there is no mathematical solution that will produce least squares estimates of the parameters. For many of these models, the loss function chosen is called maximum likelihood. A likelihood is a conditional probability (e.g., P(Y|X), the probability of Y given X). We can pick the parameters of the model (a and b of the logistic curve) at random or by trial-and-error and then compute the likelihood of the data given those parameters (actually, we do better than trail-and-error, but not perfectly). We will choose as our parameters, those that result in the greatest likelihood computed. The estimates are called maximum likelihood because the parameters are chosen to maximize the likelihood (conditional probability of the data given parameter estimates) of the sample data. The techniques actually employed to find the maximum likelihood estimates fall under the general label numerical analysis. There are several methods of numerical analysis, but they all follow a similar series of steps. First, the computer picks some initial estimates of the parameters. Then it will compute the likelihood of the data given these parameter estimates. Then it will improve the parameter estimates slightly and recalculate the likelihood of the data. It will do this forever until we tell it to stop, which we usually do when the parameter estimates do not change much (usually a change .01 or .001 is small enough to tell the computer to stop). [Sometimes we tell the computer to stop after a certain number of tries or iterations, e.g., 20 or 250. This usually indicates a problem in estimation.]

Where on Earth Did This Stuff Come From?

Suppose we only know a person's height and we want to predict whether that person is male or female. We can talk about the probability of being male or female, or we can talk about the odds of being male or female. Let's say that the probability of being male at a given height is .90. Then the odds of being male would be

(Odds can also be found by counting the number of people in each group and dividing one number by the other. Clearly, the probability is not the same as the odds.) In our example, the odds would be .90/.10 or 9 to one. Now the odds of being female would be .10/.90 or 1/9 or .11. This asymmetry is unappealing, because the odds of being a male should be the opposite of the odds of being a female. We can take care of this asymmetry though the natural logarithm, ln. The natural log of 9 is 2.217 (ln(.9/.1)=2.217). The natural log of 1/9 is -2.217 (ln(.1/.9)=-2.217), so the log odds of being male is exactly opposite to the log odds of being female. The natural log function looks like this:

Note that the natural log is zero when X is 1. When X is larger than one, the log curves up slowly. When X is less than one, the natural log is less than zero, and decreases rapidly as X approaches zero. When P = .50, the odds are .50/.50 or 1, and ln(1) =0. If P is greater than .50, ln(P/(1-P) is positive; if P is less than .50, ln(odds) is negative. [A number taken to a negative power is one divided by that number, e.g. e^-10 = 1/e^10.A logarithm is an exponent from a given base, for example ln(e¹⁰) = 10.]

Back to logistic regression.

In logistic regression, the dependent variable is a logit, which is the natural log of the odds, that is,

So a logit is a log of odds and odds are a function of P, the probability of a 1. In logistic regression, we find

logit(P) = a + bX,

Which is assumed to be linear, that is, the log odds (logit) is assumed to be linearly related to X, our IV. So there's an ordinary regression hidden in there. We could in theory do ordinary regression with logits as our DV, but of course, we don't have logits in there, we have 1s and 0s. Then, too, people have a hard time understanding logits. We could talk about odds instead. Of course, people like to talk about probabilities more than odds. To get there (from logits to probabilities), we first have to take the log out of both sides of the equation. Then we have to convert odds to a simple probability:

The simple probability is this ugly equation that you saw earlier. If log odds are linearly related to X, then the relation between X and P is nonlinear, and has the form of the S-shaped curve you saw in the graph and the function form (equation) shown immediately above.

An Example

Suppose that we are working with some doctors on heart attack patients. The dependent variable is whether the patient has had a second heart attack within 1 year (yes = 1). We have two independent variables, one is whether the patient completed a treatment consistent of anger control practices (yes=1). The other IV is a score on a trait anxiety scale (a higher score means more anxious).

Our data:

Person	2^nd Heart Attack	Treatment of Anger	Trait Anxiety
1	1	1	70
2	1	1	80
3	1	1	50
4	1	0	60
5	1	0	40
6	1	0	65
7	1	0	75
8	1	0	80
9	1	0	70
10	1	0	60
11	0	1	65
12	0	1	50
13	0	1	45
14	0	1	35
15	0	1	40
16	0	1	50
17	0	0	55
18	0	0	45
19	0	0	50
20	0	0	60

Our correlation matrix:

	Heart	Treat	Anx
Heart	1
Treat	-.30	1
Anx	.59**	-.23	1
Mean	.50	.45	57.25
SD	.51	.51	13.42

Note that half of our patients have had a second heart attack. Knowing nothing else about a patient, and following the best in current medical practice, we would flip a coin to predict whether they will have a second attack within 1 year. According to our correlation coefficients, those in the anger treatment group are less likely to have another attack, but the result is not significant. Greater anxiety is associated with a higher probability of another attack, and the result is significant (according to r).

Now let's look at the logistic regression, for the moment examining the treatment of anger by itself, ignoring the anxiety test scores. SAS prints this:

Response Variable: HEART

Response Levels: 2

Number of Observations: 20

Link Function: Logit

Response Profile

Ordered

Value HEART Count

1 0 10

2 1 10

SAS tells us what it understands us to model, including the name of the DV, and its distribution.

Then we calculate probabilities with and without including the treatment variable.

Model Fitting Information and Testing Global Null Hypothesis BETA=0

Criterion Intercept Intercept Chi-sq

Only and

Covariates

-2 LOG L 27.726 25.878 1.848

1df (p=.17)

The computer calculates the likelihood of the data. Because there are equal numbers of people in the two groups, the probability of group membership initially (without considering anger treatment) is .50 for each person. Because the people are independent, the probability of the entire set of people is .50²⁰, a very small number. Because the number is so small, it is customary to first take the natural log of the probability and then multiply the result by -2. The latter step makes the result positive. The statistic -2LogL (minus 2 times the log of the likelihood) is a badness-of-fit indicator, that is, large numbers mean poor fit of the model to the data. SAS prints the result as -2 LOG L. For the initial model (intercept only), our result is the value 27.726. This is a baseline number indicating model fit. This number has no direct analog in linear regression. It is roughly analogous to generating some random numbers and finding R² for these numbers as a baseline measure of fit in ordinary linear regression. By including a term for treatment, the loss function reduces to 25.878, a difference of 1.848, shown in the chi-square column. The difference between the two values of -2LogL is known as the likelihood ratio test.

When taken from large samples, the difference between two values of -2LogL is distributed as chi-square:

Recall that multiplying numbers is equivalent to adding exponents (same for subtraction and division of logs).

This says that the (-2Log L) for a restricted (smaller) model - (-2LogL) for a full (larger) model is the same as the log of the ratio of two likelihoods, which is distributed as chi-square. The full or larger model has all the parameters of interest in it. The restricted is said to be nested in the larger model. The restricted model has one or more of parameters in the full model restricted to some value (usually zero). The parameters in the nested model must be a proper subset of the parameters in the full model. For example, suppose we have two IVs, one categorical and once continuous, and we are looking at an ATI design. A full model could have included terms for the continuous variable, the categorical variable and their interaction (3 terms). Restricted models could delete the interaction or one or more main effects (e.g., we could have a model with only the categorical variable). A nested model cannot have as a single IV, some other categorical or continuous variable not contained in the full model. If it does, then it is no longer nested, and we cannot compare the two values of -2LogL to get a chi-square value. The chi-square is used to statistically test whether including a variable reduces badness-of-fit measure. This is analogous to producing an increment in R-square in hierarchical regression. If chi-square is significant, the variable is considered to be a significant predictor in the equation, analogous to the significance of the b weight in simultaneous regression.

For our example with anger treatment only, SAS produces the following:

Analysis of Maximum Likelihood Estimates
Variable	DF	Par Est	Std Err	Wald Chisq	Pr > Chi- sq	Stand. Est	Odds Ratio
Intercept	1	-.5596	.6268	.7972	.3719	.	.
Treatment	1	1.2528	.9449	17566	.1849	.3525	3.50

The intercept is the value of a, in this case -.5596. As usual, we are not terribly interested in whether a is equal to zero. The value of b given for Anger Treatment is 1.2528. the chi-square associated with this b is not significant, just as the chi-square for covariates was not significant. Therefore we cannot reject the hypothesis that b is zero in the population. Our equation can be written either:

Logit(P) = -.5596+1.2528X

The main interpretation of logistic regression results is to find the significant predictors of Y. However, other things can sometimes be done with the results.

The Odds Ratio

Recall that the odds for a group is :

Now the odds for another group would also be P/(1-P) for that group. Suppose we arrange our data in the following way:

	Anger Treatment
Heart Attack	Yes (1)	No (0)	Total
Yes (1)	3 (a)	7 (b)	10 (a+b)
No (0)	6 (c)	4 (d)	10 (c+d)
Total	9 (a+c)	11 (b+d)	20 (a+b+c+d)

Now we can compute the odds of having a heart attack for the treatment group and the no treatment group. For the treatment group, the odds are 3/6 = 1/2. The probability of a heart attack is 3/(3+6) = 3/9 = .33. The odds from this probability are .33/(1-.33) = .33/.66 = 1/2. The odds for the no treatment group are 7/4 or 1.75. The odds ratio is calculated to compare the odds across groups.

If the odds are the same across groups, the odds ratio (OR) will be 1.0. If not, the OR will be larger or smaller than one. People like to see the ratio be phrased in the larger direction. In our case, this would be 1.75/.5 or 1.75*2 = 3.50.

Now if we go back up to the last column of the printout where is says odds ratio in the treatment column, you will see that the odds ratio is 3.50, which is what we got by finding the odds ratio for the odds from the two treatment conditions. It also happens that e^1.2528= 3.50. Note that the exponent is our value of b for the logistic curve.

K-fold Cross Validation

2007-10-04T12:26:00.000+01:00

Cross validation is a method for estimating the true error of a model. When a model is built from training data, the error on the training data is a rather optimistic estimate of the error rates the model will achieve on unseen data. The aim of building a model is usually to apply the model to new, unseen data--we expect the model to generalise to data other than the training data on which it was built. Thus, we would like to have some method for better approximating the error that might occur in general. Cross validation provides such a method.

Cross validation is also used to evaluate a model in deciding which algorithm to deploy for learning, when choosing from amongst a number of learning algorithms. It can also provide a guide as to the effect of parameter tuning in building a model from a specific algorithm.

Test sample cross-validation is often a preferred method when there is plenty of data available. A model is built from a training set and its predictive accuracy is measured by applying the model a test set. A good rule of thumb is that a dataset is partitioned into a training set (66%) and a test set (33%).

To measure error rates you might build multiple models with the one algorithm, using variations of the same training data for each model. The average performance is then the measure of how well this algorithm works in building models from the data.

The basic idea is to use, say, 90% of the dataset to build a model. The data that was removed (the 10%) is then used to test the performance of the model on ``new'' data (usually by calculating the mean squared error). This simplest of cross validation approaches is referred to as the holdout method.

For the holdout method the two datasets are referred to as the training set and the test set. With just a single evaluation though there can be a high variance since the evaluation is dependent on the data points which happen to end up in the training set and the test set. Different partitions might lead to different results.

A solution to this problem is to have multiple subsets, and each time build the model based on all but one of these subsets. This is repeated for all possible combinations and the result is reported as the average error over all models.

In k-fold cross validation, sometimes called rotation estimation, the dataset D is randomly split into k mutually exclusive subsets (the folds) D1 ; D2 ; ...; Dk of approximately equal size. The inducer is trained and tested k times; each time t from {1; 2;... ; k}, it is trained on D - Dt and tested on Dt . The crossvalidation estimate of accuracy is the overall number of correct classifica
tions, divided by the number of instances in the dataset. Formally, let D (i) be the test set that includes instance x i = (vi ; yi), then the crossvalidation estimate of accuracy

The crossvalidation estimate is a random number that depends on the division into folds. Complete cross validation is the average of all

possibilities for choosing m-k instances out of m, but it is usually too expensive. Except for leaveoneone (nfold crossvalidation), which is always complete, kfold cross validation is estimating complete kfold crossvalidation using a single split of the data into the folds. Repeating crossvalidation multiple times using different splits into folds provides a better MonteCarlo estimate to the complete crossvalidation at an added cost. In stratified crossvalidation, the folds are stratified so that they contain approximately the same proportions of labels as the original dataset.

Linear regression

2007-09-03T13:33:00.000+01:00

In statistics, linear regression is a regression method that models the relationship between a dependent variable Y, independent variables X_i, i = 1, ..., p, and a random term ε. The model can be written as

Example of linear regression with one dependent and one independent variable.

where $β 1$ is the intercept ("constant" term), the $β i$ s are the respective parameters of independent variables, and $p$ is the number of parameters to be estimated in the linear regression. Linear regression can be contrasted with nonlinear regression.

This method is called "linear" because the relation of the response (the dependent variable $Y$ ) to the independent variables is assumed to be a linear function of the parameters. It is often erroneously thought that the reason the technique is called "linear regression" is that the graph of $Y = β 0 + β x$ is a straight line or that $Y$ is a linear function of the $X$ variables. But if the model is (for example)

the problem is still one of linear regression, that is, linear in $x$ and $x 2$ respectively, even though the graph on $x$ by itself is not a straight

Sample data.

Say we have a set of data, , shown at the left. If we have reason to believe that there exists a linear relationship between the variables x and y, we can plot the data and draw a "best-fit" straight line through the data. Of course, this relationship is governed by the familiar equation . We can then find the slope, m, and y-intercept, b, for the data, which are shown in the figure below.

Let's enter the above data into an Excel spread sheet, plot the data, create a trendline and display its slope, y-intercept and R-squared value. Recall that the R-squared value is the square of the correlation coefficient. (Most statistical texts show the correlation coefficient as "r", but Excel shows the coefficient as "R". Whether you write is as r or R, the correlation coefficient gives us a measure of the reliability of the linear relationship between the x and y values. (Values close to 1 indicate excellent linear reliability.))

Enter your data as we did in columns B and C. The reason for this is strictly cosmetic as you will soon see.

Linear regression equations.

If we expect a set of data to have a linear correlation, it is not necessary for us to plot the data in order to determine the constants m (slope) and b (y-intercept) of the equation . Instead, we can apply a statistical treatment known as linear regression to the data and determine these constants.

Given a set of data with n data points, the slope, y-intercept and correlation coefficient, r, can be determined using the following:

(Note that the limits of the summation, which are i to n, and the summation indices on x and y have been omitted.)

Implicitly applying regression to the sample data.

It may appear that the above equations are quite complicated, however upon inspection, we see that their components are nothing more than simple algebraic manipulations of the raw data. We can expand our spread sheet to include these components.

First, add three columns that will be used to determine the quantities xy, x² and y², for each data point.

Next, use Excel to evaluate the following: Sx, Sy, S(xy), S(x²), S(y²), (Sx)², (Sy)². Recall that the symbol, S, means "summation". Additionally, the term xy is the product of x and y, that is: x * y. Also, the term S(x²) is very different than the term (Sx)². Be careful with your order of operations!

Now use Excel to count the number of data points, n. (To do this, use the Excel COUNT() function. The syntax for COUNT() in this example is: =COUNT(B3:B8) and is shown in the formula bar in the screen shot below.

Finally, use the above components and the linear regression equations given in the previous section to calculate the slope (m), y-intercept (b) and correlation coefficient (r) of the data. If you are careful, your spread sheet should look like ours. Note that our equations for the slope, y-intercept and correlation coefficient are highlighted in yellow.

2007-09-03T13:32:00.000+01:00

Linear function

In mathematics, the term linear function can refer to either of two different but related concepts.

Usage in elementary mathematics

In elementary algebra and analytic geometry, the term linear function is often used to mean a first degree polynomial function of one variable. These functions are called "linear" because they are precisely the functions whose graph in the Cartesian coordinate plane is a straight line.

Such a function can be written as

f (x) = m x + b

where $m$ and $b$ are real constants and $x$ is a real variable. The constant $m$ is often called the slope while $b$ is the y-intercept, which gives the point of intersection between the graph of the function and the $y$ -axis. Changing $m$ makes the line steeper or shallower, while changing $b$ moves the line up or down.

Three geometric linear functions — the red and blue ones have same slope (m), while red and green ones have same y-intercept (b).

Examples of functions whose graph is a line include the following:

$f 1 (x) = 2 x + 1$
$f 2 (x) = x / 2 + 1$
$f 3 (x) = x / 2 - 1$

The graphs of these are shown in the image at right.

Usage in advanced mathematics

In advanced mathematics, a linear function often means a function that is a linear map, that is, a map between two vector spaces that preserves vector addition and scalar multiplication.

A function $f (x) = m x + b$ is a linear map if and only if $b = 0$ . For other values of $b$ this falls in the more general class of affine maps.

Spearman Correlation - Example

2007-09-03T13:07:00.000+01:00

Example of Statistical Test using the Spearman Rank-Difference Correlation Coefficient

Is there a significant positive correlation between the rankings of 10 children on a reading test and their teacher's ranking of their reading ability? In this problem we are relating a set of scores (interval level of measurement) with the teacher's ranking of the children in reading (ordinal level of measurement). To do this we first convert the reading test scores to ranks by assigning the highest score a rank of 1, the next highest a rank of 2, etc. Now we are looking at rankings on two variables and can use the Spearman Rank-Difference Correlation Coefficient to test the significance of the relationship. The two set of ranks, as well as the difference between the pairs of ranks (D) and the differences squared (D²), are shown in the following table.

**Worksheet to Calculate the Correlation between Students Ranks on a Reading Test and Teacher's Ranking of the Students on Reading**
Reading Test Score Rank	Teacher's Ranking on Reading	D	D²
1	3	-2	4
2	2	0	0
3	1	2	4
4	4	0	0
5	5	0	0
6	6	0	0
7	8	-1	1
8	7	1	1
9	10	-1	1
10	9	1	1
Total			12

From the table we can see that:

df = N - 2 = 10 - 2 = 8

We now have the information we need to complete the six step process for testing statistical hypotheses for our research problem.

State the null hypothesis and the alternative hypothesis based on your research question.
H₀: r_S = 0

H₁: r_S > 0

Note: Our null hypothesis states that there is no significant relationship between the two variables. The alternative hypothesis states that there is a significant positive correlation between the two variables.
Set the alpha level.

Note: As usual we will set our alpha level at .05, we have 5 chances in 100 of making a type I error.
Calculate the value of the appropriate statistic. Also indicate the degrees of freedom for the statistical test if necessary.
r_S = .93

df = N - 2 = 10 - 2 = 8
Write the decision rule for rejecting the null hypothesis.
Reject H₀ if r_S >= .549

Note: To write the decision rule we had to know the critical value for r_S, with an alpha level of .05, and 8 degrees of freedom. We can do this by looking at Appendix Table B, this is the same table we used for the Pearson r, and noting the tabled value for the column for the .10 level and the row for 8 df (.549).

Note: We used the .10 column because we are doing a one-tailed test with an alpha of .05 As noted in our problem above the the Pearson r, in the table of critical values for r, the .10 column is used for alpha = .10 (two-tailed test) and for alpha = .05 (one-tailed test).
Write a summary statement based on the decision.
Reject H₀, p < .05, one-tailed
Note: Since our calculated value of r_S (.93) is greater than .549, we reject the null hypothesis and accept the alternative hypothesis.
Write a statement of results in standard English.
There is a significant positive correlation between the children's ranks on a reading test and their teacher's ranking of them on reading.

Choosing the Proper Statistical Test

2007-09-03T13:04:00.000+01:00

Let's finish our discussion of inferential statistics with a summary of all the inferential statistics we have discussed and look at the conditions under which we would use each of these statistics. Generally if we know the number of groups or samples in our research design and the level of measurement of the dependent variable we will know which inferential statistic to use.

First let us look at statistical hypotheses in research designs where the dependent variable is at the interval or ratio level. These statistics are known as parametric statistics and we have used the following:

If we are testing a statistical hypothesis, involving a single score (we are comparing the score with the population mean) we will use the z-score test (see lesson 9).
If we are testing a statistical hypothesis involving a single group (we are comparing the mean of the group with the population mean) and the standard deviation of the population is know use the z test (see lesson 10).
If we are testing a statistical hypothesis involving a single group (we are comparing the mean of the group with the population mean) and the standard deviation of the population is not known use the single sample t-test (see lesson 10).
If we are testing a statistical hypothesis involved two groups of subjects (we are comparing the means of the two groups) and the two groups are independent of one another, we use the independent t-test (see lesson 11).
If we are testing a statistical hypothesis involved two groups of subjects (we are comparing the means of the two groups) and the two groups are dependent on one another (pretest/posttest or matched samples), we use the dependent t-test (see lesson 12).
If we are testing a statistical hypothesis involved three or more groups of subjects (we are comparing the means of three or more groups) and there is a single dependent variable in the study, we use one-way analysis of variance (see lesson 13).
If we are testing a statistical hypothesis involved the relationship between two variables for one sample (we are measuring the relationship between the two variables) and the data is at the interval or ratio level of measurement), use the Pearson product moment correlation coefficient.

We also looked at two other statistics we could use with data that was not at the interval or ratio level of measurement. These statistics are called non-parametric statistics.

If we are testing a statistical hypothesis for one, two, or more groups with one or two variables where the data is catagorical (frequencies). The data is at the nominal level of measurement. For this type of study use chi-square (see lesson 14). We have discussed three different variants of the chi-square statistic.
1. one variable chi-square with equal expected frequencies
2. one variable chi-square with unequal (predetermined) expected frequencies
3. two variable chi-square
If we are testing a statistical hypothesis involved the relationship between two variables for one sample (we are measuring the relationship between the two variables) and the data is at the ordinal level of measurement (ranks), use the Spearman rank-difference correlation coefficient (see lesson 15).

The information we have discussed above can be put into the following table. The table also includes other statistics that we have not included in this course. If you think you may need one of the statistics we did not cover in your research design, please send e-mail to the instructor and I will give you a reference to the calculation and interpretation of that statistic. I wish you the best as you complete the final examination for this course and as you apply the information from this course to your own research design.

**Selecting a Statistical Test**
Level of Measurement	Sample Characteristics
	One-Sample Statistical Tests	Two-Sample Statistical Tests		Multiple Sample Statistical Tests	Measures of Association (one-sample, more than one variable)
	One-Sample Statistical Tests	Independent Samples	Non-independent Samples	Multiple Sample Statistical Tests
Nominal or Categorical (frequencies)	Chi-Square	Chi-Square	McNemar Change Test	Chi-Square	Phi Coefficient
Ordinal (Ranks)	Kolmagorov-Smirnov One-Sample Test	Mann Whitney U-Test	Wilcoxon Matched Pairs Signed-Rank Test	Krushcal-Wallis One-Way Analysis of Variance	Spearman rho r_S
Interval or Ratio	Z test One-Sample t-Test	Independent t-test	Dependent t-test	Simple Analysis of Variance Factorial Analysis of Variance Scheffe Tests Analysis of Covariance	Pearson r Multiple Regression

Spearman Correlation

2007-09-01T19:00:00.000+01:00

Spearman's rank correlation coefficient, named after Charles Spearman and often denoted by the Greek letter ρ (rho), is a non-parametric measure of correlation – that is, it assesses how well an arbitrary monotonic function could describe the relationship between two variables, without making any assumptions about the frequency distribution of the variables. Unlike the Pearson product-moment correlation coefficient, it does not require the assumption that the relationship between the variables is linear, nor does it require the variables to be measured on interval scales; it can be used for variables measured at the ordinal level.

In principle, ρ is simply a special case of the Pearson product-moment coefficient in which the data are converted to rankings before calculating the coefficient. In practice, however, a simpler procedure is normally used to calculate ρ. The raw scores are converted to ranks, and the differences d between the ranks of each observation on the two variables are calculated.

If there are no tied ranks, i.e.

then ρ is given by:

where:

$d i$ = the difference between each rank of corresponding values of x and y, and

$n$ = the number of pairs of values.

If tied ranks exist, classic Pearson's correlation coefficient between ranks has to be used instead of this formula. You have to assign the same rank to each of the equal values. It is an average of their positions in the ascending order of the values:

An Example of Averaging Ranks

Variable	Position in the decending order	Rank
0.8	5	5
1.2	4
1.2	3
2.3	2	2
18	1	1

Spearman's rank correlation coefficient is equivalent to Pearson correlation on ranks. The formula above is a short-cut to its product-moment form, assuming no tie. The product-moment form can be used in both tied and untied cases.

A version of this correlation is called Spearman's rho. In this case ranks are calculated as above, but in the formula of Pearson's correlation a standard deviation is taken as there were no ties.

Another popular method for computing rank correlation is the Kendall tau rank correlation coefficient.

Example

The raw data used in this example is shown below.

IQ	Hours of TV per week.
106	7
86	0
100	27
101	50
99	28
103	29
97	20
113	12
112	6
110	17

The first step is to sort this data by the first column. Next, two more columns are created. Both of these are for ranking the first two columns. Notice how the rank of values that are the same is the mean of what their ranks would otherwise be. Then a column "d" is created to hold the differences between the two rank columns. Finally another column "d²" should be created. This is just column d squared.

After doing this process with the example data you should end up with something like:

IQ (i)	Hours of TV per week (t)	rank (i)	rank (t)	d	d²
86	0	1	1	0	0
97	20	2	6	4	16
99	28	3	8	5	25
100	27	4	7	3	9
101	50	5	10	5	25
103	29	6	9	3	9
106	7	7	3	4	16
110	17	8	5	3	9
112	6	9	2	7	49
113	12	10	4	6	36

The values in the d² column can now be added to find . The value of n is 10. So these values can now be substituted back into the equation,

which evaluates to $ρ = - 0.175758$ . In the case of ties in the original values, then this formula should not be used. Instead, the Pearson correlation coefficient should be calculated on the ranks (where ties are given ranks, as described above).

Statistics: Another Syllabus

2007-09-01T18:58:00.000+01:00

Statistics
Descriptive statistics	Mean (Arithmetic, Geometric) - Median - Mode - Power - Variance - Standard deviation
Inferential statistics	Hypothesis testing - Significance - Null hypothesis/Alternate hypothesis - Error - Z-test - Student's t-test - Maximum likelihood - Standard score/Z score - P-value - Analysis of variance
Survival analysis	Survival function - Kaplan-Meier - Logrank test - Failure rate - Proportional hazards models
Probability distributions	Normal (bell curve) - Poisson - Bernoulli
Correlation	Pearson product-moment correlation coefficient - Rank correlation (Spearman's rank correlation coefficient, Kendall tau rank correlation coefficient)
Regression analysis	Linear regression - Nonlinear regression - Logistic regression

STATISTICAL METHODS FOR RESEARCH WORKERS

2007-09-01T18:00:00.000+01:00

Referenced from Christopher D. Green's Website: http://psychclassics.yorku.ca/Fisher/Methods/

STATISTICAL METHODS FOR RESEARCH WORKERS

By Ronald A. Fisher (1925)
Originally published in Edinburgh by Oliver and Boyd.

Posted March 2000
Modified Sept. 2005

TITLE PAGE

PREFACES

Image of original TABLE OF CONTENTS

I. INTRODUCTION

II. DIAGRAMS

III. DISTRIBUTIONS

Image of TECHNICAL APPENDIX: p. 74, p. 75

IV. TESTS OF GOODNESS OF FIT, INDEPENDENCE AND HOMOGENEITY; WITH TABLE OF χ²

V. TESTS OF SIGNIFICANCE OF MEANS, DIFFERENCE OF MEANS, AND REGRESSION COEFFICIENTS

VI. THE CORRELATION COEFFICIENT

VII. INTRACLASS CORRELATIONS AND THE ANALYSIS OF VARIANCE

VIII. FURTHER APPLICATIONS OF THE ANALYSIS OF VARIANCE

SOURCES USED FOR DATA AND METHODS INDEX

INDEX

TABLES

I. and II. NORMAL DISTRIBUTION

III. TABLE OF c²

IV. TABLE OF t

V.A. CORRELATION COEFFICIENT -- SIGNIFICANT VALUES

V.B. CORRELATION COEFFICIENT -- TRANSFORMED VALUES

VI. TABLE OF z

Correlation ratio

2007-09-01T17:11:00.000+01:00

In statistics, the correlation ratio is a measure of the relationship between the statistical dispersion within individual categories and the dispersion across the whole population or sample.

Suppose each observation is y_xi where x indicates the category that observation is in and i is the label of the particular observation. We will write n_x for the number of observations in category x (not necessarily the same for different values of x) and

and

then the correlation ratio η (eta) is defined so as to satisfy

which might be written as

It is worth noting that if the relationship between values of and values of is linear (which is certainly true when there are only two possibilities for x) this will give the same result as the square of the correlation coefficient; if not then the correlation ratio will be larger in magnitude, though still no more than 1 in magnitude. It can therefore be used for judging non-linear relationships.

Retrieved from "http://en.wikipedia.org/wiki/Correlation_ratio"

Correlation, Pearson Correlation

2007-09-01T17:07:00.000+01:00

In probability theory and statistics, correlation, also called correlation coefficient, indicates the strength and direction of a linear relationship between two random variables. In general statistical usage, correlation or co-relation refers to the departure of two variables from independence. In this broad sense there are several coefficients, measuring the degree of correlation, adapted to the nature of data.

A number of different coefficients are used for different situations. The best known is the Pearson product-moment correlation coefficient, which is obtained by dividing the covariance of the two variables by the product of their standard deviations. Despite its name, it was first introduced by Francis Galton.

Pearson's product-moment coefficient

Main article: Pearson product-moment correlation coefficient

Mathematical properties

The correlation coefficient ρ_{X, Y} between two random variables X and Y with expected values μ_X and μ_Y and standard deviations σ_X and σ_Y is defined as:

where E is the expected value operator and cov means covariance. Since μ_X = E(X), σ_X² = E(X²) − E²(X) and likewise for Y, we may also write

The correlation is defined only if both of the standard deviations are finite and both of them are nonzero. It is a corollary of the Cauchy-Schwarz inequality that the correlation cannot exceed 1 in absolute value.

The correlation is 1 in the case of an increasing linear relationship, −1 in the case of a decreasing linear relationship, and some value in between in all other cases, indicating the degree of linear dependence between the variables. The closer the coefficient is to either −1 or 1, the stronger the correlation between the variables.

If the variables are independent then the correlation is 0, but the converse is not true because the correlation coefficient detects only linear dependencies between two variables. Here is an example: Suppose the random variable X is uniformly distributed on the interval from −1 to 1, and Y = X². Then Y is completely determined by X, so that X and Y are dependent, but their correlation is zero; they are uncorrelated. However, in the special case when X and Y are jointly normal, independence is equivalent to uncorrelatedness.

A correlation between two variables is diluted in the presence of measurement error around estimates of one or both variables, in which case disattenuation provides a more accurate coefficient .

Positive linear correlations between 1000 pairs of numbers. The data are graphed on the lower left and their correlation coefficients listed on the upper right. Each square in the upper right corresponds to its mirror-image square in the lower left, the "mirror" being the diagonal of the whole array. Each set of points correlates maximally with itself, as shown on the diagonal (all correlations = +1).

The sample correlation

If we have a series of n measurements of X and Y written as x_i and y_i where i = 1, 2, ..., n, then the Pearson product-moment correlation coefficient can be used to estimate the correlation of X and Y . The Pearson coefficient is also known as the "sample correlation coefficient". It is especially important if X and Y are both normally distributed. The Pearson correlation coefficient is then the best estimate of the correlation of X and Y . The Pearson correlation coefficient is written:

where and are the sample means of X and Y , s_x and s_y are the sample standard deviations of X and Y and the sum is from i = 1 to n. As with the population correlation, we may rewrite this as

Again, as is true with the population correlation, the absolute value of the sample correlation must be less than or equal to 1. Though the above formula conveniently suggests a single-pass algorithm for calculating sample correlations, it is notorious for its numerical instability (see below for something more accurate).

The square of the sample correlation coefficient, which is also known as the coefficient of determination, is the fraction of the variance in y_i that is accounted for by a linear fit of x_i to y_i . This is written

where s_y|x² is the square of the error of a linear regression of x_i on y_i by the equation y = a + bx:

and s_y² is just the variance of y:

Note that since the sample correlation coefficient is symmetric in x_i and y_i , we will get the same value for a fit of x_i to y_i :

This equation also gives an intuitive idea of the correlation coefficient for higher dimensions. Just as the above described sample correlation coefficient is the fraction of variance accounted for by the fit of a 1-dimensional linear submanifold to a set of 2-dimensional vectors (x_i , y_i ), so we can define a correlation coefficient for a fit of an m-dimensional linear submanifold to a set of n-dimensional vectors. For example, if we fit a plane z = a + bx + cy to a set of data (x_i , y_i , z_i ) then the correlation coefficient of z to x and y is

Geometric Interpretation of correlation

The correlation coefficient can also be viewed as the cosine of the angle between the two vectors of samples drawn from the two random variables.

Caution: This method only works with centered data, i.e., data which have been shifted by the sample mean so as to have an average of zero. Some practitioners prefer an uncentered (non-Pearson-compliant) correlation coefficient. See the example below for a comparison.

As an example, suppose five countries are found to have gross national products of 1, 2, 3, 5, and 8 billion dollars, respectively. Suppose these same five countries (in the same order) are found to have 11%, 12%, 13%, 15%, and 18% poverty. Then let x and y be ordered 5-element vectors containing the above data: x = (1, 2, 3, 5, 8) and y = (0.11, 0.12, 0.13, 0.15, 0.18).

By the usual procedure for finding the angle between two vectors (see dot product), the uncentered correlation coefficient is:

Note that the above data were deliberately chosen to be perfectly correlated: y = 0.10 + 0.01 x. The Pearson correlation coefficient must therefore be exactly one. Centering the data (shifting x by E(x) = 3.8 and y by E(y) = 0.138) yields x = (-2.8, -1.8, -0.8, 1.2, 4.2) and y = (-0.028, -0.018, -0.008, 0.012, 0.042), from which

as expected.

Interpretation of the size of a correlation

Correlation	Negative	Positive
Small	−0.29 to −0.10	0.10 to 0.29
Medium	−0.49 to −0.30	0.30 to 0.49
Large	−1.00 to −0.50	0.50 to 1.00

Several authors have offered guidelines for the interpretation of a correlation coefficient. Cohen (1988),^[1] for example, has suggested the following interpretations for correlations in psychological research, in the table on the right.

As Cohen himself has observed, however, all such criteria are in some ways arbitrary and should not be observed too strictly. This is because the interpretation of a correlation coefficient depends on the context and purposes. A correlation of 0.9 may be very low if one is verifying a physical law using high-quality instruments, but may be regarded as very high in the social sciences where there may be a greater contribution from complicating factors.

Non-parametric correlation coefficients

Pearson's correlation coefficient is a parametric statistic, and it may be less useful if the underlying assumption of normality is violated. Non-parametric correlation methods, such as Chi-square, Point biserial correlation, Spearman's ρ and Kendall's τ may be useful when distributions are not normal; they are a little less powerful than parametric methods if the assumptions underlying the latter are met, but are less likely to give distorted results when the assumptions fail.

Other measures of dependence among random variables

To get a measure for more general dependencies in the data (also nonlinear) it is better to use the correlation ratio which is able to detect almost any functional dependency, or mutual information/total correlation which is capable of detecting even more general dependencies.

Copulas and correlation

The information given by a correlation coefficient is not enough to define the dependence structure between random variables; to fully capture it we must consider a copula between them. The correlation coefficient completely defines the dependence structure only in very particular cases, for example when the cumulative distribution functions are the multivariate normal distributions. In the case of elliptic distributions it characterizes the (hyper-)ellipses of equal density, however, it does not completely characterize the dependence structure (for example, the a multivariate t-distribution's degrees of freedom determine the level of tail dependence).

Correlation matrices

The correlation matrix of n random variables X₁, ..., X_n is the n × n matrix whose i,j entry is corr(X_i, X_j). If the measures of correlation used are product-moment coefficients, the correlation matrix is the same as the covariance matrix of the standardized random variables X_i /SD(X_i) for i = 1, ..., n. Consequently it is necessarily a positive-semidefinite matrix.

The correlation matrix is symmetric because the correlation between $X i$ and $X j$ is the same as the correlation between $X j$ and $X i$ .

Removing correlation

It is always possible to remove the correlation between zero-mean random variables with a linear transform, even if the relationship between the variables is nonlinear. Suppose a vector of n random variables is sampled m times. Let X be a matrix where $X i, j$ is the jth variable of sample i. Let $Z r, c$ be an r by c matrix with every element 1. Then D is the data transformed so every random variable has zero mean, and T is the data transformed so all variables have zero mean, unit variance, and zero correlation with all other variables.

where an exponent of -1/2 represents the matrix square root of the inverse of a matrix. The covariance matrix of T will be the identity matrix. If a new data sample x is a row vector of n elements, then the same transform can be applied to x to get the transformed vectors d and t:

Common misconceptions about correlation

Correlation and causality

The conventional dictum that "correlation does not imply causation" means that correlation cannot be validly used to infer a causal relationship between the variables. This dictum should not be taken to mean that correlations cannot indicate causal relations. However, the causes underlying the correlation, if any, may be indirect and unknown. Consequently, establishing a correlation between two variables is a not sufficient condition to establish a causal relationship (in either direction).

Here is a simple example: hot weather may cause both crime and ice-cream purchases. Therefore crime is correlated with ice-cream purchases. But crime does not cause ice-cream purchases and ice-cream purchases do not cause crime.

A correlation between age and height in children is fairly causally transparent, but a correlation between mood and health in people is less so. Does improved mood lead to improved health? Or does good health lead to good mood? Or does some other factor underlie both? Or is it pure coincidence? In other words, a correlation can be taken as evidence for a possible causal relationship, but cannot indicate what the causal relationship, if any, might be.

Correlation and linearity

Four sets of data with the same correlation of 0.81

While Pearson correlation indicates the strength of a linear relationship between two variables, its value alone may not be sufficient to evaluate this relationship, especially in the case where the assumption of normality is incorrect.

The image on the right shows scatterplots of Anscombe's quartet, a set of four different pairs of variables created by Francis Anscombe.^[2] The four $y$ variables have the same mean (7.5), standard deviation (4.12), correlation (0.81) and regression line ( $y = 3 + 0.5 x$ ). However, as can be seen on the plots, the distribution of the variables is very different. The first one (top left) seems to be distributed normally, and corresponds to what one would expect when considering two variables correlated and following the assumption of normality. The second one (top right) is not distributed normally; while an obvious relationship between the two variables can be observed, it is not linear, and the Pearson correlation coefficient is not relevant. In the third case (bottom left), the linear relationship is perfect, except for one outlier which exerts enough influence to lower the correlation coefficient from 1 to 0.81. Finally, the fourth example (bottom right) shows another example when one outlier is enough to produce a high correlation coefficient, even though the relationship between the two variables is not linear.

These examples indicate that the correlation coefficient, as a summary statistic, cannot replace the individual examination of the data.

Computing correlation accurately in a single pass

The following algorithm (in pseudocode) will estimate correlation with good numerical stability

sum_sq_x = 0
sum_sq_y = 0
sum_coproduct = 0
mean_x = x[1]
mean_y = y[1]
for i in 2 to N:
  sweep = (i - 1.0) / i
  delta_x = x[i] - mean_x
  delta_y = y[i] - mean_y
  sum_sq_x += delta_x * delta_x * sweep
  sum_sq_y += delta_y * delta_y * sweep
  sum_coproduct += delta_x * delta_y * sweep
  mean_x += delta_x / i
  mean_y += delta_y / i
pop_sd_x = sqrt( sum_sq_x / N )
pop_sd_y = sqrt( sum_sq_y / N )
cov_x_y = sum_coproduct / N
correlation = cov_x_y / (pop_sd_x * pop_sd_y)

For an enlightening experiment, check the correlation of {900,000,000 + i for i=1...100} with {900,000,000 - i for i=1...100}, perhaps with a few values modified. Poor algorithms will fail.

Currency correlation

Currency correlation is correlation between two currency pairs, or more generally, correlations between values of commodities, stocks and bonds markets. It is used as a tool to predict changes in market value.

Overview of Statistical Test [recommended]

2007-08-30T16:07:00.000+01:00

Choosing a statistical test

This is chapter 37 of Intuitive Biostatistics (ISBN 0-19-508607-4) by Harvey Motulsky. Copyright © 1995 by Oxford University Press Inc. All rights reserved. You may order the book from GraphPad Software with a software purchase, from any academic bookstore, or from amazon.com.

Learn how to interpret the results of statistical tests and about our programs GraphPad InStat and GraphPad Prism.

REVIEW OF AVAILABLE STATISTICAL TESTS

This book has discussed many different statistical tests. To select the right test, ask yourself two questions: What kind of data have you collected? What is your goal? Then refer to Table 37.1.

All tests are described in this book and are performed by InStat, except for tests marked with asterisks. Tests labeled with a single asterisk are briefly mentioned in this book, and tests labeled with two asterisks are not mentioned at all.

Table 37.1. Selecting a statistical test

	Type of Data
Goal	Measurement (from Gaussian Population)	Rank, Score, or Measurement (from Non- Gaussian Population)	Binomial (Two Possible Outcomes)	Survival Time
Describe one group	Mean, SD	Median, interquartile range	Proportion	Kaplan Meier survival curve
Compare one group to a hypothetical value	One-sample t test	Wilcoxon test	Chi-square or Binomial test **
Compare two unpaired groups	Unpaired t test	Mann-Whitney test	Fisher's test (chi-square for large samples)	Log-rank test or Mantel-Haenszel*
Compare two paired groups	Paired t test	Wilcoxon test	McNemar's test	Conditional proportional hazards regression*
Compare three or more unmatched groups	One-way ANOVA	Kruskal-Wallis test	Chi-square test	Cox proportional hazard regression**
Compare three or more matched groups	Repeated-measures ANOVA	Friedman test	Cochrane Q**	Conditional proportional hazards regression**
Quantify association between two variables	Pearson correlation	Spearman correlation	Contingency coefficients**
Predict value from another measured variable	Simple linear regression or Nonlinear regression	Nonparametric regression**	Simple logistic regression*	Cox proportional hazard regression*
Predict value from several measured or binomial variables	Multiple linear regression* or Multiple nonlinear regression**		Multiple logistic regression*	Cox proportional hazard regression*

REVIEW OF NONPARAMETRIC TESTS

Choosing the right test to compare measurements is a bit tricky, as you must choose between two families of tests: parametric and nonparametric. Many -statistical test are based upon the assumption that the data are sampled from a Gaussian distribution. These tests are referred to as parametric tests. Commonly used parametric tests are listed in the first column of the table and include the t test and analysis of variance.

Tests that do not make assumptions about the population distribution are referred to as nonparametric- tests. You've already learned a bit about nonparametric tests in previous chapters. All commonly used nonparametric tests rank the outcome variable from low to high and then analyze the ranks. These tests are listed in the second column of the table and include the Wilcoxon, Mann-Whitney test, and Kruskal-Wallis tests. These tests are also called distribution-free tests.

CHOOSING BETWEEN PARAMETRIC AND NONPARAMETRIC TESTS: THE EASY CASES

Choosing between parametric and nonparametric tests is sometimes easy. You should definitely choose a parametric test if you are sure that your data are sampled from a population that follows a Gaussian distribution (at least approximately). You should definitely select a nonparametric test in three situations:

• The outcome is a rank or a score and the population is clearly not Gaussian. Examples include class ranking of students, the Apgar score for the health of newborn babies (measured on a scale of 0 to IO and where all scores are integers), the visual analogue score for pain (measured on a continuous scale where 0 is no pain and 10 is unbearable pain), and the star scale commonly used by movie and restaurant critics (* is OK, ***** is fantastic).
• Some values are "off the scale," that is, too high or too low to measure. Even if the population is Gaussian, it is impossible to analyze such data with a parametric test since you don't know all of the values. Using a nonparametric test with these data is simple. Assign values too low to measure an arbitrary very low value and assign values too high to measure an arbitrary very high value. Then perform a nonparametric test. Since the nonparametric test only knows about the relative ranks of the values, it won't matter that you didn't know all the values exactly.
• The data ire measurements, and you are sure that the population is not distributed in a Gaussian manner. If the data are not sampled from a Gaussian distribution, consider whether you can transformed the values to make the distribution become Gaussian. For example, you might take the logarithm or reciprocal of all values. There are often biological or chemical reasons (as well as statistical ones) for performing a particular transform.

CHOOSING BETWEEN PARAMETRIC AND NONPARAMETRIC TESTS: THE HARD CASES

It is not always easy to decide whether a sample comes from a Gaussian population. Consider these points:

• If you collect many data points (over a hundred or so), you can look at the distribution of data and it will be fairly obvious whether the distribution is approximately bell shaped. A formal statistical test (Kolmogorov-Smirnoff test, not explained in this book) can be used to test whether the distribution of the data differs significantly from a Gaussian distribution. With few data points, it is difficult to tell whether the data are Gaussian by inspection, and the formal test has little power to discriminate between Gaussian and non-Gaussian distributions.
• You should look at previous data as well. Remember, what matters is the distribution of the overall population, not the distribution of your sample. In deciding whether a population is Gaussian, look at all available data, not just data in the current experiment.
• Consider the source of scatter. When the scatter comes from the sum of numerous sources (with no one source contributing most of the scatter), you expect to find a roughly Gaussian distribution.
When in doubt, some people choose a parametric test (because they aren't sure the Gaussian assumption is violated), and others choose a nonparametric test (because they aren't sure the Gaussian assumption is met).

CHOOSING BETWEEN PARAMETRIC AND NONPARAMETRIC TESTS: DOES IT MATTER?

Does it matter whether you choose a parametric or nonparametric test? The answer depends on sample size. There are four cases to think about:

• Large sample. What happens when you use a parametric test with data from a nongaussian population? The central limit theorem (discussed in Chapter 5) ensures that parametric tests work well with large samples even if the population is non-Gaussian. In other words, parametric tests are robust to deviations from Gaussian distributions, so long as the samples are large. The snag is that it is impossible to say how large is large enough, as it depends on the nature of the particular non-Gaussian distribution. Unless the population distribution is really weird, you are probably safe choosing a parametric test when there are at least two dozen data points in each group.
• Large sample. What happens when you use a nonparametric test with data from a Gaussian population? Nonparametric tests work well with large samples from Gaussian populations. The P values tend to be a bit too large, but the discrepancy is small. In other words, nonparametric tests are only slightly less powerful than parametric tests with large samples.
• Small samples. What happens when you use a parametric test with data from nongaussian populations? You can't rely on the central limit theorem, so the P value may be inaccurate.
• Small samples. When you use a nonparametric test with data from a Gaussian population, the P values tend to be too high. The nonparametric tests lack statistical power with small samples.

Thus, large data sets present no problems. It is usually easy to tell if the data come from a Gaussian population, but it doesn't really matter because the nonparametric tests are so powerful and the parametric tests are so robust. Small data sets present a dilemma. It is difficult to tell if the data come from a Gaussian population, but it matters a lot. The nonparametric tests are not powerful and the parametric tests are not robust.

ONE- OR TWO-SIDED P VALUE?

With many tests, you must choose whether you wish to calculate a one- or two-sided P value (same as one- or two-tailed P value). The difference between one- and two-sided P values was discussed in Chapter 10. Let's review the difference in the context of a t test. The P value is calculated for the null hypothesis that the two population means are equal, and any discrepancy between the two sample means is due to chance. If this null hypothesis is true, the one-sided P value is the probability that two sample means would differ as much as was observed (or further) in the direction specified by the hypothesis just by chance, even though the means of the overall populations are actually equal. The two-sided P value also includes the probability that the sample means would differ that much in the opposite direction (i.e., the other group has the larger mean). The two-sided P value is twice the one-sided P value.

A one-sided P value is appropriate when you can state with certainty (and before collecting any data) that there either will be no difference between the means or that the difference will go in a direction you can specify in advance (i.e., you have specified which group will have the larger mean). If you cannot specify the direction of any difference before collecting data, then a two-sided P value is more appropriate. If in doubt, select a two-sided P value.

If you select a one-sided test, you should do so before collecting any data and you need to state the direction of your experimental hypothesis. If the data go the other way, you must be willing to attribute that difference (or association or correlation) to chance, no matter how striking the data. If you would be intrigued, even a little, by data that goes in the "wrong" direction, then you should use a two-sided P value. For reasons discussed in Chapter 10, I recommend that you always calculate a two-sided P value.

PAIRED OR UNPAIRED TEST?

When comparing two groups, you need to decide whether to use a paired test. When comparing three or more groups, the term paired is not apt and the term repeated measures is used instead.

Use an unpaired test to compare groups when the individual values are not paired or matched with one another. Select a paired or repeated-measures test when values represent repeated measurements on one subject (before and after an intervention) or measurements on matched subjects. The paired or repeated-measures tests are also appropriate for repeated laboratory experiments run at different times, each with its own control.

You should select a paired test when values in one group are more closely correlated with a specific value in the other group than with random values in the other group. It is only appropriate to select a paired test when the subjects were matched or paired before the data were collected. You cannot base the pairing on the data you are analyzing.

FISHER'S TEST OR THE CHI-SQUARE TEST?

When analyzing contingency tables with two rows and two columns, you can use either Fisher's exact test or the chi-square test. The Fisher's test is the best choice as it always gives the exact P value. The chi-square test is simpler to calculate but yields only an approximate P value. If a computer is doing the calculations, you should choose Fisher's test unless you prefer the familiarity of the chi-square test. You should definitely avoid the chi-square test when the numbers in the contingency table are very small (any number less than about six). When the numbers are larger, the P values reported by the chi-square and Fisher's test will he very similar.

The chi-square test calculates approximate P values, and the Yates' continuity correction is designed to make the approximation better. Without the Yates' correction, the P values are too low. However, the correction goes too far, and the resulting P value is too high. Statisticians give different recommendations regarding Yates' correction. With large sample sizes, the Yates' correction makes little difference. If you select Fisher's test, the P value is exact and Yates' correction is not needed and is not available.

REGRESSION OR CORRELATION?

Linear regression and correlation are similar and easily confused. In some situations it makes sense to perform both calculations. Calculate linear correlation if you measured both X and Y in each subject and wish to quantity how well they are associated. Select the Pearson (parametric) correlation coefficient if you can assume that both X and Y are sampled from Gaussian populations. Otherwise choose the Spearman nonparametric correlation coefficient. Don't calculate the correlation coefficient (or its confidence interval) if you manipulated the X variable.

Calculate linear regressions only if one of the variables (X) is likely to precede or cause the other variable (Y). Definitely choose linear regression if you manipulated the X variable. It makes a big difference which variable is called X and which is called Y, as linear regression calculations are not symmetrical with respect to X and Y. If you swap the two variables, you will obtain a different regression line. In contrast, linear correlation calculations are symmetrical with respect to X and Y. If you swap the labels X and Y, you will still get the same correlation coefficient.

Statistical Data Analysis: Elementary Concepts

2007-08-30T15:17:00.000+01:00

Understanding Statistical Inference

Statistical inference is based upon mathematical laws of probability. The following example will give you the basic ideas.

Statistical Inference & The Coin Toss

Suppose we want to do a few coin tosses (sample) so that we can decide if a particular coin is equally likely to land head or tail over an infinite number of tosses (population).

If we toss the coin ten times and get 6 heads and 4 tails, we might suspect the coin is biased towards heads, but we wouldn't be very confident about this, because it's not that unusual (not that improbable) to get 6 heads out of 10.

On the other hand, if we toss the coin ten times and get 10 heads - we would be more confident that the coin is biased towards heads, because it is very unusual (not very probable at all) that we would get this result from an unbiased coin.

Statistical Data Analysis: Hypothesis Testing

The most common kind of statistical inference is hypothesis testing. Statistical data analysis allows us to use mathematical principles to decide how likely it is that our sample results match our hypothesis about a population. For example, if our research hypothesis is that the coin is not fair, but is actually biased towards heads - we can use principles of statistics to tell us how likely it is that we could get our sample results even if the coin were fair after all (null hypothesis).

If the probability of getting our sample results from a fair coin is very low, we feel confident in rejecting the null hypothesis (that the coin is fair). Even though we can't say for sure (because even a fair coin could produce our sample results), we can say that the results of our study support the hypothesis that the coin is indeed biased.

When we make this decision based on statistical data analysis, this is statistical inference.

Statistical Data Analysis: p-value

In statistical hypothesis testing we use a p-value (probability value) to decide whether we have enough evidence to reject the null hypothesis and say our research hypothesis is supported by the data.

The p-value is a numerical statement of how likely it is that we could have gotten our sample data (e.g., 10 heads) even if the null hypothesis is true (e.g., fair coin). By convention, if the p-value is less than 0.05 (p <>

KOLMOGOROV SMIRNOV Test (TWO SAMPLE) II

2007-08-30T14:10:00.001+01:00

Purpose:

Perform a Kolmogorov-Smirnov two sample test that two data samples come from the same distribution. Note that we are not specifying what that common distribution is.

Description: The one sample Kolmogorov-Smirnov (K-S) test is based on the empirical distribution function (ECDF). Given N data points Y₁ Y₂ ..., Y_N the ECDF is defined as

where n(i) is the number of points less than Y_i This is a step function that increases by 1/N at the value of each data point. We can graph a plot of the empirical distribution function with a cumulative distribution function for a given distribution. The one sample K-S test is based on the maximum distance between these two curves. That is,

where F is the theoretical cumulative distribution function.

The two sample K-S test is a variation of this. However, instead of comparing an empirical distribution function to a theoretical distribution function, we compare the two empirical distribution functions. That is,

where E₁ and E₂ are the empirical distribution functions for the two samples. Note that we compute E₁ and E₂ at each point in both samples (that is both E₁ and E₂ are computed at each point in each sample).

More formally, the Kolmogorov-Smirnov two sample test statistic can be defined as follows.

H₀:	The two samples come from a common distribution.
H_a:	The two samples do not come from a common distribution.
Test Statistic:	The Kolmogorov-Smirnov two sample test statistic is defined as where E₁ and E₂ are the empirical distribution functions for the two samples.
Significance Level:
Critical Region:	The hypothesis regarding the distributional form is rejected if the test statistic, D, is greater than the critical value obtained from a table. There are several variations of these tables in the literature that use somewhat different scalings for the K-S test statistic and critical regions. These alternative formulations should be equivalent, but it is necessary to ensure that the test statistic is calculated in a way that is consistent with how the critical values were tabulated.

Nonparametric Methods II (Brief Overview)

2007-08-30T12:42:00.000+01:00

Basically, there is at least one nonparametric equivalent for each parametric general type of test. In general, these tests fall into the following categories:

Tests of differences between groups (independent samples);
Tests of differences between variables (dependent samples);
Tests of relationships between variables.

Differences between independent groups.

Usually, when we have two samples that we want to compare concerning their mean value for some variable of interest, we would use the t-test for independent samples; nonparametric alternatives for this test are the Wald-Wolfowitz runs test, the Mann-Whitney U test, and the Kolmogorov-Smirnov two-sample test.
If we have multiple groups, we would use analysis of variance see ANOVA/MANOVA; the nonparametric equivalents to this method are the Kruskal-Wallis analysis of ranks and the Median test.

Differences between dependent groups.

If we want to compare two variables measured in the same sample we would customarily use the t-test for dependent samples ( for example, if we wanted to compare students' math skills at the beginning of the semester with their skills at the end of the semester). Nonparametric alternatives to this test are the Sign test and Wilcoxon's matched pairs test.
If the variables of interest are dichotomous in nature (i.e., "pass" vs. "no pass") then McNemar's Chi-square test is appropriate.
If there are more than two variables that were measured in the same sample, then we would customarily use repeated measures ANOVA. Nonparametric alternatives to this method are Friedman's two-way analysis of variance and Cochran Q test (if the variable was measured in terms of categories, e.g., "passed" vs. "failed"). Cochran Q is particularly useful for measuring changes in frequencies (proportions) across time.

Relationships between variables. To express a relationship between two variables one usually computes the correlation coefficient. Nonparametric equivalents to the standard correlation coefficient are Spearman R, Kendall Tau, and coefficient Gamma (see Nonparametric correlations).

If the two variables of interest are categorical in nature (e.g., "passed" vs. "failed" by "male" vs. "female") appropriate nonparametric statistics for testing the relationship between the two variables are the Chi-square test, the Phi coefficient, and the Fisher exact test.
In addition, a simultaneous test for relationships between multiple cases is available: Kendall coefficient of concordance. This test is often used for expressing inter-rater agreement among independent judges who are rating (ranking) the same stimuli.

Descriptive statistics. When one's data are not normally distributed, and the measurements at best contain rank order information, then computing the standard descriptive statistics (e.g., mean, standard deviation) is sometimes not the most informative way to summarize the data. For example, in the area of psychometrics it is well known that the rated intensity of a stimulus (e.g., perceived brightness of a light) is often a logarithmic function of the actual intensity of the stimulus (brightness as measured in objective units of Lux). In this example, the simple mean rating (sum of ratings divided by the number of stimuli) is not an adequate summary of the average actual intensity of the stimuli. (In this example, one would probably rather compute the geometric mean.)

Nonparametrics and Distributions will compute a wide variety of measures of location (mean, median, mode, etc.) and dispersion (variance, average deviation, quartile range, etc.) to provide the "complete picture" of one's data.

Customer Name	P(x)	Actual Response
Alex	88	Y
Amy	87	N
Hilary	81	Y
Philip	80	Y
Bob	79	Y
Catherine	77	N
Nancy	76	Y
Jessica	75	Y
Preston	75	N
Laura	71	Y
Elizabeth	70	Y
Kim	69	N
Erin	65	Y
Alan	61	N
Chris	58	N
Fred	52	N
Margot	49	N
Trent	45	N
Sean	35	Y
John	24	N

Total Customers Contacted	Number of Responses	Response Rate
2	1	10%
4	3	30%
6	4	40%
8	6	60%
10	7	70%
12	8	80%
14	9	90%
16	9	90%
18	9	90%
20	10	100%

Johnny Deng's Column

Autoregressive Moving-Average Process

White Noise

ARCH and GARCH Processes

Greek alphabet, again

Dummy Variables

Why use dummies?

Nominal variables with multiple levels

Interpreting results

Levels of Measurement

Why is Level of Measurement Important?

AUC: Z distribution

What is overfitting and how can I avoid it?

Overfitting and Underfitting

Data Mining Techniques

Supervised learning

Supervised learning

From Wikipedia, the free encyclopedia

Overfitting

Bias-Variance trade-off

Preventing overfitting

Early stopping

Weight decay

Training with noise

Algebra - Imperial College

M2P2 Algebra

Lecturer: Prof M W Liebeck

Recommended books

M2S1 PROBABILITY AND STATISTICS II - Imperial College

Confusion Matrix

Timetable II

Autumn 2007

MSc Advanced Computing (Weeks 2 - 11)

Timetable

Autumn 2007

MSc Computing Science (Weeks 2 - 11)

ROC Graph

Cumulative Gains and Lift Charts

Example Problem 1

Evaluating a Predictive Model

Example Problem 2

Introduction to ROC Curves

Introduction to ROC Curves

Plotting and Intrepretating an ROC Curve

The Area Under an ROC Curve

Logistic Regression

K-fold Cross Validation

Linear regression

Linear function

Usage in elementary mathematics

Usage in advanced mathematics

Spearman Correlation - Example

Example of Statistical Test using the Spearman Rank-Difference Correlation Coefficient

Choosing the Proper Statistical Test

Spearman Correlation

Example

Statistics: Another Syllabus

STATISTICAL METHODS FOR RESEARCH WORKERS

Correlation ratio

Correlation, Pearson Correlation

Pearson's product-moment coefficient

Mathematical properties

The sample correlation

Geometric Interpretation of correlation

Interpretation of the size of a correlation

Non-parametric correlation coefficients

Other measures of dependence among random variables

Copulas and correlation

Correlation matrices

Removing correlation

Common misconceptions about correlation

Correlation and causality

Correlation and linearity

Computing correlation accurately in a single pass

Currency correlation

Overview of Statistical Test [***recommended***]

Statistical Data Analysis: Elementary Concepts

Statistical Data Analysis: Hypothesis Testing

Statistical Data Analysis: p-value

KOLMOGOROV SMIRNOV Test (TWO SAMPLE) II

Overview of Statistical Test [recommended]