11.. _lda_qda :
22
33==========================================
4- Linear and quadratic discriminant analysis
4+ Linear and Quadratic Discriminant Analysis
55==========================================
66
77.. currentmodule :: sklearn
88
9- Linear discriminant analysis (:class: `lda.LDA `) and
10- quadratic discriminant analysis (:class: `qda.QDA `)
11- are two standard classifiers, with, as their names suggest, a linear and a
12- quadratic decision surface, respectively.
9+ Linear Discriminant Analysis
10+ (:class: `discriminant_analysis.LinearDiscriminantAnalysis `) and Quadratic
11+ Discriminant Analysis
12+ (:class: `discriminant_analysis.QuadraticDiscriminantAnalysis `) are two classic
13+ classifiers, with, as their names suggest, a linear and a quadratic decision
14+ surface, respectively.
1315
1416These classifiers are attractive because they have closed-form solutions that
15- can be easily computed, are inherently multiclass, have proven to work well in practice and have
16- no hyperparameters to tune.
17+ can be easily computed, are inherently multiclass, have proven to work well in
18+ practice and have no hyperparameters to tune.
1719
1820.. |ldaqda | image :: ../auto_examples/classification/images/plot_lda_qda_001.png
1921 :target: ../auto_examples/classification/plot_lda_qda.html
2022 :scale: 80
2123
2224.. centered :: |ldaqda|
2325
24- The plot shows decision boundaries for LDA and QDA. The first row shows that,
25- when the classes covariances are the same, LDA and QDA yield the same result
26- (up to a small difference resulting from the implementation). The bottom row demonstrates that in general,
27- LDA can only learn linear boundaries, while QDA can learn
28- quadratic boundaries and is therefore more flexible.
26+ The plot shows decision boundaries for Linear Discriminant Analysis and
27+ Quadratic Discriminant Analysis. The bottom row demonstrates that Linear
28+ Discriminant Analysis can only learn linear boundaries, while Quadratic
29+ Discriminant Analysis can learn quadratic boundaries and is therefore more
30+ flexible.
2931
3032.. topic :: Examples:
3133
32- :ref: `example_classification_plot_lda_qda.py `: Comparison of LDA and QDA on synthetic data.
34+ :ref: `example_classification_plot_lda_qda.py `: Comparison of LDA and QDA
35+ on synthetic data.
3336
34- Dimensionality reduction using LDA
35- ==================================
36-
37- :class: `lda.LDA ` can be used to perform supervised dimensionality reduction, by
38- projecting the input data to a linear subspace consisting of the directions which maximize the
39- separation between classes (in a precise sense discussed in the mathematics section below).
40- The dimension of the output is necessarily less that the number of classes,
41- so this is a in general a rather strong dimensionality reduction, and only makes senses
42- in a multiclass setting.
37+ Dimensionality reduction using Linear Discriminant Analysis
38+ ===========================================================
4339
44- This is implemented in :func: `lda.LDA.transform `. The desired
45- dimensionality can be set using the ``n_components `` constructor
46- parameter. This parameter has no influence on :func: `lda.LDA.fit ` or :func: `lda.LDA.predict `.
40+ :class: `discriminant_analysis.LinearDiscriminantAnalysis ` can be used to
41+ perform supervised dimensionality reduction, by projecting the input data to a
42+ linear subspace consisting of the directions which maximize the separation
43+ between classes (in a precise sense discussed in the mathematics section
44+ below). The dimension of the output is necessarily less that the number of
45+ classes, so this is a in general a rather strong dimensionality reduction, and
46+ only makes senses in a multiclass setting.
47+
48+ This is implemented in
49+ :func: `discriminant_analysis.LinearDiscriminantAnalysis.transform `. The desired
50+ dimensionality can be set using the ``n_components `` constructor parameter.
51+ This parameter has no influence on
52+ :func: `discriminant_analysis.LinearDiscriminantAnalysis.fit ` or
53+ :func: `discriminant_analysis.LinearDiscriminantAnalysis.predict `.
4754
4855.. topic :: Examples:
4956
50- :ref: `example_decomposition_plot_pca_vs_lda.py `: Comparison of LDA and PCA for dimensionality reduction of the Iris dataset
57+ :ref: `example_decomposition_plot_pca_vs_lda.py `: Comparison of LDA and PCA
58+ for dimensionality reduction of the Iris dataset
5159
5260Mathematical formulation of the LDA and QDA classifiers
5361=======================================================
5462
55- Both LDA and QDA can be derived from simple probabilistic models
56- which model the class conditional distribution of the data :math: `P(X|y=k)`
57- for each class :math: `k`. Predictions can then be obtained by using Bayes' rule:
63+ Both LDA and QDA can be derived from simple probabilistic models which model
64+ the class conditional distribution of the data :math: `P(X|y=k)` for each class
65+ :math: `k`. Predictions can then be obtained by using Bayes' rule:
5866
5967.. math ::
6068 P(y=k | X) = \frac {P(X | y=k) P(y=k)}{P(X)} = \frac {P(X | y=k) P(y = k)}{ \sum _{l} P(X | y=l) \cdot P(y=l)}
6169
6270:math: `k` which maximizes this conditional probability.
6371
64- More specifically, for linear and quadratic discriminant analysis, :math: `P(X|y)`
65- is modelled as a multivariate Gaussian distribution with density:
72+ More specifically, for linear and quadratic discriminant analysis,
73+ :math: `P(X|y)` is modelled as a multivariate Gaussian distribution with
74+ density:
6675
6776.. math :: p(X | y=k) = \frac{1}{(2\pi)^n |\Sigma_k|^{1/2}}\exp\left(-\frac{1}{2} (X-\mu_k)^t \Sigma_k^{-1} (X-\mu_k)\right)
6877
69- To use this model as a classifier, we just need to estimate from the training data
70- the class priors :math: `P(y=k)` (by the proportion of instances of class :math: `k`), the
71- class means :math: `\mu _k` (by the empirical sample class means) and the covariance matrices
72- (either by the empirical sample class covariance matrices, or by a regularized estimator: see the section on shrinkage below).
78+ To use this model as a classifier, we just need to estimate from the training
79+ data the class priors :math: `P(y=k)` (by the proportion of instances of class
80+ :math: `k`), the class means :math: `\mu _k` (by the empirical sample class means)
81+ and the covariance matrices (either by the empirical sample class covariance
82+ matrices, or by a regularized estimator: see the section on shrinkage below).
7383
74- In the case of LDA, the Gaussians for each class are assumed
75- to share the same covariance matrix: :math: `\Sigma _k = \Sigma ` for all :math: `k`.
76- This leads to linear decision surfaces between, as can be seen by comparing the the log-probability ratios
77- :math: `\log [P(y=k | X) / P(y=l | X)]`:
84+ In the case of LDA, the Gaussians for each class are assumed to share the same
85+ covariance matrix: :math: `\Sigma _k = \Sigma ` for all :math: `k`. This leads to
86+ linear decision surfaces between, as can be seen by comparing the the
87+ log-probability ratios :math: `\log [P(y=k | X) / P(y=l | X)]`:
7888
7989.. math ::
8090 \log \left (\frac {P(y=k|X)}{P(y=l | X)}\right ) = 0 \Leftrightarrow (\mu _k-\mu _l)\Sigma ^{-1 } X = \frac {1 }{2 } (\mu _k^t \Sigma ^{-1 } \mu _k - \mu _l^t \Sigma ^{-1 } \mu _l)
8191
82- :math: `\Sigma _k` of the Gaussians,
83- leading to quadratic decision surfaces. See [#1 ]_ for more details.
92+
93+ :math: `\Sigma _k` of the Gaussians, leading to quadratic decision surfaces. See
94+ [#1 ]_ for more details.
8495
8596.. note :: **Relation with Gaussian Naive Bayes**
8697
87- If in the QDA model one assumes that the covariance matrices are diagonal, then
88- this means that we assume the classes are conditionally independent,
89- and the resulting classifier is equivalent to the Gaussian Naive Bayes classifier :class: `GaussianNB `.
98+ If in the QDA model one assumes that the covariance matrices are diagonal,
99+ then this means that we assume the classes are conditionally independent,
100+ and the resulting classifier is equivalent to the Gaussian Naive Bayes
101+ classifier :class: `naive_bayes.GaussianNB `.
90102
91103Mathematical formulation of LDA dimensionality reduction
92- ===========================================================
104+ ========================================================
93105
94106To understand the use of LDA in dimensionality reduction, it is useful to start
95107with a geometric reformulation of the LDA classification rule explained above.
96- We write :math: `K` for the total number of target classes.
97- Since in LDA we assume that all classes have the same estimated covariance :math: `\Sigma `, we can rescale the
98- data so that this covariance is the identity:
108+ We write :math: `K` for the total number of target classes. Since in LDA we
109+ assume that all classes have the same estimated covariance :math: `\Sigma `, we
110+ can rescale the data so that this covariance is the identity:
99111
100112.. math :: X^* = D^{-1/2}U^t X\text{ with }\Sigma = UDU^t
101113
102- Then one can show that to classify a data point after scaling is equivalent to finding the estimated class mean :math: `\mu ^*_k` which is
103- closest to the data point in the Euclidean distance. But this can be done just as well after projecting on the :math: `K-1 ` affine subspace :math: `H_K`
104- generated by all the :math: `\mu ^*_k` for all classes. This shows that, implicit in the LDA classifier, there is
105- a dimensionality reduction by linear projection onto a :math: `K-1 ` dimensional space.
106-
107- We can reduce the dimension even more, to a chosen :math: `L`, by projecting onto the linear subspace :math: `H_L` which
108- maximize the variance of the :math: `\mu ^*_k` after projection (in effect, we are doing a form of PCA for the transformed class means :math: `\mu ^*_k`).
109- This :math: `L` corresponds to the ``n_components `` parameter in the :func: `lda.LDA.transform ` method. See [#1 ]_ for more details.
114+ Then one can show that to classify a data point after scaling is equivalent to
115+ finding the estimated class mean :math: `\mu ^*_k` which is closest to the data
116+ point in the Euclidean distance. But this can be done just as well after
117+ projecting on the :math: `K-1 ` affine subspace :math: `H_K` generated by all the
118+ :math: `\mu ^*_k` for all classes. This shows that, implicit in the LDA
119+ classifier, there is a dimensionality reduction by linear projection onto a
120+ :math: `K-1 ` dimensional space.
121+
122+ We can reduce the dimension even more, to a chosen :math: `L`, by projecting
123+ onto the linear subspace :math: `H_L` which maximize the variance of the
124+ :math: `\mu ^*_k` after projection (in effect, we are doing a form of PCA for the
125+ transformed class means :math: `\mu ^*_k`). This :math: `L` corresponds to the
126+ ``n_components `` parameter used in the
127+ :func: `discriminant_analysis.LinearDiscriminantAnalysis.transform ` method. See
128+ [#1 ]_ for more details.
110129
111130Shrinkage
112131=========
@@ -115,10 +134,11 @@ Shrinkage is a tool to improve estimation of covariance matrices in situations
115134where the number of training samples is small compared to the number of
116135features. In this scenario, the empirical sample covariance is a poor
117136estimator. Shrinkage LDA can be used by setting the ``shrinkage `` parameter of
118- the :class: `lda.LDA ` class to 'auto'. This automatically determines the
119- optimal shrinkage parameter in an analytic way following the lemma introduced
120- by Ledoit and Wolf [#2 ]_. Note that currently shrinkage only works when setting the
121- ``solver `` parameter to 'lsqr' or 'eigen'.
137+ the :class: `discriminant_analysis.LinearDiscriminantAnalysis ` class to 'auto'.
138+ This automatically determines the optimal shrinkage parameter in an analytic
139+ way following the lemma introduced by Ledoit and Wolf [#2 ]_. Note that
140+ currently shrinkage only works when setting the ``solver `` parameter to 'lsqr'
141+ or 'eigen'.
122142
123143The ``shrinkage `` parameter can also be manually set between 0 and 1. In
124144particular, a value of 0 corresponds to no shrinkage (which means the empirical
@@ -154,12 +174,13 @@ a high number of features.
154174
155175.. topic :: Examples:
156176
157- :ref: `example_classification_plot_lda.py `: Comparison of LDA classifiers with and without shrinkage.
177+ :ref: `example_classification_plot_lda.py `: Comparison of LDA classifiers
178+ with and without shrinkage.
158179
159180.. topic :: References:
160181
161182 .. [#1 ] "The Elements of Statistical Learning", Hastie T., Tibshirani R.,
162- Friedman J., Section 4.3, p.106-119, 2008.
183+ Friedman J., Section 4.3, p.106-119, 2008.
163184
164- #2 ] Ledoit O, Wolf M. Honey, I Shrunk the Sample Covariance Matrix. The Journal of Portfolio
165- Management 30(4), 110-119, 2004.
185+ #2 ] Ledoit O, Wolf M. Honey, I Shrunk the Sample Covariance Matrix.
186+ The Journal of Portfolio Management 30(4), 110-119, 2004.
0 commit comments