Beyond Feature Selection: A Term Weighting Approach in Text Classification
As you may know, there are a lot of techniques in feature extraction for text classification. You might be familiar with count-vectorizer, term-frequency, term-frequency and inverse document frequency (TF-IDF). If you have already used scikit-learn and often play with text classification. I guarantee you, those terms are used in almost your notebooks.
I have already written about feature selection, if you have not yet read it, please click the link below to get the sense of feature selection. The article discusses briefly feature selection in general, and SOTA (kind of, I guess :D ) for filter method.
Liu et. al. define term weighting as to assign a numeric value to each term to weight its contribution which helps a document stand out from others. On the other hand, term selection a.k.a feature selection is defined as to select a subset of terms from all terms occurring in the collection to represent the document in a better way, either to faster computing or to achieve better effectiveness in classification.
The term feature selection also needs another parameter to tune how many terms that needed to be trained. Unlike term weighting, you do not need parameters to process the data into a term weighting algorithm.
According to Debole and Sebastiani, term weighting algorithms are said to be a good algorithm if they fulfill the 3 fundamental assumptions, they are:
- Rare terms are no less important than frequent terms
- Multiple appearances of a term in a document are no less important than single appearance
- For the same quantity of term matching, long documents are no less important than short documents
These characters are written on a paper of Liu et al. Those characters also similar to what Uysal said in feature selection. Uysal defines a good filter method in feature selection should have the following requirements:
- the algorithm should assign a high score to the feature if it frequently occurs in a single class or in some classes.
- the algorithm should assign a lower score to the feature if it rarely occurs in a single class or frequently occurs in all classes.
Because term weighting and term selection have the same goal, there are a lot of term weighting algorithms are inspired by feature selection algorithms. there are a lot of mix algorithms in term weighting such as TF-Chi-square, TF-Correlation coefficient, etc. As a result, Liu et. al experimented this kind of term weight algorithms in order to find which algorithms gives the best accuracy for text classification.
Get AC’s stories in your inbox
Join Medium for free to get updates from this writer.
The experiment settings that are conducted by Liu et. al in “Imbalance text classification: A term weighting approach” as follows: They used 2 data sets, Reuters-21578 and MCV1. They compared eight probability based term weighting schemes such as TF-chi-square, tf-correlation coefficient, tf-odds ratio, tf-information gain, tf-idf, tfidf-ltc, normalized ltc, and probability based. Moreover, they also use two popular algorithms which are Support Vector Machine and Complement Naive Bayes. In their experiments, they used Weka 3.5.3 to get the result.
In overall performance, Probability-based which the algorithms proposed, give higher F1 Values to all scenario. The highest F1 Score in scenario falls into SVM with Reuters data set which give almost all term weighting algorithms above 80 percent.
The author noticed another paper that gives a novel algorithm, namely Term Frequency-Inverse Gravity Moment (TF-IGM) by Chen et. al. The experiments were done with several popular data sets such as 20 Newsgroups, Reuters-21578, and TanCorp. They also compared the algorithm with Support Vector Machine and K-Nearest Neighbor. Another important point in their experiments is that they benchmark the popular term weighting schemes.
Debole and Sebastiani called this as supervised term weighting (STW), weighting terms by exploiting the known categorical information in training corpus. They compared their proposed algorithm (TF-IGM) with TF-IDF, TF-RF, TF-IGM, RTF-IGM. Moreover, Chen et. al. also give a clear how many feature used in these experiments, in 20 Newsgroups, they used the top features {500, 1000, 2000, 4000, 6000, 9000, 12000, 16000}, {100, 300, 500, 700, 1000, 1500, 2000, 3000, 5000, 8436} terms are used for Reuters-21578, and last {200, 500, 1000, 2000, 4000, 6000} for TanCorp.
The experimental results are as follows: They breakdown the performance in each data set. In 20 Newsgroups, using Support Vector Machine gives RTF-IGM, followed by TF-IGM. as the best performance measured in macro-F1, the results are also the same in K-NN with K = 20. In Reuters-21578, using SVM, RTF-IGM and TF-IGM is closed tight as the best term weighting, on the other hand, the experiment is done with KNN with K = 15 with the same data set. it gives really close result to TF-IGM, RTF-IGM, and TF-RF. The last, TanCorp corpus also give a similar result to the other dataset, RTF-IGM and TF-IGM are out as the top two terms weighting algorithms which proposed by Chen et. al.
The author would like to implement all algorithms for term-weighting, and the code can be found here. The current development is only available for TF-IDF
In conclusion, this article is trying to give information that there are a lot of techniques to improve classification accuracy, especially, in an imbalanced data set. The use of all terms for text classification is extremely big followed by how broad the corpus. the broader corpus, the larger terms are. The author hopes that after reading this article, you are not only implementing the basic method as available in popular machine learning library. The author also has the desire to write all algorithms in Python.
If you enjoyed this post, feel free to hit the clap button 👏🏽 and if you’re interested in posts to come, make sure to follow me on medium
References:
K. Chen, Z. Zhang, J. Long, H. Zhang, “Turning from TF-IDF to TF-IGM for term weighting in text classification,” Expert Systems with Application, vol. 66, pp. 245–260, 2016
Y. Liu, H. T. Loh, A. Sun, “Imbalanced text classification: A term weighting apporach,” Expert Systems with Applications, vol. 36, pp. 690–701, 2009
