A Statistical Model
How do we categorize the questions? This is an easy problem if you say, ok let’s categorize all the past paper problems manually and that’s it but that’s boring. That’s tedious work and I don’t wanna do that. That’s a bad idea.
So, I am proposing to build a statistical model (fancy name) which will be fed enough textbook data so, it can learn and identify which words belong to which topic, and later, when we feed it a question, it will check the words and calculate the probability of being in, say one or two topics. Sounds good to me for now. Maybe in the future, I will use the word Embeddings and deep learning to do this but for now, the previous idea should be good. It will be a prototype so it will definitely be bad in terms of functionality.
Get Open Past Paper’s stories in your inbox
Join Medium for free to get updates from this writer.
I’ve wrote a python program to do this for now and it seems promising. I am using my physics text book (a pdf version) to build the words model (another fancy name). I can definitely see the words frequency which correlates to the topic. Let’s see what i can do more.
Update 1
I’ve written a python program to do this for now and it seems promising. I am using my physics textbook (a pdf version) to build the words model (another fancy name). I can see the frequency of the word which correlates to the topic. Let’s see what I can do more.
Update 2
But Wait! why did I not think about this. So, I was thinking about using a topical question bank for testing the model but I just realized, most people took screenshots of the questions in the past papers and merged all of them together to make that book. I mean it’s okay to do that but for me, that’s a problem.
My program can’t read that book because it’s an image, not a text. Although I can build an image to text program using OpenCV and others stuffs to read what’s written on that image and print the corresponding text but that’s just a lot of work right now. Maybe I will consider doing that in the future while testing the validity/accuracy of the model.
Using NLTK to categorize most of the words so I can collect those words which actually have meanings, so Adjectives. But something weird I’ve found that is the NLTK library is classifying words like adjectives and the same word again as something else like a noun. I will let the NLTK tagging use unique words only for now. As far as I can see the model gives me good enough words if I collect adjectives and nouns. But more interestingly, the frequency of nouns in the chapter is quite high compared to adjectives. Looks good. If I see these words in any questions, I can pretty much classify which topic the question was taken from. Good progress.
Originally published at http://manishgotame.wordpress.com on June 18, 2020.
