Text Summarization

Using Term Frequency

Brief

Sentence scoring using term frequency is one of the extractive approaches for text summarization. It is based on the frequency of the word. The sentences with the scores above threshold value are extracted to generate a summary.

Steps

1. Covert text to sentences : Converting a single text to list of sentences.
2. Pre-process text : Clean the sentences by removing unnecessary words, stopwords, punctuations, etc.
3. Create word frequency table : Word frequency table shows the frequency of distinct words present in the text excluding the unimportant words like the stopwords.
4. Calculate sentence scores : Sentence score is the sum of frequency of words present in the sentence.
5. Determine threshold : Threshold is the average value of the scores of the sentences. It is calculated as S / s where,
   S = Total sum of scores of sentences
   s = Number of sentences
6. Generate Summary : Generate a summary by extracting the sentences having scores greater than the threshold value.

Implementation

Requirement
Python, NLTK library

Code

Get Ashin Shakya’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

1.Convert text to sentences

from nltk.tokenize import sent_tokenize
text = "Your text goes here."
sentences = sent_tokenize(text)

2. Pre-process text

from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer
ps = PorterStemmer()
def text_preprocessing(sentences):
    “””
    Pre processing text to removeunnecessary words.
    “”” 
    print(‘Preprocessing text’)
    stop_words = set(stopwords.words('english'))
    
    clean_words = []
    for sent in sentences:
        words = word_tokenize(sent)
        words = [ps.stem(word.lower()) for word in words if word.isalnum()]
        clean_words += [word for word in words if word not in stop_words]
    return clean_words

3.Create word frequency table

def create_word_frequency_table(words: list) -> dict:
    """
    Creating a frequency table to keep the score of each word.
    """
    print('Creating word frequency table')
    freq_table = dict()
    for word in words:
        if word in freq_table:
            freq_table[word] += 1
        else:
            freq_table[word] = 1

    return freq_table

4.Calculate sentence scores

def create_sentence_score_table(sentences, freq_table) -> dict:
    """
    Creating a dictionary to keep the score of each sentence.
    """
    print('Creating sentence score table')
    sent_value = dict()
    for sentence in sentences:
        for word, freq in freq_table.items():
            if ps.stem(word) in sentence.lower():
                if sentence[:15] in sent_value:
                    sent_value[sentence[:15]] += freq
                else:
                    sent_value[sentence[:15]] = freq

    return sent_value

5.Determine threshold

def find_average_score(sent_value) -> int:
    """
    Calculate average value of a sentence form the original text.
    """
    print('Finding average score')
    sum_values = 0
    for sentence in sent_value:
        sum_values += sent_value[sentence]

    average = int(sum_values / len(sent_value))

    return average

6.Generate Summary

def generate_summary(sentences, sent_value, avg) -> str:
    """
    Generate a sentence for sentence score greater than average.
    """
    print('Generating summary')
    sentence_count = 0
    summary = ''

    for sent in sentences:
        if sent[:15] in sent_value and sent_value[sent[:15]] >= avg:
            summary += sent + " "
            sentence_count += 1

    return summary

Full Code Implementation

Get the full code here.