Simple Text Data Handling

In this blog, we will take sample text and clean it. Then we will convert them into words and then calculate the words length, sum, average, median, mode etc. After that, we will see the frequencies of the words and create a word cloud.

Photo by Hitesh Choudhary on Unsplash

Declare input text

text = """

Once, there were two friends who were crossing the jungle.
After some time, they saw a bear coming towards them.
Then, one of the friends quickly climbed the nearby tree, and the other one did not know how to climb the tree.
So he lays down on the ground, holding his breath.
The bear reaches towards him and sniffs him in the ear.
After some time, the bear left the place, thinking the man was dead.
Now the other friend climbs down and asks his friend, What did bear say to him in his ear?
He replied, "
To be safe from the fake friends."

"""

text

Output:

print(text)

Output:

Replace , (comma) .(dot), ? (question mark) and \ (slash) from the text

mod_text = text.replace(",", "").replace(".", "").replace("?", "").replace('"', ""
            ).replace("\n", " ")
print(mod_text)

Output:

mod_text

Output:

Check the type of mod_text

type(mod_text)

Output: str

The type of the mod_text is string.

Split the mod_text by space

words = mod_text.split(" ")

print(words)

Output:

Print the each word from words

[ word  for word in words  if len(word) == 0  ]

Output:

mod_words = [ word  for word in words  if len(word) != 0  ]

print(mod_words)

Output:

Count the length of mod_words

word_lengths = [len(x)  for x in mod_words ]

print(word_lengths)

Output:

How many word_lenghts

n = len(word_lengths)
n

Output: 104

What is sum of all of them

s = sum(word_lengths)
s

Output: 417

Average

avg = s / n
avg

Output: 4.009615384615385

We can check mean from numpy and statistics package also

import numpy as np
import statistics as st

np.mean(word_lengths)

Output: 4.009615384615385

st.mean(word_lengths)

Output: 4.009615384615385

Median

Median is the middle element in the sorted list.

sorted_lengths = sorted(word_lengths)
print(sorted_lengths)

Output:

Middle position

n / 2

Output: 52.0

Get Nutan’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

As it is even, 52 and 53 are both in the middle

sorted_lengths[52]

Output: 4

sorted_lengths[53]

Output: 4

(sorted_lengths[52] + sorted_lengths[53]) / 2

Output: 4.0

median = (sorted_lengths[52]   + sorted_lengths[53]) / 2
median

Output: 4.0

np.median(word_lengths)

Output: 4.0

st.median(word_lengths)

Output: 4.0

What is mode?

Mode is the item that appears highest number of times

st.mode(word_lengths)

Output: 3

word_lengths.count(0)

Output: 0

word_lengths.count(1)

Output: 0

word_lengths.count(2)

Output: 11

word_lengths.count(3)

Output: 37

min(word_lengths)

Output: 1

max(word_lengths)

Output: 8

for x in range(1, 9):
    #print(x)
    print(x, word_lengths.count(x))

Output:

Mode is 3 as it appears 37 times, the most.

Get the frequency using the Counter class from collections package

import collections  as cl

cl.Counter(word_lengths)

Output:

So we got the frequency of items.

Plot frequency

word_freqs_1 = cl.Counter(word_lengths)
word_freqs_1

Output:

import matplotlib.pyplot as plt

tuples = tuple(   word_freqs_1.items()     )
tuples

Output:

plt.plot(tuples)
plt.show()

Output:

Create a word cloud

from wordcloud import WordCloud

print(mod_words)

Output:

Convert mod_words to word_frequencies

word_freqs_2 = cl.Counter(mod_words)
print(word_freqs_2)

Output:

word_cloud_1 = WordCloud( 
    width = 800, height = 500, background_color ='white').generate_from_frequencies(word_freqs_2)

#show word cloud in matplotlib
plt.imshow(word_cloud_1, interpolation='bilinear')
plt.axis("off")
plt.show()

Output:

Convert word_lengths key into str() to create a word cloud

print(word_freqs_1)

Output:

word_freqs_1 = {str(k):int(v) for k,v in word_freqs_1.items()}
word_freqs_1

Output:

word_cloud_2 = WordCloud( 
    width = 800, height = 500, background_color ='white').generate_from_frequencies(word_freqs_1)

#show word cloud in matplotlib
plt.imshow(word_cloud_2, interpolation='bilinear')
plt.axis("off")
plt.show()

Output:

That’s it, in this blog. If you have questions related to my blog, you can email me at nutanbhogendrasharma@gmail.com.

Thanks for reading, Happy reading 😊😊😊.