First thing first: Why did I write this article? Well, while working on a machine learning project, I was confronted with this problem and I didn’t know which one to choose. So I had to take several minutes of my time to Google and find the reasons why one form could be preferred over the other. I found some useful information and decided to write this article to share what I learned. Hopefully, it will help others who are facing the same question.

But before starting, what are actually one-hot encoding and integer encoding? These two methods are options to numerically encode your data which take place during the preprocessing phase. Indeed, for the computer to understand the input you provide, it must be transformed into numerical values and this is where the one-hot encoding and the integer encoding come in. To illustrate this, I’ll use the example that led me to ask myself that very question. Using data from rotten tomatoes, I wanted to build a model that predicted whether a film would be classified as ‘Rotten’, ‘Fresh’, or ‘Certified-Fresh’. You can find the complete project with the data on StrataScratch. In my data frame, I had several features that I had to transform into numerical values, for example, the content rating (‘PG’, ‘R’, etc.), and audience status (‘Spilled’ or ‘Upright’).

There I had the choice between either using one-hot encoding or integer encoding. In the latter, each category is assigned a unique integer, and the integer values are used to represent the categories in the dataset. At that time, that’s the method I used — to be honest, the only reason I favoured that one is simply because I didn’t want to have too many columns in my df. To do so, I used the LabelEncoder() from sklearn.preprocessing and ended up with this result:

The last two columns are my new categorical data represented as integers. Its advantage is that it is pretty straightforward to implement. The code here was simply :

LE = LabelEncoder()
movies_cl['audience'] = LE.fit_transform(movies_cl['audience_status']

It has as well the advantage of preserving an ordinal relationship, however, that is as well one of the reasons you shouldn’t be using it. Indeed, Integer encoding can introduce a bias into the data by implying a natural ordering between the categories. This can lead to poor performance or unexpected results in machine learning models that use the encoded data. For example, the model may make predictions that are halfway between the encoded categories, which may not accurately reflect the original data. Furthermore, it implies that one category is more similar than another. For example, if we encode the categories “red”, “green”, and “blue” as 1, 2, and 3 respectively, the encoded values will imply that “red” is closer to “green” than it is to “blue”. It can lead to incorrect assumptions and conclusions.

In this case, where there is no ordinal relationship, it would be better to use One-hot encoding. This will create a new column for each category, and each column will have a value of 0 or 1 to indicate the presence or absence of that category in a given sample.

One advantage of one-hot encoding is that it allows the model to learn more easily and effectively. This is because each input value is represented as a binary vector, where only a single element of the vector is set to 1 and the rest are set to 0. This makes it easier for the model to learn the relationship between the input values and the target output because the model can easily distinguish between the different input values based on their unique binary representation.

To do so easily you can use the pandas’ get_dummies() which will give you this result:

You can as well use the sklearn.preprocessing OneHotEncoder() which is apparently better for machine learning. You can then put this feature into your original data frame and get rid of the old columns.

So, we can see that it is not really more complicated to implement than the integer encoding and One-hot encoding has some more advantages.

• First, one-hot encoding is often considered to be more expressive than integer encoding, because it can more accurately represent the data and its relationships. Indeed, it can better represent the presence or absence of a category. . For example, if a sample belongs to the “Red” and “Green” categories, the one-hot encoded representation of that sample would be [1, 0, 1] (with a 1 in the first and third columns and a 0 in the second column). This accurately reflects the fact that the sample belongs to the “Red” and “Green” categories, but not to the “Blue” category. In contrast, the integer encoded representation of the same sample would be [1, 2], which does not provide any information about the presence or absence of the “Blue” category.
• The model can as well more easily handle new input values that it hasn’t seen before. For example, if the model has been trained on a set of input values that includes the values “red”, “green”, and “blue”, and then it receives a new input value “yellow”, it can easily represent this new value as a binary vector and make a prediction based on its training.
• Finally, one-hot encoding can also be more efficient in terms of memory and computational cost, because the binary vectors are typically much shorter and sparser than the corresponding integer encodings. This can be especially important for large datasets or complex models.

However, you have to note that one-hot encoding might lead to high dimensional data and to multicollinearity which might lower the model’s accuracy.

But why did I ask myself so many questions about those two methods? Well, simply because I saw that the Stratascratch solution was using a one-hot encoding method for the content rating when I had chosen to represent it with integer encoding and I couldn’t understand why it was doing so there when it was using integer encoding for the other categorical values. To be honest, I am not 100% sure that in this case, it was absolutely necessary to use one-hot encoding as my results were as good as Stratascratch’s (probably because content rating is perhaps not a very important feature in the construction of this model and we could as well find some sense of ordinality in it).

I hope that this article helped you better understand what is the difference between the two methods and what are its advantages and disadvantages of each. I mainly hope that you choose one version over another because you truly understand what it will imply for your model and not, as I did at first, because you don’t want to have too many columns in your dataframe… I’m at least happy that it piqued my interest enough to want to look into it.

Machine learning: one-hot encoding vs integer encoding was originally published in Geek Culture on Medium, where people are continuing the conversation by highlighting and responding to this story.

Here is the second and last part of my article about time series. In this part, I will dive into data filtering, the analysis of the components and prediction. You’ll find the first part which focuses on visualisation and statistics by following this link:

Time series: what tools are available in Python to analyse them?

As a reminder, I was interested in analysing time series (which can be defined by the chronological evolution of a quantity, regularly spaced in time). The most common time series you can think of would be the stock market. After the visualisation and statistical analysis of my data (which were left intentionally abstract), I wanted to filter my data since it seemed to be noisy.

Data Filtering

A filter ‘should eliminate all the undesired components’(Haslwanter, 2021, p. 72). Indeed, if the aim of the analysis is to create a model, ‘[t]he presence of noise in a data set can increase the model complexity and time of learning which degrades the performance of learning algorithms.’(Gupta & Gupta, 2019, p. 466). Noisy data can refer to ‘errors (incorrect examples), as well as outliers (correct examples representing some relatively rare subconcept of the target theory)’ (Gamberger et al., 2000, p. 206). Being able to handle such data is crucial if you want to recognise and analyse the pattern present. It is, however, really important to understand that it might reduce the precision of the data, but in some contexts, you might be ‘more concerned with establishing whether the data exhibits a meaningful relationship, rather than establishing its precise character.’(Janert, 2011, p. 48). Certainly, ‘A smooth curve such as a spline or LOESS approximation is only a rough approximation to the data set — and, by the way, contains a huge degree of arbitrariness in the form of the smoothing parameter (α or h respectively).’(Janert, 2011, p. 57). Of course, you should pay attention to the context and, according to it, make the best judgment. Here, I will use two smoothing algorithms: one that I will implement by myself, and one available in the library SciPy, I will compare the results and see how a filter can drastically change the function and its components.

To choose the best fit for the filter, it is important to consider some criteria. The first one would be to be sure that the data has been sampled at equal intervals which is true for those values. In such a case, ‘FIR- or IIR-filters can be used.’(Haslwanter, 2021, p. 90) otherwise, I would need to use more complex smoothing algorithms such as LOESS or splines which handle the filtering of irregularly sampled data.

An Infinite Impulse Response (IIR) filter makes more sense in a dynamic system, I will therefore be using a Finite Impulse Response (FIR) filter. The FIR filter has no feedback in its equation which makes it very stable. First, I will be using the moving average filter, which is a special case of the FIR filter, and, as the name suggests, they both have a finite impulse response. This method places more importance on the recent data and is very good to reduce random white noise.

Here is the function weighted_moving_average() that was used.

def weighted_moving_average(x,y,step_size=0.05,width=1):
    bin_centers = np.arange(np.min(x),np.max(x)-0.5*step_size,step_size)+0.5*step_size
    bin_avg = np.zeros(len(bin_centers))

    def gaussian(x,amp=1,mean=0,sigma=1):
        return amp*np.exp(-(x-mean)**2/(2*sigma**2))

    for index in range(0,len(bin_centers)):
        bin_center = bin_centers[index]
        weights = gaussian(x,mean=bin_center,sigma=width)
        bin_avg[index] = np.average(y,weights=weights)

    return (bin_centers,bin_avg)

In the visualisation below, we can compare the result between our function untouched and the result of the weighted moving average algorithm.

Another kind of FIR filter is the scipy.signal lfilter whose result is given in the graph below.

It is visible that the two algorithms give different answers which vary as well greatly according to the parameters input. Choosing which one is the best and which parameters to use will depend vastly on the context. We can see with those two results that smoothing ‘reveals behaviour that would otherwise not be visible’ (Janert, 2011, p. 52) and confirms what the visualisation had suggested that there is a recurrence of rises followed by drops around the mean.

Some interesting insights from our lfilter results are given from the summary statistics. If I use again the describe() function, I obtain those results:

Compared to the original function, it is visible that our mean and median are closer even to zero and that the minimum and maximum have been drastically reduced. Therefore, the range of the function is considerably narrowed. As for the distribution, I get the following histogram which seems to be closer to a normal distribution but slightly right skewed with more data in the negative.

To know if those results are in concordance with what you want to achieve you will once again need to check what are the objectives of the study and the model.

Analysis of the main components of the time series

Stationarity

An important part of the analysis is to know if the data is stationary that is, if its ‘statistical properties do not change over time’ (Peixeiro, 2022, p. 39). This can be said true if it has a constant mean and constant variance over time. To calculate and visualise this, I used this code:

yy_average = smoothed_function.rolling(window=50).mean()
yy_std = smoothed_function.rolling(window=50).std()[2]
plt.ylim([-0.5, 0.5 ])
yy_std.plot()
yy_average.plot()
plt.show()

Plotting it using matplotlib, we get the result visible for the original function in the figure on the left and on the right for the smoothed function.

Neither the mean nor the variance is changing too drastically, except in the smoothed function where the mean goes slightly more in the negative around the last values. We can therefore say that our data is stationary which is important to know since many forecasting models assume that our properties don’t change over time. This indeed makes sense because if the statistical properties change over time so should the properties of the model. If the data is not stationary ‘We cannot possibly derive a function of future values as a function of past values, since the coefficients change at each point in time’ (Peixeiro, 2022, p. 39). If the data is not stationary, one possibility to stabilise the mean is to use differentiation and for stabilising the variance to apply a logarithmic function.

Autocorrelation and Seasonality

Another useful tool to understand time series and more specifically its seasonality is autocorrelation which is the matching of a signal with itself over different time delays. It ‘can be used to detect periodicity in a signal which may be impossible to see otherwise’(Haslwanter, 2021, p. 114).

The autocorrelation shows if there are periodical events that impact the data and the strength of the relation between the current data and the past ones. To do that I will use the statsmodels library and specifically its plot_acf() function, the plot on the left shows the result for the original function while the one on the right shows it for the filtered version.

The cone in transparent blue gives the confidence interval (here set at 95%). Therefore, if a point is outside the blue zone, we can say with 95% probability that it has an impact on the current value while if it is inside the blue cone we can consider it equal to zero (Peixeiro, 2022, p. 47). The point at lag 0 is 1 since it reflects how much the current value explains itself. For the original function, we can see that there seems to be no seasonality visible as the coefficient decrease directly at lag 1 under the confidence interval. For the smoothed function, it is a bit different: we see a correlation in the first values but, as we progress, the current values are explained less and less by the previous ones. We can see a dip in the negative that is statistically relevant which says that in the function ‘if a past value is above average the newer value is more likely to be below average (or the other way round)’ (Drelczuk, 2020). This could be the indicator that we are confronted to a mean reversion where values tend to revert towards previous levels after large moves. We can see as well that it has seasonality even though it is not significant enough to take any conclusion and, of course, without a time measure given to the x-axis it is difficult to conclude, but a cycle seems to be forming where the data goes from positive to negative in what can be described as 15 seconds/days/months/years…

Seasonal Decompose

The statsmodels package has another very useful feature, called seasonal_decompose(), that actually lets you decompose the time series into its main features: trend, seasonality and residual. Applying this to the smoothed data, I obtain the results visible in this figure below.

The ‘seasonal’ plot shows how our data deviate from the general trend. It has regular spikes going above the trend while the main part of the data is staying below. As for the trend, which represents the ‘slow-moving changes in a time series’ (Peixeiro, 2022, p. 6), we see that it seems to stay around the mean (a bit below zero for the smoothed function), even though the last 50 points seem to show a tendency to decrease.

Prediction

Those precise descriptions of the function will help go toward the final step which is forecasting i.e. ‘the prediction of the future using historical data and knowledge of future events that might affect our forecasts’ (Peixeiro, 2022, p. 6). In order to do so we will need to take into account the features that we have underlined, especially seasonality which is known to have a significant impact on time series.

One of the methods used to predict future results is a random walk. To be able to use a random walk the data must be stationary and I have shown previously that the original function is purely random with no visible trend or seasonality with tests such as the ACF plot, the running average, etc. Therefore, a random walk would be possible in this context. However, those values seem to be too noisy to really see a point in doing so.

Another option that is suitable if the data is autocorrelated is the moving average. Indeed, in that context, ‘The present value is linearly dependent on current and past error terms.’ (Peixeiro, 2022, p. 88). However, for this smoothed data, the autocorrelation coefficient falls too slowly to use this method and I have repeatedly shown that it seems to have a sinusoidal pattern which makes this model inappropriate. Therefore, taking into account those characteristics, an autoregressive model could be the best answer to predict this set of data. This process ‘establishes that the output variable depends linearly on its own previous values’ (Peixeiro, 2022, p. 91) which means that I will use lagged values to make our prediction.

In this case, I will be using the SARIMA model which is a more powerful extension of the ARIMA (Autoregressive Integrated Moving Average) that supports the seasonal component of the data. Ideally, to find the best parameters for the model, I would need to use all the elements gathered previously. However, it is, of course, quite difficult to set proper parameters without a real time frame on the x-axis. But I still managed to get a realistic result by using the pmdarima library which helps to set the best parameters for the Arima models. The result is visible below. It does seem to make sense since the predictions for the value above the 400th point seem to regress toward the mean which has been a feature of the data that I have shown previously.

And that’s it for the time series. I tried in those two articles to show how to analyse such data with the help of some of the great libraries available in Python. Follow me for more!

Time series: what tools are available in Python to analyse them? was originally published in Geek Culture on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article, which I will divide into two parts, I would like to focus on time series, which is the chronological evolution of a quantity, regularly spaced in time (hourly, monthly, yearly…). They can represent the evolution of the temperature over time, the evolution of the stock market, the monitoring of the heart rate, etc. In any data analyst’s career, processing the information contained in time series is a crucial and recurring part of their work. However, these can sometimes be confusing and difficult to read. So how do you proceed with this data?

Faced with this challenge, I decided to turn to the literature written on this topic, so this article will be sprinkled with many references that helped me to better understand this subject.

Since its creation, Python has managed to gather around it a growing community of scientists and analysts. That can be attributed to its easiness to code, open-source frameworks and libraries that have been developed which allows Python ‘to perform statistical computing, machine learning tasks, develop websites, and much more’ (Peixeiro, 2022, p. 1). Indeed, several of those libraries greatly facilitate the work of any researcher by automating complex procedures and offering many tools to visualise and calculate data. This is, of course, true as well for time series.

To help you understand the process better and to make it as concrete as possible, I propose working with concrete figures taken without context. I chose to leave these values deliberately abstract to be as universal as possible, although obviously knowledge of the sector in which they were taken would be part of a thorough and complete analysis and would likely allow me to conclude more quickly which method would be most relevant and effective. As Chatfield, in his book The Analysis of time series: an introduction, confirms ‘The context of a given problem is crucial in time-series analysis’ (Chatfield, 2004, p. 11).

1. Visualisation

In any analysis, the visualisation of data should be the first step. This is confirmed by many professionals: ‘Making plot and static or interactive visualizations is one of the most important tasks in data analysis.’(McKinney, 2018, p. 219) Janert in his book Data analysis with open source tools explains: ‘Looking at data, you will notice things — the way data points are distributed, or the manner in which one quantity varies with another, or the large number of outliers, or the total absence of them.’(Janert, 2011, p. 1)

This is revealed to be of even more importance in time series as Chatfield confirms: ‘Anyone who tries to analyse a times series without plotting it first is asking for trouble. A graph will not only show up trend and seasonal variation, but will also reveal any’ wild’ observation or outliers that do not appear to be consistent with the rest of the data.’ (Chatfield, 2004, p. 6)

If I plot the data into a scatter plot (figure 1), it seems that no specific trend is visible at first glance.

However, if I use a line graph instead as shown in the graph below, I get a totally different picture. The line graph, by showing the change between two points, lets us glimpse at what seems to be a ‘regularly recurring feature’ (Janert, 2011, p. 80), looking almost like a sinusoidal function. This, combined with the amount of surrounding noise, makes the scatter plot confusing. The line graph can be useful when we need not only to see a general trend (which in this case is too confusing) but as well to distinguish the local trend between a group of points (‘Line Graphs and Scatter Plots,’ 2005). Here it allows us to see the repetitive pattern. Choosing the visualisation correctly is therefore an important step and different criteria have to be taken into account, such as ‘the amount of noise and outliers in the data’ (Wang et al., 2018) or if the data is discrete or continuous.

The visualisation gives us the first hint at one of the main components of our function: the presence of seasonality. As for the trend (another important component of the time series) which is defined as a ‘long-term change in the mean level’ (Chatfield, 2004, p. 12), it doesn’t seem to have one. However, we will need to dive a bit more into our function to be able to make that clear.

2. Statistics

Halswanter says in his book that ‘[s]tatistics provides us with the tools to extract the maximum amount of knowledge from a given data.’ (Haslwanter, 2021, p. 123) Summary statistics can therefore be of great help in our understanding of the function. It is, however, important to take this information with caution as Chatfiled warns to be very careful with summary statistics of time series as they can be misleading (Chatfield, 2004, p. 11).

The data consists of 400 points, with the x-axis going from -20 to 20 and the y-axis from -0.511 to 0.486. It has a mean value of -0.019. Each value on the x-axis has only one corresponding value on the y-axis and they are equally spaced. Using the describe() function which returns the statistical summary of our dataframe, I get the following results:

The mean (-0,019) and the median (-0,010) are close together which implies that our distribution is symmetrical and might be normally distributed. However, a quick glance at a histogram plot tells us differently: the highest number of values are found at the extremities but with a general homogeneous distribution in each of the bins. We therefore have data that are symmetrically distributed around zero and oscillating between -0.5 and 0.5.

However, thanks to the line graph, it is obvious that the data seems noisy. To further understand the graph, I would need to get rid of this feature by filtering it. And that's what I will be doing in the next article. Be sure to follow to get the next parts where we will focus on filtering, analysis and prediction!

Thank you for reading!

Time series: what tools are available in Python to analyse them? was originally published in Geek Culture on Medium, where people are continuing the conversation by highlighting and responding to this story.

By the way, if you’re interested in the easiest French songs, my article can be found here:

How can NLP help French learners find songs that match their current level?

I have more or less followed the same process, therefore I won’t dwell so much on the code in question, but rather on the results. Some new features though: I used the MusiXmatch API to find the music's genre and get the singers' nationalities. The procedure to register for an API key is available by following this link.

Exploratory Data Analysis

My dataframe consists of 4582 songs by artists from 38 different countries. The most represented genres are pop (1139 songs), Latin (505 songs) and Latin Urban genres (275 songs).

As mentioned earlier, the novelty in this analysis is the addition of nationalities. Despite a large number of missing data, we can still see a clear top three with the USA (862 songs), Spain (671 songs) and Mexico (596 songs).

In the same way as for French songs, the vocabulary remains linked to love (‘amor’, ‘amar’), life (‘vida’) and desire/love (‘quiero’). Compared to French lyrics, I do have the impression though that there is a greater emphasis on the body in these songs with a regular occurrence of terms like ‘labios’ (‘lips’), ‘besos’ (‘kisses’), ‘piel’ (‘skin’), ‘sentir’ (‘feel’), ‘boca’ (‘mouth’), ‘cuerpo’ (‘body’).

The artists with the most songs are Julio Iglesias (78 songs), Shakira (62 songs), Daddy Yankee (58 songs), Wisin and Yandel (53 songs) and Enrique Iglesias (53 songs).

Classification

This time for my vocabulary I used the frequency list offered by the Real Academia Espanola (REAL ACADEMIA ESPAÑOLA: Banco de datos (CORDE) [en línea]. Corpus diacrónico del español. http://www.rae.es [22.06.2022]). A major difference between this list and the one used in French is that it contains not only the basic form of a word but all its forms classified according to their occurrence. Thus, we find “soy” (‘I am’) as well as “fuimos” (‘we were’) or “ser” (‘to be’), whereas my previous list was limited to the infinitive verb (here it would have been ‘ser’).

In order to estimate how many words I should include in each level, I based myself on this article. In this way, I have again created four lists of words divided into A1, A2, B1 and B2. For the rest, the process is quite identical to the French songs (creation of a matrix with CountVectorizer, the insertion of the level of each word in this same matrix, creation of a dictionary with the percentage of words per level for each song, …)

Results

Here are the results of the analysis. First, you can see here a graph showing the distribution of songs according to their lexical range: either A (A1 to A2) or A + B (A1 to B2).

And finally here are the three lists of songs: one with the highest percentage of known words for A1 level, then for A2 level and finally for B2 level.

TOP 10 songs for A1 level

• Sueño de Noche by Gipsy Kings (genre = Latin)
• ¡Qué bien! [Hot Dog Dance] (Latin Spanish) by Mickey Mouse Clubhouse (OST) (genre = Children’s Music)
• La despedida by Shakira (genre = Folk)
• Te quiero tanto by OV7 (genre = Pop)
• Toma mi vida by Milly Quezada (genre = Merengue)
• ¿Dónde Están, Corazón? by Enrique Iglesias (genre = Pop)
• Así es la mujer que amo by Victor Manuelle (genre = Latin)
• Más by Nelly Furtado (genre = Pop)
• 5 razones by Manu Chao (genre = World)
• Nada ni nadie by Soy Luna (OST)

Again I created a Spotify playlist with some A1 level songs from the corpus (almost 3 hours of songs).

TOP 10 songs for A2 level

• Tú estás aquí by Jesús Adrián Romero (genre = Christian & Gospel)
• Reflejo de Luna by Alacrán (genre = Electronica)
• Eres tú by Mocedades (genre = Pop)
• A Lo Mejor by Banda MS (genre = Pop)
• Mi primer amor by Ender Thomas (genre = Latin)
• Lo que yo sé de ti by Ha*Ash (genre = Pop)
• Me voy by Yasmin Levy (genre = Pop)
• Por qué será by Xtreme (genre = Salsa y Tropical)
• Amor Genuino by Zion & Lennox (genre = Latin Urban)
• Renuevame by Christian Hymns & Songs (genre = Christian & Gospel)

Here is the playlist for the A2 level.

TOP 10 songs for B2 level

• Porque by Yasmin Levy (genre = Raices)
• Dame tu mano by Glorya (genre = Electronica)
• Los adolescentes by Dënver (genre = Pop)
• La incondicional by Luis Miguel (genre = Pop)
• Vayamos compañeros by Marquess (genre = Pop)
• Años by Mercedes Sosa (genre = Latin)
• Amor, amor, amor by Julio Iglesias (genre = Pop)
• You are my hiding place by Selah (genre = Christian & Gospel)
• Vai (LLP Remix) by Muneca (genre = Pop)
• Amada amante by Roberto Carlos (genre = Brazilian)

Finally, the playlist for the B2 level.

If you rather want to search by yourself according to your preferences, you can use the table below to browse the whole corpus. You can either search by artist, title, genre or nationality. You can also classify the list according to the lexical range: A1-A2 or A1-B2.

If you are interested in the full code here is the GitHub repository. Thank you very much for reading!

Top Songs to Learn Spanish According to Data Science was originally published in Geek Culture on Medium, where people are continuing the conversation by highlighting and responding to this story.

After reading the series of articles written by Frank Andrade who used data science tools to classify more than 3000 films and tv shows according to their language level, I thought I would try to do the same with French songs.

The aim of this analysis is therefore to classify the songs according to their CEFR level. However, how do I evaluate the level of a text? Obviously, there are many ways to work on this problem (grammar, length, readability, pronunciation,…) As I see this as a tool to develop language comprehension and improve vocabulary, I will focus on the lexicon used. In this article, I will show you how I worked on this challenge before revealing the most appropriate songs per level. If you’re only interested in the song list, you can head directly to the result section.

What information do we need?

So I have to go looking for the data, but what information is essential? At the very least, I need the text, the artist and the title of each song. To this, I would also like to add the genre, as learners don’t realise the actual diversity of the French music scene, believing that it’s relatively small and limited to the artists that cross the borders of French-speaking countries. Generally, the learners are only presented with pop songs or older songs from famous singers like Edith Piaf. The fact that you’re learning French shouldn’t force you to listen to pop songs for hours if you like metal. In addition to all these elements, I also need a list of words that are classified according to their level (i.e. A1 to C2).

Getting the Data

For the first part, I have collected all the songs categorised as ‘French’ on the Lyricstranslate website. I realise, however, that this implies an important bias in my corpus, which is that these songs have been already translated into other languages by the users (which would therefore imply that they are already popular with non-French speakers). Despite this, I still favoured it because it provided a piece of essential information: the popularity of the song. Indeed, when sorting the songs, you can choose to order the list by popularity. The index of my corpus would thus serve as a basis to know their popularity level.

After collecting the songs, I was still missing the genres. To do this I created several lists: rock, pop, chanson francaise, metal, electro, folk/children’s music, and film music. In each list, I added the artists corresponding to the genre.

Finally, all I needed was a list of words classified by level. However, unlike English where it’s quite easy to find a vocabulary list classified by levels from A1 to C1, in French, I couldn't get one and had to create it. To do this, I discovered two lists on the internet that would give me an overview of the vocabulary that a learner of French should know. The first one is from the French Ministry of National Education and Youth which gives a list of the 1500 most frequent words in French, and the other is from Memrise where you can find a list of 5000 most used words (according to the creator of the list, so it has to be taken with caution). I will explain below how I used those two lists to classify the words by level.

So I started my work with a list of 5000 songs, several lists of artists classified according to their genre and a list of over 6000 French words ranked according to their frequency. All of them in their own CSV files.

Data wrangling

The first step is to load the songs' lyrics CSV file into a DataFrame and check its information with df.info() and how it looks like with df.head() here are the results:

I notice that I’m missing some lyrics and many album titles. I as well have a column ‘Unnamed: 0’ that I will get rid of. Looking more specifically at the text of the songs, I can see that there are some elements that would have to disappear before being able to lemmatise the text correctly (HTML remains, line breaks, punctuation, …) I used the replace method with a regular expression pattern to do so (for example, <.*> would be the regex pattern to delete all the remaining HTML elements like <div>or <em>. Here is the complete code:

I had to add the 178 missing songs manually. However, I limited myself to the songs that were in the top 300 since I was mainly interested in the ones that are popular. It was principally songs from Stromae, Gims and Jacques Brel. I as well added the genre corresponding to each artist. Also, by looking at some of the songs I quickly noticed that some of them weren’t in French. This isn’t dramatic since they would simply be at the bottom of the ranking but I still wanted to see if a Python library could tell me the language of the text. I was happy to discover the Langdetect library which allows you to do just this. Thanks to langdetect, I created a function that returns True if the song is in French and False if it’s in another language. This allows me to create a new column on my df that stores this. The next step is simply to remove from my df all rows whose value for this French column is False.

Before moving on to lemmatisation, I also wanted to create a final column that contained the number of words in each song. This would be useful to better understand my data and to get the percentage of words in each level.

The next step is lemmatising my text, but what is lemmatising?

The action […] of giving (a word) the canonical neutral form that it has, for example, in a dictionary (Le Robert)

Component for assigning base forms to tokens using rules based on part-of-speech tags, or lookup tables. (Spacy)

Indeed, to avoid our classifier counting ‘bon’ (good, masculine singular), ‘bons’ (good, masculine plural), ‘bonnes’ (good, feminine plural) as different words, it seems more coherent to keep only the neutral form. Of course, this creates some problems, such as the fact that a student may know the indicative form ‘vouloir’ but not its subjunctive ‘veuille’. However, I think the positive impacts outweigh the negatives.

In order to be able to lemmatise each of the song lyrics, I used Spacy’s lemmatizer which offers nice performances in French and I created a function to add a new column called ‘lemmatised ’ to my data frame.

To give a concrete example, the first words from the song ‘La vie en rose’ by Edith Piaf go from “Des yeux qui font baisser les miens un rire qui se perd sur sa bouche” to “un, oeil, qui, faire, baisser, le, mien, un, rire, qui, se, perdre, sur, son, bouche”. After all this preliminary work, my data frame is ready for analysis. Here is what it looks like after this first part. Now we have 4632 songs (after dropping the nan) distributed among these 7 columns:

Understanding the data

Before starting the analysis of the levels per song, let’s try to understand our corpus a little better. First, we can look at the most used words in our list. To do this I used a word cloud and removed from the list the so-called stop words, i.e. words that are used frequently in the corpus and don’t add any meaning. I used the stop-word list given by Spacy to which I appended a few words that were high in the ranking but didn’t add weight to it (‘j’, ‘c’est’, …) I ended up with this words cloud.

We can see that words such as ‘amour’, ‘aime’ (‘love’), ‘vie’ (‘life’), temps (‘time’), ‘veux’(‘want’) are very prominent in the songs.

I then created a bar chart ranking the artists by the number of songs they had in the corpus (Note that technically Jacques Brel is in the lead with more than 100 songs, but since many of his songs weren’t ranked in the top 300, I didn’t add them to the corpus and therefore removed them from the data frame before lemmatisation).

Then I tried to find out which music genre had the most words on average. We can see that it’s rap music which is largely in the lead. Indeed if we look at the mean number of words per artist, we can see that Fonky Family has an average of 1186 words per song, followed by Kery James (1105 words) and Mafia K’1 Fry (1089 words), the three of them are rappers or groups of rappers.

Classification

After these first inputs that allow us to better understand our dataset, we can move on to the classification part. However, we’re still missing an important element which is the list of words per level. As noted earlier, I already have a list of 1500 words classified by the frequency of use. In addition to this list, I also have another list of 5000 of the most used words in French made by a Memrise user. But, how can I classify them by level?

After some research on the internet, I found this table which could help me:

We’re in a comprehension exercise so I will be more interested in the ‘passive vocabulary’ part and I will limit myself to levels A1, A2, B1, and B2 since I have around 5000 words. I think it’s logical to assume that for A1 and A2 levels, it’s better to know words that are widely used. I will therefore populate these first lists with the word frequency list of the French Ministry of Education. Then, after removing the words that are already present in the frequency list, I take the Memrise list and complete what is missing to reach the quotas of the different CEFR levels. As the Memrise list wasn’t made by a professional, the difference between levels B1 and B2 will be taken with a pinch of salt.

So now I have four lists of words which add up to 5000 words divided between all four levels as shown below.

I can finally start working on the classification.

In order to do this, I’m going to use the scikit-learn count vectorizer that I discovered thanks to Frank Andrade’s posts. The count Vectorizer ‘convert a collection of text documents to a matrix of token counts’ (scikit-learn documentation).

Once I have this matrix (26551 rows over 4633 columns!), I add to it the level linked to each word and I get the table below. We can see, for example, that the word ‘alors’ (then) appears thrice in the song at index 3 and twice in the one at index 5 and is ranked level A1.

Then I create a table grouping the sum of words for each category using df_dtm.groupby(by=’level').sum() and I get a list of each of the songs with the number of words per category.

You can see in this table that the song at index 4 has 193 words from the A1 list, 17 from the A2 list, 6 from the B1 list and 4 from the B2 list. The last part consists of associating this information with that of the songs. To do this I create a dictionary which takes the index of the song as its key and a list of the four levels for each song as its value. The index allows me to link those pieces of information to the song that has the same index in the original df. I as well, divide the words per level by the total number of words in the song and get a percentage for each level.

You can see in the code above that I have also created two new columns: one called ‘A’ which covers the lexical range for A1 and A2 and a column called ‘A+B’ which covers the vocabulary from levels A1 to B2.

Results

Finally, we can analyse the results! Firstly, I have created a graph that allows us to see the distribution of the songs according to their level coverage, either A or A+B.

In the graph below you will find the average A1 to B2 level per music genre. Not surprisingly, poetry and chanson francaise are ranked first, while rap and rai have fewer words covered by the vocabulary lists created. This can be explained for rai (an Algerian musical genre) because many Arabic words are used in the lyrics. As for rap, this can be attributed to the use of many slang words, neologisms or words of foreign origin.

Then I created three lists of songs: one with the highest percentage of known words for A1 level, then for A2 level and finally for B2 level. To do this, I collected the songs that ranked highest in terms of the level chosen, and from A2 level onwards I also removed the songs that were already in the previous lists. I thus obtained these three lists.

Top 10 best songs to learn French if you’re at A1 level

• ‘Il faut manger’ by Manu Chao (genre = pop)
• ‘Qu’avons-nous fait de vous?’ by Le Roi Soleil (genre = comedie musicale/film)
• ‘La poupée qui fait non’ by Mylène Farmer (genre = pop, chanson francaise)
• ‘Elle’ by Melissa M (genre = R&B)
• ‘Tu me manques’ by Mia Martina (genre = pop)
• ‘Reste-là’ by Keren Ann (genre = pop,chanson francaise)
• ‘Tu te reconnaîtras’ by Anne-Marie David( genre = pop)
• ‘Attendre ’ by Céline Dion (genre = pop, chanson francaise)
• ‘Que sont devenues les fleurs’ by Dalida (genre = pop)

I as well created a playlist on Spotify with a selection of A1 level songs.

Top 10 best songs to learn French if you’re at A2 level

• ‘Je te laisserai des mots’ by Patrick Watson (genre = pop,classique)
• ‘Pas avec toi’ by Plastiscines (genre = pop)
• ‘Pour une fois’ by Marie-Mai (genre = pop)
• ‘L’intranquilité ’ by Louise Attaque (genre = pop,rock,chanson francaise)
• ‘Te passe pas de moi lyrics’ by Judith (genre = pop)
• ‘Tu me manques’ by Sheryfa Luna (genre = pop)
• ‘Le temps des fleurs lyrics’ by Vicky Leandros (genre = pop)
• ‘Tes yeux noirs’ by Indochine (genre = pop,rock,chanson francaise)
• ‘Fais ce que tu voudras lyrics’ by Céline Dion( genre = pop,chanson francaise)
• ‘Sous le vent’ by Garou (genre = pop,chanson francaise)

Here is the Spotify playlist for the level A2.

Top 10 best songs to learn French if you’re at B2 level

• ‘Ne partez pas sans moi lyrics’ by Céline Dion (genre = pop,chanson francaise)
• ‘Si la vie est cadeau ’ by Corinne Hermès (genre = pop,chanson francaise)
• ‘Mon Dieu’ by Édith Piaf (genre = chanson francaise)
• ‘La taille de mon âme’ by Daniel Darc (genre = rock)
• ‘Maman me dit’ by Angélina (genre = pop)
• ‘Un ange est tombé’ by Lara Fabian (genre = pop)
• ‘Ce que je sais’ by Johnny Hallyday (genre = pop,rock)
• ‘Pourquoi tu vis et où tu vas’ by Jeanette (genre = pop)
• ‘Ma référence’ by Jena Lee (genre = pop)
• ‘Le verger aux petits’ by Bastien Lallemant (genre = chanson francaise)

And the final playlist.

If you rather want to search by yourself according to your preferences, you can use the table below to browse the whole corpus

I’m quite happy with the result. However, there are a few things to consider that could improve this list. Firstly, to rate the level of a song only on the lexicon is rather reductive. It could be interesting to add grammar points, and verbal tenses into the criteria to evaluate the level of a song. Secondly, pronunciation is an important element that has been overlooked. Finally, ideally, it would be interesting to be able to classify the songs by theme. If we’re working on the theme of ecology, having a few songs that correspond to both the level of the learner and the topic being addressed would be an invaluable tool.

You can find the complete code on my Github: https://github.com/StMaCre/french_song_level_analysis. Thank you for reading!

How can NLP help French learners find songs that match their current level? was originally published in Geek Culture on Medium, where people are continuing the conversation by highlighting and responding to this story.

For my work, I needed concrete data on the children’s literature sector in Lithuania. However, despite some research on the net, apart from a few articles (available in the sources of this document), I could not find any precise figures.

I am also training in data analysis with Python and specifically wanted to learn more about web scraping. I could therefore get around this lack of resources while developing my understanding of beautiful soup (a Python library for parsing HTML) and selenium (a tool for controlling web browsers).

In writing this article, I want to show a concrete case of web scraping and how it can help us to obtain data that is otherwise difficult to acquire. Those data being the basis for developing a relevant analysis. Clearly, my topic was very specific and I could not limit myself to a generic idea of the field.

I had concrete questions I wanted to answer: what is the average price of a children’s book, how many categories exist, which books are the most popular, and what is the proportion of Lithuanian authors in the selected books? In addition, I wanted to know if people tended to buy more children’s books during the Covid crisis. Since parents had to stay at home with their children during the quarantine, it is likely that they would purchase more books to find activities to do with their children.

To obtain this information I decided to retrieve the data available on the website knygos.lt (a Lithuanian version of Amazon). Indeed, on a single page, we can find almost 200 books for children and teenagers divided into 11 categories as well as a lot of other relevant information for our analysis.

1. Scraping the data

The first step is to check that the site can indeed be scraped by checking the robots.txt file. Once this is confirmed, I can start the actual scraping. I’ll begin with importing the needed libraries and creating the soup object.

Looking at the HTML file from the website, you can see that the class for the properties of the books is called “book-properties” divided into “.book-author”, “.book-title” and “.book-price”. To limit the selection to the grid containing youth literature, I need to select the class “.col-12” (otherwise we would include the promotions that are not related to our topic).

There is a star rating system for each book (going from 0 to 5). However, those evaluations are not very diverse (between 4 and 5 stars for the vast majority). In order to analyse the popularity of a book, the number of reviews given seems more relevant. This is available under the class “.badge-secondary”.

For data storage, I used a dictionary whose key is the index and the value is a list containing the name of the author, the title, the number of reviews, and the price.

The books are as well divided into 11 categories: Knygos mažiausiems (books for toddlers), Knygos vaikams (books for children), Knygos paaugliams (books for teenagers), Pažintinė literatūra vaikams (Educational literature for children), Pasakos (Fairy tales), Kakė Makė (a popular Lithuanian children’s series featuring the character “Nelly Jelly”), Lavinamosios, užduočių knygelės (Educational and activity books), Veiklos knygelės (Workbooks), Kalėdinės knygelės (Christmas books), Mokiniams rekomenduojamos knygos (Books recommended by schools), Smagioji edukacija (Fun Education).

On this web page, for each category there are 16 books, so we can easily add those categories to the dictionary we created earlier.

Great, we now have everything to create our DataFrame! However, we need first to clean the data a bit (fill in the missing reviews with “0” and change the data type to float).

We now are left with this DataFrame which contains 176 rows and 5 columns.

2. Analysing the data

Using the describe() function will help us obtain some solid information regarding our quantitative data (reviews and price).

2.1. Analysing the reviews

Let’s check what books and what categories are the most reviewed in our selection.

We can see that the most reviewed book is Between Shades of Gray by American-Lithuanian author, Ruta Sepetys. The categories receiving the most reviews are the ones for toddlers and the ones recommended for school.

2.2. Analysing the price

The next part of our analysis will look at the average price of a book according to its category and which books are the most expensive.

The most expensive category is fairy tales (pasakos), and logically the most expensive book is also in this category. However, it should be noted that this book is actually a collection of volumes, hence the high price.

2.3. Correlation between price and reviews

I’ve noticed that the more expensive books don’t seem to get a lot of ratings. So I would like to see if there is a correlation between the two with df_lithuania_no_outliers.plot.scatter(x=”price”, y=”review”, c=”DarkBlue”). Before that, I removed the collection of volumes which was a clear outlier with df_lithuania_no_outliers = df_lithuania.drop([73]).

We can see that the most reviewed books are between the 5 and 15 euros price range. However, as we have noticed with the describe() function, almost 50% of our books are in this range. It is still interesting, though, that the books that do not fall into this interval receive almost no reviews.

3. Nationality of the author

One question that interested me at the beginning of the analysis was to know the proportion of Lithuanian authors in this corpus. This seems to be a bit complex, without having to track the Lithuanian authors one by one. However, we can try to get around this by using one of the “quirks” of the Lithuanian language which is its pretty unique diacritical marks (“ąčęėįšųūž”). These may indeed help me to note which authors are likely to be Lithuanian (or of Lithuanian descent). I used the regular expression module (import re) to classify the authors into two groups.

I noticed that the search did indeed seem to admit mainly Lithuanians although at least two persons whose nationality is not Lithuanian were included in the selection (Pavla Hanáčková and Ester Dobiášová, both of Czech origin). However, I kept this approach as it gave me a good approximation. I removed Mrs Hanáčková and Dobiášová with the .remove() function.

The last change to the list is the addition of some common names particular to Lithuania (Vytautas, Linas) and finally the addition of Ruta Sepetys (author of Between Shades of Gray) whose dual nationality is perhaps responsible for the absence of “ū”.

Despite the fact that she lives in the United States, the general theme of her book is directly related to Lithuania. Therefore, it seems logical to me to include her among the Lithuanian authors. Here is the complete code :

Thanks to this list, we now know that the percentage of Lithuanian authors in our corpus is 19,32%. The next step would be to add a new column in our DataFrame which would have a “Y” if the author is Lithuanian and “N” otherwise. This new column allows us to see if Lithuanian authors tend to receive more reviews than non-Lithuanians.

With this calculation we see that books written by Lithuanians receive on average 9.56 reviews were non-Lithuanians receive only 4.39 reviews. However, we do know that there is an outlier: the book Between Shades of Gray which has 118 reviews. Let’s see how it changes if we take it out of our corpus.

Without Sepetys’ book, the average drops to 6.27 reviews which is still higher than the average for foreign authors.

4. Comparing the top seller's books from 2019 to the ones from 2021

On knygos.lt, you can see that a fairly large number of children's literature books are among the top-selling books of this year. I would like to know if the percentage has changed over the years and more specifically if the covid crisis has had any impact. However, our site does not give the top 100 from previous years. So we have to turn to another website www.knyguklubas.lt. This site requires JavaScript to be enabled, which Beautiful Soup does not permit. So we have to turn to another python library: Selenium.

To compare 2019 to 2021, I decided to get a list of children’s authors from this site and then analyse the percentage of authors in the top sales of 2019 and 2021.

So I went to the page dedicated to children’s literature (set to 96 books sorted by popularity). Then I scraped the authors from the first two pages, which gave me a list of 192 popular authors in the field of children’s literature. I got rid of the “Nėra Autoriaus” (no author) with a list comprehension which gave me a final list of 172 authors.

I then went to the page giving the most popular books of 2019 and added all the authors to a list and did the same for 2021.

Finally, I’m checking to see if there are any children’s authors in 2019 or 2021.

So, drum roll: out of a total of 95 books for 2019, there are no children’s authors present, whereas, out of 97 books, there are 18 books written by children’s authors in the list of best-selling books in 2021.

To be honest, this part with Selenium was quite time-consuming, and, to be completely frank, the result is obvious from a quick glance at the two lists on the website. However, this process allowed me to discover and get to grips with Selenium, which is still a positive point.

This brings us to the end of the analysis. I am quite happy with the result. Thanks to these web-scraping tools, I got data that was otherwise unavailable to me.

Sources

• Literatūra vaikams ir jaunimui
• Išaugo knygų pardavimas internetu, daugiausia - vaikų literatūros

https://www.knyguklubas.lt/literatura-vaikams

Here is the code on Github :

GitHub - StMaCre/web_scrapping_youth_literature_lithuania

How to use web scraping tools to gain insight into a specific sector? was originally published in Geek Culture on Medium, where people are continuing the conversation by highlighting and responding to this story.