Looking into MAL Database
Still alive exile? I was going to do a blog post on my Phase 1 Flatiron Bootcamp project regarding movie data and creating a suggestion on what the most profitable movie would be, or the time to release it. Unfortunately, it seemed like everyone was going far and beyond with it, writing about stats, web-scraping, or analyzing other databases. As a result, I felt that I also had to do something spicy, so I thought of something random such as “I wonder if the standards of manga or anime have gone down in recent times or are we just boomers who like stuff of the past?”.
To begin, luckily my fellow cohort members recommended this nifty website called Kaggle (Sounds like kegel heh) that contains countless databases to look and work with. I decided to use the “MyAnimeList Anime and Manga Datasets” since the information it sources from is probably the most reliable and largest database for data pertaining to manga and anime. For the most part, the data was really clean and consistent to work with so there wasn’t too much cleaning to do.
Overall, pretty nice just had to delete a lot of columns that I thought were irrelevant to what I was looking for. I kept the Title, Mean (Score), Num_Scoring_Users, Start_Date, Num_Chapters, and Genre's columns since I was looking to find a relation between the start years of manga in relation to the mean scores, number of users, and number of chapters. It was easily quantifiable data and it’s nice to see how manga has changed over the course of the years. I was also looking to find the relation between genres and the number of manga written for that genre, average rating, and chapter count. Both things tackled different aspects, so I had to create two different tables based off the main database that I had already filtered.
Before I step into all the crude oil of how I went about it, I’ll just show the fruits of my research since you might just be interested in what I’ve extracted out of it. Who cares about the coding anyways too, it’s all simple stuff!
First, we can see the average score by year chart, and it definitely does coincide with my belief that manga has become worse over the years. Of course, there is a bit of an issue as most manga in 2020–2022 don’t really have the time to develop into something amazing such as the older manga do. Nonetheless, new manga sucks bring back the old authors and illustrators who spent hours scribbling with a pencil to make one panel!
Average Manga Length by Years, while you can definitely see a trend of manga getting shorter and shorter; we also have to factor in that the year is the written start date. Something written in 2020 probably can’t even surpass 200 chapters in the first place unless it was written daily! But you can definitely see that manga in the past went far and beyond in the number of chapters; perhaps publishers weren’t so cut-throat or maybe the stories were written with longevity in mind.
Over here, we can see that manga really exploded near the start of 2010 and there was just a crazy number of manga being published during that time. It’s interesting to see how the market slowly grew and became bloated; since then, it seems like either the passion has died or the markets too small to sustain all these authors.
Alright, not much to see here but fun to know. Average rating doesn’t really differ by too much and award winning is a bit of a scam of a category. Now that I do see this graph, I should have set the Y parameter to be between 6 and 8 so there would be more of a visual difference between the genres.
Award winning you scammer, there’s a reason why you’re so high up there anyways. It’s nice to see that most sports manga tend to last a long time, and not surprising to see that Hentai (Whatever that may be) and Boys Love have the lowest average girth, I mean length. Whatever content is written in it probably isn’t written with longevity in mind…
Now we just have the total manga count by Genre. Comedy and Romance are the top two, while Sports and Girls Love are the bottom two. Award winning makes sense since there can only be so many award winners out there. It’s quite interesting that Boys Love has more than double of Girls Love, not that it implies anything…
ALRIGHT NOW THAT’S OUT OF THE WAY!
Let’s just step through the initial steps to create the database cleaned before it diverges into two separate databases that we used for those charts.
Step 1.
manga_df = pd.read_csv{'/Data/manga.csv')
#Reads the file and creates the databaseStep 2.
manga2 = manga_df[['title','mean','num_scoring_users','num_chapters','start_date','genres','authors']]
#Taking the columns, I need since deleting would probably take longerStep 3.
manga3 = manga2.dropna()
#Removing all null values, I did check to make sure a decent chunk of the database was still left!Step 4.
m1 = manga3.loc[manga3['num_chapters'] >= 10]
m1 = m1.loc[m1['start_year'].astype(int) >= 1980]
#Dropping stuff thats less than 10 chapters, we don't care about one shot or failures. Also dropping stuff older than 1980.Step 5. m1['start_year'] = m1.start_date.str[:4]
#This makes the year in a new column, just so it'll be easier to groupby later
Pretty nifty, everything is now set up in terms of what I wanted. There is a slight issue of genres containing every single genre out there and I won’t be able to immediately turn it to a chart for the genre relationships yet. But for now, I can focus on my first goal of just creating charts that show the relation of manga start years and the rest of the shenanigans.
Get Charles Pan’s stories in your inbox
Join Medium for free to get updates from this writer.
Well, more coding blocks
Step 1.
m1_years = m1.groupby(['start_year']).mean().sort_values(by='start_year', ascending=True).reset_index()
#This groups everything by the year, sorted and finds the mean value of everythingStep 2.
m1_yearscount = m1.groupby(['start_year']).count().sort_values(by='start_year', ascending=True).reset_index()
m1_yearscount2 = m1_yearscount[['start_year','title']]
#This groups everything by the year, but checks the count of each year; essentially it tells us the number of manga started that year, we also just want to remove every unnecessary column afterwardsStep 3.
m1_years2 = pd.merge(m1_years, m1_yearscount2, on='start_year', how='inner')
m1_years2['num_of_mangas'] = m1_years2['title']
del m1_years2['title']
Merging the two databases to get the complete picture and basically renaming the title column to the num of manga.
Now our first database for graphing is complete! It looks like this, pretty cool. All the data looks good, 2022 only has one manga written during that time but it seems reasonable since that’s the current year and we’ve filtered out trash; I mean low chapter manga.
Alright, let’s take a step back and go back to the initial database before we made m1_years2 because we need to fix it for the genres. Here are the steps taken below.
Step 1.
m1['genres'] = m1.genres.str.replace('[','')
m1['genres'] = m1.genres.str.replace(']','')
m1['genres'] = m1.genres.str.replace("'",'')
m1['genres'] = m1.genres.str.replace(" ",'')
m1['genres'] = m1['genres'].str.split(",")
# Removing all the unnecessary characters, then splitting the string afterwards to turn those genres into a ListStep 2.
m1 = m1.explode('genres')
# Explode is a real nifty function that takes the list and turns it into multiple columns for each entry of that listStep 3.
m2_g = m1.groupby('genres').filter(lambda x: len(x) > 200)
#Gotta remove the tiny genres cause I was getting like 100+ genresStep 4.
m2_genres = m2_g.groupby(['genres']).mean().sort_values(by='mean', ascending=False).reset_index()
#Just grouping by the genres that are now separate and getting the mean value for other stuffStep 5.
m2_genrescount = m2_g.groupby(['genres']).count().sort_values(by='mean', ascending=False).reset_index()
m2_genrescount2 = m2_genrescount[['genres','title']]
#We did this to the last one, just gets us the count of manga for each genre basicallyStep 6.
m2_genres2 = pd.merge(m2_genres, m2_genrescount2, on='genres', how='inner')
m2_genres2['count'] = m2_genres2['title']
del m2_genres2['title']
#Merging the databases together and renaming it, we're ready now!
Now below is our second database that we can start generating these charts off. It looks pretty good to work with and everything seems about correct.
Using these two databases, the charts were made above that displayed the results. Honestly nothing too big although the explode was kind of nice courtesy to my phase 1 project member [REDACTED]. It was fun to kind of see the data.
Below is my repository I used if you’re curious in how many times I wrote .describe(), .head(), and other repetitive things. Sorry, this wasn’t anything too complicated! I’ll do better next time!!!
charlesmpan/WeebStuff: Material for Blog Post 2 (github.com)
