Engineering Journal: SDC
Day: 3/3 SDC
Challenge: Data needs to be loaded into the database
Action: I spent nearly half of my day trying to figure out different ways to load all 4 of the csv files into the database.
Get jesse kovash’s stories in your inbox
Join Medium for free to get updates from this writer.
I explored building schemas with mongoose. Some preliminary research down this path showed that mongoose can lead to poor performance at scale.
I considered using a csv parser and a script to help load all of the data into the DB. This also seemed to be an inefficient process since the csv files were huge. One of the files had >22,000,000 lines.
Fortunately, I stumbled upon mongoImport. This command is simply run in the command line of the terminal. It takes the csv file and creates a collection automatically. It takes a few parameters (database name, collection name, file type, headerline, and file location). The headerline parameter tells mongoImport to create fields based on the column names present in the CSV file. mongoImport also streams the write operation so that the memory is not exceeded.
Result: I was pleasantly surprised with the performance and ease of use with mongoImport. It took less than 45 minutes to load all 4 of the huge csv files (>3GB) total.
Next: Now the fun part. I need to rearrange my data a bit so that multiple queries won’t be needed.
