What Working with Real-World Data Taught Me About Data Science
One aspect that is often overlooked is how different classroom work can be from getting hands-on experience with real-world data.
I remember interviewing for the position of Associate Data Analyst at a startup focused on climate technology. The company was working on converting waste sugarcane bagasse into biochar through a process called pyrolysis.
I knew I loved working with data, but honestly, I was nervous. I made it through the first round, answering questions as best as I could.
How I made it to the second round is a story for another day.
But I did make it, and, with time, I not only grew into the role but was eventually promoted to a higher position.
Well, never mistake nervousness for being unprepared.
The next several months would test my resilience in ways I couldn’t have imagined.
There were machines breaking down despite being barely a year old. There were processes that needed to be created, improved and documented. I progressively built SOPs to keep everything running.
And somewhere in the middle of all this was one of the most important things: collecting accurate data.
That was when I began to understand something I had not fully appreciated in the classroom.
Real-world data is messy.
It does not always arrive neatly formatted in a CSV file.
Sometimes information is missing. Sometimes a measurement doesn’t make sense. At times, the IoT system responsible for generating the data stops working. Other times, the problem is simply operator error.
Get Olivia A.O’s stories in your inbox
Join Medium for free to get updates from this writer.
And yet, within that mess, there were patterns. The more data we collected, the clearer those patterns became, obvious and hard to ignore.
The data could reveal trends in machine performance, production trajectory, operator activity and equipment downtime.
It became a way to understand machine health. Did the equipment need preventive maintenance? Was production becoming less efficient? Where were operational problems occurring?
Data had become an integral part of our major decision-making processes.
This was different from working with a dataset in a classroom.
I had to deal with missing information, operational problems, data quality, decision-making and communicating results to people who didn’t necessarily care about the code behind the analysis.
They cared about what the numbers meant.
And that, to me, was one of the biggest differences between learning data science and doing data science.
It’s been about a year now.
I have spent time building dashboards using SQL queries, writing Python scripts and working with operational data to understand what is happening on the ground.
In all these, what have I learned?
Data is not just about numbers.
Data is about turning numbers into information that someone can actually use to make a decision.
For me, it’s always been data and my story is data.
And somewhere along the way, it has also become about building beautiful, aesthetic dashboards that turn that data into something people can understand and use.
This is only the beginning of that story.
