In my opinion, it’s a must-read for all data analysts or those looking to enter this field. It is also more relevant than ever given the recent development of AI and tools like ChartGPT can easily perform the tasks of entry-level analysts.

Here are a few key things I learned from Cassie’s articles on how to be a professional and valuable analyst.

1. Tools are easy to learn, but adopting the analytics mindset is not.

Cassie talks about what she called “data science bias” — trusting information more when it smells of the data sciences. Adding a fancy chart to the data or building a sophisticated model from it doesn’t make it any more special.

It’s important to keep in mind that data does not inherently possess truth or wisdom. We the analysts should use it as a tool to help understand the business and not treat it as an infallible source of truth.

Data is created by humans with biases and inaccuracies. A good analyst should always maintain a healthy scepticism, carefully vetting and questioning the data — understanding the sources, limitations and assumptions that are built into it.

A good analyst also knows how important domain knowledge is. We can only make meaningful interpretations of the data when we understand the real-world context that generated the data.

Recently I’ve spent months working on a fancy Marketing Mix Model which I was so excited about. Neglecting dataset accuracy during the project led to wasting months of time and effort when senior managers questioned me during the product presentation.

2. Understand that analytics and statistics are very different fields.

While every analyst should have a grasp of fundamental statistics knowledge, the main role of an analyst is to explore the data, to inspire and to ask good questions, within the content of the given dataset, but not beyond.

On the other hand, a statistician’s job is to infer what’s beyond the data.

In short, analysts come up with good questions (hypotheses) to ask while statisticians answer the questions through vigorous testing.

There is an overlap between the two roles and they should work closely together, but they are fundamentally different.

I have to admit that I have never thought about this before, and looking back, I might, quite a few times, act as a “Data Charlatan” in the team — someone that is worse than an amateur, as Cassie argued.

Being a Charlatan is dangerous because we use data to prove what we already believe and make predictions without enough evidence.

Charlatan: a person who pretends to have skills or knowledge that they do not have. — Cambridge Dictionary

3. Knowing how to beat one of the biggest enemies of analysts — confirmation bias.

We all know what it is and we all know (hopefully) that we should resist it. But in practice, it’s not easy.

As a marketing data analyst, analysing the performance of marketing campaigns is a significant part of my job. The stakeholders of these analyses are mostly people who design and manage the campaigns. Naturally, nobody wants to hear that the campaigns ran failed to meet the expectations, so we are inclined to, intentionally or unintentionally, look for evidence from the data to support the hypothesis that the campaign is a success and generates incremental revenue. We believe that showing a “good” result will make our stakeholders happy and give us a good rating.

But if we think about it, if every campaign or product they produce is successful, why do they even need us in the first place? Our job is to help the stakeholders figure out what is working and, more importantly, what is not, so that they can take away the insights and make improvements. That will ultimately benefit the business, the stakeholders themselves and eventually the analysts.

This leads to the last and most important point: how do analysts add value?

Even today, I feel like I am seen as a second-class citizen in the organisation as I don’t do complex statistical analysis or fancy machine learning modelling.

However, as Cassie pointed out, while analysts are not decision-makers (and they shouldn’t be), we inspire them. We explore the data to “hunt for unknown unknowns worth knowing about”. We inspire the stakeholders to think from different angles. In other words, we do proactive analytics.

It’s like Apple. They knew we wanted and needed a better portable music player, so they created the iPod, that’s reactive. But they also created iPad, something we didn’t even know that we needed and inspire the app developers around the world to come up with wonderful applications that utilised the big screen. That’s proactive.

Of course, it’s not easy to be a good and professional data analyst. It might only take a couple days to learn how to write basic SQL and make nice charts in Tableau, but what differentiates a pro analyst from an amateur one would take years of deliberate practice to master.

But your effort won’t be wasted. As Cassie states in her article, analysts are the most misunderstood heroes and we are “the most-likely heir to the throne”.

The insights from Cassie Kozyrkov help me rethink my career as a data analyst and encourage me to continue this path. I hope it helps you too.

Thanks for reading!

Here you can find the original articles from Cassie Kozyrkov and If you would like to read more articles on analytics, data visualisation and AI in the future, consider following me.

Graduate from an Amateur Analyst: An Insider’s Perspective was originally published in The Analytics Journal on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Supermarkets have always been a business driven by data, but it’s not until recent years where the storage capacity and computational power allow the business to store and analyse the large volume of data it generates and the result could be transformational.

The supermarket and grocery industry in Australia is worth $121B in 2021 and growth is expected to continue in coming years thanks to an upswing in the business cycle and increase in population (IBISWorld, 2021). The sector is highly concentrated with four top players controlling over 83% of the market share, according to IBISWorld (2021). The concentration is notably high when comparing to other developed markets, in the US for example, only 26% of the market is controlled by major companies (IBISWorld, 2021). Despite solid growth in revenue for the industry, the profit margin has been declining. While Covid-19 has boosted the bottom line for the grocery stores due to strong demand and less discount offering, long term challenges facing by the industry such as intense competition and increasing consumer sophistication remains.

To secure its market position, Woolworth Group, the Sydney-based company that runs the country’s largest supermarket chain and the focus of this report, has invested heavily in advanced analytics infrastructure and aims to leverage its vast volume of data to solidify its lead in the industry. While the implementation is not without challenges if successful it will strengthen the company’s position and more importantly, transform the way Australians’ grocery shopping.

Challenges of Woolworth

Though revenue has been steadily growing for Woolworth over the last decade along with the broader industry and its grip of market position is stable in the foreseeable future with a marketing share of 37.4% in 2021 (IBIS, 2021), challenges should not be ignored.

First, competition is intense as Coles, the 2nd largest player is scrambling to catch up and the growing popularity of budget stores have been eating up the market share — ALDI alone has doubled its share since 2010. As a result, the profit margin of Woolworth has been squeezed thanks to intense price-based competition, despite growing revenue, Second, improved technology has given consumers easier access to different retailers, usually at their fingertips. Shoppers are able to pick stores that provide the lowest price and the fastest delivery if they buy online. This poses challenges for retailers to maintain consumer stickiness. Shoppers also expect a more personalised experience that is tailored to their needs and budget.

How data can help to address the challenges

With an average of 30 million customers served per week and 1.6 billion transactions executed per year (Woolworth, 2019), Woolworth generates one of the largest volumes of consumer data in Australia. The company clearly sees the value of the data and tapped into the capability of advanced analytics ahead of its rival by investing in Quantium, an analytics consultancy in 2013. In 2020, Woolworth doubled down on its bet on data science and raised its stake in Quantium from 47% to 75%. Internally, the company set up a seperate division called Woolies X to drive data and AI effort and rolled out third-party analytics platforms such as Tableau and Google Cloud across the business to allow employers to analyse and act on customer insights (Patrick 2019).

By using the data ineffective can help Woolworth stay ahead of the game in several ways:

Dynamic pricing strategy

Setting the price strategy right is critical to retail businesses, as a McKinsey study shows, a percentage increase in price can translate to a jump of 8.7% in profit (Solum, 2021). Pricing is particularly important for supermarkets where the margin is thin and competitors nearby can easily win over customers by selling exactly the same product at a much lower price.

By setting the right price is tricky for businesses like Woolworth where there are thousands of products on the shelves and different discount rates are applied to different items every day and week. Sensitively to price can also be vastly different for each type of product and for different customers. Thus, a one-size pricing strategy is far from ideal but achieving an optimal price point for every product is nearly impossible with traditional and manual methodology. But the problem could be solved by data science.

Big data analytics will allow the algorithm to analyse the past sales data of each product at each location at different price points and help the business set the price that will maximise sales of the item at a specific store. The process will factor in shoppers’ sensitivity to price as well as competitors’ pricing for the same product in each neighbourhood.

The strategy can also be applied at a more granular level for eCommerce and provide dynamic pricing for individual customers. According to Woolworth’s report about the future of grocery shopping (2014), Australians are shifting their supermarket shopping online at a fast pace. In fact, 8.2% of total sales of the company was generated online in H2, 2021, that’s an increase of 77.9% compared to the previous quarter (Woolworth, 2021).

Through pricing optimisation, a boost in profit margin can be expected for Woolworth.

Personalised Shopping Experience

As mentioned previously, advanced analytics can provide differentiated pricing based on each customer’s shopping habits and preferences. But personalisation can go much further. By analysing customers’ past shopping and web browsing behaviour, Woolworth can show, at the customer level, the types of products that he or she typically buying when they shop or the product on promotion that might be interested in when they land on the website, much like what K-Mart has successfully implemented in the US (Alkhaldi, 2021).

Personalised experience also does not limit to online shopping. In fact, Woolworth in its own The Future of Fresh report has envisioned what in-store shopping will be like in 2034 — customers will be guided by applications that facilitate their shopping experience. When detecting that a customer is in-store, the application can provide real-time recommendations and discounts based on his/her lifestyle and eating habits (Woolworth, 2014).

The experience is further extended to our home. With the growing prevalence of smart homes and smart appliances, the system will monitor our consumption of groceries and detect items that are running low and sending automatic orders to Woolworth online stores (Woolworth, 2014).

By studying customer behaviour can also make Woolworth’s loyalty program much more attractive with personalised offers and rewards from both the grocery stores and other products offered by the group such as alcohols, telecom, insurance and electronics.

The more data Woolworth gathers about the customers the better experience it can provide to them further increases loyalty, retention and yield more up-sale opportunities.

Sophisticated Inventory Management

Stock shortages and supply chain disruptions in grocery stores during Covid-19 has demonstrated the importance of Inventory management when demand for certain products skyrocketed. But even in normal times, decide when to replenish which item(s) is a daily struggle in supermarkets where products come and go quickly. This could lead to high inventory management costs and unhappy customers that could turn to nearby competitors.

The potential solution is predictive analytics. Real-time data will be able to predict the potential sales and demand of the products and inform the stores to ensure those items of high demand will always be in stock (Almazing, 2019).

The benefit is also significant for the opposite scenario. Unsold fresh food often goes wasted and creates economic and environmental problems. By properly manage inventory such waste will be reduced. This could be achieved by trim down the stocks of items with predicted low demand or sending out real-time marketing offers for bargain hunters.

Potential Issues of Applying Data Science

While the potential power of data and advanced analytics are unlimited and the impact it could have on Woolworth’s bottom line and customer experience are promising, the implementation is not without hurdles.

Data management

For data science, garbage in means garbages out. For algorithms, data scientists and analysts to function properly, accessibility to high-quality data is essential.

With the hefty volume of data generated by Woolworth on a daily or even hourly basis, along with the complexity of the business structure and associated IT system, ensuring the data is collected timely, smoothly, properly cleaned, structured and stored is no small task. To achieve such will require the hard work of a team of sophisticated solution architects, data engineers and project managers, the process will often involve experiments and failures. Patient and support from the business especially the senior executives are essential to success.

Privacy Protection

Data privacy has emerged as one of the most important issues in the data science field. While consumers largely applaud personalised shopping experiences and marketing offers, they are also increasingly concerned about how much companies know about them. Regulation is also catch up. General Data Protection Regulation, the EU law on data protection and privacy, the first of such laws in the world, had a profound impact globally on data collection since it came into effect in 2016. Similar regulations are expected to be introduced in other parts of the world, including Australia.

Woolworth should allow customers to control what type of data about themselves are being collected. Between profit, experience, privacy and legal compliance, there is a very delicate balance that Woolworth needs to manage.

Data Culture

While a new data and analytics platform can be rolled out across the business in a relatively short period of time, training and upskilling thousands of employees to maximise the value of the technology will be a daunting task for Woolworth.

More importantly, truly leveraging the power of data science and analytics requires a change of mindset across the business. Every person from executives at the top to the store managers on the floor has to trust data and incorporate insights from data in daily decision making. Cultivating a data-driven culture is a long process and requires proper strategy from management and dedicated effort from the whole team.

Conclusion

The potential of applications data that Woolworth holds about its stores, suppliers and customers that can transform the way the business operates and the way Australians do grocery shopping are boundless. The examples discussed in this paper barely scratch the surface. In fact, in 2020, Woolworth even used data analytics to decide which of its stores would benefit the most from refurbishment (Crozier, 2020).

Data is the secret weapon of Woolworth against its rivals, both domestic and foreign. It will be a long and complicated process to fully unleash the potential of data with many challenges ahead but the company has made a promising start and solidify its position as an industry leader.

“Those who rule data will rule the entire world”, once said Masayoshi Son, founder and chief executive of SoftBank. The executives at Woolworth clearly believe in the statement.

References:

Alkhaldi, N., 2021. Supermarkets of the Future: Deploying AI in Your Grocery Store — ITRex. [online] ITRex. Available at: <https://itrexgroup.com/blog/supermarkets-of-the-future-deploying-ai-in-grocery-store/#header> [Accessed 15 September 2021].

Aimazing.co. 2021. How Grocery Stores and Supermarkets Are Making Use of Data Analytics. [online] Available at: <https://www.aimazing.co/post/how-grocery-stores-and-supermarkets-are-making-use-of-data-analytics> [Accessed 15 September 2021].

Hatch, P., 2019. Woolworths, Coles to use data to fight off Aldi competition. [online] Smh.com.au. Available at: <https://www.smh.com.au/business/companies/coles-and-woolies-secret-weapon-in-war-with-aldi-is-data-20190529-p51sej.html> [Accessed 15 September 2021].

Woolworth Group, 2014. The future of fresh. Sydney: Woolworth Group.

Europe, S., 2021. How Does Big Data Help Improve Retail Price Points?. [online] SoluM Europe. Available at: <https://solumesl.com/en/insights/how-to-improve-your-retail-price-points-through-big-data> [Accessed 16 September 2021].

IBISWorld, 2021. Supermarkets and Grocery Stores in Australia. IBISWorld.

Crozier, R., 2020. Woolworths uses analytics to find stores for trials, refurbishment. [online] iTnews. Available at: <https://www.itnews.com.au/news/woolworths-uses-analytics-to-find-stores-for-trials-refurbishment-536523> [Accessed 15 September 2021].

Woolworth Group, 2021. Woolworth Group Annual Report 2021. Sydney: Woolworth Group.

van der Schaaf, K., 2017. How Big Data Analytics can Contribute to the Marketing Performance of Supermarkets. Bachelor. Wageningen University.

Australia has gone 27 years without a recession, a record among developed countries. In terms of GDP per capita, the country has been outperforming all G7 members since the 2008 financial crisis.

Australian dollar had a wilde ride over the last two decades agaisnt the US Dollar. After hitting the post-floating low of $0.4792 in 2001, the currency began a multi-years upward run, largely driven by commodity boom, high interest rate and strong global, in particular Chinese economy. Optimism was evident and predictions of a strong Aussie went wild.

Then came the Global Financial Cricis in 2008, which knocked down the value of Aussie Dollar by nearly 40% in a matter of 3 months as investors fled from risky currencies. Yet the tumbled was short-lived. The strong Australian economy, coupled with a weakened USD, lifted the pair above parity for the first time on 3 Nov 2010.

Fast forward to 2019, the Aussie Dollar is stuck in a multi-year downtrend and trading aroud $0.7 agaisnt USD. Domestic interest rate is at rock bottom, economies are slowing down everwhere while king USD returns on Fed tightening. The outlook for AUD in the near future looks rather gloomy and parity seems so far away……