Web-data Superpowers

Web-data is data that can be found openly on the web. It is a public resource that is not behind a log-in, doesn’t require registration and is open to all search engines to index.

I wish to share how you can harness the superpowers of the open web to create a competitive advantage for your business.

Meta + Bright Data

I previously worked at Facebook, now Meta, for 5 years, leading the research platform team, located in the TLV R&D center, and recently joined Bright Data as vice president of product.

• Meta collects and analyze huge amount of data for competitive analysis, understanding the apps landscape and surveys its 3B users to understand people sentiment towards the brand
• Bright data is democratizing data collection for all companies, to empower small and large companies and give access to similar superpowers that used to require expensive infrastructure and developers. All fully comply with data protection laws.

Big players, sure, but now also growing businesses joined the revolution

All companies are using web data in some form or another, so it is no surprise that big brands like McDonald’s Canada will have data departments, and are using Bright Data to gain access to detailed competitor insights from pricing, menu offerings, and distributions, all found online. Then Bright Data structures and delivers the data to McDonalds daily for analysis

Now that access to data services is becoming easier, affordable and with high quality, thousand of small and medium companies also learned that they can use structured public web data to their advantage and help determine things such as pricing strategy — which is changing by the hour -understand their ranking and trends in search engine results pages, listen to public chatter on social media, uncover customer needs and sentiment towards their products and their competitors products. The list goes on and on.

At first it was hard for me to imagine what could be achieved using web data, and how can it be applied to research, so let me start off with a cool example (next slide)

Be Creative with web-data: the ‘Yankees Candle’ example

In 2020, Researcher Kate Petrova, a PhD student at Stanford University, read a tweet about angry reviews over Yankee Candle’s that didn’t have any smell at all. She wondered if Scented candles are an unexpected victim of the Covid-19 pandemic.

Kate collected 20K scented and unscented candles Amazon reviews, from 2017 to 2020, and saw a sharp drop in the avg rating, compared to unscented candles starting Jan 2020.

Another research repeated the experiment with UK Amazon perfume ratings and reviews, with similar results.

Easily available data from one source can have a strong link to seemingly unrelated topics.

Being creative and thinking “outside of the box” is crucial especially when time is of the essence, and web data provides a needed proxy metric that moves faster and provides a more comprehensive view than just measuring the variable alone.

Get Erez Naveh’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Let’s say we owned a Brewery, how would we leverage web-data?

We have few breweries in the audience today, let’s imagine that we want to harness the superpowers of web data to our brewery advantage:

• Perform market research and discover the best suppliers
• Tackle supply chain issues
• Keep track of the latest regulation updates
• Discover resellers across the world
• Get authentic feedback from social media users

Market research & Market Intelligence — using Google Maps

• We can use public data from Google Maps, for example, to find business on the map, plan the expansion into a new country, understand the market share of the competitors and map residential areas to optimize delivery.
• As one example, you might have seen this cool feature in google maps that lets you see a business “Live” status, if it’s busy or not. If collected in a systematic manner, at scale, for all bars and restaurants for example, you can build an understanding of an area’s demand, how busy the bars are, and if this should be your next place for store or dealership location.

MAP monitoring (Minimum Advertised Price), Brand sentiment, Market Intelligence — using Instacard, Amazon, Walmart, etc.

• As you can imagine, Amazon has vast amounts of retail related data, you can find data on your competitors products rating and customer feedback, about the ranking of products in your category (trends over time), and even estimate inventory and demand!
• How? If you add a product to your cart, and edit the quantity, at some point you’ll get an error, out of stock, that is the upper limit. Collect that data the following days to check if the quantity changed and by how much. Overtime you can see trends in product stock availability and project the associated demand.
• It’s important to note that collecting such data requires the ability to enter the public site from specific locations. Depending on your zipcode, you’ll get different products available, different prices and delivery options.

Regulation monitoring

• Beer, and many other industries are regulated. Keeping track of the ever changing landscape can be automated and even set to trigger internal processes in your business operations.
• Another example would be collecting data about businesses that went under, chapter 11, in your market. It can have a massive effect on your business and supply chain, and that data can also be collected with filters that are tailored to your search terms.

Share of Voice + Brand sentiment — Social Media

• The major social media platforms have an open version of their applications, which is limited, that doesn’t require any login credentials.
• Tapping into the social chatter allows you to be proactive about what’s being said about your brand and your market. Understanding trends and taste of customers, ‘is organic beer trending up/down?’, and who are the top influencers that are talking about it?
• A lot of the power is hidden and requires combining a few data sources.
• Does the influencer have a lot of reach on all platforms or just one? How many followers, likes per post, comments, location and so on…
• By combining recruiting data from Indeed and LI, together with a company’s industry category which is found on LI company page or Crunchbase, you can analyze an entire industry’s hiring trends. Is hiring IT engineers trending up or down?
• People web data such as this can also help recruiters find and retain diverse top talent for their organizations, by collecting web data from the different resume fields of public profiles on LinkedIn.

DaaS — Data as a Service

• Enriching data, matching and finding companies, products and people across a dataset, is hard and sophisticated work, even for a company without a dedicated data team to perform on its own.
• Not to mention, collecting the data, validating its quality, running the operation at scale across the open web, then cleaning and structuring the data, this very resource heavy and time consuming for any organization.
• But today, there are data providers (such as Bright Data) who are providing access to pre-collected, large-scale datasets, snapshots of entire websites, which update regularly.
• This solution is democratizing data collection, providing customers access to new data sources.
• This is the revolution from big data, to Data as a service (DaaS)
• It provides businesses with the ability to request continuous pipelines of fresh, complete and high quality ready-to-use data that will feed directly into an organization’s data storages and strengthen the datasets which they base their insights on whenever they need a refresh.
• DaaS platforms hold large amounts of data in a single location, compiled from different sources, including web data scraped from the internet as well as data from specialized third-party data providers or other organizations that share data. So there is never a shortage of data, and new data is always being added to these storages.
• Typically organizations who perform data collection alone in-house do so with a lot of resources, including IT, development teams and so on, each responsible for collecting, cleaning, structuring/indexing and analyzing the data.
• DaaS platforms, however, perform all of these actions for their customers. They regularly refresh the datasets, they discover new relevant data to add to the datasets as well as offer enrichment options for some fields. Furthermore, the platform cleans and structures the data so that they may integrate seamlessly with customer data lakes, warehouses and other storages.
• Therefore customers always get the data they want, when they need it. This is where I see the future of data heading.

The Bright Initiative

• Working at Bright Data, I have seen many ways in which this data can be put to good use, for social good and to drive positive change across the globe.
• The Bright Initiative offers free access to our datasets, our products and to our infrastructure to help solve the most crucial issues of our time. I would like to share a few examples of this.
• When Covid vaccine first came to the US, it was hard to find an appointment. We helped collect data for 25K locations nationwide, to build a real-time updating dataset that was used to help thousands of Americans find an appointment, instead of needing to search at multiple local pharmacies.
• On the climate front, Subak is a UK-based climate accelerator that selects, funds and scales organizations combating climate change, using shared data, infrastructure and tools. The Bright Initiative provides Subak’s member organizations with access to Bright Data’s technology, support and expertise, helping them tackle climate change by providing access to large amounts of relevant, publicly available data across a range of different climate projects.
• Human Trafficking Initiative (HTI) Labs fights human trafficking in the commercial sex industry and addresses issues of domestic violence through data-driven research. HTI Labs uses Bright Data tools to counter online sex advertisements and access the data needed to combat sex trafficking on a widespread level. It uses the tools to combat the different blocking techniques of these advertisements, which have developed ways, such as CAPTCHA, to block web scraping tools.
• Furthermore, we recently collaborated with the UK government, to help assess whether or not ppl work in what they’ve studied. Such study affects the future of education. We are also providing guidance when needed to determine the future use of data in Uk society.
• If you are a non-profit research organization, or a university with the need for such data, please reach out to me to collaborate.

Be Data Creative

• I hope this has inspired you to be creative with using web data, and you’ll find new ways to use the abundance of online resources to your advantage.