<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by James Kehoe on Medium]]></title>
        <description><![CDATA[Stories by James Kehoe on Medium]]></description>
        <link>https://medium.com/@jamesjmkehoe?source=rss-2069a916d726------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*MkgwB65c1tZy47V2cTU-zQ.jpeg</url>
            <title>Stories by James Kehoe on Medium</title>
            <link>https://medium.com/@jamesjmkehoe?source=rss-2069a916d726------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 12:08:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jamesjmkehoe/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[An Overview of E-commerce Product Data at Scale]]></title>
            <link>https://medium.com/@jamesjmkehoe/an-overview-of-e-commerce-product-data-at-scale-ec37ece39060?source=rss-2069a916d726------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec37ece39060</guid>
            <category><![CDATA[product-data]]></category>
            <category><![CDATA[marketplaces]]></category>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[James Kehoe]]></dc:creator>
            <pubDate>Thu, 03 Mar 2022 23:29:38 GMT</pubDate>
            <atom:updated>2022-03-03T23:29:38.579Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Authors:</strong> Comber, D., FitzGerald, K., &amp; Kehoe, J.</p><h3>TL;DR</h3><p>E-commerce companies collect product data from competitors’ websites to inform; pricing decisions, competitor &amp; market strategy, and vendor management.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0NxGRRHTpmDyWjEOyFrIIg.png" /></figure><h3>Introduction</h3><p>This article gives a very high level overview of:</p><ol><li>Who wants e-commerce product data?</li><li>Why do they want e-commerce product data?</li><li>What e-commerce product data do they get?, and</li><li>How do they get e-commerce product data?</li></ol><p>We’ll focus on the perspective of e-commerce companies. This article is your map of this amazing and dynamic world of ec-ommerce product data.</p><h3>Who wants e-commerce product data?</h3><p>Typically it’s organizations that also sell the same or similar products. For simplicity we break these organizations into three groups:</p><ol><li>Marketplaces</li><li>Retail — e-commerce</li><li>Retail — Brick &amp; Mortar</li></ol><h3>Marketplaces</h3><p>These organizations are often digital natives (i.e. were born digital), and data indigenous (i.e. data has always been at their core). Marketplaces have the largest teams of data scientists and at their heart, at least for C2C Marketplaces, are almost entirely data companies.</p><p>Because they are not significantly hindered by the inertia of stock and traditional merchandising models, they can move more freely in the world of data. Marketplaces’ need for e-commerce product data certainly overlaps with their brethren in retail, but they also have some nuanced data desires that we’ll look at below.</p><h3>Retail — e-commerce</h3><p>These are e-commerce first retailers, think B2C, they have the warehouse but not the shopfront. Naturally these are also digital natives, but as they have often evolved out of traditional retail backgrounds they may not be data indigenous, but are getting there. Interestingly in the last few years we have seen some of these companies start branching out to Brick &amp; Mortar stores.</p><h3>Retail — Brick &amp; Mortar</h3><p>These are traditional, often long established, retailers that over the past decade have heavily invested in not being left behind. Because these organizations are neither digital natives nor data indigenous you often see them accelerating their e-commerce journey by purchasing data companies (e.g. Homedepot bought <a href="https://techcrunch.com/2012/12/17/home-depot-acquires-data-driven-pricing-analytics-startup-blacklocus/?guccounter=1&amp;guce_referrer=aHR0cHM6Ly93d3cuZ29vZ2xlLmNvbS8&amp;guce_referrer_sig=AQAAADc3CjGkR7aFAoICsmkDDjRFmmUyklwgm2Kzx7k53lnGzb6Idx1hh-lbKKg3Uc69og_jS2d9cGxDk9hy5KF_oiA8LxVe8ygnSmxbwE7zJokyMMdO8a4SeMIfv52pVs25NIRyMze8_Wzr5fsBjDlhA5LwpJgm3rFZ8hgepcWzwuUa">BlackLocus</a> in 2012, and Lowes bought <a href="https://www.digitalcommerce360.com/2019/05/22/lowes-buys-boomerang-commerces-pricing-technology/">Boomerang Commerce</a> in 2019).</p><h3>Alternative Data — Finance</h3><p>Another group that we see seeking e-commerce product data are Financial organizations that see this data as <a href="https://en.wikipedia.org/wiki/Alternative_data_(finance)#">Alternative Data</a>, but that is another day’s story, for today let’s focus on e-commerce companies.</p><h3>Why do they want e-commerce product data?</h3><p>From the perspective of e-commerce organizations the primary use cases for this data are as follows:</p><ol><li>Price intelligence</li><li>Competitor intelligence</li><li>Market analysis</li><li>Vendor Management</li><li>Compliance</li><li>Improving the sellers’ experience</li><li>Internal barriers to data</li></ol><p>Below we’ll touch briefly on each of these use cases and the questions they help answer, but we will go into much more depth in separate use case articles.</p><h3>Price intelligence</h3><p>If your pricing strategy is in any way relative to your competitors (e.g. always match price, or always be 1 cent cheaper, etc.) then you need to get their prices.</p><h3>Competitor intelligence</h3><p>Who is moving in on your market (by geography, category, etc.), or perhaps who is doing well that maybe you could learn from.</p><h3>Market analysis</h3><p>What products or sellers are trending, what gaps are there in product offerings, do you have a dominant stock position that you can leverage.</p><h3>Vendor Management</h3><p>Are your vendors providing you with their full range of products, most competitive price, equally high quality collateral (e.g. images, copy, etc.)? Not sure, you need to get that data.</p><h3>Compliance</h3><p>This more often applies to brands, where they want to ensure compliance with MAP (minimum advertised pricing), branding guidelines, and marketing copy. Other considerations could also be the discoverability of products, via categories or keyword searches.</p><h3>Improving the sellers’ experience</h3><p>Make it easier for your sellers by pre-populating your database with products they might sell so as to speed up the uploading process, and ensure better meta data quality. Also, recommend a selling price based on competitor data. Another fountain of information here is buyer reviews, are you getting better reviews than competitors for the same product? If so what is it you are doing well?</p><h3>Internal barriers to data</h3><p>One of the most surprising uses is to eliminate all the internal barriers to data within an organization. Surprisingly enough, many clients actually scrape data from their own websites so as to avoid all of the internal friction around accessing data, and to ensure the data will better match the structure of product data from target websites.</p><h3>What e-commerce product data do they get?</h3><p>There are four main types of e-commerce product data that are collected:</p><ol><li>Product details</li><li>Product lists</li><li>Product reviews</li><li>Product seller details</li></ol><h3>Product details</h3><p>This is the essence of product data and is usually called the “Product Details Page” (PDP). It’s where you find information like the following:</p><ul><li>name</li><li>price</li><li>availability (i.e. in stock)</li><li>unique identifier (i.e. sku, mpn, gtin, asin, etc.)</li><li>images</li><li>description</li><li>rating</li><li>seller information</li><li>delivery information (i.e. costs, timelines, etc.)</li><li>etc.</li></ul><p>You can see a more detailed list <a href="https://docs.zyte.com/unified-schema.html#operation/product">at this link</a>.</p><h3>Product lists</h3><p>Think of this as the product category page, or search result page, where you can see a list (or grid) of products from a handful to hundreds. It includes limited data, but perhaps enough depending on your use case. Common data fields would be:</p><ul><li>name</li><li>price</li><li>page</li><li>position</li><li>rating (i.e. Stars, etc.)</li><li>delivery information</li></ul><h3>Product reviews</h3><p>Though the top, or most recent, or most popular reviews may be on the PDP, the full list is typically elsewhere and contains much more information. The available fields are few, but can be rich picking.</p><ul><li>rating</li><li>date</li><li>review</li><li>reviewer</li></ul><h3>Product seller details</h3><p>Not as commonly targeted, but could be where you find your next power seller, or identify what’s becoming hot. There is huge variation here in terms of availability of these pages, the types of data fields, sub-pages, etc.</p><h3>How do they get e-commerce product data?</h3><p>At a simple level a computer replicates a browser, visits a web page, and collects just the data you want. It’s easy. At least until you’re trying to copy data for 5 to 20 thousand products per hour.</p><p>If we look at the process of obtaining this data there are several steps and, depending on a number of factors, these may be completed in-house or outsourced.</p><ol><li>Identify the pages that you need to collect data from</li><li>Visit these pages and collect the data</li><li>Extract the data into a format you can process (e.g CSV, JSON, etc.)</li><li>Process the data (e.g. cleaning, deduping, matching, etc.)</li><li>Visualize the data (though not always)</li><li>Connect the data into your business systems</li></ol><p>Naturally the above is simplified and overlooks a lot of complexities, especially how hard it can be to get the raw data in the first place. Many popular ecommerce sites have advanced systems to protect them, these systems can perceive your data extraction as an attack and shut you out. Extracting web data must be done in a compliant and sustainable way that in no way negatively impacts target sites.</p><h3>Conclusion</h3><p>The above points are an extreme simplification of what in fact is an entire industry that has evolved with both the web and e-commerce, however the pragmatic challenge of shifting more product remains it’s true self.</p><p>There are many shortcuts and trap doors when it comes to the world of acquiring and leveraging e-commerce product data, we know because we discovered many of them in our decade of extracting e-commerce data at scale. I hope this article helps prepare you for your journey. If you want to learn more check out some of our <a href="https://www.zyte.com/results/price-intelligence-resources/">other resources</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec37ece39060" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>