<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jay M. Patel on Medium]]></title>
        <description><![CDATA[Stories by Jay M. Patel on Medium]]></description>
        <link>https://medium.com/@jay-68918?source=rss-8dc22d6b3ace------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*70u282k_i8w3OU7qSnjKVQ.jpeg</url>
            <title>Stories by Jay M. Patel on Medium</title>
            <link>https://medium.com/@jay-68918?source=rss-8dc22d6b3ace------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 19:28:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jay-68918/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Airbnb vs. Mid-Term vs. Long-Term Rental: The Real Math on the Same Property]]></title>
            <link>https://jay-68918.medium.com/airbnb-vs-mid-term-vs-long-term-rental-the-real-math-on-the-same-property-0b6de1b82484?source=rss-8dc22d6b3ace------2</link>
            <guid isPermaLink="false">https://medium.com/p/0b6de1b82484</guid>
            <category><![CDATA[real-estate-investments]]></category>
            <category><![CDATA[airbnb]]></category>
            <category><![CDATA[property-management]]></category>
            <category><![CDATA[short-term-rentals]]></category>
            <category><![CDATA[investing]]></category>
            <dc:creator><![CDATA[Jay M. Patel]]></dc:creator>
            <pubDate>Tue, 07 Jul 2026 00:00:35 GMT</pubDate>
            <atom:updated>2026-10-06T11:59:38.984Z</atom:updated>
            <content:encoded><![CDATA[<h4>Most comparisons stop at gross revenue and get the answer wrong. We run one property through all three strategies — expenses, vacancy, and the owner’s own time included — and show where the profit actually lands.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*iTsdpR3vvp1bsN7wVt4nZQ.png" /></figure><p>“Is Airbnb still worth it?” It’s one of the most searched questions in short-term rentals, and the way most people answer it is broken. They take a property’s Airbnb payout, compare it to a long-term rent check, see a number twice as big, and declare victory. Or they read one saturation headline, compare a slow month against a lease, and declare the whole model dead.</p><p>Both answers make the same mistake: they compare gross revenue across strategies with completely different cost structures, workloads, and risk profiles. Gross revenue is where the comparison starts. It is nowhere near where it ends.</p><p>So let’s do it properly. One property, three strategies, all the way down to the number that actually pays your mortgage.</p><p><strong>A note before you read further:</strong> every figure in this piece is illustrative, built for a hypothetical mid-tier Southeast drive-to market to demonstrate the method. Swap in your own comp set’s real numbers before treating any figure here as a claim about your property — that’s exactly what a proper audit does.</p><h3>The setup</h3><p>Our test case is a 3 bedroom, 2 bath home in a mid-tier Southeast drive-to leisure market — think of the outer ring around a major cabin or lake destination: real weekend demand, real seasonality, not a trophy market. The numbers below are deliberately unheroic. If you own in a top-decile market, your spread will look better. If you own in a saturated one, worse. The method is the point; the method is what transfers to your property.</p><p>Three strategies for the same house:</p><p><strong>Short-term rental (STR):</strong> nightly stays on Airbnb and Vrbo, dynamic pricing, 2-night minimum.</p><p><strong>Mid-term rental (MTR):</strong> furnished stays of 30 days or more, marketed to travel nurses, relocations, and corporate stays through channels like Furnished Finder.</p><p><strong>Long-term rental (LTR):</strong> a standard 12-month unfurnished lease.</p><p>One housekeeping note so the math stays honest: we treat cleaning fee income as a pass-through that roughly covers cleaning cost, we exclude property taxes and debt service from every scenario because they’re identical across strategies, and we compare everything at the level of net operating income (NOI) — what the property actually earns its owner before the mortgage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*I_UKrOEpgyAxVan9ec3_ow.png" /></figure><p>The STR runs a blended average daily rate of $230. That blend hides a steep curve: peak weekends well above $300, shoulder season in the low $200s, off-season weeknights struggling to clear $165. At 60% occupancy, that’s 219 booked nights — <strong>$50,370 per year.</strong></p><p>The MTR places at $3,300 per month furnished. Mid-term tenants don’t stack back to back perfectly; between placements you carry vacancy while you find the next nurse or relocation. Call it 10.5 effective months — <strong>$34,650 per year.</strong></p><p>The LTR leases at $2,150 per month. A full year is $25,800, but even long-term rentals have turnover months and the occasional collection problem, so apply a standard 5% economic vacancy — <strong>about $24,500 collected per year.</strong></p><p>Scoreboard after round one: STR earns roughly double the LTR and lands about 45% ahead of the MTR. The three strategies now go on to spend that revenue very differently.</p><h3>Round two: what each strategy actually costs</h3><p>Here’s the mistake people make. STR costs aren’t “a bit higher” than LTR costs. They’re a different category of business expense, and they scale with every single booking.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*vKFCInrZzAylNpzex04o0A.png" /></figure><p>Furnishing is a real capital outlay for both STR and MTR — commonly five figures for a 3 bedroom done to guest standards, and it wears out faster than owners budget for. Turnover is the STR’s structural cost: every 2-night booking triggers a full clean, laundry, and restock, which is why short minimum stays quietly eat margin. Insurance is the line people discover too late: many homeowner policies exclude short-term rental activity outright, so a proper STR policy costs real money and isn’t optional.</p><p>Roll those lines up and you get the operating expense ratios experienced operators and property managers commonly discuss: self-managed STRs typically run 40–55% of gross revenue in operating costs, PM-managed STRs 50–65%, MTRs materially less, LTRs less again.</p><h3>Operating Expenses as a Share of Gross Revenue</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*CvVzs_toRrZex-64hrijMw.png" /></figure><h3>Round three: NOI, the number that actually matters</h3><p>Now subtract. This table contains the entire argument.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*Za318ejBxyjbFlLlmrO2qw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*ZTdgkf4Wge5AWu7OqFRZzA.png" /></figure><p><strong>Read it slowly:</strong> the STR’s 2x gross advantage over the LTR shrinks to a 33% NOI advantage; and only if the owner does the managing. Hand the same property to a full-service manager and STR’s NOI drops <em>below</em> the long-term lease. The mid-term rental, which lost the gross revenue contest badly, finishes second on profit with a fraction of the workload, and it beats the PM-managed STR outright.</p><h3>The fourth number: what your time is worth</h3><p>The self-managed STR wins the table above, so let’s price the asterisk on that win. Self-managing this property realistically costs 5 to 8 hours a week across guest messages, pricing checks, scheduling cleaners, restocking, and the occasional 10pm hot tub emergency. Call it 6 hours a week, roughly 310 hours a year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*skIxVnAuZQKXwMX3kzlfnw.png" /></figure><p>That’s not an argument against short-term rentals. In strong seasonal markets, on well-bought properties with the right amenities and sharp pricing, the spread over LTR is multiples of this example and the hourly math becomes genuinely attractive. It’s an argument for knowing which case you’re actually in — the difference between a $22-an-hour STR and a $90-an-hour STR is invisible in a gross revenue comparison and obvious the moment you run the full stack of numbers against your real comp set.</p><h3>The two risks the spreadsheet still isn’t showing</h3><p><strong>Vacancy behaves differently in each strategy.</strong> STR vacancy is loud: you watch empty calendar dates approach in real time — stressful, but at least measurable and fixable with price. MTR vacancy hides between placements; one slow tenant search can erase a month. LTR vacancy is the quietest and most underestimated: forum comparisons routinely assume a rent check appears every month forever, then a turnover month, a repaint, and a slow re-lease take a 10% bite out of the year nobody modeled.</p><p><strong>Regulatory risk is wildly asymmetric.</strong> A long-term rental faces landlord-tenant law, which evolves slowly. A short-term rental faces local ordinances, which can change in one county commission meeting. Permit caps, primary-residence requirements, and zoning changes have ended profitable STR operations overnight in plenty of markets — and in the Southeast the rules can differ sharply from one county to the next, sometimes across two shores of the same lake. Mid-term rentals typically sit outside STR ordinances entirely, a structural advantage that never appears in a revenue projection.</p><h3>The strategy nobody prices correctly: the hybrid</h3><p>In strongly seasonal markets there’s a fourth option, and it’s often the best one: run the property as an STR through peak season, then take a 30-plus day furnished placement through the deep off-season instead of chasing discounted nightly bookings.</p><p>The math has a crossover point. When your off-season effective ADR — after the deeper discounts, the extra turnovers, and the per-booking costs — drops below what a flat monthly furnished rate nets you, the mid-term placement wins those months. Operators in mountain markets around the Smokies and along the Florida panhandle run this play every year. The only way to find your crossover point is to know your true off-season effective ADR and your realistic MTR rate — a data question before it’s a strategy question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*2GXSXUhyXmOtS9pNGcgxng.png" /></figure><h3>The takeaway</h3><p>Gross revenue comparisons make short-term rentals look unbeatable. Expense-adjusted, time-adjusted, risk-adjusted comparisons tell the truth, and the truth is property-specific: the same house can be a clear STR win, a sneaky MTR win, or an LTR-with-less-drama situation depending on its market, its rules, and its owner.</p><p>The gap between those outcomes is not luck. It’s a set of numbers: your real blended ADR, your real occupancy against your actual competitors, your off-season crossover point, your full expense stack, your break-even occupancy. Every one of them is knowable before you commit to a strategy, and most operators have never seen them for their own property.</p><p>About this analysis</p><h3>How to Read These Numbers</h3><p>Every figure above exists to demonstrate the method, not to describe your property. Your market’s real blended ADR, occupancy against your actual comp set, and true expense stack are exactly what a listing audit produces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*FCuhpjw2sTl-bre3n96ZHg.png" /></figure><h3>Frequently asked questions</h3><h3>Does Airbnb make more money than renting long-term?</h3><p>Usually on gross revenue, not always on profit. Short-term rentals commonly gross 1.5x to 2.5x a long-term lease on the same property, but operating costs of 40–65% of revenue and the owner’s own labor shrink or erase the gap. The honest comparison is net operating income per strategy, plus an hourly value on self-management time.</p><h3>Is a mid-term rental more profitable than an Airbnb?</h3><p>Sometimes, and more often than people expect. Mid-term rentals gross less but keep far more of it, because they eliminate per-stay cleaning, most platform fees, and most of the workload. In heavily regulated markets, off-seasons, and hospital-adjacent locations, MTR frequently beats a professionally managed STR on net.</p><h3>What expenses do people forget when comparing STR vs LTR?</h3><p>The usual omissions: furnishing and its replacement cycle, host-paid utilities, STR-specific insurance, the software stack, capital expenditure reserves, and the owner’s time. Comparing an Airbnb payout to a rent check skips every one of them.</p><h3>What is the hybrid rental strategy?</h3><p>Running a property as a short-term rental in peak season and switching to 30-plus day furnished stays in the off-season. It wins whenever the flat monthly rate beats your discounted off-season nightly revenue after per-booking costs, which is common in seasonal mountain and beach markets.</p><p>Run This on Your Own Property</p><h3>Which Strategy Actually Wins on Your Listing?</h3><p>This analysis used illustrative numbers to show the method. If you own or manage a short-term rental in the Southeast, <a href="https://www.specrom.com/str-intel/teardown/">we’ll run your listing and your real comp set through this exact framework — free</a>.</p><p><em>Originally published at </em><a href="https://www.specrom.com/str-intel/insights/airbnb-vs-mid-term-vs-long-term-rental-math/"><em>https://www.specrom.com</em></a><em> on July 7, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0b6de1b82484" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Sweet Divide: How Plant and Whey Proteins Take Radically Different Approaches to Sweeteners]]></title>
            <link>https://jay-68918.medium.com/the-sweet-divide-how-plant-and-whey-proteins-take-radically-different-approaches-to-sweeteners-05c5fee3709d?source=rss-8dc22d6b3ace------2</link>
            <guid isPermaLink="false">https://medium.com/p/05c5fee3709d</guid>
            <category><![CDATA[health]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[food]]></category>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[nutrition]]></category>
            <dc:creator><![CDATA[Jay M. Patel]]></dc:creator>
            <pubDate>Thu, 18 Sep 2025 10:39:54 GMT</pubDate>
            <atom:updated>2025-09-18T10:41:00.575Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. Introduction: Why Protein Powders Need a Sweet Solution</h3><p>Protein powder faces a fundamental palatability challenge. Pure whey isolate tastes like watered-down milk with metallic notes.</p><p>Unflavored plant proteins? Think freshly mowed grass mixed with chalk. Without sweeteners, most consumers would abandon their protein shakes faster than a January gym membership.</p><p>The sweetener serves three critical functions: masking inherent off-flavors, creating an enjoyable taste experience that encourages consistent use, and maintaining the product’s position as a “healthy” alternative to sugary foods.</p><p>Yet how brands approach this challenge reveals a fascinating divide between plant and whey protein markets.</p><h4>1.1 Data Scraping Methodology</h4><p>At Specrom Analytics, we have been crawling large ecommerce stores such as Amazon, Walmart, Target apart from half dozen other specialty ecommerce stores for over 6 years now.</p><p>We clean and package that data to clients who use it for competitor analysis, market research and for driving decisions about product development and innovation.</p><p>For this article, we randomly sampled our database for plant and whey protein products in the American market while ensuring that the product studied was still on sale right now at atleast 3 major US retailers.</p><p>Our database contains product variations for package weight, flavor etc. We deduplicated the data for package weight since that will not affect our analysis here but we chose to keep unique flavors of same products as different entries since in our experience, brands use different sweeteners for different flavors.</p><h4>1.2 Machine Vision models for Nutritional Labels and Ingredients List</h4><p>Our clients need nutritional and ingredient list analysis on a very large scale (hundreds of thousands of images), hence, we have developed proprietary machine vision models that extracts nutritional labels and ingredients list with 98% accuracy.</p><p>We combine that data for each product SKU with raw crawl data (that we talked in section 1.1) from ecommerce store &amp; get an aggregate dataset that is used here for analysis.</p><h4>1.3 Previous Articles in this Series</h4><p>This article is part of our larger ongoing series where we have looked at serving sizes, protein concentrations, and packaging size variations of <a href="https://www.linkedin.com/pulse/shrinkflation-action-hidden-evolution-whey-protein-packaging-patel-sq3jf/">whey proteins</a> and <a href="https://www.linkedin.com/pulse/plant-protein-paradox-when-premium-pricing-meets-reality-jay-m-patel-u1kcf/">plant proteins</a> respectively.</p><h3>2. The Data Tells a Striking Story</h3><p>Figure 1 shows our sweetener analysis for whey protein powders on the market.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e_N50UnZw1rashyRiFl-2Q.png" /><figcaption>Figure 1: Whey Protein Sweetener Analysis (Specrom Analytics/Jay M. Patel © 2025)</figcaption></figure><p>We define basic, premium and standard product tiers on basis of the product pricing within its segment as well as use of premium ingredients such as whey protein hydrolysate vs whey protein isolate vs whey protein concentrates.</p><p>We have performed a similar analysis with plant protein powders in figure 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4PJBi5I_16OxZjxDxf-YuA.png" /><figcaption>Figure 2: Plant Protein Sweetener Analysis (Specrom Analytics/Jay M. Patel © 2025)</figcaption></figure><p>Our analysis of current market offerings reveals two completely different sweetener philosophies:</p><p><strong>Plant Proteins: The Natural Purists</strong></p><ul><li>87.9% use natural sweeteners</li><li>Only 1.9% contain artificial sweeteners</li><li>Stevia dominates at 55.6% usage rate</li><li>Monk fruit follows at 24.9%</li></ul><p><strong>Whey Proteins: The Artificial Pragmatists</strong></p><ul><li>51.9% use artificial sweeteners</li><li>Sucralose appears in 51.1% of products</li><li>Acesulfame potassium in 27.5%</li><li>Natural sweeteners present but secondary</li></ul><p><strong><em>This isn’t a minor preference , it’s a fundamental philosophical divide that reflects different consumer bases, price points, and brand positioning.</em></strong></p><h4>2.1. The Rise of the “Pure” Movement</h4><p>Nearly 20% of plant proteins now contain zero sweeteners, compared to just 9.5% of whey proteins.</p><p>This “pure” positioning appeals to consumers who view any sweetener — natural or artificial — as unnecessary processing.</p><p>These products often command premium prices, marketed to purists who add their own fruits or blend with naturally sweet ingredients.</p><p>The emergence of unsweetened options represents a maturing market where brands can succeed by subtraction rather than addition.</p><p>It’s the protein powder equivalent of black coffee , an acquired taste that signals sophistication.</p><h4>2.2 The Sweetener Evolution: Following Health Headlines</h4><p>The past five years have witnessed seismic shifts in sweetener preferences:</p><ul><li><strong>The Aspartame Exodus</strong>: Once ubiquitous, aspartame has virtually disappeared from new formulations following WHO’s 2023 classification as “possibly carcinogenic.” Products reformulated overnight, with most migrating to sucralose or stevia.</li><li><strong>Sugar Alcohol Retreat</strong>: Erythritol, xylitol, and maltitol are declining rapidly. Recent studies linking erythritol to cardiovascular events, combined with long-standing digestive complaints, have made these once-popular “natural” alternatives marketing liabilities. Plant proteins show 10.2% usage while whey sits at just 0.9%.</li><li><strong>Monk Fruit Moment:</strong> Pretty much unknown to mass market about five years ago, monk fruit (luo han guo) now appears in 24.9% of plant proteins. Its appeal? Zero calories, natural origin, and no bitter aftertaste. The premium pricing hasn’t deterred adoption among quality-focused brands.</li><li><strong>Stevia’s Refinement:</strong> Early stevia products suffered from metallic aftertastes. Modern formulations use specific steviol glycosides (Reb A, Reb M) that eliminate bitterness. The result: stevia leads plant protein sweetening at 55.6%.</li></ul><h4>2.3 Same Sweetener, Different Names: The Marketing Shell Game</h4><p>Consumers face a bewildering array of names for identical ingredients:</p><ul><li>Steviol glycosides = Stevia leaf extract = Reb A = Rebiana</li><li>Monk fruit = Luo han guo = Swingle fruit extract</li><li>Sucralose = Splenda (brand name often used generically)</li><li>Acesulfame potassium = Ace-K = Acesulfame K</li></ul><p>This nomenclature confusion isn’t accidental. Brands choose names that align with their positioning , “steviol glycosides” sounds more scientific, while “stevia leaf extract” emphasizes natural origins.</p><h4>2.4 The Plant vs Whey Divide: Why Such Different Approaches?</h4><ul><li><strong>Consumer Demographics</strong>: Plant protein consumers skew more health-conscious, environmentally aware, and willing to pay premiums for “clean” ingredients. They chose plant proteins partially to avoid artificial ingredients — artificial sweeteners would betray that promise.</li><li><strong>Flavor Masking Requirements</strong>: Plant proteins have stronger, more complex off-flavors requiring more sophisticated sweetening strategies. Natural sweeteners’ subtle profiles can complement rather than mask these notes.</li><li><strong>Price Positioning</strong>: Plant proteins average 40% higher prices than whey. This premium positioning demands premium ingredients. Using sucralose in a $50 plant protein would create cognitive dissonance.</li><li><strong>Regulatory Anticipation</strong>: Plant protein brands, often newer and more nimble, anticipate future restrictions on artificial sweeteners. Building with natural sweeteners avoids future reformulation costs.</li></ul><h4>2.5 Where Sweeteners Appear Matters</h4><p>Whey and plant protein categories position sweeteners around the 7th position in ingredient lists; late enough to signal they’re not primary ingredients, early enough to ensure adequate sweetness.</p><p>This positioning represents a careful balance between efficacy and clean label perception.</p><h4>2.6 The Caloric Consideration</h4><p>Figure 3 and figure 4 shows caloric sweeteners for plant proteins and whey protein respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/1*tkOKffCVtheKJ_OM8SM1dg.png" /><figcaption>Figure 3: Plant Protein Product Sweeteners: Caloric vs Non caloric (Specrom Analytics/Jay M. Patel © 2025)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*WkI-U3SStIIaJSOqLBjCUQ.png" /><figcaption>Figure 4: Whey Protein Product Sweeteners: Caloric vs Non caloric (Specrom Analytics/Jay M. Patel © 2025)</figcaption></figure><p>Surprisingly, 41% of plant proteins contain caloric sweeteners versus just 25% of whey proteins.</p><p>This counterintuitive finding reflects plant proteins’ emphasis on “natural” over “zero calorie.”</p><p>Coconut sugar, date syrup, and agave nectar add 20–40 calories per serving but align with whole-food philosophies.</p><h4>2.7 The Multi-Sweetener Strategy: Why Brands Stack Their Sweet</h4><p>Figure 5 and figure 6 shows percentage of products by distinct sweetener count.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*uqZ5AnstHGAo3-0HtkTUlA.png" /><figcaption>Figure 5: % of Plant Protein Products By Sweetener Count (Specrom Analytics/Jay M. Patel © 2025)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*8FYmvc-YhETCv2HXzUTMXg.png" /><figcaption>Figure 6: % of Whey Protein Products By Sweetener Count (Specrom Analytics/Jay M. Patel © 2025)</figcaption></figure><p>Our data reveals that 46.5% of whey proteins and 37.3% of plant proteins use two sweeteners. <strong><em>Why this complexity?</em></strong></p><ul><li><strong>Taste Synergy</strong>: Sucralose provides initial sweetness, acesulfame-K reduces aftertaste. Stevia offers natural credentials, monk fruit smooths the flavor profile. Combined, they create a more sugar-like experience than any single sweetener.</li><li><strong>Cost Optimization</strong>: High-intensity sweeteners like sucralose are expensive. Combining with cheaper alternatives maintains sweetness while controlling costs.</li><li><strong>Hedging Bets</strong>: Using multiple sweeteners reduces dependency on any single ingredient that might face future regulatory scrutiny or supply chain issues.</li></ul><h3><strong>3. Conclusion: The Future is Selectively Sweet</strong></h3><p>The sweetener landscape reveals two parallel universes in protein supplementation. Whey proteins optimize for taste and cost using artificial sweeteners, while plant proteins pursue premium positioning through natural alternatives.</p><p>We recommend the readers to fully embrace our <a href="https://www.linkedin.com/pulse/why-you-shouldnt-brand-loyal-protein-powders-most-jay-m-patel-zqy3f/">nutritional portfolio theory and do aggressive brand rotation</a> to stay safe, healthy while avoiding exposure to one type of sweetener.</p><p>Consumers who hedged their bets with aspartame and used bunch of other products stayed safe while others who went all in on aspartame and consumed a lot were later found to be at a greater risk of cancer.</p><p>Looking forward, expect:<br>- Continued decline of sugar alcohols across both categories<br>- Growth in unsweetened options as consumers become more sophisticated<br>- Novel natural sweeteners (allulose, thaumatin) entering the market<br>- Potential regulatory changes affecting artificial sweeteners<br>- More brands offering multiple sweetness levels of the same product</p><p>The sweetener strategy has become a proxy for brand philosophy. In choosing between sucralose and stevia, brands aren’t just selecting ingredients — they’re choosing their tribe. As the market matures, success will come not from universal solutions but from precisely matching sweetener strategies to target consumer values.</p><p><strong><em>The sweet truth? There’s no single right answer, only the right answer for your specific consumer.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=05c5fee3709d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Traveling Local Matters: The Environmental Toll of Going the Distance]]></title>
            <link>https://jay-68918.medium.com/why-traveling-local-matters-the-environmental-toll-of-going-the-distance-03e21930e7f0?source=rss-8dc22d6b3ace------2</link>
            <guid isPermaLink="false">https://medium.com/p/03e21930e7f0</guid>
            <category><![CDATA[sustainability]]></category>
            <category><![CDATA[environment]]></category>
            <category><![CDATA[culture]]></category>
            <category><![CDATA[world]]></category>
            <category><![CDATA[travel]]></category>
            <dc:creator><![CDATA[Jay M. Patel]]></dc:creator>
            <pubDate>Wed, 20 Aug 2025 12:05:18 GMT</pubDate>
            <atom:updated>2025-08-20T12:05:18.491Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rbNn5kof6C3ZJfFC" /><figcaption>Photo by <a href="https://unsplash.com/@dubai_love_story?utm_source=medium&amp;utm_medium=referral">Agnieszka Stankiewicz</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the earlier part of the sustainability series we talked about <a href="https://medium.com/p/bcffb7e51dfd">“eat global, travel local”</a> and mentioned why eating global can help you lead healtier, happier lives.</p><p>In this part, let us focus on why traveling local is the way to go if you want to minimize environmental impact and lead a sustainable life.</p><p>Let us re-define local travel as someplace you can reach easily using rail, road or waterways generally in your own country as compared to traveling in airplanes.</p><p>For our American and other North American readers, that can basically mean traveling thousands of miles. However, lets not be pedantic and agree that for someone living in Florida even though they can technically drive all the way to Alaska, they will not say that their alaska roadtrip is “traveling local” and most likely a Floridian when traveling local would mean somewhere in their homestate or if they are from the panhandle, than somewhere in the south US.</p><p>We will use common sense definition of what local travel is but in most cases it should be under few hundred miles.</p><h4><strong>Current Carbon Emissions from Aviation</strong></h4><p>Aviation represents one of the most carbon-intensive forms of transportation available today. The industry accounts for approximately 2.4% to 2.5% of global CO₂ emissions from fossil sources, but when including non-CO₂ climate impacts such as contrails, water vapor, and nitrogen oxides, aviation contributes around 3.5% to 4% of human-induced global warming. This figure is particularly concerning given that only a small fraction of the global population flies regularly.</p><p>The emissions per mile are staggering. On average, a commercial aircraft produces approximately 53.3 pounds of CO₂ per mile traveled. To put this in perspective, a single round-trip flight from London to San Francisco emits around 5.5 tonnes of CO₂ equivalent per person — more than twice the emissions produced by a family car in an entire year.</p><p>Even shorter flights carry enormous environmental costs; a round-trip flight from London to Berlin generates approximately 0.6 tonnes of CO₂ per passenger.</p><h4><strong>The Inequality of Flight Frequency</strong></h4><p>The environmental impact of aviation is disproportionately driven by a small segment of the population.</p><p>Research indicates that only 12–15% of people in wealthy countries like the UK and US are frequent fliers, while globally, just 3% of the population takes regular flights. This means that a tiny fraction of humanity is responsible for a massive share of transportation-related emissions.</p><p>Per capita flight statistics reveal stark disparities.</p><p>In 2018, Americans took an average of 2.717 flights per person, while countries like India averaged only 0.126 flights per capita.</p><p>Ireland topped the list with 34.507 flights per person, highlighting how certain populations have vastly disproportionate aviation footprints.</p><h4><strong>Projected Growth in Aviation Emissions</strong></h4><p>Despite growing environmental awareness, aviation emissions are projected to continue rising dramatically. Current projections suggest the aviation industry will emit 79 million tonnes of CO₂ annually by 2050 — representing only a 3% reduction from 2019 levels. This minimal improvement occurs despite anticipated increases in sustainable aviation fuel adoption.</p><p>Air passenger traffic is expected to more than double by 2050, with European aviation emissions potentially increasing by 46% by 2040. Even conservative growth projections would result in aviation emissions that are 46% higher in 2040 compared to 1990 levels, nowhere near the reductions needed for climate targets.</p><p><strong>The Acceleration Problem</strong></p><p>Aviation emissions demonstrate concerning acceleration trends. Between 1990 and 2019, global aviation emissions approximately doubled despite efficiency improvements that made aircraft more than twice as energy efficient. The industry’s emissions peaked at over 920 million tons of CO₂ in 2019, marking a 30% increase from 2013 levels in just six years.</p><p>This growth pattern reveals a fundamental problem: efficiency gains are being overwhelmed by increased demand. While individual flights have become more fuel-efficient, the total number of flights has grown exponentially, resulting in higher overall emissions.</p><h4>Ground Transportation is Far More Sustainable</h4><p>Local travel options demonstrate dramatically lower carbon footprints compared to aviation. Walking and cycling produce virtually no emissions, while public transportation offers highly efficient alternatives for longer distances within local regions.</p><p>Trains are particularly effective, with emissions ranging from as low as 6g CO₂e per passenger kilometer for efficient services like the Eurostar, to an average of 41g CO₂e per passenger kilometer for national rail services.</p><p>Ground transportation comparisons reveal the environmental advantages of staying local:</p><ul><li>Intercity buses: 0.15 lbs CO₂ per passenger mile</li><li>Trains: 0.30 lbs CO₂ per passenger mile</li><li>Cars (single occupancy): nearly 0.5 lbs CO₂ per passenger mile</li></ul><p>Taking a bus instead of flying can decrease your carbon footprint by up to 77.5%, while using trains instead of domestic flights reduces emissions by approximately 86%.</p><p>Even driving alone is typically more environmentally friendly than flying for domestic distances.</p><h4><strong>Other Benefits of Local Travel and Tourism</strong></h4><p>So it’s pretty straightforward that further away you travel, higher is your carbon footprint. However, there are some other additional advantages of traveling local.</p><h4><strong>Supporting Local Conservation</strong></h4><p>Tourism dollars spent locally directly fund conservation efforts in nearby areas.</p><p>Many local destinations use entry fees and tourism income to support wildlife conservation, maintain parks, and rehabilitate natural areas.</p><p>This creates a positive feedback loop where local recreation supports environmental protection in your immediate region.</p><p>Local tourism often overlaps with eco-tourism practices, where visitors encounter businesses that prioritize environmental stewardship, waste reduction, and sustainable operations.</p><p>Supporting these local enterprises reinforces the importance of sustainability in tourism while building community resilience.</p><h4><strong>Economic and Social Benefits</strong></h4><p>Local travel supports community-based tourism that ensures economic benefits remain within local communities while maintaining cultural diversity and environmental integrity.</p><p>This model creates jobs and income opportunities for people in your region while reducing the negative impacts associated with mass tourism in distant locations.</p><h4>Final thoughts</h4><p>I hope we might have convinced you to take the next holiday somewhere local instead of hopping on a low cost airline and flying off to some exotic locales.</p><p>In the next part of the series we will talk about how sase of travel has diminshed local stewardships in some locations that had lead to severe local air and water pollution which in turn drives local population to not only holiday somewhere else but in fact migrate out of the area, which in turn makes the problem even worse.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=03e21930e7f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sustainable Living Through “Eat Global, Travel Local”: Balancing Nutritional Diversity On…]]></title>
            <link>https://jay-68918.medium.com/sustainable-living-through-eat-global-travel-local-balancing-nutritional-diversity-on-bcffb7e51dfd?source=rss-8dc22d6b3ace------2</link>
            <guid isPermaLink="false">https://medium.com/p/bcffb7e51dfd</guid>
            <category><![CDATA[nutrition]]></category>
            <category><![CDATA[food]]></category>
            <category><![CDATA[health]]></category>
            <category><![CDATA[healthy-lifestyle]]></category>
            <category><![CDATA[sustainability]]></category>
            <dc:creator><![CDATA[Jay M. Patel]]></dc:creator>
            <pubDate>Wed, 20 Aug 2025 11:35:07 GMT</pubDate>
            <atom:updated>2025-08-20T11:35:07.952Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Sustainable Living Through “Eat Global, Travel Local”: Balancing Nutritional Diversity On Individual Level.</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Xko-qQcqfepRB_V4" /><figcaption>Photo by <a href="https://unsplash.com/@milo_photo?utm_source=medium&amp;utm_medium=referral">Michael Lock</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The pre-industrial maxim of “eating local” is no longer a sufficient framework for sustainable living in our interconnected world.</p><p>Modern environmental challenges, regulatory inconsistencies, climate change impacts on traditional agriculture, and contamination risk management necessitate a more nuanced approach: <strong>eating global while traveling local</strong>.</p><p>This strategy maximizes nutritional benefits and safety through diversified food sourcing while offsetting environmental impacts through reduced leisure travel emissions.</p><h4>The Limitations of Modern Local Food Systems</h4><p><strong>The contemporary local food movement, while well-intentioned, faces significant limitations that challenge its universal applicability.</strong></p><p><strong>Research demonstrates that local food systems are not necessarily more sustainable than systems operating at larger scales.</strong></p><p>The sustainability of local food depends heavily on regional agricultural capabilities, regulatory frameworks, and environmental conditions rather than proximity alone.</p><p><a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC8147589/">Stein and Santini (2021)</a> analyzed whether local food increases sustainability or lowers greenhouse emission, and they found it does neither and sustainability should be looked at based on overall lifecycle of the food and not just at the transportation portion of it.</p><h4>Why should we eat globally sourced food items?</h4><h4><strong>1. Regulatory Weakness in Local Food Systems</strong></h4><p>Many regions suffer from weak regulatory frameworks that fail to ensure food safety.</p><p>Developing countries particularly struggle with poor infrastructure, inadequate transportation and storage facilities, and weak food safety regulations.</p><p>Among the developed world, US has almost invariably lagged behind food safety regulations in past two decades and their pesticide and PFAS lobby has been super effective at slowing down regulatory process.</p><p>Food contamination is now a major concern, with poor agricultural practices leading to high levels of pesticide residues.</p><p>This issue is even more pronounced if you dont live in US. <a href="https://documents1.worldbank.org/curated/en/484371545400065950/pdf/133154-PUB-PUBLIC-9781464813450.pdf">Steven Jaffe and team (2019) from the world Bank</a> reported that most low- and middle-income countries have weak food safety systems in terms of scientific evidence, necessary infrastructure, trained human resources, and enforceable regulations.</p><p>For example, pesticide contamination patterns vary dramatically by region and regulatory strength.</p><p><a href="https://www.sciencedirect.com/science/article/pii/S2589004225011228">Huang and Li (2025)</a> research on global freshwater contamination found that Africa was identified as a priority for freshwater pesticide control, with most countries showing relatively high contamination scores.</p><p>The study revealed significant disparities in contamination among countries, with robust pesticide regulations associated with reduced surface freshwater pollution.</p><p>Regional variations in pesticide residue detection rates demonstrate this disparity clearly. In a recent study by Alokail and team (2023) from Saudi Arabia, pesticide residue detection rates ranged from 70.7% in the western region to 89.7% in the central region.</p><h4><strong>2. Climate Change and the Loss of Local Agricultural Heritage</strong></h4><p>Climate change has fundamentally altered the viability of traditional local food systems.</p><p>The UN’s Food and Agriculture Organization estimates that globally 75% of crop diversity was lost between 1900 and 2000. This dramatic biodiversity loss makes agriculture more vulnerable to climate change, pests, and diseases.</p><p>Heirloom varieties that once thrived in specific regions are increasingly threatened by changing climate conditions.</p><p>These traditional cultivars, which survived climate variations and extreme conditions for generations, were abandoned in favor of a handful of commercial cultivars. However, the genetic diversity preserved in these varieties represents crucial adaptive capacity for climate change.</p><p>Climate change is altering the geographic distribution of optimal growing conditions. <strong>Crops that once flourished in certain regions may now grow better thousands of miles away under different climate conditions.</strong></p><p>For example, farmers in the Eastern Himalayas have adopted varieties grown at lower altitudes in response to rising temperatures. This suggests that accessing genetic diversity from global sources may be more sustainable than forcing unsuitable local production.</p><h4><strong>3. Reduce Long Term Contamination Exposure Risks by Global Sourcing</strong></h4><p>The concept of eating global aligns with the <strong>nutritional portfolio theory, which applies diversification principles to reduce contamination risks</strong>.</p><p>Just as financial portfolio diversification reduces investment risk by spreading exposure across different assets, <strong>sourcing food from diverse geographic regions and production systems reduces long term exposure to region-specific contamination patterns</strong>.</p><p>Our idea is that its almost impossible to completely avoid exposure to potentially harmful contaminants just like its almost impossible to avoid risks when trading in stock markets.</p><p>However, diversification can help mitigate risks in stock markets, similarly, if you adopt a portfolio theory and diversify sources of your food items, than you get similar benefits.</p><p>The hypothesis is that when you eat a globally sourced diet, chances are you will only be exposed to potentially harmful contaminants in a limited way and hopefully one single harmful origin of food will not dominate your diet so you will have less chances of getting sick due to it (which wont be the case if you eat local and your local water/soil itself is contaminated).</p><h4><strong>4. Accessing Heirloom Diversity Through Global Sourcing</strong></h4><p>Many regions have preserved traditional crop varieties that are no longer viable in their original locations due to climate change. The communities of the Potato Park in Cusco, Peru, hold about 1,500 varieties of the world’s 4,000 potato varieties.</p><p>Similarly, indigenous farmers in the Karst mountains of Southwest China, coastal Kenya, and the Bolivian Andes maintain traditional varieties with unique climate-adapted characteristics.</p><p>Heirloom varieties contribute to biodiversity by maintaining crop diversity, preserving genetic resources, and supporting ecosystem services.</p><p>These varieties often exhibit greater phenotypic plasticity, meaning they can adjust their physical characteristics in response to environmental changes. For instance, an heirloom tomato variety grown in a hotter, drier climate might produce smaller fruit or develop deeper roots to access water.</p><p><strong>Global access to these diverse genetic resources provides nutritional and safety benefits that local production cannot match</strong>.</p><h4><strong>5. Carbon Footprint Reality of Food Miles vs. Production Methods</strong></h4><p>The environmental impact of global food sourcing is often overestimated due to misconceptions about food miles.</p><p>Our world in data <a href="https://ourworldindata.org/food-choice-vs-eating-local">did a detailed analysis</a> on increased greenhouse gas emissions on bringing beef from across the world, and they found increase in greenhouse gas emissions was less than less than 1% of beef’s total greenhouse gas emissions.</p><p><strong>Transport-related greenhouse gas emissions represent only 5–6% of total greenhouse gas emissions from global food production.</strong></p><p>What you eat matters far more for your carbon footprint than where your food comes from. <a href="https://bigthink.com/the-present/eating-local/">A comprehensive study</a> found that substituting less than one day per week’s worth of calories from beef and dairy products for chicken, fish, eggs, or plant-based alternatives reduces greenhouse gas emissions more than buying all food from local sources.</p><h4><strong>6. Local Production Can Increase Emissions</strong></h4><p>Several studies demonstrate cases where local production actually increases environmental impact.</p><p>Importing Spanish lettuce to the UK during winter months results in three to eight times lower emissions than producing it locally.</p><p>Similarly, tomatoes produced in greenhouses in Sweden used 10 times as much energy as tomatoes imported from Southern Europe where they were in season.</p><p>Local food systems significantly reduce the need for long-distance transportation, but may employ energy-intensive production methods that prioritize soil health and biodiversity.</p><p>The trade-off between transportation emissions and production emissions often favors efficient global production systems over local alternatives.</p><h4>Final Thoughts</h4><p>I hope we have convinced you why a path to truly sustainable living is by eating healthy, good, nutritious food products that can be sourced from around the world.</p><p>We are not saying that stop advocating for stricter regulation in your own local area for a cleaner pollution free lakes and rivers or help local communities clean up local rivers, lakes and streams. <strong>However, till the time that happens, we want to give ideas on mitigating your own personal exposure risk and best way to do it is by eating global.</strong></p><p>If enough of us do this than there will be a financial incentive for the local growers, regulators and government to prove to us consumers that local products are as good (if not better) than globally sourced ones and this may drive them to real change instead of them hiding behind “eat local” slogan.</p><p>If you truly want to offset impact of food miles in bringing the food to you, than you can seriously take a look at our next two articles that talk about traveling local and living sustainably.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bcffb7e51dfd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyzing User Sentiment & Insights from DeepSeek iOS App Reviews]]></title>
            <link>https://jay-68918.medium.com/analyzing-user-sentiment-insights-from-deepseek-ios-app-reviews-8331f91a9603?source=rss-8dc22d6b3ace------2</link>
            <guid isPermaLink="false">https://medium.com/p/8331f91a9603</guid>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[mobile-app-development]]></category>
            <category><![CDATA[apps]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Jay M. Patel]]></dc:creator>
            <pubDate>Thu, 20 Feb 2025 10:56:36 GMT</pubDate>
            <atom:updated>2025-02-24T11:40:13.068Z</atom:updated>
            <content:encoded><![CDATA[<p>DeekSeek has been making the top app charts for last few weeks since the app was released on both Apple app store and Google play store.</p><p>We thought to analyze Deepseek’s user reviews using <a href="https://www.specrom.com/appstoregenie/">AppStoreGenie</a>.</p><p>We used the <a href="https://www.specrom.com/ios-app-reviews/">AppStoreGenie review tracker</a> to download every new review that appeared on the Apple app store for Deepseek and later used the data report feature to generate the graphs.</p><p>Firstly, lets get word counts for all the reviews and plot that against rating value and against review date.</p><figure><img alt="this plot shows word count vs rating value. It shows an even distribution of word counts over 1,2,3,4,5 star reviews" src="https://cdn-images-1.medium.com/max/615/1*A41g9A_COfpRXWJJiym-3A.png" /><figcaption>word count vs rating value</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*O27Nfm3bbYbHz9O9vgpwbQ.png" /><figcaption>word count vs review date</figcaption></figure><p>There are a bunch of 1 star reviews with lots of text and that might indicate some issue with the app.</p><p>There is also definitely an uptick in review text count after 30 jan 2025; that could indicate either some fraudulent reviews trying to sink the review score, or on other hand, it can just be due to app gaining more popularity.</p><p>We can check if there was any flooding of reviews with highly positive or highly negative reviews by plotting the review star rating against time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*l6LqR0Ohwmb5FFweyLP-Zg.png" /><figcaption>Review Score vs Review Date</figcaption></figure><p>This seems pretty well balanced except that it seems that in the initial days of the app (pre 23 Jan 2025), it mostly had neutral and positive reviews, however, the week of January 29, the reviews per day really ramped up and as we can see, it started getting a lot of 1 star review (blue area) during that week.</p><p>Let us look at overall rating distribution or the histogram that is available on the app details page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*Nr5sb1zzpHciN89Jvnd04A.png" /><figcaption>Rating Distribution (Overall)</figcaption></figure><p>Rating Distribution (Overall) shows combined rating from users that have left a detailed text review (like the one we are analyzing here) and just a star rating.</p><p>Let us look at only users who actually left a text review.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*iuOQayLwti6Roe8K2aZnBw.png" /><figcaption>Rating Distribution (only considering those that left review text)</figcaption></figure><p>So our sample contains more 1 star reviews than what the overall ratings indicate.</p><p>Let us look at how the overall rating score has been dropping since last 6 weeks due to a bunch of 1 star reviews.</p><p>The blue line is the average rating till date. So every new review that is published changes the overall rating score; this curve visualizes the overall impact on the rating value with each new review.</p><p>If we see a string of “bad” reviews, than the average rating of this recent reviews sample (green line) will plunge below the overall rating score (orange line) and vice versa.</p><p>This could mean that user satisfaction in recent past is less than overall user sentiments across all app versions.</p><p>This decrease in the average rating till date does have a negative effect on the overall rating score and we can see that here (the orange line).</p><p>It seems that there was a strong onslaught of negative reviews around the week of Jan 30 (2025) making the blue line dipping below green and it has led in some decrease in overall rating, but now the overall rating has stabilized for now.</p><figure><img alt="Avg rating till date" src="https://cdn-images-1.medium.com/max/503/1*2-eWzaipZGLQr0I9_N_P9g.png" /><figcaption>Avg rating till date</figcaption></figure><p>Let us look at wordcloud of the review text itself.</p><figure><img alt="wordcloud from all the review text" src="https://cdn-images-1.medium.com/max/811/1*n-Qez-38BBzhGeQPNHrdTg.png" /><figcaption>wordcloud from all the review text</figcaption></figure><p>Its better if we plot just the 1 star reviews. Most one star reviewers are concerned about deepseek being a Chinese app, about their data or information being secure and lastly, about server being busy. Tiananmen square and propaganda also features in this wordcloud.</p><figure><img alt="wordcloud from the review text of 1 star reviews" src="https://cdn-images-1.medium.com/max/811/1*HoQ1m8_1605hPl5f-3hq6g.png" /><figcaption>Wordcloud of 1 star reviews</figcaption></figure><p>Let us instead look at 4 and 5 star reviews</p><figure><img alt="wordcloud from all the text of 4 and 5 star reviews" src="https://cdn-images-1.medium.com/max/811/1*i3w4Q7CtwEmYCGGnmjd-7A.png" /><figcaption>Wordcloud from all the text of 4 and 5 star reviews</figcaption></figure><p>In this case most reviewers seem to have compared deepseek to chatgpt, and it seems like the model is good or great.</p><p>The next step will be to analyze the Android version of the DeepSeek app and see if the user reviews are saying the same thing or not.</p><p>All of the pltos and data are generated on our <a href="https://www.specrom.com/apple-app-store-explorer/">AppStoreGenie platform</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8331f91a9603" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scraping Indeed job postings]]></title>
            <link>https://medium.com/web-data-extraction/scraping-indeed-job-postings-dae14b5967e3?source=rss-8dc22d6b3ace------2</link>
            <guid isPermaLink="false">https://medium.com/p/dae14b5967e3</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[jobs]]></category>
            <category><![CDATA[indeed]]></category>
            <dc:creator><![CDATA[Jay M. Patel]]></dc:creator>
            <pubDate>Fri, 14 May 2021 03:17:56 GMT</pubDate>
            <atom:updated>2021-05-14T03:17:56.062Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_aj0ReE3ash6TG8m" /><figcaption>Photo by <a href="https://unsplash.com/@kobuagency?utm_source=medium&amp;utm_medium=referral">KOBU Agency</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>So, what is the easiest way to get all the job posting details from Indeed.com?</p><h3>Option 1: Subscribe to Specrom’s Indeed Scraper API</h3><p>We have an <a href="https://www.specrom.com/indeed-scraper-api/">Indeed Scraper API</a> that will extract all the pertinent job posting information such as company name, city, snippet, job title etc. by just specifying a search query and a location.</p><p>The API is available on RapidAPI and there is a free trial with no credit card information required. Paid plans start at just $10.</p><h3>Option 2: Full service web scraping service.</h3><p>If you just need job postings data as a CSV or excel file, than simply contact us for our <a href="https://www.specrom.com/web-scraping-service/">full service web scraping service</a>. You can simply sit back and let us handle all the backend issues to get the data you need.</p><h3>Option 3: Scrape indeed.com on your own</h3><p>Python is great for web scraping and we will be using a library called Selenium to extract Job postings from Indeed for Atlanta, GA.</p><pre>### Using Selenium to extract Indeed.com&#39;s raw html source from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import Select import time from bs4 import BeautifulSoup import numpy as np import pandas as pd test_url = &#39;https://www.indeed.com/&#39; option = webdriver.ChromeOptions() option.add_argument(&quot;--incognito&quot;) chromedriver = r&#39;chromedriver.exe&#39; browser = webdriver.Chrome(chromedriver, options=option) browser.get(test_url) text_area = browser.find_element_by_id(&#39;text-input-what&#39;) text_area.send_keys(&quot;Web scraping&quot;) text_area2=browser.find_element_by_id(&#39;text-input-where&#39;) text_area.send_keys(&quot;Atlanta, GA&quot;) element = browser.find_element_by_xpath(&#39;//*[@id=&quot;whatWhereFormId&quot;]/div[3]/button&#39;) element.click() html_source = browser.page_source browser.close()</pre><h3>Using BeautifulSoup to extract Indeed job postings</h3><p>Once we have the raw html source, we should use a Python library called BeautifulSoup for parsing the raw html files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MqgPcnCDIPSSgqii.PNG" /></figure><p><strong>Figure 1: Inspecting the source of HTML source code for Indeed.com search query webpage</strong>.</p><h4>Extracting job titles</h4><p>From inspecting the html source, we see that job titles have h2 tags and belong to class ‘title’.</p><pre># extracting job titles soup=BeautifulSoup(html_source, &quot;html.parser&quot;) job_title_src = soup.find_all(&#39;h2&#39;, {&#39;class&#39;,&#39;title&#39;}) job_title_list = [] for val in job_title_src: try: job_title_list.append(val.get_text()) except: pass job_title_list #Output [&#39;Data Analyst&#39;, &#39;Data Analyst&#39;, &#39;Data/Reporting Analyst&#39;, &#39;Sr. Data Analyst&#39;, &#39;Data Analyst&#39;, &#39;Data Analyst&#39;, &#39;Behavior Data Analyst - Marcus Autism Center - Behavioral Analysis Core&#39;, &#39;Data Analyst&#39;, &#39;Data Analyst&#39;, &#39;Police Analyst&#39;, &#39;Data Analyst (2021-1614)&#39;, &#39;Employee Data Analyst&#39;, &#39;Data Analyst&#39;, &#39;Data and Research Analyst&#39;, &#39;Data Analyst (13255)&#39;]</pre><h4>Extracting company names</h4><p>The next step is extracting company names. We see that it is span tag of class ‘company’.</p><pre># extracting Indeed addresses company_name_src = soup.find_all(&#39;span&#39;,{&#39;class&#39;, &#39;company&#39;}) company_name_list = [] for val in company_name_src: company_name_list.append(val.get_text()) company_name_list # Output [&#39;KIPP Foundation&#39;, &#39;Emory University&#39;, &#39;City of Atlanta, GA&#39;, &#39;The Coca-Cola Company&#39;, &#39;KIPP Metro Atlanta Schools&#39;, &#39;Spartan Technologies&#39;, &quot;Children&#39;s Healthcare of Atlanta&quot;, &#39;ARK Solutions&#39;, &#39;Anthem&#39;, &#39;City of Forest Park, GA&#39;, &#39;Atrium CWS&#39;, &#39;Salesforce&#39;, &#39;Sovos Compliance&#39;, &#39;Southern Poverty Law Center&#39;, &#39;Baer Group&#39;]</pre><h4>Extracting snippets</h4><p>Snippets are couple of sentences of text that briefly explain the job postings. Along with job title and company name, these are one of the most important pieces of information to extract from each Indeed job posting result.</p><p>As a reference, refer to the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/0*RmvA1V47uT3BloRk.PNG" /></figure><p><strong>Figure 2: individual job postings from Indeed.com search results</strong>.</p><pre># extracting snippets from each job postings snippet_src = soup.find_all(&#39;div&#39;, {&#39;class&#39;, &#39;summary&#39;}) snippet_list = [] for val in snippet_src: snippet_list.append(val.get_text()) snippet_list[:3] # Output [&#39;\n\nMaintain and troubleshoot the integrity of data linkages between data source systems and data warehouse.\nCollaborate with other Data Team members to develop and...\n&#39;, &#39;\n\nCreates and maintains a data dictionary and meta data.\nAnalyzing data reporting data for clinical outcomes, qualitative and other types of research.\n&#39;, &#39;\n\n3 years of work experience in creation, reporting, and/or management of data or closely related tasks (not including data entry).\n&#39;]</pre><h3>Converting into CSV file</h3><p>You can take the lists above, and read it as a pandas DataFrame. Once you have the Dataframe, you can convert to CSV, Excel or JSON easily without any issues.</p><h3>Scaling up to a full crawler for extracting all Indeed job postings</h3><ul><li>Once you scale up to make thousands of requests to fetch all the pages, the indeed.com servers will start blocking your IP address outright or you will be flagged and will start getting CAPTCHA.</li><li>To make it more likely to successfully fetch data for all USA, you will have to implement:</li><li>rotating proxy IP addresses preferably using residential proxies.</li><li>rotate user agents</li><li>Use an external CAPTCHA solving service like 2captcha or anticaptcha.com</li></ul><p>After you follow all the steps above, you will realize that our pricing for <a href="https://www.specrom.com/web-scraping-service/">managed web scraping</a> or our <a href="https://www.specrom.com/indeed-scraper-api/">Indeed scraper API</a> is one of the most competitive in the market.</p><p><em>Originally published at </em><a href="https://www.specrom.com/blog/web-scraping-job-postings-on-indeed-using-python/"><em>https://www.specrom.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dae14b5967e3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/web-data-extraction/scraping-indeed-job-postings-dae14b5967e3">Scraping Indeed job postings</a> was originally published in <a href="https://medium.com/web-data-extraction">Web Data Extraction</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scraping Best Buy Stores Location]]></title>
            <link>https://medium.com/web-data-extraction/scraping-best-buy-stores-location-551c237cb1e1?source=rss-8dc22d6b3ace------2</link>
            <guid isPermaLink="false">https://medium.com/p/551c237cb1e1</guid>
            <category><![CDATA[store-locator]]></category>
            <category><![CDATA[best-buy]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[Jay M. Patel]]></dc:creator>
            <pubDate>Fri, 14 May 2021 03:14:09 GMT</pubDate>
            <atom:updated>2021-05-14T03:14:09.013Z</atom:updated>
            <content:encoded><![CDATA[<p>So, what is the easiest way to get CSV file of all the Best Buy store locations data in the USA?</p><h3>Buy the Best Buy store data from our data store</h3><p>There are over 950 Best Buy stores in USA and you buy the CSV file containing address, city, zip, latitude, longitude of each location in <a href="https://www.specrom.com/best-buy-store-locations-in-the-usa/">our data store for $50</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ADimHMUADoCgT0_4.png" /></figure><p><strong>Figure 1: Best Buy store locations.</strong> Source: <a href="https://www.specrom.com/best-buy-store-locations-in-the-usa/">Best Buy Store Locations dataset</a></p><p>If you are instead interested in scraping for locations on your own than continue reading rest of the article.</p><h3>Scraping Best Buy stores locator using Python</h3><p>We will keep things simple for now and try to web scrape Best Buy store locations for only one zipcode.</p><p>Python is great for web scraping and we will be using a library called Selenium to extract Best Buy store locator’s raw html source for zipcode 94536 (Fremont, CA area).</p><pre>### Using Selenium to extract Best Buy store locator&#39;s raw html source from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import Select import time from bs4 import BeautifulSoup import numpy as np import pandas as pd test_url = &#39;https://www.bestbuy.com/site/store-locator&#39; option = webdriver.ChromeOptions() option.add_argument(&quot;--incognito&quot;) chromedriver = r&#39;chromedriver_path&#39; browser = webdriver.Chrome(chromedriver, options=option) browser.get(test_url) time.sleep(5) html_source = browser.page_source browser.close()</pre><h3>Using BeautifulSoup to extract Best Buy store details</h3><p>Once we have the raw html source, we should use a Python library called BeautifulSoup for parsing the raw html files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-L-VexNBqbbTTQyW.PNG" /></figure><p><strong>Figure 2: Inspecting the source of Best buy store locator</strong>.</p><ul><li>We will extract store names.</li><li>The data will still require some cleaning to extract out store names, address line 1,address line 2, city, state, zipcode and phone numbers but thats just basic Python string manipulation and we will leave that as an exercise to the reader.</li></ul><pre># extracting Best Buy store names soup=BeautifulSoup(html_source, &quot;html.parser&quot;) store_name_list_src = soup.find_all(&#39;h2&#39;, {&#39;class&#39;,&#39;location-card-heading c-section-title heading-6 v-fw-medium &#39;}) store_name_list = [] for val in store_name_list_src: try: store_name_list.append(val.find(&#39;button&#39;).get_text()) except: pass #Output [&#39;Union City&#39;, &#39;Milpitas&#39;, &#39;Mountain View&#39;, &#39;Dublin&#39;, &#39;El Camino Real&#39;, &#39;San Carlos&#39;, &#39;Santana Row&#39;, &#39;The Plant&#39;, &#39;Blossom Hill&#39;, &#39;Emeryville&#39;, &#39;Pleasant Hill&#39;, &#39;Colma&#39;, &#39;San Francisco (13th and Harrison St.)&#39;, &#39;Slatten Ranch&#39;, &#39;Tracy&#39;]</pre><p>The next step is extracting addresses. Referring back to the inspect in the chrome browser, we see that each address text is in fact of the class name c-address so we just use the BeautifulSoup find_all method to extract that into a list.</p><pre># extracting best buy addresses addresses_src = soup.find_all(&#39;span&#39;,{&#39;class&#39;, &#39;loc-address&#39;}) addresses_src address_list = [] for val in addresses_src: address_list.append(val.get_text()) address_list[:10] # Output [&#39;31350 Courthouse DrUnion City, CA 94587&#39;, &#39;63 Ranch DrMilpitas, CA 95035&#39;, &#39;2460 E Charleston RdMountain View, CA 94043&#39;, &#39;4820 Dublin BlvdDublin, CA 94568&#39;, &#39;715 E El Camino RealMountain View, CA 94040&#39;, &#39;1127 Industrial RdSan Carlos, CA 94070&#39;, &#39;3090 Stevens Creek BlvdSan Jose, CA 95128&#39;, &#39;181 Curtner AveSan Jose, CA 95125&#39;, &#39;5065 Almaden ExpySan Jose, CA 95118&#39;, &#39;3700 Mandela PkwyOakland, CA 94608&#39;]</pre><p>We will extract Phone number of each store using a similar approach.</p><pre>phone_number_list = [] for val in phone_number_src: phone_number_list.append(val.get_text()) phone_number_list # Output [&#39;Phone Number510-441-2130&#39;, &#39;Phone Number408-942-0201&#39;, &#39;Phone Number650-903-0591&#39;, &#39;Phone Number925-829-7041&#39;, &#39;Phone Number408-738-8680&#39;, &#39;Phone Number650-622-0050&#39;, &#39;Phone Number408-241-6040&#39;, &#39;Phone Number408-297-0701&#39;, &#39;Phone Number408-979-1591&#39;, &#39;Phone Number510-420-0323&#39;, &#39;Phone Number925-988-0256&#39;, &#39;Phone Number650-756-8711&#39;, &#39;Phone Number415-626-9682&#39;, &#39;Phone Number925-513-4995&#39;, &#39;Phone Number209-832-2166&#39;]</pre><h3>Converting into CSV file</h3><p>You can take the lists above, and read it as a pandas DataFrame.</p><pre>import pandas as pd df = pd.DataFrame({&quot;store_name&quot;:store_name_list,&quot;address&quot;:address_list, &quot;phone_number&quot;:phone_number_list}) df.to_csv(&quot;best_buy_store_locations.csv&quot;) df.head() #Output address phone_number store_name 0 31350 Courthouse DrUnion City, CA 94587 Phone Number510-441-2130 Union City 1 63 Ranch DrMilpitas, CA 95035 Phone Number408-942-0201 Milpitas 2 2460 E Charleston RdMountain View, CA 94043 Phone Number650-903-0591 Mountain View 3 4820 Dublin BlvdDublin, CA 94568 Phone Number925-829-7041 Dublin 4 715 E El Camino RealMountain View, CA 94040 Phone Number408-738-8680 El Camino Real</pre><h3>Scaling up to a full crawler for extracting all Best Buy store locations in USA</h3><ul><li>Once you have the above scraper that can extract data for one zipcode/city, you will have to iterate through all the US zip codes.</li><li>it depends on how much coverage you want, but for a national chain like Best Buy you are looking at running the above function 100,000 times or more to ensure that no region is left out.</li><li>Once you scale up to make thousands of requests, the bestbuy.com servers will start blocking your IP address outright or you will be flagged and will start getting CAPTCHA.</li><li>To make it more likely to successfully fetch data for all USA, you will have to implement:</li><li>rotating proxy IP addresses preferably using residential proxies.</li><li>rotate user agents</li><li>Use an external CAPTCHA solving service like 2captcha or anticaptcha.com</li></ul><p>After you follow all the steps above, you will realize that our pricing($50) for web scraped store locations data for <a href="https://www.specrom.com/best-buy-store-locations-in-the-usa/"><strong>all Best Buy stores locations</strong></a> is one of the most competitive in the market.</p><p><em>Originally published at </em><a href="https://www.specrom.com/blog/scraping-best-buy-stores-location/"><em>https://www.specrom.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=551c237cb1e1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/web-data-extraction/scraping-best-buy-stores-location-551c237cb1e1">Scraping Best Buy Stores Location</a> was originally published in <a href="https://medium.com/web-data-extraction">Web Data Extraction</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Web Scraping search engine result pages (SERP) from Google.com]]></title>
            <link>https://medium.com/web-data-extraction/web-scraping-search-engine-result-pages-serp-from-google-com-4aaef55f9d69?source=rss-8dc22d6b3ace------2</link>
            <guid isPermaLink="false">https://medium.com/p/4aaef55f9d69</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[erp]]></category>
            <dc:creator><![CDATA[Jay M. Patel]]></dc:creator>
            <pubDate>Fri, 14 May 2021 03:09:51 GMT</pubDate>
            <atom:updated>2021-05-14T03:09:51.808Z</atom:updated>
            <content:encoded><![CDATA[<p>So, what is the easiest way to scrape Google search result pages (SERP) ?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lTPO1F_T4WEpOmAT.PNG" /></figure><p><strong>Figure 1: Inspecting the source of HTML source code for Google search result pages (SERP) webpage</strong>.</p><h3>Option 1: Subscribe to Specrom’s Google Scraper API</h3><p>We have an <a href="https://www.specrom.com/google-scraper-api/">Google Scraper API</a> that will extract all the relevant information such as SERP position, title, url, snippet by simply specifying a keyword.</p><p>The API is available on RapidAPI and there is a free trial with no credit card information required. Paid plans start at just $10.</p><p>This option can allow you to fetch results for hundreds of keywords without any complex scraping logic.</p><h3>Option 2: Full service web scraping service.</h3><p>If you just need SERP data as a CSV or excel file, than simply contact us for our <a href="https://www.specrom.com/web-scraping-service/">full service web scraping service</a>. You can simply sit back and let us handle all complexities of web scraping a site like Google that has plenty of anti-scraping protections built in to try and dissuade from people scraping it in bulk.</p><h3>Option 3: Scrape Google.com on your own</h3><p>Python is great for web scraping and we will be using a library called Selenium to extract results for the keywords “web scraping”.</p><pre>### Using Selenium to extract Google.com&#39;s raw html source from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import Select import time from bs4 import BeautifulSoup import numpy as np import pandas as pd test_url = &#39;https://www.google.com&#39; option = webdriver.ChromeOptions() option.add_argument(&quot;--incognito&quot;) chromedriver = r&#39;chromedriver.exe&#39; browser = webdriver.Chrome(chromedriver, options=option) browser.get(test_url) text_area = browser.find_element_by_id(&#39;gLFyf gsfi&#39;) text_area.send_keys(&quot;Web scraping&quot;) element = browser.find_element_by_xpath(&#39;//*[@id=&quot;tsf&quot;]/div[1]/div[1]/div[2]/button/div/span&#39;) element.click() html_source = browser.page_source browser.close()</pre><h3>Using BeautifulSoup to extract Google SERP data</h3><p>Once we have the raw html source, we should use a Python library called BeautifulSoup for parsing the raw html files.</p><p>As a reference, refer to the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/937/0*CyRNrl-I8GNG2moj.PNG" /></figure><p><strong>Figure 2: individual search results from Google SERP page</strong>.</p><h4>Extracting titles for each SERP result</h4><p>From inspecting the html source, we see that titles have h3 tags and belong to class ‘LC20lb DKV0Md’.</p><pre># extracting titles soup=BeautifulSoup(html_source, &quot;html.parser&quot;) serp_title_src = soup.find_all(&#39;h3&#39;, {&#39;class&#39;,&#39;LC20lb DKV0Md&#39;}) serp_title_list = [] for val in serp_title_src: try: serp_title_list.append(val.get_text()) except: pass serp_title_list #Output [&#39;Web scraping - Wikipedia&#39;, &#39;Web Scraper - The #1 web scraping extension&#39;, &#39;Beautiful Soup: Build a Web Scraper With Python - Real Python&#39;, &#39;What is Web Scraping and What is it Used For? | ParseHub&#39;, &#39;ParseHub | Free web scraping - The most powerful web scraper&#39;, &#39;Tutorial: Web Scraping with Python Using Beautiful Soup - Dataquest&#39;, &#39;Web Scraper - Free Web Scraping&#39;, &#39;Web Scraping Explained - WebHarvy&#39;, &#39;Web Scraping with Python - Beautiful Soup Crash Course - YouTube&#39;, &#39;What Is Web Scraping and 5 Findings about Web Scraping | Octoparse&#39;, &#39;What Is Web Scraping And How Does Web Crawling Work?&#39;, &#39;Data Scraping | Web Scraping | Screen Scraping | Extract - Import.io&#39;, &#39;Web Scraping Made Easy: The Full Tutorial - OutSystems&#39;, &#39;Web scraping with JS | Analog Forest&#39;, &#39;Web Scraping, Data Extraction and Automation · Apify&#39;, &#39;Web Scraping with Python: Everything you need to know (2021)&#39;, &#39;What is Web Scraping and How to Use It? - GeeksforGeeks&#39;, &#39;Web Scraping: The Comprehensive Guide for 2020 - ProWebScraper&#39;, &#39;Introduction to web scraping - Library Carpentry&#39;]</pre><h4>Extracting URLs</h4><p>The next step is extracting URLs of individual SERP results. We see that it is div tag of class ‘yuRUbf’.</p><pre># extracting URLs from Google SERP results URLs_src = soup.find_all(&#39;div&#39;,{&#39;class&#39;, &#39;yuRUbf&#39;}) url_list = [] for val in URLs_src: url_list.append(val.find(&#39;a&#39;)[&#39;href&#39;]) url_list # Output [&#39;https://en.wikipedia.org/wiki/Web_scraping&#39;, &#39;https://webscraper.io/&#39;, &#39;https://realpython.com/beautiful-soup-web-scraper-python/&#39;, &#39;https://www.parsehub.com/blog/what-is-web-scraping/&#39;, &#39;https://www.parsehub.com/&#39;, &#39;https://www.dataquest.io/blog/web-scraping-python-using-beautiful-soup/&#39;, &#39;https://chrome.google.com/webstore/detail/web-scraper-free-web-scra/jnhgnonknehpejjnehehllkliplmbmhn%3Fhl%3Den&#39;, &#39;https://www.webharvy.com/articles/what-is-web-scraping.html&#39;, &#39;https://www.youtube.com/watch%3Fv%3DXVv6mJpFOb0&#39;, &#39;https://www.octoparse.com/blog/what-is-web-scraping&#39;, &#39;https://www.zyte.com/learn/what-is-web-scraping/&#39;, &#39;https://www.import.io/product/extract/&#39;, &#39;https://www.outsystems.com/blog/posts/web-scraping-tutorial/&#39;, &#39;https://qoob.cc/web-scraping/&#39;, &#39;https://apify.com/&#39;, &#39;https://www.scrapingbee.com/blog/web-scraping-101-with-python/&#39;, &#39;https://www.geeksforgeeks.org/what-is-web-scraping-and-how-to-use-it/&#39;, &#39;https://prowebscraper.com/blog/what-is-web-scraping/&#39;, &#39;https://librarycarpentry.org/lc-webscraping/&#39;]</pre><h4>Extracting snippets</h4><p>Snippets are couple of sentences of text that briefly explain the content in each of the result webpages.</p><p>Along with title and URL, snippets are one of the most important pieces of information to extract from each Google SERP page.</p><pre># extracting snippets from each SERP listing snippet_src = soup.find_all(&#39;span&#39;, {&#39;class&#39;, &#39;aCOpRe&#39;}) snippet_list = [] for val in snippet_src: snippet_list.append(val.get_text()) snippet_list[:3] # Output [&#39;Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites. The web scraping software may directly access the \xa0...\nHistory · Techniques · Software · Legal issues&#39;, &#39;Web Scraper allows you to build Site Maps from different types of selectors. This system makes it possible to tailor data extraction to different site structures.\nVideo Tutorials · Web Scraper · Cloud · Test Sites&#39;, &#39;What Is Web Scraping? Web scraping is the process of gathering information from the Internet. Even copy-pasting the lyrics of your favorite song is a form of web\xa0...&#39;]</pre><h3>Converting into CSV file</h3><p>You can take the lists above, and read it as a pandas DataFrame. Once you have the Dataframe, you can convert to CSV, Excel or JSON easily without any issues.</p><h3>Scaling up to a full crawler for extracting Google results in bulk</h3><ul><li>Once you scale up to make thousands of requests to fetch all the pages, the Google.com servers will start blocking your IP address outright or you will be flagged and will start getting CAPTCHA.</li><li>To make it more likely to successfully fetch data for all USA, you will have to implement:</li><li>rotating proxy IP addresses preferably using residential proxies.</li><li>rotate user agents</li><li>Use an external CAPTCHA solving service like 2captcha or anticaptcha.com</li></ul><p>After you follow all the steps above, you will realize that our pricing for <a href="https://www.specrom.com/web-scraping-service/">managed web scraping</a> or our <a href="https://www.specrom.com/google-scraper-api/">Google scraper API</a> is one of the most competitive in the market.</p><p><em>Originally published at </em><a href="https://www.specrom.com/blog/web-scraping-search-engine-result-pages-serp-from-google-dot-com/"><em>https://www.specrom.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4aaef55f9d69" width="1" height="1" alt=""><hr><p><a href="https://medium.com/web-data-extraction/web-scraping-search-engine-result-pages-serp-from-google-com-4aaef55f9d69">Web Scraping search engine result pages (SERP) from Google.com</a> was originally published in <a href="https://medium.com/web-data-extraction">Web Data Extraction</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Web scraping for Marshalls stores location data]]></title>
            <link>https://medium.com/web-data-extraction/web-scraping-for-marshalls-stores-location-data-58831d7e279c?source=rss-8dc22d6b3ace------2</link>
            <guid isPermaLink="false">https://medium.com/p/58831d7e279c</guid>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[Jay M. Patel]]></dc:creator>
            <pubDate>Wed, 12 May 2021 02:27:37 GMT</pubDate>
            <atom:updated>2021-05-12T02:27:37.814Z</atom:updated>
            <content:encoded><![CDATA[<p>So, what is the easiest way to get CSV file of all the Marshalls store locations data in the USA?</p><h3>Easy Method: Buy the Marshalls store data from our data store</h3><p>There are over 1000 Marshalls stores in USA and you buy the CSV file containing address, city, zip, latitude, longitude of each location in <a href="https://www.specrom.com/marshalls-store-locations-in-the-usa/">our data store for $50</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*400TcRYaNSdAQfUV.png" /></figure><p><strong>Figure 1: Marshalls store locations. Source: </strong><a href="https://www.specrom.com/marshalls-store-locations-in-the-usa/"><strong>Marshalls Store Locations dataset</strong></a></p><p>If you are instead interested in scraping for locations on your own than continue reading rest of the article.</p><h3>Method 2: Scraping Marshalls stores locator using Python</h3><p>We will keep things simple for now and try to web scrape Marshalls store locations for only one zipcode.</p><p>Python is great for web scraping and we will be using a library called Selenium to extract Marshalls store locator’s raw html source for zipcode 30301 (Atlanta, GA).</p><pre>### Using Selenium to extract Marshalls store locator&#39;s raw html source from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import Select import time from bs4 import BeautifulSoup import numpy as np import pandas as pd query_url = &#39;https://www.marshalls.com/us/store/stores/storeLocator.jsp&#39; option = webdriver.ChromeOptions() option.add_argument(&quot;--incognito&quot;) chromedriver = r&#39;chromedriver_path&#39; browser = webdriver.Chrome(chromedriver, options=option) browser.get(test_url) text_area = browser.find_element_by_id(&#39;store-location-zip&#39;) text_area.send_keys(&quot;30301&quot;) element = browser.find_element_by_xpath(&#39;//*[@id=&quot;findStoresForm&quot;]/div[3]/input[9]&#39;) element.click() time.sleep(5) html_source = browser.page_source browser.close()</pre><h3>Using BeautifulSoup to extract Marshalls store details</h3><p>Once we have the raw html source, we should use a Python library called BeautifulSoup for parsing the raw html files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yu2-CwVJJ2Pf2DrD.PNG" /></figure><p><strong>Figure 2: Inspecting the source of Marshalls store locator</strong>.</p><h4>Extracting store names</h4><p>As a first step, we will start by extracting each store names</p><pre># extracting Marshalls store names store_name_list_src = soup.find_all(&#39;h3&#39;, {&#39;class&#39;,&#39;address-heading fn org&#39;}) store_name_list = [] for val in store_name_list_src: try: store_name_list.append(val.get_text()) except: pass store_name_list[:10] #Output [&#39;East Point&#39;, &#39;College Park&#39;, &#39;Fayetteville&#39;, &#39;Atlanta&#39;, &#39;Decatur&#39;, &#39;Atlanta&#39;, &#39;Atlanta (Brookhaven)&#39;, &#39;McDonough&#39;, &#39;Smyrna&#39;, &#39;Lithonia&#39;]</pre><h4>Extracting Marshall store addresses</h4><p>Look at inspect in the chrome browser to find tags and class names for each address.</p><pre># extracting marshalls addresses addresses_src = soup.find_all(&#39;div&#39;,{&#39;class&#39;, &#39;adr&#39;}) addresses_src address_list = [] for val in addresses_src: address_list.append(val.get_text()) address_list[:10] # Output [&#39;\n3606 Market Place Blvd.\nEast Point,\n\t\t\t\t\t\t\t\t\t\t\t\tGA\n30344\n&#39;, &#39;\n 6385 Old National Hwy \nCollege Park,\n\t\t\t\t\t\t\t\t\t\t\t\tGA\n30349\n&#39;, &#39;\n109 Pavilion Parkway\nFayetteville,\n\t\t\t\t\t\t\t\t\t\t\t\tGA\n30214\n&#39;, &#39;\n2625 Piedmont Road NE\nAtlanta,\n\t\t\t\t\t\t\t\t\t\t\t\tGA\n30324\n&#39;, &#39;\n2050 Lawrenceville Hwy\nDecatur,\n\t\t\t\t\t\t\t\t\t\t\t\tGA\n30033\n&#39;, &#39;\n3232 Peachtree St. N.E. \nAtlanta,\n\t\t\t\t\t\t\t\t\t\t\t\tGA\n30305\n&#39;, &#39;\n150 Brookhaven Ave\nAtlanta (Brookhaven),\n\t\t\t\t\t\t\t\t\t\t\t\tGA\n30319\n&#39;, &#39;\n1930 Jonesboro Road\nMcDonough,\n\t\t\t\t\t\t\t\t\t\t\t\tGA\n30253\n&#39;, &#39;\n2540 Hargroves Rd.\nSmyrna,\n\t\t\t\t\t\t\t\t\t\t\t\tGA\n30080\n&#39;, &#39;\n8080 Mall Parkway \nLithonia,\n\t\t\t\t\t\t\t\t\t\t\t\tGA\n30038\n&#39;]</pre><h4>Extract phone numbers</h4><p>We will extract phone numbers.</p><pre># extracting phone numbers phone_src = soup.find_all(&#39;div&#39;,{&#39;class&#39;, &#39;tel&#39;}) phone_list = [] for val in phone_src: phone_list.append(val.get_text()) phone_list[:10] # Output [&#39;404-344-6703&#39;, &#39;770-996-4028&#39;, &#39;770-719-4699&#39;, &#39;404-233-3848&#39;, &#39;404-636-5732&#39;, &#39;404-365-8155&#39;, &#39;404-848-9447&#39;, &#39;678-583-6122&#39;, &#39;770-436-6061&#39;, &#39;678 526-0662&#39;]</pre><h4>Extracting individual Marshalls store URL</h4><p>Each local store location has its own URL where additional information is available.</p><pre># extracting store url for each store store_list_src = soup.find_all(&#39;li&#39;,{&#39;class&#39;,&#39;store-list-item vcard address&#39;}) store_list = [] for val in store_list_src: try: store_list.append(val.find(&#39;a&#39;)[&#39;href&#39;]) except: pass store_list[:10] #Output [&#39;https://www.marshalls.com/us/store/stores/East+Point-GA-30344/463/aboutstore&#39;, &#39;https://www.marshalls.com/us/store/stores/College+Park-GA-30349/1207/aboutstore&#39;, &#39;https://www.marshalls.com/us/store/stores/Fayetteville-GA-30214/687/aboutstore&#39;, &#39;https://www.marshalls.com/us/store/stores/Atlanta-GA-30324/353/aboutstore&#39;, &#39;https://www.marshalls.com/us/store/stores/Decatur-GA-30033/1098/aboutstore&#39;, &#39;https://www.marshalls.com/us/store/stores/Atlanta-GA-30305/621/aboutstore&#39;, &#39;https://www.marshalls.com/us/store/stores/Atlanta+Brookhaven-GA-30319/49/aboutstore&#39;, &#39;https://www.marshalls.com/us/store/stores/McDonough-GA-30253/859/aboutstore&#39;, &#39;https://www.marshalls.com/us/store/stores/Smyrna-GA-30080/644/aboutstore&#39;, &#39;https://www.marshalls.com/us/store/stores/Lithonia-GA-30038/836/aboutstore&#39;]</pre><h3>Geo-encoding</h3><ul><li>You will need latitudes and longitudes of each stores if you want to plot it on map like figure 1.</li><li>Lats and longs are also necessary to calculate distances between points, driving radius etc. all of which are important part of location analysis.</li><li>We recommend that you use a robust geocoding service like <a href="https://developers.google.com/maps/documentation/geocoding/overview">Google maps</a> to convert the address into coordinates (latitudes and longitudes). It costs $5 for 1000 addresses but in our view its totally worth it.</li><li>There are some free alternatives for geocoding based on openstreetmaps but none that matches the accuracy of Google maps.</li></ul><h3>Scaling up to a full crawler for extracting all Marshalls store locations in USA</h3><ul><li>Once you have the above scraper that can extract data for one zipcode/city, you will have to iterate through all the US zip codes.</li><li>it depends on how much coverage you want, but for a national chain like Marshalls you are looking at running the above function 100,000 times or more to ensure that no region is left out.</li><li>Once you scale up to make thousands of requests, the Marshalls.com servers will start blocking your IP address outright or you will be flagged and will start getting CAPTCHA.</li><li>To make it more likely to successfully fetch data for all USA, you will have to implement:</li><li>rotating proxy IP addresses preferably using residential proxies.</li><li>rotate user agents</li><li>Use an external CAPTCHA solving service like 2captcha or anticaptcha.com</li></ul><p>After you follow all the steps above, you will realize that our pricing($50) for web scraped store locations data for <a href="https://www.specrom.com/marshalls-store-locations-in-the-usa/"><strong>all Marshalls stores locations dataset</strong></a> is one of the most competitive in the market.</p><p><em>Originally published at </em><a href="https://www.specrom.com/blog/web-scraping-marshalls-stores-location-data/"><em>https://www.specrom.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58831d7e279c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/web-data-extraction/web-scraping-for-marshalls-stores-location-data-58831d7e279c">Web scraping for Marshalls stores location data</a> was originally published in <a href="https://medium.com/web-data-extraction">Web Data Extraction</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Information about UPC, ASIN, Walmart Product Codes]]></title>
            <link>https://medium.com/web-data-extraction/information-about-upc-asin-walmart-product-codes-b03eb29c5862?source=rss-8dc22d6b3ace------2</link>
            <guid isPermaLink="false">https://medium.com/p/b03eb29c5862</guid>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[Jay M. Patel]]></dc:creator>
            <pubDate>Sat, 08 May 2021 02:55:53 GMT</pubDate>
            <atom:updated>2021-05-08T02:55:53.932Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xvLDx-SSZSFO9X_8" /><figcaption>Photo by <a href="https://unsplash.com/@heftiba?utm_source=medium&amp;utm_medium=referral">Toa Heftiba</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>What are product codes? What are its types? Why are product codes used?</strong></p><p>A product code describes a specific product and contains a combination of five to seven numbers and letters. These are the codes that are usually on the back of the product. These codes are important mainly due to its unique nature.</p><h3>Types of Product Codes</h3><p><strong>UPC</strong>: Universal Product Code (UPC) is a 12-digit bar code used extensively for retail packaging in the United States. A typical process of obtaining a 12-digit UPC number is as follows: — License a unique Company Prefix from your local GS1 office. GS1 is a not-for-profit organisation that develops and maintains global standards for business communication. — Assign product number(s) to unique products making your number equal 11 digits — Using a check digit calculator with your 11 digit number, generate your check digit.</p><p><strong>EAN</strong>: The European Article Number (EAN) is a barcode standard, a 12- or 13-digit product identification code. Each EAN uniquely identifies the product, manufacturer, and its attributes; typically, the EAN is printed on a product label or packaging as a bar code. We require EAN codes to improve the quality of search results and the quality of the catalogue as a whole.</p><p><strong>ISBN</strong>: The International Standard Book Number (ISBN) is a unique commercial book identifier barcode. Each ISBN code identifies uniquely a book. ISBN have either 10 or 13 digits. Typically, the ISBN is printed on the back cover of the book.</p><p><strong>ASINs</strong>: Amazon Standard Identification Numbers (ASINs) are unique blocks of 10 letters and/or numbers that identify items. You can find the ASIN on the item’s product information page at Amazon.com. For books, the ASIN is the same as the ISBN, but for all other products, a new ASIN is created when the item is uploaded to our catalogue.</p><p>These are the commonly used types of codes. But many stores have product codes of their own. For Example, the product <em>Invicta Men’s Pro Diver Scuba 48mm Gold Tone Stainless Steel Quartz Watch</em> is available on Walmart with a product code- 39825323. But this product has a UPC-843836069830. The UPC code is used to find the product regardless of the platform from which it’s being sold on.</p><h3>How to Get UPC Product Codes?</h3><h3>Online Converter:</h3><p>There are several data providers that take product URLs and convert them the corresponding Code to a UPC code. Some of these are:</p><p><strong>algopix.com</strong></p><p>By simply entering a keyword or product description, you can view product images and metadata, which can help you better describe your merchandise. You also receive a complete summary of how similar products are being sold on various marketplaces.</p><p>This product contains a free plan for amazon approved sellers. Rest must opt for a $27.99/month package.</p><p><strong>synccentric.com</strong></p><p>This is primarily ASIN to UPC or UPC to ASIN converter. Import your UPCs or ASINs along with any other custom data. The system then searches for the products that match the uploaded ASINs or UPCs and returns as much data as possible including identifiers (ASIN, UPC, MPN, SKU, EAN, ISBN), pricing (buy box, shipping, lowest pricing for FBA and Merchant, list pricing), sales rank, and other product listing data. This conversion feature is free although a fee of $39.99/ month must be paid for the site to provide analytics and access to the API.</p><p><strong>asinscope.com</strong></p><p>This is similar to the synccentric, wherein they do various bulk conversions. The price to access this API is $25/month</p><h3>UPC Databases</h3><p>The above sites provide product analytics and information of product across the various platforms it is being sold on. There are also large databases that contain UPCs and we can search through them. One particular example of it is <a href="https://www.specrom.com/blog/scraping-product-codes/barcodelookup.com">barcodelookup.com</a> Here all we need to do is type in the UPC and we get details on the product.</p><p>Some other websites that do a similar function are <a href="https://www.specrom.com/blog/scraping-product-codes/www.upcitemdb.com">www.upcitemdb.com</a>, <a href="https://www.specrom.com/blog/scraping-product-codes/scandit.com/upc-lookup">scandit.com/upc-lookup</a>, <a href="https://www.specrom.com/blog/scraping-product-codes/barcodesdatabase.org">barcodesdatabase.org</a></p><h3>How to scrape UPC codes?</h3><p>UPC codes can be found on all the product websites if you look close enough. Every product contains a dictionary which includes all product info- name, price, description, product code, upc etc.</p><p>In the case of Walmart, this can be found in the product reviews page source. We can then call a parser to parse this and collect the dictionary and collect the upc from this.</p><p>The python code block on scraping the first 10 pages of walmart products is illustrated below:</p><p>Here we scrape the watches UPC from walmart:</p><pre>my_header = {&quot;User-Agent&quot;: &quot;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36&quot; + &quot;(KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36&quot;} querystring = {&quot;page&quot;:&quot;10&quot;, &#39;product_url&#39;:&#39;https://www.walmart.com/search/?query=watch&#39;} item_url=[] total_pages=int(querystring[&quot;page&quot;]) for i in range(1,total_pages+1): url=&#39;https://www.walmart.com/search/?page=&#39;+ str(i) +&#39;&amp;ps=48&amp;query=watches&#39; r= requests.get(url, headers= my_header) price_per_page=[] if r.status_code==200: soup_main= BeautifulSoup(r.content, &#39;html&#39;) summary=soup_main.find(&#39;div&#39;, {&#39;class&#39;:&#39;search-product-result&#39;, &#39;id&#39;:&#39;searchProductResult&#39;}) product_list= summary.find_all(&#39;li&#39;) for prod in product_list: try: item_url.append(prod.find(&#39;a&#39;, {&quot;class&quot;:&quot;product-title-link line-clamp line-clamp-2 truncate-title&quot;}).get(&#39;href&#39;)) except: pass else: print(&quot;Error-&quot;,r.status_code) product_code=[u.split(&#39;/&#39;)[-1] for u in item_url]</pre><p>The variable Product code contains all the Walmart product codes of the products in the first 10 pages. Now We will loop through these pages to get the UPCs.</p><pre>upc=[] item_ID=[] product_name=[] for prod_code in product_code: item_url= &#39;https://www.walmart.com/reviews/product/&#39;+ prod_code +&#39;?page=2&#39; r = requests.get(item_url,headers=proxy_headers) r.status_code soup = BeautifulSoup(urlopen(item_url),&#39;html.parser&#39;) for val in soup.find_all(&quot;script&quot;): #print(val) if &#39;upc&#39; in str(val): val=str(val) prob_dict = val.split(&#39;upc&#39;)[1] UPC=prob_dict.split(&#39;,&#39;)[0] UPC=UPC[3:-1] upc.append(UPC) prob_dict = val.split(&#39;usItemId&#39;)[1] item_id=prob_dict.split(&#39;,&#39;)[0] item_id=item_id[3:-1] item_ID.append(item_id) prob_dict = val.split(&#39;productName&#39;)[1] product=prob_dict.split(&#39;,&#39;)[0] product=product[3:-1] product_name.append(product)</pre><p>Doing this will give us a data base of product name and its corresponding Walmart code and UPC.</p><pre>df_dict={&quot;Product Name&quot;: product_name, &quot;Walmart Product Code&quot;: item_ID, &quot;UPC&quot;: upc} df=pd.DataFrame(df_dict) df.to_csv(&quot;UPCs of Walmart Watches&quot;)</pre><p>Here we saw what Product codes are and its different types- UPC, EAN, ASIN, ISBN. We also saw data providers that analyze products from various platforms to see its trends and gain insights. We also looked the various UPC databases and checked how we can create these databases of our own.</p><p><em>Originally published at </em><a href="https://www.specrom.com/blog/scraping-product-codes/"><em>https://www.specrom.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b03eb29c5862" width="1" height="1" alt=""><hr><p><a href="https://medium.com/web-data-extraction/information-about-upc-asin-walmart-product-codes-b03eb29c5862">Information about UPC, ASIN, Walmart Product Codes</a> was originally published in <a href="https://medium.com/web-data-extraction">Web Data Extraction</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>