<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by George Vyshnya on Medium]]></title>
        <description><![CDATA[Stories by George Vyshnya on Medium]]></description>
        <link>https://medium.com/@gvyshnya?source=rss-11881363dca------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*0xmgruopko3p2fJ2NRJmJw.jpeg</url>
            <title>Stories by George Vyshnya on Medium</title>
            <link>https://medium.com/@gvyshnya?source=rss-11881363dca------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 03:59:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gvyshnya/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What Can Payment Transactions Tell You About the Real AI Adoption?]]></title>
            <link>https://medium.com/@gvyshnya/what-payment-transactions-can-tell-you-about-the-real-ai-adoption-208fb1168a11?source=rss-11881363dca------2</link>
            <guid isPermaLink="false">https://medium.com/p/208fb1168a11</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm-agent]]></category>
            <category><![CDATA[aiproviders]]></category>
            <category><![CDATA[ai-automation]]></category>
            <dc:creator><![CDATA[George Vyshnya]]></dc:creator>
            <pubDate>Fri, 07 Aug 2026 16:11:06 GMT</pubDate>
            <atom:updated>2026-08-07T18:00:44.665Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lF9zAQuucWqvvcd_3USCTg.png" /></figure><p>The question from the title of this blog post came to my mind when I first approached <a href="https://ramp.com/data/ai-index">Ramp AI Index</a> data.</p><p>Ramp’s transaction data — built from actual corporate card and bill-pay spend across more than 70,000 U.S. businesses — paints a picture of AI adoption that looks nothing like the one self-reported surveys describe: faster, more concentrated, and structurally different across firm size, sector, and geography.</p><p>Some of the findings really stand out.</p><h3>Real spend shows roughly 2.7× more AI adoption than surveys report</h3><p>Ramp’s transaction-based index put AI adoption at 54.95% of businesses in June 2026, more than double the 20.6% reported by the U.S. Census Bureau&#39;s Business Trends and Outlook Survey (BTOS) for the same month. The two series have never converged since Census data became available in late 2023 — the gap has instead widened as Ramp&#39;s measured rate keeps climbing. This is consistent with the paper this dataset builds on (Bonney et al., 2024): self-reported surveys likely undercount AI use because free tools, personal accounts, and quiet adoption inside existing subscriptions don&#39;t register as a clear &quot;yes&quot; on a questionnaire, while a paid transaction leaves a trail.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*n5C4x-zmxBkAkuBX3RMNVQ.png" /></figure><p>Growth wasn’t linear. Adoption rocketed from 7.5% to ~31% in ChatGPT&#39;s first year (Jan–Dec 2023), then slowed to a steady crawl through most of 2024–2025 (roughly 0.5–1.5 points a month). It re-accelerated sharply starting January 2026 — up nearly 8 points in five months (46.96% to 54.95%) — the fastest sustained monthly gains since the initial 2023 surge, and worth watching as a possible second adoption wave.</p><h3>The relative gap between large and small firms has stabilized — but the absolute gap keeps widening</h3><p>Large firms have led adoption throughout the dataset, but the shape of that lead has changed. In January 2023, large firms were adopting AI at 2.5× the rate of small firms (13.45% vs. 5.39%); by mid-2024 that ratio had compressed to about 1.25×, as small firms briefly caught up during the early ChatGPT boom. Since then the ratio has held fairly steady around 1.3–1.4×, but because both curves keep rising, the absolute percentage-point gap has more than doubled — from about 7 points in mid-2024 to nearly 17 points by June 2026 (65.49% for large firms vs. 48.65% for small firms). Proportionally, small firms are keeping pace; in raw terms, the divide is growing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*JM-1olK-GqqTAQs7JPuqcw.png" /></figure><h3>Late-adopting sectors are now growing the fastest</h3><p>Technology &amp; media (79.75%) and finance &amp; insurance (73.55%) remain the most AI-saturated sectors as of June 2026, as expected. But the sectors gaining the most ground right now are the ones that started furthest behind. Over the trailing 12 months, manufacturing added 17.8 points, retail added 15.0, and construction added 14.5 — each outpacing technology &amp; media&#39;s 9.0-point gain. Construction is the standout: from a base of just 1.27% adoption in January 2023, it has grown 32-fold to 40.80%, now on par with health care. The AI adoption curve is starting to broaden out beyond its original knowledge-work base.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*iDGHpwHAvtoQJCSocsTwPw.png" /></figure><h3>Adoption is geographically top-heavy, led by tech, finance, and corporate hubs.</h3><p>Delaware (72.83%), New York (65.55%), and California (64.41%) lead all states as of June 2026. Delaware&#39;s rank is worth a caveat: it is the default state of incorporation for a large share of U.S. companies, so firms may be counted there based on legal registration rather than where employees actually work — treat it as a data artifact rather than evidence that Delaware businesses use AI most.</p><p>New York and California, by contrast, plausibly reflect real concentrations of finance, media, and tech employment. At the other end, Alaska (29.11%), Vermont (37.61%), and New Mexico (37.91%) trail the pack, roughly 25–35 points behind the leaders — still meaningfully higher than where the national average stood as recently as 2024.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*FspIsQiiPXEP1bc8MB5RGw.png" /></figure><h3>Anthropic overtook OpenAI in business adoption share in spring 2026.</h3><p>OpenAI held a commanding, near-uncontested lead in the share of businesses paying for its models from 2023 through most of 2025 — peaking near 41% of adopting businesses while Anthropic sat in the high teens. That changed fast: Anthropic&#39;s adoption share climbed from 21.7% in January 2026 to 42.4% by June 2026 — essentially doubling in five months — while OpenAI&#39;s share stayed flat to slightly down (41.1% → 39.5%). The crossover happened between April and May 2026, the first time Anthropic has led any major provider-adoption metric in this dataset&#39;s history.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*Tv_ZHqdAuQJkr6TnC2iY9w.png" /></figure><p>The shift is even sharper in dollar terms. Among spend Ramp can attribute to a specific named model version (a smaller, more volatile slice of total AI spend, weighted toward API and coding-tool usage), Anthropic’s share swung from roughly 37–68% through most of 2025 to a stable ~90% by May–June 2026, with OpenAI collapsing to under 7% and Cursor (largely running on Claude models) picking up the remainder. Named Claude releases — Opus 4.7, Opus 4.6, and Sonnet 4.6 — account for nearly two-thirds of all identifiable named-model spend in June 2026 on their own.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*hTDP62VHAaEf7fjchC1wQQ.png" /></figure><h3>AI spend per employee is compounding, and it’s compounding fastest for the biggest spenders</h3><p>Median monthly AI spend per employee grew from $2.00 in July 2023 to $10.66 in June 2026 — a 5.3× increase. However, the growth is not evenly paced. It roughly doubled in just the <strong>last six months</strong> alone (from $5.01 in December 2025 to $10.66 in June 2026), mirroring the same early-2026 acceleration seen in adoption rates.</p><p>Spend is also concentrating: the ratio of the top-10%-of-firms’ median spend to the overall median has climbed from about 27× in mid-2023 to roughly 48× in mid-2026, meaning the heaviest AI spenders are pulling further ahead of the typical firm even as everyone&#39;s spending rises.</p><p>The bottom line is, the median monthly AI spend per employee (‘$10.66’ as of June 2026) is still lower then a spend on coffee per employee. Not only the heaviest AI spenders pull further ahead from typical companies, they drive the entire growth of the AI industry as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*TDIiqQNEcJqSIE80g4HHlQ.png" /></figure><h3>Small firms adopt AI less often, but spend far more intensively per employee</h3><p>This is the sharpest counter-intuitive finding in the dataset. Large firms have the highest adoption rate, but among firms that do spend, small firms spend 7× more per employee than large firms — $20.06 vs. $2.86 per employee per month as of June 2026.</p><p>The likely explanation is <strong>compositional</strong>: a large enterprise’s adoption rate reflects a handful of departments buying AI tools, diluted across a payroll of thousands, while a small firm that adopts AI is disproportionately likely to be an AI-native or software team where most employees hold a paid seat. In other words, large-firm adoption is broad but shallow; small-firm adoption, where it happens, is narrow but deep.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*mk7_55D4uD69oNb9rSMvOA.png" /></figure><h3>VC-backed companies are in a different spending league — and pulling further away</h3><p>VC-backed firms spent $66.67 per employee per month on AI by June 2026 — 9.3× what non-VC/non-PE firms spent ($7.18) and 10.7× what PE-backed firms spent ($6.25). VC-backed spend has grown 27× since July 2023 ($2.51 → $66.67), and more than <strong>doubled in the six months</strong> from December 2025 to June 2026 alone. This tracks with what&#39;s likely happening on the ground: VC-backed companies skew toward software and AI-native business models. Many of them are directly building on top of frontier models rather than just using AI tools internally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*_PerkYxUhqa-aCMNtjV6-w.png" /></figure><h3>Spend is shifting from generic subscriptions toward infrastructure and coding tools</h3><p>The composition of AI spend has matured. Through 2023, essentially all identifiable AI spend fell into an undifferentiated “Other AI” bucket. Starting mid-2024, as Ramp’s categorization improved and usage diversified, API usage and coding-agent subscriptions became visible and have steadily gained share — API usage alone regularly accounts for 40–55% of spend since 2025, and coding-agent subscriptions have climbed from under 1% of spend in 2024 to 4–5% by mid-2026, reflecting the rise of tools like Cursor and Claude Code. This is a shift from paying for AI as a packaged product toward paying to build with AI directly.</p><p>The split from the generic AI subscriptions to more specialized instruments shows up clearly by sector.</p><p>As of Jun 2026, professional, scientific &amp; technical services spends 56% of its AI budget on API usage plus another 23% on model-serving and GPU infrastructure — a sector building AI-powered products.</p><p>Retail and accommodation &amp; food services, by contrast, put 30%+ of spend into chat subscriptions and support bots — sectors consuming AI as an off-the-shelf tool.</p><p>Manufacturing is the outlier: 31% of its spend goes to GPU cloud, the highest of any sector, suggesting a meaningful share of manufacturers running their own inference or training workloads rather than just calling hosted APIs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*_mSouqVlvAInOnAhOAg1Zg.png" /></figure><h3>Advance of Small Business Through AI</h3><p>Based on what’s in the data, there’s a real reason for optimism (with caveats though).</p><p>The strongest evidence is the spend-intensity finding: small firms that adopt AI already spend 7x more per employee than large firms, and VC-backed companies (disproportionately small, agent-native operations) have grown their AI spend 27x since 2023, doubling again in just the last six months. That&#39;s not incidental—it suggests capital-efficient small teams are already using AI to substitute for headcount rather than just augment it.</p><p>Agents in particular attack the thing that’s always most disadvantaged small companies: needing enough people to cover sales, support, ops, and research simultaneously. If a five-person company can run agents that do the work of twenty, the “minimum viable team” for competing in a given market shrinks, and that’s a genuine structural advantage for lean, unencumbered operators. Small firms also adopt faster once they commit — no procurement cycles, no legacy workflows to unwind, no internal politics about whether the sales team “owns” the CRM process.</p><p>The limits are real too, though. Large firms still have the highest adoption rate in this dataset, and they’re not sitting still — they have the capital to buy the same agents, plus proprietary data, distribution, brand trust, and balance sheets that let them outspend in an actual arms race if it comes to that. Agents themselves are becoming commoditized fast: if everyone has access to roughly the same frontier models, the differentiator stops being “who has AI” and reverts to whatever advantages existed before AI — capital, network effects, regulatory relationships, existing customer trust. In heavily regulated or capital-intensive industries (finance, health care, manufacturing at scale), the liability and compliance overhead of agentic AI may actually favor incumbents who can absorb that risk.</p><p>So, in the evidence of the mentioned above, we can expect that, in software, professional services, agencies, content, and other knowledge-work-heavy segments, smart small teams using agents well would plausibly outcompete much larger, slower incumbents on cost and speed. This is already visible in the VC-backed spend data. In capital- or trust-intensive industries where size itself is the moat, agents narrow the gap but probably don’t close it.</p><h3>Deeper Look at the US Manufacturing</h3><p>Two real patterns showed up in the Ramp AI Index data for Manufacturing sector</p><ul><li>manufacturing sends 31% of AI spend to GPU Cloud (vs. single digits elsewhere), and</li><li>among identifiable named-model spend, OpenAI holds 23.6% of manufacturing vs. only 5–7% in finance, tech, and professional services (where Anthropic dominates at 88–93%). A few explanations line up with what’s happening outside the Ramp dataset.</li></ul><p>The GPU spend is largely a workload-type story, not a preference story. Manufacturing’s AI investment is concentrated in computer vision for defect detection and quality control, predictive maintenance on sensor time-series data, and digital twins — over half of large industrial facilities had deployed at least one digital twin for maintenance simulation as of 2025. Those are custom-trained perception and forecasting models that need dedicated GPU training and inference infrastructure, structurally different from the chat/API-call pattern that dominates knowledge-work sectors. A finance or tech company’s AI spend is mostly “call a hosted LLM”; a manufacturer’s is more often “train and run a vision model on a production line,” which shows up as infrastructure spend rather than subscription spend.</p><p>The OpenAI tilt likely traces back to enterprise software lock-in. Manufacturing runs on legacy ERP, PLM, and MES vendors, and Microsoft has pushed hard into that stack specifically — Microsoft Cloud for Manufacturing and Copilot for Manufacturing are being positioned as the 2026 entry point for agentic AI on the factory floor, and both are delivered through Azure OpenAI Service. So a lot of “OpenAI spend” in manufacturing may not be a deliberate choice between ChatGPT and Claude — it’s bundled into an existing Microsoft/SAP/Siemens relationship the company already has for its core operations software. That’s a different buying motion than a finance or tech firm evaluating AI vendors directly. There’s also a concrete large-scale example: Samsung Electronics, a major manufacturer, signed one of OpenAI’s largest enterprise deals, rolling ChatGPT Enterprise and Codex out company-wide.</p><p>Timing may compound this. Manufacturing is earlier on the adoption curve than finance or tech — one industry estimate puts only 42% of manufacturers with any AI deployment and just 12% at enterprise scale as of 2026, with agentic AI specifically projected to roughly quadruple (6% to 24%) this year. Anthropic’s recent surge in this dataset has been driven heavily by coding and developer tooling (Claude Code, Cursor), and manufacturing simply has a smaller software-engineering population per employee than tech or finance to drive that channel. Sectors with more developers picked up the Anthropic wave earlier; manufacturing, still catching up on basic deployment, is more likely still running on whatever it adopted first — which for most of 2023–2024 was OpenAI.</p><p>One caveat: the “named-model spend share” metric is based on a thin, high-variance slice of total AI spend (it only captures transactions where the specific model version is identifiable), and manufacturing’s Cursor share in that same table (7.9%, higher than tech’s 2.1%) looks like it could be small-sample noise rather than a real signal — worth treating the sector-level model splits as directional rather than precise.</p><h3>More Focused View on States with higher OpenAI Model Share Rates</h3><p>Confirmed in the data are:</p><ul><li>Virginia (20.1%) and Texas (20.0%) have the highest OpenAI share of any state</li><li>Florida is elevated too (11.6%, essentially tied with Colorado for third),</li></ul><p>It is oppose to a 3–7% OpenAI share in California, Washington, Delaware, and New York, where Anthropic runs 91–96%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gxY0_2YF2iqzb2MFNYgufw.png" /></figure><p>Notably, Virginia and Texas also sit near the bottom on dollar spend per employee (9.09, among the lowest of the states Ramp breaks out) — so this isn’t “more AI spend,” it’s “different AI spend.”</p><p>Three separate regional economic patterns line up with that.</p><p>Virginia is the center of gravity for federal government and defense contracting — Northern Virginia is home to Booz Allen Hamilton, Leidos, General Dynamics, CACI, and Northrop Grumman, among others. OpenAI has pushed aggressively into that exact market: ChatGPT Gov launched in 2025 for agencies needing FedRAMP High/IL5-level compliance, and OpenAI struck a deal with the GSA giving the entire federal executive branch ChatGPT Enterprise access for $1 per agency through 2026. Azure OpenAI Service has since been authorized up through DoD IL6 and Top Secret cloud, and Microsoft has struck parallel Azure OpenAI deals with defense contractors like Anduril and Palantir. A federally-subsidized, near-free OpenAI rollout across the government/contractor ecosystem would show up exactly like this: high adoption share, low dollar intensity, concentrated in Virginia.</p><p>Florida’s tilt likely traces to its insurance industry, which is unusually large there given the state’s hurricane and property-insurance exposure. A May 2026 industry survey (IA Capital, reported by The Insurer) found OpenAI present in 91% of insurance company AI stacks, often delivered through Microsoft Copilot, with Anthropic present in about 55% but typically as a secondary model rather than the primary one. Florida’s growing “Wall Street South” financial-services cluster reinforces the same pattern, since Microsoft’s own financial-services materials describe Copilot/OpenAI as the default entry point for that industry.</p><p>Texas fits a similar legacy-enterprise story: it’s home to large, Microsoft-anchored incumbents (Dell, AT&amp;T, the state’s energy majors), and Dell has been actively pushing Copilot+ PCs and Microsoft 365 Copilot deployment services as of Dell Technologies World 2026, explicitly targeting commercial and middle-market customers moving from AI experimentation to production. That’s a direct channel for OpenAI-powered tools to reach Texas businesses bundled through hardware and IT-services relationships they already have, rather than a deliberate model choice.</p><p>The common thread across manufacturing, Virginia, Texas, and Florida is the same one from the manufacturing question: places with deep, pre-existing Microsoft enterprise relationships (government contracts, ERP/office software agreements, hardware partnerships) tend to get OpenAI bundled in as part of Copilot, while software-forward, VC-funded, developer-heavy pockets (California, New York, Washington, tech and finance generally) have shifted more deliberately to Anthropic, largely on the strength of coding tools like Claude Code and Cursor.</p><h3>References</h3><p>In case you like to follow the analysis line as well as make your hands dirty with the data of Ramp AI Index, you can review the respective source code in</p><ul><li>My <a href="https://github.com/gvyshnya/ramp-ai-index">GitHub repo</a></li><li>My <a href="https://www.kaggle.com/code/gvyshnya/ramp-ai-index-jun-2026-eda">EDA notebook publication </a>on Kaggle</li></ul><p>Apart from it, you can review a bunch of the industry publications related to adoption of AI in manufacturing</p><ul><li><a href="https://www.microsoft.com/en-us/microsoft-cloud/blog/manufacturing/2026/03/16/manufacturing-at-the-2026-inflection-point-how-frontier-companies-are-entering-the-agentic-era/">Manufacturing at the 2026 inflection point — Microsoft Cloud Blog</a></li><li><a href="https://aufaittechnologies.com/blog/microsoft-copilot-for-manufacturing/">How Microsoft Copilot for Manufacturing Is Reshaping Factory Operations in 2026</a></li><li><a href="https://voxel51.com/blog/visual-ai-in-manufacturing-2025-landscape">Visual AI in Manufacturing: 2025 Landscape — Voxel51</a></li><li><a href="https://www.oxmaint.com/blog/post/future-factory-ai-predictive-maintenance-smart-manufacturing">Future of Smart Factories: AI Predictive Maintenance</a></li><li><a href="https://thinking.inc/en/industry-service/ai-in-manufacturing/">AI in Manufacturing — Complete 2026 Guide</a></li><li><a href="https://www.gsmarena.com/openai_signs_one_of_its_largest_enterprise_deals_with_samsung_electronics-news-73386.php">OpenAI signs one of its largest enterprise deals with Samsung Electronics — GSMArena</a></li><li><a href="https://techcrunch.com/2026/01/22/openai-is-coming-for-those-sweet-enterprise-dollars-in-2026/">OpenAI is coming for those sweet enterprise dollars in 2026 — TechCrunch</a></li></ul><p>You can also grab some insights from the publications below to see why AI adoption in Texas, Florida, and Virginia is different from the rest of the U. S. states.</p><ul><li><a href="https://openai.com/global-affairs/introducing-chatgpt-gov/">Introducing ChatGPT Gov — OpenAI</a></li><li><a href="https://www.gsa.gov/about-gsa/newsroom/news-releases/gsa-announces-new-partnership-with-openai-delivering-deep-discount-to-chatgpt-08062025">GSA Announces New Partnership with OpenAI, Delivering Deep Discount to ChatGPT Gov-Wide Through MAS</a></li><li><a href="https://devblogs.microsoft.com/azuregov/azure-openai-fedramp-high-for-government/">Azure OpenAI Service Approved within FedRAMP High Authorization for Azure Government</a></li><li><a href="https://www.theinsurer.com/ti/news/openai-dominates-ai-stacks-as-insurance-industry-moves-from-pilot-to-production-2026-05-06/">OpenAI dominates AI stacks as insurance industry moves from pilot to production: IA Capital survey — The Insurer</a></li><li><a href="https://www.microsoft.com/en-us/microsoft-cloud/blog/financial-services/2025/12/18/ai-transformation-in-financial-services-5-predictors-for-success-in-2026/">AI transformation in financial services: 5 predictors for success in 2026 — Microsoft Cloud Blog</a></li><li><a href="https://siliconangle.com/2026/05/27/agentic-ai-pc-strategy-dell-microsoft-delltechworld/">Agentic AI PC strategy at Dell and Microsoft — SiliconANGLE</a></li><li><a href="https://investors.delltechnologies.com/news-releases/news-release-details/dell-technologies-launches-new-services-microsoft-365-copilot">Dell Technologies Launches New Services for Microsoft 365 Copilot</a></li></ul><h3>Data notes and caveats</h3><p>Ramp AI Index measures adoption via paid transactions on Ramp’s corporate card and bill-pay platform across 70,000+ U.S. businesses; a firm counts as an “adopter” if it has a positive AI-related transaction in a given month. This almost certainly understates true adoption, since it can’t see free-tier usage or employees expensing AI tools on personal accounts. Sector and business-size assignments come from internal predictive models rather than firm self-reports, and state-level figures can be skewed by where a company is legally headquartered (see Delaware, Section 4) rather than where its employees sit. The “named-model spend share” metric (Section 5) is a narrower, higher-variance slice of total spend — limited to transactions where Ramp can identify the exact model version — and should be read as directional rather than a precise market-share figure. Where monthly changes look erratic (e.g., the spend-mix categories in Figure 10), that reflects genuine volatility in a growing, unevenly-sampled space rather than data error.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=208fb1168a11" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Year 2023 in AI/ML Retrospective]]></title>
            <link>https://medium.com/@gvyshnya/year-2023-in-ai-ml-retrospective-3b57513f7c84?source=rss-11881363dca------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b57513f7c84</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[George Vyshnya]]></dc:creator>
            <pubDate>Sat, 30 Dec 2023 16:11:13 GMT</pubDate>
            <atom:updated>2023-12-30T16:11:48.441Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>“The soul is amused by congenial work” — Hryhorii Savych Skovoroda.</blockquote><blockquote>As the year 2023 comes to its end, it would be nice to look at it back in retrospective. IMHO it has been a remarkable year for AI, with lots of progress and excitement yet with the decoration of the threat reflections and risk assessments equally.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*HK7KpaeDqvRsHHWWwAnBug@2x.jpeg" /><figcaption>Courtesy to GenAI</figcaption></figure><p>Below are my subjective takeaways and reflections from 2023 in AI/ML industry.</p><h3>Key Highlights</h3><p><strong>Generative AI Everywhere:</strong> This year, AI became virtually synonymous with generative AI (at least, in the wide public discussions). Inspired by OpenAI, BigTech giants scrambled to launch their own chatbots and rushed cutting-edge natural language processing research to market at a furious pace. Text-to-image generators (also sparked by OpenAI with DALL·E in early 2021) continued to improve and ultimately began to merge with their text-generator counterparts.</p><p><strong>LLMs and diffusion models as innovation triggers:</strong> LLMs and diffusion models continue to drive real-world breakthroughs, especially in the life sciences, with meaningful steps forward in both molecular biology and drug discovery.</p><p><strong>Compute as new oil:</strong> 2023 has manifested the amazing progress with various AI-centric hardware innovations (good review of it conveyed by the nice essay “<a href="https://www.kaggle.com/code/jonbown/acceleration-is-not-all-you-need-ai-hardware">Acceleration Is Not All You Need: AI Hardware</a>” by Jon Bown). Additionally, compute as a resource has become the instrument of geopolitics. With NVIDIA printing record earnings and startups wielding their GPUs as a competitive edge. As the US tightens its restrictions on trade with China as well as mobilizes its allies in the chip wars, NVIDIA, Intel, and AMD have started to sell export-control proof chips at scale.</p><p><strong>Salvaging VC industry:</strong> GenAI saves the VC world, as amid a slump in tech valuations, AI startups focused on generative AI applications (including video, text, and coding), raised over $18 billion from VC and corporate investors. My analysis of <a href="https://www.kaggle.com/code/gvyshnya/forbes-ai50-insights">Forbes AI 50 trends</a> provides the additional perspective on it.</p><h3>Key Lowlights</h3><p><strong>Generative AI Everywhere (yes, sadly):</strong> In 2023, the wave of generative AI washed over everything. Its expanding capabilities raised fears that intelligent machines might render humanity obsolete, defeating the good old “Three Laws of Robotics” formulated by <a href="https://en.wikipedia.org/wiki/Three_Laws_of_Robotics">Isaac Asimov</a>.</p><p><strong>Cybersecurity challenges to AI systems</strong>: 2023 demonstrated the Adversarial attacks on AI systems, along with the classical cybersecurity threats can ‘wash up’ the innovative potential of the proliferated AI solutions (my essay on <a href="https://www.kaggle.com/code/gvyshnya/cybersecurity-ai-in-2022-2023">Cybersecurity AI in 2022–2023</a> provides the comprehensive review on the topics related).</p><p><strong>AI Safety and Ethics at the Crossroad</strong>: The AI safety and ethics debate has exploded into the mainstream, prompting action from governments and regulators around the world. However, this flurry of activity conceals profound divisions within the AI community and a lack of concrete progress towards global governance, as governments around the world pursue conflicting approaches.</p><h3>Year 2024 in Perspective: Challenges and Opportunities</h3><p>As we approach 2024, a pivotal year for AI/ML industry, let’s try to ‘see the forest behind the trees’ as it comes out of the fog of uncertainty.</p><blockquote>The universe is entering the age of huge turbulence. Forecasts are harder to plot than ever. Therefore, I would only indicate the problem areas to be addressed as well as potential opportunities to be considered in 2024.</blockquote><p><strong>Job Market Impact.</strong> The concerns about AI’s impact on employment, stirring debates over job displacement and the urgency of re-skilling initiatives will soar in 2024. Many analysts/insiders keep talking about the number of jobs to be fully / partially replaced by AI-driven automation (both in the blue- and white-collar segments — see <a href="https://www.nexford.edu/insights/how-will-ai-affect-jobs">Nexford’s review</a>, for instance). However, there is no sound strategy for the skills overhaul / leverage of people who lose their current jobs/occupations in other areas. It is naive to think it would resolve on its own, nor should it be believed in establishing something like <a href="https://en.wikipedia.org/wiki/Universal_basic_income">UBI</a> for people affected.</p><p>Addressing such an issue would be one of the tangible AI-driven challenges in 2024.</p><p><strong>GenAI / LLM proliferation: hype or solid industry shift?</strong> I think the level of ‘hype’ around GenAI/LLM boom will be coherent with the number of sound practical use cases or even proven ‘productized’ end-user-facing solutions that consume GenAI/LLM capabilities. This area has recently been pumped with a large amount of venture capital (VC), and it opens the avenue for the potential new innovations.</p><p><strong>Cybersecurity AI challenges</strong>. Year 2024 promises the ‘singular storm’ in cybersecurity. Rise in ransomware attacks, increased AI-backed attacks, IoT vulnerabilities, electric vehicle hack ‘apocalypse’, and sophisticated adversarial attacks on AI systems are just a few threats to be mentioned for the area (you can find more on it in the <a href="https://www.marketsandmarkets.com/Market-Reports/artificial-intelligence-ai-cyber-security-market-220634996.html">summary review</a> , for instance).</p><p>There is a good opportunity there though as well. AI can become a relevant cybersecurity tool, if applied/integrated properly (more on it discussed in <a href="https://www.kaggle.com/code/gvyshnya/cybersecurity-ai-in-2022-2023">my essay</a>).</p><p><strong>AI/ML Industry shifts.</strong> Whether we like it or not, the proliferation of GenAI/LLM will drastically change the industry. The advent of ‘<a href="https://towardsdatascience.com/is-decision-science-quietly-becoming-the-new-data-science-5616a12fa9e8">decision science</a>’ , vital requirements for more <a href="https://towardsdatascience.com/revisiting-the-death-of-data-science-2f1398ed2cbb">specialized roles/niches for Data Scientists</a>, and new technologies / workflows would crack the landscape of the ML/Data Science as we used to know it until recently.</p><h3>Post-Scriptum</h3><p>In any case, the changes (both in the AI/ML industry and beyond) are going to be tremendous in 2024. Many of us will have to redefine our career paths/goals in future. We will have to find internal drivers to stay efficient. Motivation only works if you try to push yourself to do something you are not entirely happy/willing to do. The true driver will be related to<a href="https://en.wikipedia.org/wiki/Hryhorii_Skovoroda"> Hryhorii Skovoroda’s</a> doctrine of “congenial work”.</p><blockquote>Once upon a time <a href="https://kpi.ua/en/skovoroda">he mentioned</a>, “Profit is not amusement, but the execution of bodily needs, and if amusement is not internal, but native amusement from heart dwells in congenial work. The sweeter it is, the more congenial it appears…The soul is amused by congenial work”.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b57513f7c84" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cloud ML In Perspective: Surprises of 2021, projections for 2022]]></title>
            <link>https://medium.com/sbc-group-blog/cloud-ml-in-perspective-surprises-of-2021-projections-for-2022-d0b3b821fa77?source=rss-11881363dca------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0b3b821fa77</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[google-cloud-ml]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[business]]></category>
            <dc:creator><![CDATA[George Vyshnya]]></dc:creator>
            <pubDate>Sat, 04 Dec 2021 21:23:55 GMT</pubDate>
            <atom:updated>2021-12-04T21:26:02.871Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*1nIJY-G7z6UUCNu4RoHB4A.jpeg" /></figure><blockquote>~ Life is just a series of conformity tests. — William Joseph Donovan</blockquote><h3>Introduction</h3><p>In my article at KDNuggets in Jan 2021, I made the following forecast as for the Cloud Computing and Cloud ML industries, for 2021</p><ul><li>The next couple of years will be crucial in the battle of Cloud Computing giants for minds, arms, and budgets in the Data Science and ML industry. Although AWS’s position still looks stronger than other top rivals, the challenges from GCP could be the intrigues part of the market reshaping in the years to come. At the same time, MS Azure seems to keep its strong positions in North America (while having little chances to penetrate other continents significantly vs. AWS and GCP).</li><li>However, we entered the age of global turbulence. 2021, the year under the Star of Kings, may expose us to unexpected surprises in every aspect of our lives.</li></ul><p>I should admit both forecasts have been fulfilled.</p><p>The Battle of Giants, with AWS, GCP and MS Azure struggling for the market shares and revenues, continued in 2021, and each giant had its strong sides and weaknesses in the positioning their products across the globe.</p><blockquote>As for the surprise, it happened indeed. In Cloud ML industry, we observed several hard-to-predict phenomena in 2021</blockquote><blockquote><strong><em>- Google </em></strong>lost its market share after the recent consolidation of its individual cloud ML products under the unified product umbrella (Google Cloud Vertex AI)</blockquote><blockquote><strong><em>- Databricks </em></strong>jumped into the top 3 products within Cloud ML market, replacing the rival offerings from Google there</blockquote><p>Let’s take a closer look on Cloud ML market in 2021 in retrospective (with occasional drills into realities of 2020, too). The next sections of this article will enlighten the industry insights.</p><p><strong>Note</strong>: When drilling for the insights, the data collected in Kaggle’s surveys of ‘State of Data Science and Machine Learning 2020’ (<a href="https://www.kaggle.com/c/kaggle-survey-2020">https://www.kaggle.com/c/kaggle-survey-2020</a>) and of ‘State of Data Science and Machine Learning 2021’ (<a href="https://www.kaggle.com/c/kaggle-survey-2021">https://www.kaggle.com/c/kaggle-survey-2021</a>) was used.</p><h3>Databricks Breaking the Wall and Other Cloud ML Trends in 2021</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*ATHpi_nh0pX-uiyG5k2Ukg.jpeg" /></figure><p>We can find a lot of captivating facts and trends about Cloud ML industry as of 2021. They are as follows</p><ul><li>Google’s move toward consolidating all their Cloud ML products under the new unified platform (that is, Google Cloud Vertex AI) in 2021 did not yield the increase in the Google’s market share in the enterprise (commercial) Cloud ML market.</li><li>Moreover, the major Cloud ML rivals within the Big Three Cloud Giants — Amazon SageMaker and Azure ML Studio — improved their positions, and they became the ultimate market leaders as of 2021 (with Amazon SageMaker taking the first place, being slightly ahead of Azure ML Studio).</li><li>We can see Databricks to take the third position in the Cloud ML product ranking as of 2021 (it is above Google Cloud Vertex AI, and it is slightly below Amazon SageMaker and Azure ML Studio)</li><li>Other Cloud ML challenger products (DataRobot, Dataiku, Alteryx, Rapidminer) are well below Databricks and Google Cloud Vertex AI.</li><li>Amazon SageMaker keeps the leading positions in the Cloud ML market for two years in a raw (in 2020–2021).</li><li>Still, a large fraction of the respondents indicate they do not use Cloud ML in their daily activities at all (the side note is, ‘None’ for 2020 and ‘None’ for 2021 could have quite different meaning, due to the drastic difference in the list of Cloud ML products exposed as the survey options in 2020 and 2021, respectively).</li><li>Databricks makes a strong offensive on Google’s unified AI platform in 2021 (so the recent decision of Google to invest in Databricks was probably the trigger to significant growth of Databricks’s market share in 2021).</li><li>Cloud ML industry leaders — Amazon SageMaker and Azure ML Studio — maintain the strong position vs. Databricks and Google Cloud Vertex AI as of the end of 2021.</li><li>India and USA are the top two countries with the biggest number of users of all Cloud ML products represented in Kaggle 2021 Survey questionnaire.</li><li>Amazon SageMaker takes leading positions in both India and USA.</li><li>Azure Machine Learning Studio takes the second place in India.</li><li>Databricks takes the second place in the USA, and it has good growing potential in the UK and EU countries in 2022 as well.</li><li>Companies with the biggest amount of ML spending prefer to work with Amazon SageMaker and Databricks, and Azure ML Studio takes the third place in such a ranking (Google Cloud Vertex AI is well behind them).</li><li>Google Cloud Vertex AI’s position is better within the companies with moderate and small ML spending (it could be an opportunity to convert them to the customers that pay more in future); it can potentially explore the opportunity to grow in the multiple locations outside North America and EU in 2022.</li></ul><p>Let’s now jump into details to see the data-driven story behind the insights listed above.</p><p><strong>Note</strong>: if you are interested in the detailed reproducing my discovery line, you are welcome to review the relevant Jupyter notebooks in the Github repo (<a href="https://github.com/gvyshnya/kaggle-2021-survey">https://github.com/gvyshnya/kaggle-2021-survey</a>).</p><h3>Usage of Cloud ML Products By Occupation and Programming Experience (2020–2021)</h3><p>Let’s review how Cloud ML products mentioned in the 2021 survey have been used by the survey participants (with the breakdown by their occupation and programming experience).</p><p>The by-occupation break-down in 2021 looks as follows</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fAhZdM1aNXb04Pt0bLaRIQ.png" /></figure><p>Regarding the programming experience of the users of the major Cloud ML products, it is pictured below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X1Kct1AUntyIgY62J5lEyQ.png" /></figure><p>From the charts above, it is evident that</p><ul><li>The big fraction of the survey respondents does not use any Cloud ML products in their daily activities</li><li>In the smaller fraction of the survey respondents who use such tools, Data Scientists predominate</li><li>Amazon SageMaker and Azure ML Studio take the leadership in terms of the number of respondents using them</li><li>The rival product from Google (that is, Google Cloud Vertex AI) is behind the leaders as well as behind the challenger product appeared in the survey just this year (that is, Databricks)</li><li>Other challenger products (DataRobot, Dataiku, Alteryx, Rapidminer) are well behind the leaders</li><li>Additionally, the most use of Cloud ML products is observed among professionals with 1–3 years and 10–20 years of programming experience</li></ul><p>Now, let’s review how Cloud ML products mentioned in the 2020 survey have been used by the survey participants (with the breakdown by their occupation and programming experience).</p><p>By-occupation break-down as of 2020 looked as follow</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0dhWWaSktoqrgnqXnL5fCg.png" /></figure><p>The spread of users of Cloud ML products by programming experience as of 2020 is provided below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7QI_w-8r5rV4FCTW1OJTlQ.png" /></figure><p>We can see that</p><ul><li>Google Cloud AI Platform / Google Cloud ML Engine lead the ML cloud products usage ‘nomination’ as of 2020</li><li>The second and third best were Amazon SageMaker and Azure Machine Learning Studio, respectively</li><li>Data Scientists were the top users of cloud ML products (for every product investigated)</li><li>There was a huge chunk of responders who indicated they do not use cloud ML products at all — this indicates the market is under-saturated, and there is a good growth potential, subject to resolving the marketing and end-user barries on the way</li><li>Additionally, we see the professionals with programming experience of 3–5 years and 5–10 year to use the cloud ML products the most, as of 2020</li></ul><p>If we consolidate the observations in 2020 and 2021 respectively, we see that</p><ul><li>Google’s move toward consolidating all of their Cloud ML products under the new unified platform (that is, Google Cloud Vertex AI) in 2021 did not yield the increase in the Google’s market share in the enterprice (commercial) Cloud ML market.</li><li>Moreover, the major Cloud ML rivals within the Big Three Cloud Giants — Amazon SageMaker and Azure ML Studio — improved their positions, and they became the ultimate market leaders as of 2021 (with Amazon SageMaker taking the first place, being slightly ahead of Azure ML Studio).</li><li>Azure Congnitive Services have been excluded from the survey in 2021 so it is hard to estimate the changes in its market position among the Kagglers vs. 2020.</li><li>It is hard to estimate the market position changes for the challenger Cloud ML products (DataRobot, Databricks, Dataiku, Alteryx, Rapidminer) in 2021 vs. 2020 since they were not listed in the survey for 2020.</li><li>However, we can see Databricks to take the third position in the Cloud ML product ranking as of 2021 (it is above Google Cloud Vertex AI, and it is slightly below Amazon SageMaker and Azure ML Studio).</li><li>Other Cloud ML challenger products (DataRobot, Dataiku, Alteryx, Rapidminer) are well below Databricks and Google Cloud Vertex AI.</li><li>Still, a large fraction of the respondents indicate they do not use Cloud ML in their daily activities at all (the side note is, ‘None’ for 2020 and ‘None’ for 2021 could have quite differnt meaning, due to the drastical difference in the list of Cloud ML products exposed as the survey options in 2020 and 2021, respectively).</li></ul><p>If we try to interpret the observations above, matched with the major business development, fund raising, and publicity progress of Databricks achieved in Feb — Aug 2021 (see Appendix below for more details), we can tell that</p><ul><li>Databricks had a strong offensive on Google’s unified AI platform in 2021 (so the recent decision of Google to invest in Databricks was probably the trigger to good growth of Databricks market share in 2021).</li><li>Cloud ML industry leaders — Amazon SageMaker and Azure ML Studio — maintained the strong position vs. Databricks and Google Cloud Vertex AI as of the end of 2021.</li></ul><h3>Usage of Cloud ML Tools by Organization Size and Industries (2021)</h3><p>As for the usage of Cloud ML in the organizations of various size as of 2021, we observe the following picture</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*XD6djt6HkAxSohs42546QQ.png" /></figure><p>It is evident that</p><ul><li>A lot of organizations in every industry do not use Cloud ML products as of 2021</li><li>The companies in Computers/Technologies use Cloud ML the most, with organizations in Academia/Education taking the second place, nand Accounting/Finance taking the third one</li><li>For every Cloud ML product under review, most of the users are in Computers/Technologies industry</li><li>For Azure Machine Learning Studio and Google Cloud Vertex AI, the second most popular industry is Academia/Education</li><li>For Amazon SageMaker and Databricks, the second most popular industry is Accounting/Finance</li></ul><p>As for the break-down of Cloud ML users by industries as of 2021, it looks as follows</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uo0whc8RNSuBg7FEZtIREg.png" /></figure><p>We find that</p><ul><li>A lot of organizations in every industry do not use Cloud ML products as of 2021</li><li>Google Cloud Vertex AI and DataRobot are the most popular with the organizations of the smallest size (0–49 employees)</li><li>Azure Machine Learning Studio and Amazon SageMaker are equally popular for both the organizations of the smallest (0–49 employees) and largest (10k+ employees) sizes</li><li>Databricks is more popular with the large organizations (with 10k+ and 1000–9999 employees, respectively)</li></ul><h3>Cloud ML and ML Spending within Organizations</h3><p>The picture as of 2021 is provided below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*JQu7_Qt1-by_R306qEG4kA.png" /></figure><p>As we can see, the industry trends as of 2021 are as follows</p><ul><li>Companies with the biggest amount of ML spending prefer to work with Amazon SageMaker and Databricks</li><li>Azure ML Studio takes the third place in such a ranking</li><li>Google Cloud Vertex AI is well behind the three leaders above for the organizations spending much money in ML spending</li><li>Google Cloud Vertex AI’s position is better within the companies with moderate and small ML spending (it could be an opportunity to convert them to the customers that pay more in 2022)</li><li>The rest of the Cloud ML products are far behind</li></ul><p>If we look back at 2020, we can see the picture below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1009/1*Lh_F_mJB7dDHI2AuiAoBrA.png" /></figure><p>It is evident that</p><ul><li>Amazon SageMaker clearly took the leading role in every ML spending category as of 2020 (like it does as of 2021).</li><li>Azure ML Studio and one of then-Google Cloud ML products (Google Cloud AI Platform/Google Cloud ML Engine) were almost on the par to take the second place after Amazon SageMaker, as of 2020.</li></ul><h3>Cloud ML Geography (2021)</h3><p>Let’s look at the geographic spread of users of the leading Cloud ML products participated in Kaggle Survey 2021.</p><p>First of all, it should be noted India and USA to be the countries with the biggest number of ML engineers.</p><p>With that said, let’s look at the geographical insights for each of the leading Cloud ML products.</p><h3>Amazon SageMaker</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*mov9i6ilJ18d1Z0VhAbCbA.png" /></figure><p>As we see, the geography of Amazon SageMaker users indicates that</p><ul><li>India takes the first place (and it looks like Azure Sagemaker is the top Cloud ML product in this country, too)</li><li>The US takes the second place</li><li>Other countries are well behind India and USA</li><li>Japan takes the third place in terms of Amazon SageMaker user base (among the respondents to Kaggle 2021 survey)</li><li>We can see the moderate popularity of the product in Brazil, UK, EU countries, and Nigeria</li></ul><h3>Azure Machine Learning Studio</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*n3t80Wn4ga7OHNr-j1-kCw.png" /></figure><p>As we see, the geography of Azure Machine Learning Studio users indicates that</p><ul><li>India is the major location of the product users (its popularity in India is only slightly below the popularity of Amazon SageMaker)</li><li>USA takes the second place (however, Azure Machine Learning Studio is quite below Amazon SageMaker in terms of the popularity)</li><li>Other countries are well behind India and USA</li><li>Nigiria takes the third place in terms of Amazon SageMaker user base (among the respondents to Kaggle 2021 survey)</li><li>We can see the moderate popularity of the product in Brazil, UK, EU countries, Pakistan, and Kenia</li></ul><h3>Google Cloud Vertex AI</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*OTjsBly1ComkE6bO5_asGg.png" /></figure><p>As we see, the geography of Google Cloud Vertex AI users indicates that</p><ul><li>India is the major location of the product users (although its popularity in India is much lower than the popularity of Amazon SageMaker and Azure Machine Learning Studio)</li><li>USA takes the second place (however, like in India, its popularity in the US is much lower than the popularity of Amazon SageMaker and Azure Machine Learning Studio)</li><li>Other countries are well behind India and USA</li><li>Surprisingly, Indonesia takes the third place in the list of the popular locations for Google Cloud Vertex AI</li><li>There is a bunch of countries across the globe that are close to Indonesia in terms of the number of survey respondents who use the product (Nigeria, Taiwan, China, Japan, South Korea, Turkey) — these areas could be potential opportunities for growing the sales and revenue from Google Cloud Vertex AI</li><li>There is very low interest in Google Cloud Vertex AI in Canada, UK and EU countries (the latter can be explained by some state- and industry-level policies that discriminate Google Cloud as a platform in favor of MS Azure and Amazon AWS, in EU and UK)</li></ul><h3>Databricks</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*Ra7SqLiCg2Y7SYPxMWOgjg.png" /></figure><p>As we see, the geography of Databricks users indicates that</p><ul><li>India is the country with the largest number of Databricks users amoung the survey respondents in 2021 (Databricks outperforms Google Cloud Vertex AI in India, and it underperforms vs. Amazon SageMaker and Azure Machine Learning AI)</li><li>USA takes the second place in terms of the number of users of the product across the globe (Databricks acaully takes the second place in the US, after Amazon SageMaker, overperforming both Azure Machine Learning and Google Cloud Vertex AI)</li><li>Other countries are well behind India and USA</li><li>Brazil takes the third place in the world’s ‘Databrick rank’ (however, UK and EU countries are quite close to it)</li></ul><h3>What Is in it for 2022?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*V6tH4mz8MOkMbOscBPjj9A.png" /></figure><p>The trends described above will shape the future of the industry in 2022 for sure. From the insights I collected in my data-driven drill, I can forecast that</p><ul><li>Being the leader of the industry in 2020–2021, Amazon SageMaker is going to retain his top position across the globe in 2022</li><li>Further proliferation of Databricks as a Cloud ML product in 2022 could be affected by the ongoing discussion whether doing ML on top of Spark is a good idea or not (while Spark ML libs are incredible powerful when it turns to the BigData-scale datasets, certain API drawbacks of Spark and Databricks leave some space for the criticism).</li><li>Despite the mentioned Spark ML debates, Databricks is likely to further improve its leading positions both in USA and China, with good growth potential in the UK and EU countries in 2022 as well.</li><li>Azure ML Studio offering is quite protected from the ‘Databricks offensive’ as Azure Cloud platform used to ‘contain’ Databricks as a solely BigData Engineering tool (with Azure Databricks service running in this role for a few years already).</li><li>Google Cloud Vertex AI can potentially explore the opportunity to grow in the multiple locations outside North America and EU in 2022, targeting medium and quickly growing companies there.</li><li>The game changer in the Google Cloud ML ecosystem in 2022 could be the massive promotion for BigQuery ML offering; since it can be a real alternative to the BigData-scale ML offering of Databricks, it may change the landscape in Cloud ML tools in the upcoming year.</li></ul><h3>Methodology and References</h3><p>This article operates with the insights from the data collected in Kaggle’s survey of ‘State of Data Science and Machine Learning 2020’ (<a href="https://www.kaggle.com/c/kaggle-survey-2020">https://www.kaggle.com/c/kaggle-survey-2020</a>) and of ‘State of Data Science and Machine Learning 2021’ (<a href="https://www.kaggle.com/c/kaggle-survey-2021">https://www.kaggle.com/c/kaggle-survey-2021</a>), respectively.</p><p>Kaggle (<a href="http://www.kaggle.com/">www.kaggle.com</a>) is a global community made up of data scientists and machine learners from all over the world with a variety of skills and backgrounds. The community has more than 3 million active members. Although it is not rigorously representative of the entire population of Data Science and ML professionals across the globe from the sociological perspective, it still constitutes the significant fraction of the practitioners and professionals in the field. Therefore, the results of the survey can really draw the projections of where the Data Science and AI/ML industry is likely to evolve in the next couple of years.</p><p>My comprehensive EDA-style notebooks for Kaggle 2020 Survey and Kaggle 2021 Survey are listed below</p><ul><li><a href="https://www.kaggle.com/gvyshnya/kaggle-2020-survey-associated-insights">https://www.kaggle.com/gvyshnya/kaggle-2020-survey-associated-insights</a></li><li><a href="https://www.kaggle.com/gvyshnya/kaggle-2021-ml-and-ds-survey-comprehensive-eda">https://www.kaggle.com/gvyshnya/kaggle-2021-ml-and-ds-survey-comprehensive-eda</a></li></ul><p>The source code for the data-driven drill to collect the insights described in this article are available in the repo in GitHub per <a href="https://github.com/gvyshnya/kaggle-2021-survey">https://github.com/gvyshnya/kaggle-2021-survey</a></p><p>My article per “Cloud Computing, Data Science and ML Trends in 2020–2022: The battle of giants” has been published by KDNuggets in Jan 2021 per <a href="https://www.kdnuggets.com/2021/01/cloud-computing-data-science-ml-trends-2020-2022-battle-giants.html">https://www.kdnuggets.com/2021/01/cloud-computing-data-science-ml-trends-2020-2022-battle-giants.html</a></p><p>You can also find a lot of interesting insights in “State of AI Report 2021” (<a href="https://www.stateof.ai/">https://www.stateof.ai/</a>).</p><h3>Appendix. History of Databricks</h3><p>Since launching its original data platform built on Apache Spark in 2015, Databricks has grown into a one-stop home for (un)structured data, automated ETL, collaborative data science notebooks, business intelligence using SQL, and full-stack machine learning built on open source MLflow. Interestingly, all three major cloud vendors — Amazon, Google, and Microsoft — invested in Databricks in February 2021.</p><p>Databricks is headquartered in San Francisco. It also has operations in Canada, UK, Netherlands, Singapore, Australia, Germany, France, Japan, China, and India.</p><p>With this company having the investments from all of three Cloud Computing giants (Amazon, Microsoft, and Google) as of Feb 2021 as well as receiving the largest valuation vs. its competitors as of Aug 2021, it becomes a strong challenger to the industry. From this perspective, Databricks products/offerings can potentially</p><p>- partially cannibalize other BigData, Cloud ML and AutoML products of Big 3 Cloud Computing providers (AWS, MS Azure, Google Cloud Platform)</p><p>- make a strong competition to the challenger products in Cloud ML and enterprise AutoML segments (H20.ai, DataRobot etc.)</p><p>The chart below displays the key points in the corporate history of the company</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*AoVTWBG6fxLgnieq_RMh7A.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0b3b821fa77" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sbc-group-blog/cloud-ml-in-perspective-surprises-of-2021-projections-for-2022-d0b3b821fa77">Cloud ML In Perspective: Surprises of 2021, projections for 2022</a> was originally published in <a href="https://medium.com/sbc-group-blog">SBC Group Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Effective Cloud ETL in Cross-Cloud Setup — Part 1: Let Big Data Flow]]></title>
            <link>https://medium.com/sbc-group-blog/effective-cloud-etl-in-cross-cloud-setup-part-1-let-big-data-flow-1690fe7a40bd?source=rss-11881363dca------2</link>
            <guid isPermaLink="false">https://medium.com/p/1690fe7a40bd</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[dataproc]]></category>
            <dc:creator><![CDATA[George Vyshnya]]></dc:creator>
            <pubDate>Wed, 21 Jul 2021 20:56:49 GMT</pubDate>
            <atom:updated>2021-08-30T14:12:16.911Z</atom:updated>
            <content:encoded><![CDATA[<h3>Effective Cloud ETL in Cross-Cloud Setup — Part 1: Let Big Data Flow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ASOpssuJLMjkYjYqzFbhhw.jpeg" /></figure><blockquote>“Data is the new science. Big Data holds the answers.” — By Pat Gelsinger</blockquote><h3>Introduction</h3><p>If you do your business as a Data Science/ML consultant, you often face a typical issue whenever you embark on a cooperation with a new customer. They dream of the cutting-edge AI and ML solutions but they may lack the basic data infrastructure with the BI-ready data that can be a common source of truth for their business operations. If it is the case, we offer the customer to build the reliable data infrastructure first. Once we put such a foundation in place, it is going to be the right time to start the discussion about ML and AI — something that the customer supposed to have from <em>Day 1</em>.</p><p>In the serious of upcoming blog posts, I would like to draw the picture of collateral data engineering journeys that data scientists/ML engineers must take sometimes.</p><p>I am going to share the experience of the data engineering rides we had to take in some of the recent projects. They will be full of practical cases and real-world problems solved during such efforts.</p><p>So let’s start the data engineering journey of the ML professionals.</p><h3>ETL/ELT in the Cross-Cloud Data Infrastructure</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*-GnTbXAlymBQnddlsPAbhg.png" /></figure><p>Nowadays, it is more and more common to see organizations operating their data infrastructures across multiple cloud providers. There are multiple reasons to drive such an architecture as follows</p><ul><li>Desire to diversify the security and cloud infrastructure availability risks across multiple transaction systems</li><li>Merges and acquisitions of the multiple businesses by a central organization (with individual businesses having their transaction systems deployed in different clouds)</li><li>Meeting legal and regulation requirements (for instance, the EU-based financial companies must run certain core systems in Azure cloud only, due to the respective legislation regulations in place)</li><li>Etc.</li></ul><p>Such a cross-cloud transactional system deployment still must cope with the provisions of having a central Data Warehouse (DWH) and/or Data Lake for BI and reporting reasons. Since such a central DWH/BI Infrastructure is hosted by a single cloud infrastructure provider, it will require to build ETL/ELT solutions capable of cross-cloud data transfers (from the data sources managed by the respective transaction systems at their cloud premises).</p><h3>Problem Statement</h3><p>With the above-mentioned considerations in mind, let us review a practical case of building a Cloud-based ETL pipeline to operate in the cross-cloud setup (actually, the idea of this case study was inspired by a recent industrial project that required implementing something similar).</p><p>Imagine that we have to build an ETL solution that</p><ul><li>Operates in Google Cloud Platform (GCP) to pull the source data</li><li>Enables export of every entity table/view from a BigQuery dataset (database) into a Google Cloud Storage bucket as either separate parquet files or separate JSON files (such a bucket will be often referred to as a reservoir)</li><li>Supports both the full export of the entire BigQuery dataset and its incremental changes from date to date</li><li>Minimize the lead time of the pipeline processing via invoking a Big Data-scaled ETL technology</li><li>Uses either fully managed or true serverless components, whenever possible</li><li>Minimizes the burning rate of the solution by using ‘fire-and-forget’ clusters/units or/and managed or even serverless deployment, whenever possible</li><li>Provides a way for third-party processes/applications to connect to the reservoir as well as pull the data into the central Data Warehouse/Data Lake on another cloud premise outside GCP</li><li>As an additional option, facilitates the active push of the exported data into Azure Blob Storage</li></ul><p>The sections below will walk you through the process of building such a cloud-based ETL solution. If using a metaphor, we could call such a solution a <em>“Sparkling Air”</em> due to the building blocks to be used in the solution as follows</p><ul><li><em>BigQuery</em></li><li><em>Cloud Composer</em></li><li><em>Cloud Dataproc</em></li><li><em>Google Cloud Storage</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nmULHrRXIm4tFa3xCLnBQg.png" /></figure><h3>Data Source: The BigQuery dataset</h3><p>The source dataset (database) is hosted by BigQuery, as mentioned above. For the purpose of this post, we imagine the following structural and data transfer requirements set force for such a dataset</p><ul><li>Every entity (that is, a physical or virtual table (view) with essential entity data) of the dataset to be enabled to export into the central DWH outside GCP</li><li>Each table and view has a large amount of data (that makes it possible to state we operate in really Big Data domain when building such an ETL solution)</li><li>A separate treatment to partitioned vs. non-partitioned entities (physical tables or virtual tables (views)) to be enabled (we will see every partitioned table or view to contain a datetime partition field named<em> ‘ingest_date’</em>)</li><li>Auxiliary tables (with certain service-level tech details irrelevant for business reporting) to be excluded from the data export (the names of such auxiliary tables/views to start with<em> ‘_’</em>)</li></ul><h3>Destination: File Reservoir in GCS</h3><p>To make the data exported from a BigQuery dataset available to pull by an external ETL system into a central DWH/BI Infrastructure outside the GCP premise, we will serialize it as parquet or JSON files in a Google Cloud Storage (GCP) bucket.</p><p>For each table in the source BigQuery dataset, we are going to create a subfolder in the destination GCS bucket. Every partitioned table in turn will further generate the partition subfolder structure corresponding to the values of the <em>ingest_date</em> partition field in it.</p><h3>Secret Sauce of BigData-Scaled ETL: The power of managed PySpark</h3><p>As we all see, Big Data becomes bigger by the day. Requirements to handle large amount of data in your ETL pipelines in a very compressed time intervals are more and more common (the case we are reviewing in this blog post is no more an exception). You cannot simply avoid tackling the problems of this sort in your real-world industrial projects.</p><p>This is where <a href="https://spark.apache.org/">Apache Spark</a> comes into the place. It is a processing framework that operates on top of the Hadoop Distributed File System (HDFS) (as well as other data stores). It features interactive shells that can launch distributed process jobs across a cluster. Spark supports programming language interfaces for Scala, Java, Python and SQL. <em>PySpark </em>is the Python interface to Spark.</p><p>PySpark is the Python API written in Python to support <em>Spark</em>.</p><p>One traditional way to handle Big Data is to use a distributed MapReduce framework like Hadoop. As a drawback, such an approach requires a lot of read-write operations on a hard disk which makes it very expensive in terms of the computational resource allocation.</p><p><em>Spark </em>(and <em>PySpark</em>, in particular), as opposed to more traditional Big Data tech stacks, deals with it in an efficient and easy-to-understand manner.</p><p>Let us see it in action as per our project case. The sections below will walk you through the topics below</p><ul><li>Building effective and speedy ETL pipeline using <em>PySpark </em>to meet our requirements</li><li>Deploying 100% managed PySpark-based solutions in Google Cloud, using <em>Dataproc</em></li><li>Debugging <em>PySpark </em>scripts in Google Cloud Platform, using <em>Dataproc</em></li></ul><p><strong>Note</strong>: It is assumed you have been already familiar with the fundamentals of PySpark. If not, you can go over the tutorials provided in References section at the end of this blog post.</p><h3>ETL Logic: PySpark scripts</h3><p>The magic of PySpark is to implement complex and computationally extensive Big Data transformations without any hassle of the complicated coding. As an outcome, the resulted code to manipulate the data via PySpark dataframes will be as simple as one you are used to see when doing the data wrangling with Pandas.</p><p>You can see the PySpark script capable of exporting the data from a particular table or view in a BigQuery dataset in action per the snippet below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1432f4c526c71f0f0fe4e8aa6cbab8d6/href">https://medium.com/media/1432f4c526c71f0f0fe4e8aa6cbab8d6/href</a></iframe><p>It is worth noting the essential details below</p><ul><li>The script has separate logic to handle partitioned and non-partitioned tables/views in <em>BigQuery</em></li><li>The script can output the data either in parquet or JSON formats</li><li>The same <em>PySpark </em>code can be applicable in both the full reload of the data and the incremental update of the diff vs. a day ago</li><li>Spark’s ‘<em>viewsEnabled</em>’ option allows for handling not only the physical tables but virtual tables (views and material views) as well</li><li>Due to <em>BigQuery</em>’s integration requirements, <em>PySpark </em>has to use a temporary materialization dataset in <em>BigQuery </em>when fetching the data from the source view and before exporting such a data as files to the reservoir (that’s the reason you see ‘<em>viewMaterializationProject</em>’ and ‘<em>viewMaterializationDataset</em>’ configuration options set)</li><li>The filter option is used to filter the data in the partitioned data tables by the <em>ingestion date</em></li></ul><h3>Deployment: Google Dataproc and its workflow templates</h3><p>As we mentioned earlier, the real power of the cloud platforms comes whenever the modern managed or even a serverless solution deployment is entertained.</p><p>The <em>Dataproc Workflow Templates API</em> provides a flexible and easy-to-use mechanism for managing and executing workflows. A <em>Workflow Template</em> is a reusable workflow configuration. It defines a graph of jobs with information on where to run those jobs.</p><p>Instantiating a Workflow Template launches a Workflow. A Workflow is an operation that runs a <em>Directed Acyclic Graph (DAG)</em> of jobs on a cluster.</p><p>The workflow can potentially use a cluster selector (that is, to run the job on a selected existing <em>Dataproc </em>server cluster) or a managed cluster. In the latter case, The workflow will create this “ephemeral” (aka ‘fire-and-forget’) cluster to run workflow jobs, and then delete the cluster when the workflow is finished.</p><p>Workflows are ideal for complex job flows. You can create job dependencies so that a job starts only after its dependencies complete successfully.</p><p>Creating a workflow template does not create a <em>Dataproc </em>cluster or submit jobs. Clusters and jobs associated with workflows are created when a workflow template is instantiated.</p><p>Technically, the workflow templates are defined as yaml templates.</p><p>In our case, we are going to encapsulate <em>PySpark </em>into <em>Dataproc </em>workflows and workflow templates.</p><p>You can review the source code of the workflow template below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/adecf3ff8272a67d83a02984ff8c66da/href">https://medium.com/media/adecf3ff8272a67d83a02984ff8c66da/href</a></iframe><p>As you see, the essential details about our workflow template are as follows</p><ul><li>It has just one job step (this step will run the <em>PySpark </em>script we reviewed in the previous section)</li><li>The “ephemeral” cluster is designed for this workflow template to minimize the costs of operating it in Google Cloud Platform</li><li>The “ephemeral” cluster we defined in the template will feature two worker nodes, with quite a low-end node machines to be used (it turns out the two-node cluster with such machines is sufficient to complete the ETL activities required thus keeping our burning rate on executing such a workflow in cloud on the relatively low level)</li><li>The additional spark plugin to ensure data exchange with <em>BigQuery </em>is mounted into the cluster for the workflow template since the <em>PySpark </em>job code is dependent on it to work properly</li><li><em>PySpark </em>script source is uploaded to a dedicated <em>Google Cloud Storage </em>bucket, to be available to the instantiated workflows to invoke it</li></ul><p><strong>Note</strong>: when defining the cluster configuration in your workflow template, you should balance within the cost-performance-execution time triangle as every Dataproc job has a maximal execution time cap of 60 min (that is, the number of your worker nodes as well as your worker machines power should be sufficient to execute your workflow within the time constraints imposed by Google Cloud Platform — the less time it spends, the better, obviously).</p><h3>Debugging PySpark Scripts in Google Cloud</h3><p>Naturally, you need to debug your <em>PySpark </em>scripts before you are ready to package them as a par of<em> Dataproc workflow templates.</em></p><p>One of the convenient methods of doing it in Google Cloud in to create a permanent <em>Dataproc </em>cluster with</p><ul><li><em>Jupyter Lab</em> and <em>Jupyter notebooks</em> enabled</li><li>External disk space (in a dedicated <em>Google Cloud Storage</em> bucket) mounted to the cluster</li></ul><p>You keep this cluster running as much time as you need to interactively debug your PySpark code while storing your <em>PySpark </em>and <em>Jupyter notebooks</em> on the <em>GCS </em>bucket (rather then on a local drive of the cluster). You shut down the cluster when it is no more needed (to optimizing your cloud operation costs) while preserving your notebooks on <em>Google Cloud Storage </em>(if you stored the notebooks on the local drive of the cluster, they would vanish as soon as you shut down the cluster).</p><p>Below is the <em>Bash </em>script automating such a permanent cluster setup</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8753e296cac540a14179eec95ca670dc/href">https://medium.com/media/8753e296cac540a14179eec95ca670dc/href</a></iframe><p>You can reuse whenever you need a new cluster to spin off for your <em>PySpark</em> development and debugging purpose (with the notebooks from the previous ‘incarnations’ of your cluster to be immediately available from the <em>GCS </em>bucket as designed).</p><h3>Useful Deployment Automation</h3><p>These days, DevOps/SRE-centric paradigm shifts the mindsets in Data Engineering, Data Science and ML more and more. I noticed this trend back to 2017 (see <a href="https://www.kdnuggets.com/2017/08/data-version-control-analytics-devops-paradigm.html">https://www.kdnuggets.com/2017/08/data-version-control-analytics-devops-paradigm.html</a>), and it proved to become the mainstream since that time, as of 2021, indeed.</p><p>One of the consequences of this mindset/philosophy is you as a Data Engineer, Data Scientist or ML Engineer is responsible for deployment of the applications/solutions you built (with minimal-to-zero help from dedicated DevOps professionals).</p><p>Therefore, I really recommend to invest some time in automating your development routines. One of my preferred ways of doing such an automation is to prepare Linux makefiles with the frequent deployment command shortcuts. The example of such a makefile is presented below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/88204a94348f8333f23a378eb281981c/href">https://medium.com/media/88204a94348f8333f23a378eb281981c/href</a></iframe><h3>Next Steps</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KzZEoc08X6ZZIImWC38c-A.jpeg" /></figure><p>In the sections above, we had a comprehensive outlook on</p><p>- The overall ETL requirements in our project case</p><p>- The data source (<em>BigQuery </em>datasets) and the destinations for the file-based data export (in the <em>GCP </em>bucket)</p><p>- Engineering the Big Data-scaled ETL using <em>PySpark </em>and managed service deployment with <em>Google Dataproc</em></p><p><strong>Note</strong>: you can review the source code files and assets discussed in this blog post in the <a href="https://github.com/gvyshnya/dataproc-pyspark-etl">Github repo</a></p><p>Our next big challenge is to orchestrate such an ETL pipeline end-to-end as well as enable its scheduled unattended execution in Google Cloud Platform. That is where <em>Google Composer</em> with <em>Airflow </em>arrives to help us.</p><p>We will be looking at the <em>Airflow</em>-based workflow orchestration inside a Google Composer environment in the next part of this blog series.</p><p>As <em>Pearl Zhu</em>, author of the “Digital Master” book series told one day, “<em>We are moving slowly into an era where big data is the starting point, not the end</em>”. This is the key driver to focused on our future Big Data journey.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1690fe7a40bd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sbc-group-blog/effective-cloud-etl-in-cross-cloud-setup-part-1-let-big-data-flow-1690fe7a40bd">Effective Cloud ETL in Cross-Cloud Setup — Part 1: Let Big Data Flow</a> was originally published in <a href="https://medium.com/sbc-group-blog">SBC Group Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is AutoML Useful for Professional Data Scientists?]]></title>
            <link>https://medium.com/sbc-group-blog/is-automl-useful-for-professional-data-scientists-7cabbe1e301?source=rss-11881363dca------2</link>
            <guid isPermaLink="false">https://medium.com/p/7cabbe1e301</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[automl]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[George Vyshnya]]></dc:creator>
            <pubDate>Thu, 22 Apr 2021 13:42:50 GMT</pubDate>
            <atom:updated>2021-04-22T13:45:10.311Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K7OoNLE5eyi0MCWTdVZ7Uw.jpeg" /><figcaption>Cover photo by <a href="https://www.pexels.com/photo/woman-writing-at-table-with-laptop-and-ring-lamp-6348142/">Liza Summer</a></figcaption></figure><h3>Introduction</h3><p>As the interest in AutoML technologies grows in the industry over the last few years, we can see quite intense discussions on how useful they could be to use in the daily routines by professional Data Scientists. The opinions are quite polar. Some <a href="https://medium.com/analytics-vidhya/6-open-source-automated-machine-learning-tools-every-data-scientist-should-know-49960c1397c9">researchers </a>indicate AutoML to improve productivity of Data Scientists whereas other professionals are quite sharp at <a href="https://www.pyimagesearch.com/2019/01/07/auto-keras-and-automl-a-getting-started-guide/">stating</a>, “There is no silver bullet for solving machine learning/deep learning with off-the-shelf algorithms. Instead, I recommend you invest in yourself as a deep learning practitioner and engineer”.</p><p>The dramatism of such a discussion can be specially highlighted if you remember AutoML technologies to be never intended to use by professional Data Scientists at the first hand. Instead, AutoML was originally <a href="https://www.automl.org/automl/">invented</a> to lower the barriers for non-ML experts to adopt good-enough ML technologies in their operations without having to hire an expensive skilled Data Scientists and/or ML Engineers.</p><p>With the context above in mind, let us review the practical cases where certain AutoML tools can be helpful in automating the daily routines of Data Science / ML Engineering professionals. The practical case studies will be provided as illustrations to the points to make, as applied to the public datasets of Kaggle Tabular Playground competitions in Jan — Mar 2021.</p><h3>Classification of Modern AutoML solutions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*XyQzAzzFS63xffdynCFlOw.jpeg" /><figcaption>“Balance” (painting by <a href="https://artedialina.com/">Alina Ciuciu</a>, 2021)</figcaption></figure><p>Since there is a huge variety of AutoML tools (both freeware and commercial ones) that exist in the market nowadays, the question about AutoML being useful for ML/Data Science professionals shall read, “What kind of automation is helpful to the Data Science and ML professionals in completing their daily routines?”</p><p>The practical classification designates AutoML tools by the specific tasks they are going to automate. From that standpoint, we can talk about</p><ul><li>Automated EDA tools aka Rapid EDA tools (e. g. AutoViz, Sweetviz, Pandas Profiling etc.)</li><li>Automated data augmentation tools (e.g. imgaug, albumentations)</li><li>Automated feature engineering/selection tools (e.g. tpot, featurewiz, boruta_py)</li><li>Automated model selection tools (e.g. auto-sklearn, xcessiv)</li><li>Automated model architecture search tools (e.g. darts, enas)</li><li>Automated hyperparameter tuning tools (e.g. hyperopt, ray.tune, Vizier)</li><li>Tools to automate the full ML pipelines (e.g. Google AutoML, H20 Driverless AI, autokeras, AutoViML etc.)</li></ul><p>In the sections below, I am going to share my own experience with using some of the AutoML tools. As I go through, I will drop my opinions on what can be really helpful for professionals in the field as oppose to the less than ML-savvy users.</p><h3>Rapid EDA Use Cases</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*vwAqf3Y_VwzdUq9Bf-AMrw.jpeg" /><figcaption>“The elective affinities” (painting by <a href="https://artedialina.com/">Alina Ciuciu</a>, 2020)</figcaption></figure><p>Rapid EDA is one of the low-hanging fruits for AutoML tools to help professional Data Scientists in their daily routines. Obviously, good enough Rapid EDA tool should meet a few basic criteria below</p><ul><li>Generate a good number of insightful charts for a human professional to delve into the data insights on the immediate note</li><li>Operate with the high performance to really save the time of professionals on EDA while delivering the helpful visualizations out of the box</li><li>Be easy to use and maintain</li></ul><p>I have already completed several solid case studies to prove usefulness of various freeware Rapid EDA tools.</p><p>Across the experiments I have made, I sticked to <a href="https://github.com/AutoViML/AutoViz">AutoViz</a> over time. I found it meeting all the three basic criteria above. It was helpful for me on several real projects as well as Kaggle competitions.</p><p>As you can see, invoking AutoViz is as simple as writing the lines of code below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c8b1c88b7c4b6b9cd2c469ec02dbe944/href">https://medium.com/media/c8b1c88b7c4b6b9cd2c469ec02dbe944/href</a></iframe><p>AutoViz has a few parameters to tune before you run it on your dataset. However, there is a comprehensive guide on how to set it up on various datasets/problems to achieve the best exploratory insights at the end.</p><p><strong>Note</strong>: to see AutoViz in action end-to-end, you can refer to the source code in the following notebooks</p><ul><li><a href="https://github.com/gvyshnya/tab-mar-21/blob/main/Mar%2021%20TPC%20-%20Express%20EDA%20with%20AutoViz.ipynb">Express EDA for Mar 2021 TPC in Kaggle</a></li><li><a href="https://github.com/gvyshnya/tab-feb-2021/blob/main/Generic%20Express%20EDA%20with%20Comprehensive%20insights.ipynb">Express EDA for Feb 2021 TPC in Kaggle</a></li><li><a href="https://github.com/gvyshnya/tab-jan-2021/blob/main/AutoViz%20Feature%20Importance.ipynb">Express EDA for Jan 2021 TPC in Kaggle</a></li></ul><h3>Automated Feature Selection: Featurewiz</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*gZ_gCOGJXtIlfnKGQ25Dtg.jpeg" /><figcaption>“Love thoughts and a pure soul” (painting by <a href="https://artedialina.com/">Alina Ciuciu</a>, 2021)</figcaption></figure><p>Automated Feature selection and feature importance detection tools are extremely helpful when you try to explain your ML model or address the curse of dimensionality with your data (when you have too many features in your dataset, and the model loses its focus to deliver the most accurate prediction).</p><p>Across the years in the industry, I sticked to using an amazing freeware feature selection product —<a href="https://github.com/AutoViML/featurewiz"> featurewiz</a>. As oppose to available feature selection automation alternatives (Boruta, Tpot etc.), it turned out to deliver more accurate feature importance suggestions that were applicable to the weak ensembling ML algorithms (GBDT and RF models) as well some of the less complicated ML algorithms provided by scikit-learn.</p><p>At the same time, its simplicity and power made it more efficient tool to use vs. the analytical feature importance detection algorithms (see one of my earlier <a href="https://medium.com/sbc-group-blog/feature-importance-in-the-age-of-explainable-ai-ml-242327a70017">blog posts</a> for more details on the analytical feature importance detection algorithms).</p><p>The additional bonus of using featurewiz is its relatively new capabilities to automate the routines in feature engineering. Not only will it help you save some time developing your feature engineering pipelines, but it will also ensure the fast execution of your data preprocessing and feature engineering flows.</p><p>Below are the step-by-step descriptions of two Featurewiz-backed feature selection experiments for the dataset of Mar 2021 Tabular Playground competition. One of them will be a considerably basic feature importance detection (without any heavy feature engineering and data preprocessing invoked against the raw dataset). The second experiment will demonstrate how featurewiz can be an instrumental assistant in the situations where you must implement complex data preprocessing and feature engineering flows.</p><h4>Mar 2021 TPC: Basic Feature Importance Experiment</h4><p>In this experiment, we are going to detect the feature importance for raw feature variables only.</p><p>First, we are going to read the competition datasets into the memory.</p><p>As a next step, we will pass the datasets with the raw features only to featurewiz to detect feature importance</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7305e97ee35438afb406a2586984168/href">https://medium.com/media/d7305e97ee35438afb406a2586984168/href</a></iframe><p>It took it less then 3 min to run it on my local machine. The featurewiz was quite instrumental at detecting the important features as quickly as that.</p><p>As a side note, featurewiz allows you to quickly assess the impact of various category variable encoding techniques on the resulted feature importance.</p><p><strong>Note</strong>: You can trace this experiment end-to-end by reviewing the source code in the respective <a href="https://github.com/gvyshnya/tab-mar-21/blob/main/Mar%2021%20TPC%20-%20Raw%20Feature%20Importance%20with%20Featurewiz.ipynb">repo</a>.</p><h4>Advanced Experiment: Additional Feature Engineering based on AutoViz insights and feature importance</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*YdT52hCOk0cUCPR-ZV91pA.jpeg" /><figcaption>“Ice moon” (painting by <a href="https://artedialina.com/">Alina Ciuciu</a>, 2021)</figcaption></figure><p>In a series of feature importance experiments (with <em>AutoViz</em> engaged in a way described above), we got the following insights regarding the feature engineering and data preprocessing for the raw dataset of Mar 21 Tabular Playground Competition in Kaggle</p><p>1/ The following continual variables to be useful in feature importance experiment, to be left ‘as is’</p><ul><li><em>cont1, cont3, cont5, cont6, and cont8</em></li></ul><p>2/ The rest of the continual variables to be binned as follows</p><ul><li><em>cont0 </em>(into 4 bins)</li><li><em>cont1 </em>(into 5 bins)</li><li><em>cont3 </em>(into 2 bins)</li><li><em>cont4 </em>(into 2 bins)</li><li><em>cont6 </em>(into 3 bins)</li><li><em>cont8 </em>(into 3 bins)</li><li><em>cont10 </em>(into 10 bins)</li></ul><p>3/ New groupby features to be added</p><ul><li>group-by <em>cont3 by cat2, cont1 by cat4, and cont3 by cat4</em></li></ul><p>4/ New interaction cat variables (feature crosses) to be added</p><ul><li><em>cat4 x cat18</em></li><li><em>cat13 x cat4</em></li><li><em>cat13 x cat2</em></li></ul><p>5/ New interaction continual variables, then to be binned as follows</p><ul><li><em>cont3 x cont7</em></li><li><em>cont3 x cont8</em></li><li><em>cont3 x cont9</em></li><li><em>cont3 x cont10</em></li><li><em>cont4 x cont5</em></li><li><em>cont4 x cont6</em></li><li><em>cont4 x cont9</em></li><li><em>cont4 x cont10</em></li></ul><p>6/ Boolean cat variables to be as is</p><ul><li><em>cat0</em></li><li><em>cat1</em></li><li><em>cat12</em></li><li><em>cat13</em></li><li><em>cat14</em></li><li><em>cat15</em></li><li><em>cat16</em></li></ul><p>7/ log transform <em>cont5, cont8, and cont7</em></p><p>As we are going to see, the above-mentioned feature engineering pipeline can be easily facilitated with <em>featurewiz</em>. In its recent versions (namely, in version 0.0.33, as of the moment of writing this blog post) there are powerful functions to help you automate all of the above-mentioned preprocessing and feature engineering steps, with just a few lines of code to compose. Let’s see it in action.</p><p>We can add the new interaction features as follows</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/41937ece8382e20156d8b89600766578/href">https://medium.com/media/41937ece8382e20156d8b89600766578/href</a></iframe><p>Binning the continuous variables can be achieved as follows</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05dc4df3a2b49b7c601414cfd4eab546/href">https://medium.com/media/05dc4df3a2b49b7c601414cfd4eab546/href</a></iframe><p>Categorical feature crosses can be added as new features in the following manner</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/95e75cfea3bdde3892bf09ab947800eb/href">https://medium.com/media/95e75cfea3bdde3892bf09ab947800eb/href</a></iframe><p>The groupby aggregate features can be added as follows</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b0fa6f8635f6114381a1dfe30199d21/href">https://medium.com/media/8b0fa6f8635f6114381a1dfe30199d21/href</a></iframe><p>Last but not least, the easy-to-apply log transform (with all these little tricks like “plus one” on negative values etc.) can also be easily facilitated with featurewiz just with a few lines of the code</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/537a43ce4a69e9028e5482a9f49d8ee4/href">https://medium.com/media/537a43ce4a69e9028e5482a9f49d8ee4/href</a></iframe><p>As we can see, featurewiz is quite instrumental at automating the common routinous steps in data preprocessing and feature engineering, along with its core mission to detect the important features for the ML modelling down the road.</p><p>After the above-mentioned feature engineering and data preprocessing, we are ready to launch feature importance detection with featurewiz, similar to what has been demonstrated in the basic experiment above.</p><p>It took less then 9 min on my local computer to get this pipeline and feature selection experiment running end-to-end. I should say, it is amazingly short time, factored in the complexity of the data transformations and the set of the new features we obtained.</p><p><strong>Note</strong>: You can trace this experiment end-to-end by reviewing the source code in the respective <a href="https://github.com/gvyshnya/tab-mar-21/blob/main/Mar%2021%20TPC%20-%20Advanced%20FE%20and%20FI%20with%20Featurewiz.ipynb">repo</a></p><h3>Automated Hyperparameter Tuning Tools</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*nTO4h5rmJrV9Yf2nsWzz9Q.jpeg" /><figcaption>“Right wind” (painting by <a href="https://artedialina.com/">Alina Ciuciu</a>, 2020)</figcaption></figure><p>Automated hyperparameter tuning tools designate one more area of the useful automation of daily routines for professional Data Scientists. They help to shrink the time spending on the tedious model parameter tuning while being more effective compared to the classical grid search algorithms.</p><p>It is especially helpful when you tune GBDT models (like <em>lightgbm, xgboost, catboost</em> etc.) where there is a huge number of essential hyper-parameters to tweak. Therefor doing it manually or via a classic grid search may not be the best way to spend your time.</p><p>I found it equally helpful to use hyperopt and optuna in this capacity.</p><p>Tools like hyperopt or optuna can be successfully leveraged to speed parameter tuning of both individual models and the ensembles of different learners.</p><p>Below is the case study to demonstrate how you can combine the power of ensemble learning with leveraging hyperopt to tune hyperparameters of each of three models in the ensemble.</p><p><strong>Note</strong>: You can trace this experiment end-to-end by reviewing the source code in the <a href="https://github.com/gvyshnya/tab-mar-21/blob/main/lgbm-xgb-catb-ensemble-hyperopt.ipynb">respective repo</a></p><p>First, let’s review the targets of this ML experiment. They are as follows</p><ul><li>Build the ensemble classification prediction model for the problem in <a href="https://www.kaggle.com/c/tabular-playground-series-mar-2021/"><em>Kaggle Tabular Playground Series — Mar 2021 contest</em></a>, using three GBDT models (<em>lightgbm, xgboost, and catboost</em>) as the ensemble members</li><li>Automate the hyperparameter tuning of every model in the ensemble using hyperopt</li><li>Use <em>AUC </em>(Area under the ROC curve) as a model performance metric to optimize the model parameters for</li></ul><p>From the software development standpoint, we will implement the ensemble as a custom Python class, with key methods of the scikit-learn Classifier predictor (<em>fit, predict, predict_proba</em>) implemented. We are also going to enable the weighted ensemble voting prediction option in case the ML modelling will justify it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5bc4663efc06ea1aaee886076bf511c3/href">https://medium.com/media/5bc4663efc06ea1aaee886076bf511c3/href</a></iframe><p>This will allow us to pass the instance of the ensemble modeler class wherever a scikit-learn Classifier object can be passed (inclusive of the search function for hyperopt).</p><p>We will rely on scikit-learn Classifier interfaces provided by maintainers of lightgbm, xgboost, and catboost libraries (rather than their native interfaces), for unification reasons and simplicity.</p><p>Our next step would be building the hyperopt search function as per the code fragment below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ad86b2151c27020464831473b0dde4ff/href">https://medium.com/media/ad86b2151c27020464831473b0dde4ff/href</a></iframe><p>There are several highlights on the search function implemented</p><ul><li>Since we optimize the model for the best AUC value (that is, to ensure the maximal value of AUC in the search rounds of hyperopt), we are forced invent the artificial metric — negative AUC (as hyperopt search works to minimize the loss function specified)</li><li>The parameter space passed to the search function of hyperopt should specify the relevant attributes for all of three models used in our ensemble (that is, <em>lightgbm, xgboost, and catboost</em>)</li></ul><p>The appropriate parameter space is facilitated via the special naming convention we adopt. Such a naming convention is supported / ensured by the constructor of our custom ensemble class (see above). As a result, we have to prepare the dictionary with the hyperopt search space attributes in a special way, as displayed below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b547314cd7f0c4493b16ddacb6dbee52/href">https://medium.com/media/b547314cd7f0c4493b16ddacb6dbee52/href</a></iframe><p>Once hyperopt detects the optimal set of hyperparameters for each model in the ensemble, we do the usual model prediction as per the scikit-learn contract.</p><h3>On the Dark Side of AutoML</h3><p>So far so good? Does it sound like AutoML is going to be the paramount to the professional Data Scientists and ML Engineers? Unfortunately, not everything seems to be so bright. We can see certain drawbacks if we touch two more types of AutoML products</p><ul><li>Automated Model Architecture Search Tools</li><li>Tools to automate the full ML pipelines</li></ul><p>I am going to review it in the next blog post of this series. At the same time, the next post will cover the eternal myth on AutoML — the one that tells about AI and AutoML to replace the highly skilled (and often highly paid) ML Engineers and Data Science professionals in the middle or long run.</p><h3>AutoML Impact on ML and Data Science Industry</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eO_m2mwspjKXSSqnW2VPvQ.jpeg" /><figcaption>“I’ll bring you to Venice” (painting by <a href="https://artedialina.com/">Alina Ciuciu</a>, 2021)</figcaption></figure><p>Despite the original intention of AutoML to serve the needs of non-Machine Learning professionals, it will affect (and is affecting) the experts in the field as well.</p><p>Some gurus on the pessimistic side predict more automation in the field is going to shift the focus away from programming skills in favour of competence in statistics and research methods. Conversely, the alternative viewpoint proclaims <a href="https://www.pyimagesearch.com/2019/01/07/auto-keras-and-automl-a-getting-started-guide/">AutoML would never beat</a> the skilled ML professional with the expertise in the appropriate technologies and the domain knowledge about the data he/she works with.</p><p>I tend to second the second point of view myself. I therefore use some kinds of AutoML instruments for what they are (that is, auxiliary tools), with no expectation to have a magic wand to solve any ML/DL problems for me. I also put the continual effort to fill my own toolbox with additional knowledge.</p><p>As shown in the sections above, some of the use cases where you can leverage AutoML tools embrace the scenarios below</p><ul><li>Utilizing Rapid EDA tools (AutoViz or similar) to speed up getting the insightful visualization of the basic statistics and variable interactions/associations in your datasets</li><li>Automating the Feature Engineering Feature Selection with featurewiz</li><li>Automating the ML Model Hyperparameter Tuning with tools like hyperopt or optuna</li></ul><p>Obviously, junior Data Scientists / ML Engineers still must learn what is under the hood with the methods being automated with AutoML tools above. In particular, it implies trying to do everything mentioned above manually, using the standard tools offered by pandas, scikit-learn and other lower-level Python libraries for ML and Data Science couple of times. In such a way, you will actually comprehend what’s going on behind the AutoML magic as well as control what tools you use to tackle a specific problem on your side.</p><p>After that you can fearlessly leverage AutoML capabilities to safe some time on doing routine operations for the sake of doing more complex problem resolutions or unstructured problem solving.</p><p>However, regardless the involvement of model and pipeline automation tools beyond the model training, the ML field is going through an explainability crisis now. That is where the real opportunity for ML professionals/Data Scientist is.</p><p>If you can establish yourself as a data professional that can understand the datasets you are working with (data cleaning, identifying data leaks etc.) as well as create models that are explainable (or/and statistically sound), you can find yourself on the right track. Creating explainable ML models, however, embraces not just using the ‘explainable’ ML algorithms alone. Nowadays it is also about using the <a href="https://medium.com/sbc-group-blog/feature-importance-in-the-age-of-explainable-ai-ml-242327a70017">analytical feature importance methods</a> or tools like <em>SHAP </em>that can bring explainability to any modern model trained with ‘non-explainable’ yet powerful algorithms (like modern neural networks, GBDT variations etc.).</p><h3>References</h3><p>If you like to delve deeper in AutoML technologies deeper, you are welcome to continue your research with the resources below</p><ul><li>Towards Automated Machine Learning: Evaluation and Comparison of AutoML Approaches and Tools — <a href="https://arxiv.org/pdf/1908.05557.pdf">https://arxiv.org/pdf/1908.05557.pdf</a></li><li>AutoML: A Survey of the State-of-the-Art — <a href="https://arxiv.org/pdf/1908.00709.pdf">https://arxiv.org/pdf/1908.00709.pdf</a></li><li>Auto-Keras and AutoML: A Getting Started Guide — <a href="https://www.pyimagesearch.com/2019/01/07/auto-keras-and-automl-a-getting-started-guide/">https://www.pyimagesearch.com/2019/01/07/auto-keras-and-automl-a-getting-started-guide/</a></li><li>AutoML Overview by AutoML.org portal — <a href="https://www.automl.org/automl/">https://www.automl.org/automl/</a></li><li>Six Levels of Auto ML — <a href="https://medium.com/@tunguz/six-levels-of-auto-ml-a277aa1f0f38">https://medium.com/@tunguz/six-levels-of-auto-ml-a277aa1f0f38</a></li><li>6 Open Source Automated Machine Learning Tools Every Data Scientist Should Know — <a href="https://medium.com/analytics-vidhya/6-open-source-automated-machine-learning-tools-every-data-scientist-should-know-49960c1397c9">https://medium.com/analytics-vidhya/6-open-source-automated-machine-learning-tools-every-data-scientist-should-know-49960c1397c9</a></li><li>PROs and CONs of Rapid EDA Tools — <a href="https://medium.com/sbc-group-blog/pros-and-cons-of-rapid-eda-tools-e1ccd159ab07">https://medium.com/sbc-group-blog/pros-and-cons-of-rapid-eda-tools-e1ccd159ab07</a></li><li>Feature Importance in the Age of Explainable AI/ML — <a href="https://medium.com/sbc-group-blog/feature-importance-in-the-age-of-explainable-ai-ml-242327a70017">https://medium.com/sbc-group-blog/feature-importance-in-the-age-of-explainable-ai-ml-242327a70017</a></li></ul><p>You can refer to the public datasets for the respective Tabular Playground competitions in Kaggle in Jan — Mar 2021 per the links below</p><ul><li><a href="https://www.kaggle.com/c/tabular-playground-series-jan-2021">https://www.kaggle.com/c/tabular-playground-series-jan-2021</a></li><li><a href="https://www.kaggle.com/c/tabular-playground-series-feb-2021">https://www.kaggle.com/c/tabular-playground-series-feb-2021</a></li><li><a href="https://www.kaggle.com/c/tabular-playground-series-mar-2021">https://www.kaggle.com/c/tabular-playground-series-mar-2021</a></li></ul><p>You can find my repositories with the code for the various experiments with the datasets of these competitions by navigating to the github repos below</p><ul><li><a href="https://github.com/gvyshnya/tab-jan-2021">https://github.com/gvyshnya/tab-jan-2021</a></li><li><a href="https://github.com/gvyshnya/tab-feb-2021">https://github.com/gvyshnya/tab-feb-2021</a></li><li><a href="https://github.com/gvyshnya/tab-mar-21">https://github.com/gvyshnya/tab-mar-21</a></li></ul><p>Cover photo by Liza Summer (<a href="https://www.pexels.com/photo/woman-writing-at-table-with-laptop-and-ring-lamp-6348142/">https://www.pexels.com/photo/woman-writing-at-table-with-laptop-and-ring-lamp-6348142/</a>)</p><p>As the section title images, the paintings by Alina Ciuciu (<a href="https://artedialina.com/">https://artedialina.com/</a>) have been used.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7cabbe1e301" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sbc-group-blog/is-automl-useful-for-professional-data-scientists-7cabbe1e301">Is AutoML Useful for Professional Data Scientists?</a> was originally published in <a href="https://medium.com/sbc-group-blog">SBC Group Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science and ML Trends in 2020–2022: Portraits of the Industry and Raise of AutoML]]></title>
            <link>https://medium.com/sbc-group-blog/data-science-and-ml-trends-in-2020-2022-portraits-of-the-industry-and-raise-of-automl-a0bc875545df?source=rss-11881363dca------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0bc875545df</guid>
            <category><![CDATA[automl]]></category>
            <category><![CDATA[trends]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[George Vyshnya]]></dc:creator>
            <pubDate>Mon, 01 Feb 2021 23:43:45 GMT</pubDate>
            <atom:updated>2021-02-02T11:48:22.385Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YGwfEtaFi1rR1Y_ojrGxFw.jpeg" /><figcaption>Knowledge is power — Francis Bacon</figcaption></figure><h3>Introduction</h3><p>The key asset of every knowledge-intensive industry (Software Development, Data Science and ML Engineering inclusive) is people. The way they operate, learn, collaborate, and share knowledge is essential part of whether the projects and initiatives are set for success.</p><p>At the other hand, the organizational environment is the industry glue that enables or equally disables human power contributions to the certain extent.</p><p>Everything mentioned above will affect the shape of Data Science and ML industry in 2021–2022.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ic6gzFoKDi0JSG1GbIKsXA.jpeg" /><figcaption>The Data Science Fellow cohort photo (reused from <a href="https://www.nceas.ucsb.edu/news/next-generation-environmental-scientists-are-data-scientists">here</a>)</figcaption></figure><p>In this post, we are going to look at ‘Portraits and Landscapes’ of Data Science and ML that would determine the industry trends in 2021–2022. It will cover</p><ul><li>Human capital of the industry (demography of ML and Data Science professionals, their education level, job responsibilities as well as the way they learn, collaborate, and share knowledge with each other)</li><li>Organizational environments to affect ML adoption in business and public sector</li><li>Usage of automation tools (AutoML products, software to manage various Data Science and ML experiments etc.) by the industry professionals</li></ul><p>The insights conveyed by this article are based on the analysis of the data collected in Kaggle’s survey of ‘State of Data Science and Machine Learning 2020’ (<a href="https://www.kaggle.com/c/kaggle-survey-2020">https://www.kaggle.com/c/kaggle-survey-2020</a>).</p><p><em>Note</em>: Kaggle (<a href="http://www.kaggle.com/">www.kaggle.com</a>) is a global community made up of data scientists and machine learners from all over the world with a variety of skills and backgrounds. The community has around 3 million active members. Although it is not rigorously representative of the entire population of Data Science and ML professionals across the globe from the sociological perspective, it still constitutes the significant fraction of the practitioners and professionals in the field. Therefore, the results of the survey can really draw the projections of where the Data Science and AI/ML industry is likely to evolve in the next couple of years.</p><h3>Who Are You, Mr. and Mrs. Data Scientists?</h3><p>As we can see from the survey results, the shape of the Data Science and ML industry was shifting in 2020. It is also predicted to see such a shift extended through 2021–2022.</p><p>The industry becomes more extensive and younger. The number of the industry practitioners grows, especially with more juniors (like students or fresh graduates) taking the path in Data Science and ML.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ody8b8MtKMy0AeaX_A39FQ.jpeg" /></figure><p>We also see the percentage of women contributing to the industry is growing in 2020 vs. 2018 and 2019, and it is likely to go on this way in 2021–2022.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IaHqX5tJ78sS8jgh7mXw1w.jpeg" /></figure><p>At the same time, it highlights the average experience level of the survey participants to be relatively low. The majority of the survey respondents fell into the following clusters</p><ul><li>Professionals with less then 1 year of programming and less then 1 year of ML experience</li><li>Professionals with 1–2 years of programming and less then 1 year of ML experience</li><li>Professionals with 3–5 years of programming and less then 1 year of ML experience</li><li>Professionals with 1–2 years of programming and 1–2 years of ML experience</li><li>Professionals with 3–5 years of programming and 1–2 years of ML experience</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D-DvLNDZC5BNP0DT26BWDA.jpeg" /></figure><p>It indicates the industry is actively expanding now. 2020–2022 are going to be the years of further proliferation of the industry. Students and junior professionals will get the relevant experience in 5 years from now, and then it will change the landscape of Data Science and ML fields.</p><h3>Where Are You, Mr. and Mrs. Data Scientists?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Rj-XGYblmRqX2Bri8hoTg.jpeg" /></figure><p>In terms of the geography, we see that</p><ul><li>The most the survey respondents reside in India</li><li>The US holds the second place in the country rank</li><li>The next places in the top rank list are held by Japan, Brazil, UK, China, Russia, and Nigiria, respectively</li><li>Developed nations in the Old World (UK; EU nations like Germany, France, Spain, Italy; Turkey), Canada and Australia are quite behind the countries in the top rank list</li><li>At the same time, some of the developing nations (Indonesia, Mexico, Pakistan) and Taiwan are on a par with the developed nations in the Old World that are not in the top rank list</li><li>South Korea and China are surprisingly under-represented among the survey participants</li></ul><p><em>Note</em>: Under-represented population of Data Scientists and ML professionals from China in Kaggle may represent some sort of ‘isolationism’ of such professionals that tend to contribute to their local Data Science platforms.</p><p>We can also see clear ‘challenger’ nations in terms of the growing number of Data Science and ML professionals (Brazil, Nigeria). When the professionals there get more experience (they are not as experienced as their counterparts in the US, Old World and India are), these countries will bring new surprising AI and ML technology innovations to the global scene.</p><h3>Age and Level of Education</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eHIkNapxpJ6AIImTEiDUiw.jpeg" /></figure><p>We can see that there are four biggest Kaggle population clusters in the space of Education level and Age as follows</p><ul><li>Bachelors of the age of 18–21 (the biggest cluster so far)</li><li>People of the age of 25–29 with the Master degree</li><li>Bachelors of the age of 22–24</li><li>Masters of the age of 22–24</li></ul><p>On top of the population cluster insight above, we can draw the additional intelligence as follows</p><ul><li>People with Bachelor degree predominate in more junior age groups</li><li>In senior age groups, the percentage of Master and Doctoral degrees grows</li><li>People with Master degrees predominate in the age groups of 25–29 and older</li><li>Starting the age group of 35–39 and older, the amount of respondents with Doctoral degrees is only a little less than the respondents with Master degree for the same age group</li></ul><h3>Gender and Level of Education</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R8yyIoJv6o52UaRVYbL77w.jpeg" /></figure><p>As we can see, there are top clusters of the survey respondents within the dimensions of Gender and Level of Educations as follows</p><ul><li>Males with Bachelor’s degree</li><li>Males with Master’s degree</li><li>Males with Doctoral degree</li><li>Females with Bachelor’s degree</li><li>Females with Master’s degree</li></ul><p>Bachelor and Master degrees are the most common levels of education for every gender.</p><h3>Level of Education and Programming Experience</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ymUJl6kCJPzJ2er12ljz_Q.jpeg" /></figure><p>As we can see there are four top clusters among the survey responders within Education and Programming experience dimensions as follows</p><ul><li>People with Bachelor’s degree having 1–2 years of programming experience</li><li>People with Master’s degree having 3–5 years of programming experience</li><li>People with Bachelor’s degree having 3–5 years of programming experience</li><li>People with Master’s degree having 1–2 years of programming experience</li></ul><p>For more experienced categories (5+ years of programming experience), we find that the ratio of people with Master’s degree exceeds one for Bachelor’s degree.</p><p>Percentage of people with Doctoral degree is higher in the most experienced categories (programming experience of 10–20 years as well as 20+ years).</p><h3>Where Are the Industry Veterans?</h3><p>The location of the industry veterans (having 20+ years of programming experience and/or having 10+ years of ML experience) does not fully correspond to the countries with the biggest number of professionals in the field.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hkf4qqIxkVS_NqbUw4JoYA.jpeg" /></figure><p>As we can see, the biggest number of programming veterans among the survey respondents reside in the US. The rest of the countries are well below the bar set by the US.</p><p>However, there are several countries that still stand out in terms of their population of the programming veterans participated in the survey. These are</p><ul><li>Japan</li><li>UK</li><li>Brazil</li><li>India</li></ul><p>As we see, India, although represented by the largest population of Kagglers among the survey respondents, does not possess the huge pool of the experts with 20+ years of programming experience vs. the rest of the leading countries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5lygIWtr9sP35HQMbBzA1A.jpeg" /></figure><p>As displayed above, the US absolutely predominates the rest of the world in terms of the number of the survey participants having 10+ years of ML experience.</p><h3>Organizational Environment and Enterprise-Scale Data Science/ML</h3><p>The important aspect of implementing Data Science/ML-related functions in the modern organization are predominated by (1) the organizational environment and (2) job responsibilities of the professionals involved.</p><p>The survey highlighted essential insights on how the factors below affect the corporate Data Science/ML processes. These are</p><ul><li>Size of the employer organization</li><li>Size of the team working in Data Science-related areas</li><li>Adoption of ML in the organization</li><li>Individual job responsibilities of the survey participants</li><li>Spending on ML and Cloud Computing in the last 5 years</li></ul><h4>Organization Size, Data Science and ML Adoption</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*foinQR3Mr8whOr-He6R_KA.jpeg" /></figure><p>As we can see, the largest cluster of the survey responders work for the small organizations (0–49 employees). Within that bucket, most of the organization has either 1–2, 0 or 3–4 employees responsible for data science workloads.</p><p>Most of such organizations is still exploring ML methods. However, some fraction of such organizations</p><ul><li>is reported to recently start using ML methods, or</li><li>do not use ML currently</li></ul><p>Small organizations with no workers dedicated to data science workloads mostly do not use ML in their daily operations (which is a kind of expected intuition).</p><p>Interesting (small-sized) subclusters of small organizations with 5+ employees dedicated to data science workloads display mature usage of ML solutions in production.</p><p>Second and third largest clusters of survey responders constitute those who work for</p><ul><li>organizations with 10000+ employees</li><li>organizations with 1000–9999 employees</li></ul><p>In such clusters, we can see quite a lot of respondents to indicate their organizations to have well established ML methods.</p><p>Overall, irrespective to the organization size, it seems that the size of a Data Science-centric team of 5+ workers indicate certain level of ML methods maturity within an organization. In organizations in every size group, the companies with 5+ workers dedicated to Data Science activities indicated to either start using ML in production recently or have the mature ML methods established.</p><h4>ML Spending</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9aFPAZHt4u5aNRTh8MkEAw.png" /></figure><p>From looking at the charts, we can see the most popular option as for ML spending is USD 0, regardless the organization size. It may be indicative of the inaccuracies in the responder inputs when they responded to the respective question of the survey (or otherwise they could fake the answers for a reason).</p><p>Still, if considering the meaningful responses, it looks like</p><ul><li>large organizations tend to spend more in ML (100+ k USD in the last 5 years)</li><li>smaller-sized organizations spent 100–999 or 1000–9999 USD in ML in the last 5 years</li></ul><h3>Will the AutoML Age Commence Soon?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*Fv4ORAqIbe8A6PkxEtJ7VA.jpeg" /></figure><p>We are going to discover how Data Science automation tools are utilized in the industry. It conveys findings as for</p><ul><li>the usage of automated machine learning tools</li><li>the usage of tools to help manage machine learning experiments</li></ul><h4>Using AutoML Tools: By-Purpose View</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dSInoXy90ApI2v_30PFQOg.jpeg" /></figure><p>We find that</p><ul><li>the vast majority of the survey participants do not use any Auto ML tools in their daily routines</li><li>automated model selection tools is the most popular type of Auto ML tools used by the kagglers at the moment</li><li>automation tools for selecting a neural network architecture are not vibrant and well known</li><li>Data Scientists are the primary users of Auto ML tools (if in use)</li></ul><h4>Using AutoML Tools By Purpose and Size of Employer Organization</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qbg-Pw3FOXYTx6jbrZZ_rg.jpeg" /></figure><p>We find that</p><ul><li>Both the large-sized and small-sized organizations are equally represented across the cluster of organizations where auto ML tools are not used now</li><li>For each type of auto ML tools, the most usage of them is observed in small-sized organizations (it is also correlated with the fact most of the survey respondents work for such organizations in fact)</li><li>Using AutoML Tools By Purpose and Size of Data Science Team in Organizations</li><li>Both the organizations with large-sized and small-sized data science teams are represented across the cluster of organizations where auto ML tools are not used at the moment</li><li>If AutoML tools used, the most use of them is observed in organizations where Data Science team has capacity of 1–2, 3–4, or 20+ workers</li></ul><h4>Utilizing AutoML Tools By Product and User Occupation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Se2ZOgtMxW_yjd-foMZzKw.jpeg" /></figure><p>In addition to the insights above, we find that</p><ul><li>For every auto ML product surveyed, professionals with 1–2, 3–5, 5–10, and 10–20 years of programming experiences are equally represented among the users</li><li>Professionals with 20+ years of programming experience are less attached to using auto ML products</li></ul><h4>Utilizing AutoML Tools By Product, Size of Organization, and DS Team Capacity</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9sfeWCVe_bV-9Y90dp2dCQ.jpeg" /></figure><p>We find that</p><ul><li>Small-sized organizations (0–49 employees) have more inclination to play around Auto-Scikitlearn and Auto-Keras</li><li>For larger-sized organization, the biggest cluster is always the subset of organizations that do not use any auto ML, regardless the size of their Data Science team</li></ul><h4>Utilization of Tools to Manage DS Experiments</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lRj27-H5JFerMdkOvMpi7A.jpeg" /></figure><p>We find that</p><ul><li>Most of the survey respondents do not use any tools to help to manage data science experiments (so the market for such tools is extensively under-saturated)</li><li>TensorBoard is the leading tool in use at the moment</li><li>Data Scientists, Software Engineers, and Data Analysts are the top occupations of the users of such tools</li></ul><h3>Knowledge and Information Sharing Channels</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dVJ6ZKtsLKplLc2bydBcSA.jpeg" /><figcaption>Sir Francis Bacon</figcaption></figure><p>Knowledge is power, as Sir Francis Bacon told one day. Additionally, information becomes the power and ‘new oil’ in the modern post-industrial age.</p><p>Therefore it is essentially interesting to draw insights on how kagglers work with information as well as share/obtain professional knowledge.</p><p>In this section, we are going to review the preferences of the survey participants as for</p><ul><li>Platforms and tools to publicly share or deploy their data analysis or machine learning applications</li><li>Platforms to take online data science courses</li><li>Primary tools they use to analyze data</li><li>Pavorite media sources that report on data science topics</li></ul><h4>Public Sharing Platform Preferences by Occupation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*onLZ0PpLazi8V4HUUoDfEw.jpeg" /></figure><p>We find that</p><ul><li>GitHub is the primary choice for Kagglers to share their work publicly</li><li>Kaggle itself as well as Colab take the next places in the top 3 list of platforms to publicly share the work assets</li><li>Good work sharing instruments like Streamlit and Plotly Dash are under-utilized by the community</li><li>Quite a big fraction of the survey respondents indicated they do not share any work publicly</li><li>The percentage of Business Analysts and people with ‘Other’ occupation who do not share their work results publicly is higher vs. the rest of the occupations</li></ul><h4>Public Sharing Platform Preferences by Occupation and Programming Experience</h4><p>In addition to the insights above, we find that</p><ul><li>GitHub and Kaggle are the preferred sharing platforms for every occupation and programming experience level groups</li><li>The percentage of professionals not sharing their work results publicly is higher with more experienced professional groups (people with 10–20 and 20+ years of programming experience) — the replied with ‘None’ more frequently then other experience level group members</li></ul><h4>Kaggle vs. Colab Usage</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ylz7yqsVJiZdSZh4cYWuEQ.jpeg" /></figure><p>After purchasing Kaggle platform, Google started to own two similar technologies for data scientists to work and share their work results publicly in the cloud at a reasonable or even no cost (these are Kaggle notebooks and Collab notebooks). Both instruments are utilized by the industry professionals now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h8jNpvkpahVhqhv4S0v9zQ.jpeg" /></figure><p>We can see that</p><ul><li>Kaggle is used more actively by males in the majority of occupations, levels of education and programming experience levels</li><li>The most tangible diff is observed for male Data Scientists with Master’s and Bachelor’s degree, with programming experience between 1 and 10 years</li><li>in a sense, we can think of Kaggle notebooks and Colab to compete over the same target audience</li></ul><p>In the evidence of the above-mentioned data-driven insights, we can assume Google could benefit from re-positioning Colab from the functional and marketing stand-points, to drive more paying users to it.</p><h4>Online Training Platform Preferences by Occupation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3KMIbvJFTWMfHLi9b0m5iQ.jpeg" /></figure><p>We find that</p><ul><li>Coursera, Kaggle Learn Courses, and Udemy are in the top 3 list of preferred learning platforms (with Coursera being far more popular then the rest of the platforms in top 3 list)</li><li>University courses leading to a formal university degree are also quite popular (it takes the 4th rank in the list)</li><li>edX is behind its primary e-learning platform rivals</li><li>only a minor fraction of respondents indicate they did not begin or complete data science courses (this is the healthy indicator of the data science community here at Kaggle to have good learning and self-learning attitudes)</li></ul><h4>Online Learning Platform Preferences by Occupation and Programming Experience</h4><p>In addition to the insights above, we find that</p><ul><li>Coursera is the top platform of choice by every occupation and programming experience level groups</li><li>Kaggle Learn Courses are more popular with the inexperienced junior professionals (have &lt;1 or 1–2 year of professional experience)</li><li>For more senior professionals (with experience level of 3+ years), Udemy is the second platform of choice (after Coursera), and they do not actively use Kaggle Learn Courses</li></ul><h4>Primary Data Analysis Tool Preferences by Occupation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9EztV0t7ojjswf1ByphB8g.jpeg" /></figure><p>We find that</p><ul><li>Local IDEs is the primary choice of the majority of the survey respondents (it highlights quite good technical skills of the respective respondents)</li><li>Basic Stas software is the second preference of the respondents</li><li>Data Scientists do not often use Basic Stas software as opposed to other analysis tool options investigated</li><li>The rest of tool types goes far behind Local IDEs and Basic Stas software</li></ul><h4>Primary Data Analysis Tool Preferences by Occupation and Programming Experience</h4><p>We can find that</p><ul><li>technically savvy occupations are more inclined to use Local IDEs as their primary ‘data analysis’ tool of choice whereas less technical occupations (like Business Analysts) tend to use basic stat tools more</li><li>it is also noted that professionals with little programming experience (&lt;1 year or zero experience) tend to choose basic stats softare whereas professionals with 2+ years of programming experience do not feel any fear of using local IDEs to serve their data analysis needs</li></ul><h4>Favorite Media Sources by Occupation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o08vcRmf9zmo4UtDFkzntQ.jpeg" /></figure><p>We find that</p><ul><li>Kaggle as a medium of a valuable Data Science-related information it the top choice with the majority of the survey respondents, and it strongly out-performs other media sources investigated</li><li>the second best is Youtube content</li><li>popular Data Science blogs take the third place in the rank</li><li>other media sources are well below the top three media listed above</li></ul><h4>Favorite Media Sources by Occupation аnd Programming Experience</h4><p>In addition to insights above, we find that</p><ul><li>Almost in every occupation, people with programming experience of less then 10 years are funs of Kaggle as the primary information source</li><li>People with 10+ years of experience tend to rank other information sources as their primary go-to medium choice, with Blogs being of the popular ones</li><li>Junior Business Analysts (with 2 or less years of programming experience) rank YouTube as their primary go-to medium whereas more senior-level Business Analysts prefer Blog posts and Kaggle</li><li>Statisticians with 20+ years of experience prefer Journal papers/publications, and Statisticians with less years of programming experience look the information up in Blogs and Kaggle as their go-to choices</li><li>Product/Project managers rank Kaggle as their go-to information source, regardless their programming experience</li></ul><h3>Summary</h3><p>In this post, we tried to draw the portrait of Data Science and ML industry in the upcoming years (2021–2022). We can see a lot of interesting trends to sharpen the industry landscape in the next two years.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*pZPOjUxiAL_rkpkjom_guA.jpeg" /></figure><p>The industry is prolificating, and its geography expands. Raise of interest in AutoML tools, along with the more pragmatic approach to them (AutoML as tools, not as a Holy Grail to tackle each ML problem without a human expert involvement), will be certainly boosting the speed of evolution of the industry.</p><p>Overall, Data Science and ML will remain one of the sexiest occupation fields in the next two years, despite the voices of sceptics.</p><h3>References</h3><p>This is the second article in my series about the insights from the data collected in Kaggle’s survey of ‘State of Data Science and Machine Learning 2020’ (https://www.kaggle.com/c/kaggle-survey-2020).</p><p>The first article in the series, “Cloud Computing, Data Science and ML Trends in 2020–2022: The battle of giants”, can be reviewed per <a href="https://medium.com/sbc-group-blog/cloud-computing-and-data-science-and-ml-trends-in-2020-2022-the-battle-of-giants-c2a174d3cd2b">https://medium.com/sbc-group-blog/cloud-computing-and-data-science-and-ml-trends-in-2020-2022-the-battle-of-giants-c2a174d3cd2b</a></p><p><em>Note</em>: you can check the repo per <a href="https://github.com/gvyshnya/state-of-data-science-and-ml-2020">https://github.com/gvyshnya/state-of-data-science-and-ml-2020</a> to see how every insight above has been discovered.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0bc875545df" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sbc-group-blog/data-science-and-ml-trends-in-2020-2022-portraits-of-the-industry-and-raise-of-automl-a0bc875545df">Data Science and ML Trends in 2020–2022: Portraits of the Industry and Raise of AutoML</a> was originally published in <a href="https://medium.com/sbc-group-blog">SBC Group Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cloud Computing and Data Science and ML Trends in 2020–2022: The battle of giants]]></title>
            <link>https://medium.com/sbc-group-blog/cloud-computing-and-data-science-and-ml-trends-in-2020-2022-the-battle-of-giants-c2a174d3cd2b?source=rss-11881363dca------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2a174d3cd2b</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[trends]]></category>
            <dc:creator><![CDATA[George Vyshnya]]></dc:creator>
            <pubDate>Sun, 10 Jan 2021 09:43:40 GMT</pubDate>
            <atom:updated>2021-01-31T21:16:55.484Z</atom:updated>
            <content:encoded><![CDATA[<h3>Cloud Computing, Data Science and ML Trends in 2020–2022: The battle of giants</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*W5eMqmQoO7DzI3qtfXyGEQ.jpeg" /><figcaption>Every time you see a giant, you must know that that giant might be just a dwarf somewhere else! ― Mehmet Murat ildan</figcaption></figure><h3>Preface</h3><p>This post got KDnuggets Gold Blog award as the most shared post in the week of <a href="https://www.kdnuggets.com/2021/01/top-news-week-0118-0124.html">Jan 18–24, 2021</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/96/1*V5R9IHs2DWoHgcXmioUaeg.png" /><figcaption>See <a href="https://www.kdnuggets.com/2021/01/cloud-computing-data-science-ml-trends-2020-2022-battle-giants.html">more</a></figcaption></figure><h3>Introduction</h3><p>I am going to dedicate a series of articles to the insights from the data collected in Kaggle’s survey of ‘State of Data Science and Machine Learning 2020’ (<a href="https://www.kaggle.com/c/kaggle-survey-2020">https://www.kaggle.com/c/kaggle-survey-2020</a>).</p><p>The survey covered a lot of diverse topics, and each of them deserves a separate post to discuss the respective trends.</p><p><em>Notes</em>:</p><ul><li>Kaggle (<a href="http://www.kaggle.com/">www.kaggle.com</a>) is a global community made up of data scientists and machine learners from all over the world with a variety of skills and backgrounds. The community has around 3 million active members. Although it is not rigorously representative of the entire population of Data Science and ML professionals across the globe from the sociological perspective, it still constitutes the significant fraction of the practitioners and professionals in the field. Therefore, the results of the survey can really draw the projections of where the Data Science and AI/ML industry is likely to evolve in the next couple of years.</li><li>You can check the repo per <a href="https://github.com/gvyshnya/state-of-data-science-and-ml-2020">https://github.com/gvyshnya/state-of-data-science-and-ml-2020</a> to see how every insight discussed in this post has been discovered.</li></ul><h3>Battle of Giants</h3><p>In this post, we are going to look at the popularity of cloud computing platforms and products among the data science and ML professionals participated in the survey. In particular, it will cover</p><ul><li>Cloud Platforms usage</li><li>Cloud Computing products usage</li><li>Cloud ML products usage</li><li>BigData platforms</li><li>BI tools (mostly, the cloud-based ones)</li></ul><p>The line of the narrative in this chapter will be often attached to the good news and opportunities for the top three cloud service providers in the market as follows</p><ul><li>Amazon Web Services (AWS)</li><li>Google Cloud Platform (GCP)</li><li>Microsoft Azure Cloud (MS Azure)</li></ul><p><em>Note:</em></p><ul><li>Survey organizers defined non-professionals as students, unemployed, and respondents that have never spent any money in the cloud. Everybody else is considered to be a professional</li></ul><h3>Usage of Cloud Service Providers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mBk6zqev4CNliPdW797PsQ.png" /></figure><p>We find that the top three list of cloud service providers among the Kaggle survey responders are</p><ul><li>AWS</li><li>GCP</li><li>MS Azure</li></ul><p>The rest of the cloud service providers seems to loose the competitive edge toward the top three provider list above at the moment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bfc7-e56MXVTF4uqicxWYw.png" /></figure><p>Also, it is notable that ‘None’ category slightly exceeds the size of MS Asure bar, and it means the market may not be saturated with the cloud service provider offerings.</p><p>We also see professionals with 3–5 years and 5–10 years of programming experince to be the largest group of users of the top 3 cloud service providers. More senior professionals (with 10+ years of experience) are less represented within the cloud service users on each of top 3 platforms (depending on the marketing priorities, special actions to educatesuch seniors could help to get the better spread of the cloud services).</p><p>As we can see, the majority of top three cloud service provider users fit into the roles below</p><ul><li>Data Scientists</li><li>Software Engineers</li></ul><p>The third posion is held by</p><ul><li>ML Engineeris (AWS, GCP)</li><li>Data Analysts (MS Azure)</li></ul><p>As noted earlier, ‘Other’ occupation group is too big itself, and it might be worth breaking it into more granular categories in the future surveys. As we see, ‘Other’ group takes a significant fraction of each cloud service platform users (although it is never seen in the top three list ffor any off the platforms).</p><p>In terms of the user occupation and programming experience, all of the top three Cloud Service Providers share the same trends below</p><ul><li>Data Scientists with 3–5 and 5–10 years of programming experience are the top user groups for AWS within the survey respondents</li><li>In Software Engineer, ML Engineer, and Data Analyst groups, professionals with 3–5 and 5–10 years of experience predominate</li><li>In Research Scientist, Data Engineer, DBA, Statistician and Other groups, professionals with 10+ years of experience are the biggest fraction of the users</li><li>In Product/Project Management group, professionals with 5+ years of experience are the biggest fraction of the users</li><li>In Business Analyst group, we see the users with 1–2 years of experience to dominate</li></ul><p>In terms of the organizational envrionments, the most Data Scince and ML professionals using Cloud Services can be found in</p><ul><li>Organizations with 0–49 employees, having 1–2 workers dedicated to Data Science workloads</li><li>Organizations with 10000+ employees, having 20+ workers dedicated to Data Science workloads</li></ul><p>So we can conclude that AWS, GCP, and MS Azure tightly compete on the same types of the organizations/users.</p><h3>Usage of Cloud Computing Products</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*beKFhIY9t_-KFpPsDSJtPg.png" /></figure><p>We find that</p><ul><li>in the segment of cloud computing engines, Amazon EC2 is more popular than its rivals from Google (Google Cloud Computing Engine) and MS Azure (Azure Cloud Services)</li><li>in the segment of cloud functions, AWS Lambda is more popular than its rivals from Google (Google Cloud Functions) and MS Azure (Azure Functions)</li><li>in the segment of cloud container runners, Amazon Elastic Container Service is more popular that its rivals from Google (Google Cloud Run) and MS Azure (MS Azure Container Instances)</li><li>Google holds the second place in cloud computing engine and cloud function segments, and it is on the third place in the cloud container runner segment</li><li>there is a huge pool of responses with ‘None’, and it is most likely to indicate the entire market of cloud computing applications is not saturated yet</li></ul><p>In terms of user roles, the most users of every cloud computing product above hold the roles below (top to bottom)</p><ul><li>Data Scientists</li><li>Software Engineers</li><li>ML Engineers</li><li>Data Analysts</li></ul><h3>Usage of Cloud Computing Products by Programming Experience</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iTls9IHmwyuQvCD3jEpg2Q.png" /></figure><p>In addition to the insights above, we see that the top number of cloud computing product users fall into the following clusters in terms of their programming experience</p><ul><li>5–10 years of experience</li><li>3–5 years of experience</li><li>10–20 years of experience</li></ul><p>Juniors and super-seniors (20+ years of programming experience) seem to be less covered by the respective knowledge/skills.</p><h3>Usage of Cloud ML Products</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QHSfLkJH5Ui1dGqqf4NLdA.png" /></figure><p>We find that</p><ul><li>Google Cloud AI Platform / Google Cloud ML Engine leads the ML cloud products usage ‘nomination</li><li>the second and third best are Amazon SageMaker and Azure Machine Learning Studio, respectively</li><li>Data Scientists are the top users of cloud ML products (for every product investigated)</li><li>There is a huge chunk of responders who indicated they do not use cloud ML products at all — it indicates the market is under-saturated, and there is a good growth potential, subject to resolving the marketing and end-user barries on the way</li></ul><h3>Usage of Cloud ML Products by Programming Experience</h3><p>In addition to the insights above, we can see that the cloud ML products are mostly used by the responders who has programming experience of</p><ul><li>3–5 years</li><li>5–10 years</li></ul><h3>Usage of Cloud ML Products by Organization Size and DS Capacity</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iNqx0rWjie_z2l1IQcGp2g.png" /></figure><p>We find that the majority of organizations in every size category does not use any Cloud ML Products at the moment.</p><p>For the tiny fraction of those who use them, there are interesting insights as follows</p><ul><li>in small organizations (0–49 employees), Google Cloud AI Platform / Google Cloud ML Engine dominates</li><li>in the middle-sized organizations (50–249 employees), Google Cloud AI Platform / Google Cloud ML Engine and Amazon SageMaker titly</li><li>for companies of bigger size (250+ employees), the size of Data Science team is often correlated with the preferred Cloud ML Product (smaller teams sticks to Google Cloud AI Platform / Google Cloud ML Engine more, and Data Science teams with 20+ headcount are more inclined to use Amazon SageMaker )</li></ul><h3>Usage of Big Data Products By Occupation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VCp2s1_-7WOUhzBW8naTfQ.png" /></figure><p>We find that</p><ul><li>Overall top 3 list is constituted by three relational DBMS platforms (MySQL, PostgreSQL, MS SQL Server)</li><li>MongoDB, a non-relational database platform, takes position 4 in the list</li><li>Other relational DBMS platforms in the list (Oracle, IBM DB2, SQLite) are behind MongoDB</li><li>In the segment of truly cloud-based Big Data products, Google BigQuery overruns its Amazon and MS Azure competitors ( Amazon Redshift, Amazon Athena, Amazon DynamoDB, and Microsoft Azure Data Lake Storage)</li><li>Google Cloud SQL instances are still less popular then ‘native’ relational database instances for MySQL and PostgreSQL</li><li>MS Access is still in use in the industry</li><li>Data Scientists are the top users of each product in this list</li></ul><h3>Big Data Product Usage Patterns by User Occupation and Programming Experience</h3><p>We find that</p><ul><li>MySQL and PostreSQL are the most popular database management platforms within each occupation</li><li>MongoDB is quite popular with Software Engineers (although less popular then MySQL and PostreSQL)</li></ul><h3>Big Data Product Usage Patterns by Organization Size and Data Science Capacity</h3><p>We find that</p><ul><li>Almost all of organizations except from the extra-large ones address their data management needs with MySQL, PostgreSQL, and MongoDB the most</li><li>Extra-large organizations (with 10000+ employees) prefer to work with MySQL, MS SQL Server, Oracle, and PostgreSQL</li></ul><h3>Usage of BI Tools</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*69L9Bx4Xk7WUfFTzZXNa7w.png" /></figure><p>We find that</p><ul><li>Tableue and MS Power BI outperforms other rivals significantly</li><li>Google Data Studio becomes a challenger to the leading BI products above, occupying the third place in the list</li><li>Data Scientists, Data Analysts, Research Scientists, and ML Engineers are the most frequent users of BI tools</li><li>Huge fraction of the survey responders indicated they do not use BI tools at all</li></ul><h3>AWS Professional Users Across the Globe</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mubk_xMftUVQA6k7DkYFPw.png" /></figure><p>We find that</p><ul><li>AWS is popular among survey respondents in India and USA the most</li><li>Brasil, Japan and UK go into tier 2 in terms of the number of respondents from these country who use AWS</li></ul><h3>GCP Professional Users Across the Globe</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ATtg45sUSgiJnMyXMr039A.png" /></figure><p>We find that</p><ul><li>India is the top country where GCP is used</li><li>USA takes the second place in the rank but it is significantly below India (unlike AWS, where India and USA were relatively on a par)</li><li>Japan and Brazil are in the tier 2 in terms of the number of respondents from these country who use AWS</li><li>GCP is less popular in UK, Canada and Australia vs. AWS</li><li>GCP outperforms AWS in Turkey, Indonesia, and Russia</li></ul><h3>MS Azure Professional Users Across the Globe</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NNEIX31G-qsh6oi3zLlA3A.png" /></figure><p>We find that</p><ul><li>Top country in terms of the number of MS Azure users is India (although MS Azure is well behind AWS and GCP there)</li><li>USA holds the second place in the rank, and the number of MS Azure users is on a par with the number of GCP users in the US</li><li>Brazil belongs to tier 2 in term of the number of MS Azure users</li><li>In the majority of the countries (except the US), the number of MS Azure users is smaller then the number of GCP and AWS users</li></ul><h3>Summary</h3><p>In this post, we reviewed the state of art with using Cloud computing platforms, products, and tools by the professionals in Data Science and ML industry. These are not just their preferences as of the end of 2020. These are cornerstones which are most likely to determine the trends for 2021–2022 as well.</p><p>The next couple of years will be crucial in the battle of Cloud Computing giants for minds, arms, and budgets in the Data Science and ML industry. Although AWS’s position still looks stronger than other top rivals, the challenges from GCP could be the intrigues part of the market reshaping in the years to come. At the same time, MS Azure seems to keep its strong positions in North America (while having little chances to penetrate other continents significantly vs. AWS and GCP).</p><p>However, we entered the age of global turbulence. 2021, the year under the Star of Kings, may expose us to unexpected surprises in every aspects of our lives.</p><p><em>Notes</em>:</p><ul><li>you can check the repo per <a href="https://github.com/gvyshnya/state-of-data-science-and-ml-2020">https://github.com/gvyshnya/state-of-data-science-and-ml-2020</a> to see how every insight above has been discovered</li><li>the post got KDnuggets Gold Blog award as the most shared post in the week of <a href="https://www.kdnuggets.com/2021/01/top-news-week-0118-0124.html">Jan 18–24, 2021</a></li></ul><figure><img alt="KDnuggets Gold Blog, week of Jan 18–24, 2021" src="https://cdn-images-1.medium.com/max/96/1*V5R9IHs2DWoHgcXmioUaeg.png" /><figcaption>Awarded KDnuggets Gold Blog Badge for the week of <a href="https://www.kdnuggets.com/2021/01/cloud-computing-data-science-ml-trends-2020-2022-battle-giants.html">Jan 18–24, 2021</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2a174d3cd2b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sbc-group-blog/cloud-computing-and-data-science-and-ml-trends-in-2020-2022-the-battle-of-giants-c2a174d3cd2b">Cloud Computing and Data Science and ML Trends in 2020–2022: The battle of giants</a> was originally published in <a href="https://medium.com/sbc-group-blog">SBC Group Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PROs and CONs of Rapid EDA Tools]]></title>
            <link>https://medium.com/sbc-group-blog/pros-and-cons-of-rapid-eda-tools-e1ccd159ab07?source=rss-11881363dca------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1ccd159ab07</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[eda]]></category>
            <dc:creator><![CDATA[George Vyshnya]]></dc:creator>
            <pubDate>Sun, 18 Oct 2020 21:05:54 GMT</pubDate>
            <atom:updated>2020-10-19T09:30:01.180Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5fZHvUmC8tYl1lzA4nXDVg.jpeg" /><figcaption>Image by Aaron Olson (<a href="https://pixabay.com/users/aaronjolson-4628445">https://pixabay.com/users/aaronjolson-4628445</a>)</figcaption></figure><blockquote>It’s better to solve the right problem approximately than to solve the wrong problem exactly.</blockquote><blockquote>John Tukey</blockquote><h3>Introduction</h3><p>Since the time <a href="https://en.wikipedia.org/wiki/John_Tukey">John Tukey</a> coined the term of <strong><em>EDA </em></strong>in his famous book, “Exploratory Data Analysis” (1977), the discipline of EDA has become the mandatory practice in industrial Data Science/ML.</p><p>Started to build on top of John Tukey’s passion, it eventually turned into one of the most resource-consuming phases of any modern ML/Data Analysis project. According to the industry statistics, EDA typically takes around 30% effort of such projects nowadays.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*BUZ4YzU0l3FQvDifQZxkhg.jpeg" /></figure><p>In the business environments (even more then in scientific ones), the time pressure is one of the factors to drive real-world ML and Data Science projects. You must deliver the results not just of a good-enough (or, even better, of excellent) quality but also within a reasonably limited time frame.</p><p>Therefore, compressing the EDA time without draining its quality could be one of the beneficial ways to improve the overall ML project performance. That is where the promise of automated rapid EDA tools starts to sound attractive.</p><p>Typically, EDA can be decomposed into two principal logical parts.</p><ul><li>finding the project-specific insights (this will always require substantial domain expertise to delve into as well as project-specific engineering to visualize data as well as do other EDA activities)</li><li>doing the routine checks of the data features (statistical checks, missing values, distributions, feature pair interactions etc.), using the extreme power of <a href="https://en.wikipedia.org/wiki/Exploratory_data_analysis">multiple EDA techniques and practices</a></li></ul><p>The second area is where we can really expect to achieve significant time savings by leveraging appropriate EDA automation.</p><p>In this case study, we are going to demonstrate the benefits and pitfalls of automating routines of EDA for Drug Mechanisms of Action data with the specialized Python packages for rapid EDA below</p><ul><li><a href="https://github.com/fbdesignpro/sweetviz">SweetViz</a></li><li><a href="https://github.com/AutoViML/AutoViz">AutoViz</a></li></ul><p><strong>Note</strong>: the full source code of the case study is provided in the <a href="https://github.com/gvyshnya/MoA/blob/main/Automating%20EDA%20and%20Feature%20Importance%20Detection%20for%20MoA%20Data.ipynb">GitHub repo</a></p><h3>Problem Statement and Dataset Details</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*F2ba1dcRRe7B9NT-IuORCA.jpeg" /><figcaption>Image credit: <a href="https://www.labiotech.eu/cancer/forx-therapeutics-cancer-treatment/">https://www.labiotech.eu/cancer/forx-therapeutics-cancer-treatment/</a></figcaption></figure><p>It has been decided to check how useful (or equally less then useful) SweetViz and AutoViz could be in a real-world problem with quite an extensive dataset to analyze. Kaggle competition of ‘<a href="https://www.kaggle.com/c/lish-moa">Mechanisms of Action (MoA) Prediction</a>’ has been selected as a problem to tackle.</p><p>The essence of MoA Prediction competition is to build a multi-label classification model to predict the MoA of drugs based on observed gene expression and cell viability effects (observation features).</p><p><strong>Note</strong>: if you like to dive into the details of the data-driven insights for this problem, you can navigate to some of the publications referred to in References section at the end of this post.</p><h3>Sweetness of SweetViz</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*L0OpQPtSBFNRFDqmGm-23Q.png" /></figure><p>In this case study, SweetViz was quite helpful in two operational modes</p><ul><li>Generating the detailed EDA report for a Pandas dataframe (in HTML format) when tried to analyze the feature-to-target label interactions</li><li>Generating the detailed EDA report to compare two dataframes with the similar structure (in HTML format)</li></ul><p>Generating a detailed EDA report for a dataframe is easy. You can see it in action below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9baa3088be550449648bedde1927f933/href">https://medium.com/media/9baa3088be550449648bedde1927f933/href</a></iframe><p>Comparing two datasets with sweetviz is equally simple, per the code fragment below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ba8709ec53032cc0ae25e128c731e16b/href">https://medium.com/media/ba8709ec53032cc0ae25e128c731e16b/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gm4oC7Oy3l8A5bdyOicvaQ.png" /></figure><p>As a result, SweetViz helped to quickly draw insights below</p><ul><li>Detect the difference in key statistical parameters of the same variables in the treated and control subsamples of training and testing datasets</li><li>Compare the skewness of numeric variables in the treated and control subsamples of training and testing datasets</li><li>Pick up important feature -to-class label interactions for the topmost frequent labels in the training dataset</li></ul><h3>Power of AutoViz</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*rSqWDNghtOvGwy4FHVuPbA.png" /></figure><p>In this case study, AutoViz was helpful to quickly discover the essential details as follows</p><ul><li>Detecting the most significant features for the ML problem</li><li>Proving the significance of the features with the comprehensive visualizations</li></ul><p>It is also accomplished with a few lines of code, as presented below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0273d7f5275c8551443c12e6f5654027/href">https://medium.com/media/0273d7f5275c8551443c12e6f5654027/href</a></iframe><p>In terms of feature importance, AutoViz performs it as follows</p><ul><li>Rejects highly correlative features</li><li>From the subset of features remained, rejects the features detected as non-significant by the internally trained xgboost model</li><li>As a result, it outputs the dataframe with only a subset of significant features it detected</li></ul><p>Such discoveries are helpful to set the stage for promising ML modelling experiments down the road.</p><p>It is worth mentioning AutoViz displayed the exemplar performance and execution speed. The AutoViz-based part of the analysis presented in <a href="https://github.com/gvyshnya/MoA/blob/main/Automating%20EDA%20and%20Feature%20Importance%20Detection%20for%20MoA%20Data.ipynb">the source code repo</a> has taken less then <em>10 </em>min whereas the operations with SweetViz took about <em>90 </em>min to run on the local machine with the following hardware and system software specification</p><ul><li>2 CPU Intel Core i7–8750H 2.2 GHz</li><li>16 GB RAM</li><li>OS Windows 10</li><li>Anaconda with Python 7</li></ul><h3>Problem-specific Visualization Instruments</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*8j3EQ1fHNkZWkxvC7HW5lg.png" /></figure><p>Despite quite a lot of insights obtained via rapid EDA tools without much time spent on the effort, not everything can be captured by them. They are not magical wands, after all. There is always something you will have to do manually, for every project specifically.</p><p>In this case, the manual effort has been put to investigate</p><ul><li>Label-to-label interactions</li><li>Class labels — to — feature interactions</li></ul><p>Below is the code sample to demonstrate the way to build a custom multi-facet plots to visualize multiple features from the training dataset interacting with a particular class label</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9a5e0ced7d2a01e08fe34284a784432b/href">https://medium.com/media/9a5e0ced7d2a01e08fe34284a784432b/href</a></iframe><p>Also, the number of domain-specific insights for the ML problem could only be captured by a human expert looking at the data AND applying the subject matter expertise (see more details on that discussed in my Kaggle thread ‘<a href="https://www.kaggle.com/c/lish-moa/discussion/190647">MoA: EDA and Feature Importance Findings</a>’).</p><h3>Additional Word on Feature Importance</h3><p>AutoViz did a great job detecting the important features for this project. However, it is always recommended to measure feature importance by multiple, preferably, analytical, methods.</p><p>That is why I also did the experiment with permutation feature importance. You can see it end-to-end in <a href="https://github.com/gvyshnya/MoA/blob/main/permutative_feature_importance.py">the source code repo</a> .</p><p>As it happens in most of the real experiments, the lists of significant features detected by AV and via permutation importance method are not identic (although overlapping on a good number of features). This will open the avenue to comparative ML study to see which set of important features leads to a better ML model (in terms of its accuracy).</p><h3>Age of Open Source Software</h3><p>Both SweetViz and AutoViz are open-source libraries, and you can start using it on your own projects for free.</p><p>All of us know the benefits of the open source software in general (that’s why it was in use in 78% of businesses as of 2015, per <a href="https://www.zdnet.com/article/its-an-open-source-world-78-percent-of-companies-run-open-source-software/">ZDNet’s article</a>, with trend going up as of this moment).</p><p>However, the adoption of a certain open software application or library can see the structural barriers. One of the most serious barriers of this sort is the lack of maintainability of the codebase.</p><p>In this sense, I was really amazed at how AutoViz maintainer team fulfills its mission. As I worked on this case study, I faced a <a href="https://github.com/AutoViML/AutoViz/issues/26">severe issue</a>. That issue has been picked up by the maintainer of AutoViz quickly, and the solid fix has been delivered in the expedited way (it signified the incremental update of AutoViz from version <em>0.0.68</em> to <em>0.0.70</em>).</p><p>The team kept further testing AutoViz with the data for MoA project to see additional improvements to add (and that’s how AutoViz <em>0.0.71</em> was released).</p><p>As a result, it enabled my work on the case study in a reliable manner.</p><h3>Summary and Conclusion</h3><p>John Tukey liked to emphasize,</p><blockquote>“If a thing is not worth doing, it is not worth doing well”.</blockquote><p>EDA is the thing to do indeed, and it is the thing to do in a right fashion, well.</p><p>However, if you can automate certain aspects of EDA without dropping the quality bar on your data research / insights obtained, it is the right step to do. It will add the edge to efficiency of your industrial ML and data analytics projects.</p><p>This is the reason why successful rapid EDA tools (like <strong><em>AutoViz </em></strong>or <strong><em>SweetViz</em></strong>) would find its niche in the Data Science and ML industry in the long run.</p><h3>References</h3><p>1. Tips for Automating EDA using Pandas Profiling, Sweetviz and Autoviz in Python — <a href="https://analyticsindiamag.com/tips-for-automating-eda-using-pandas-profiling-sweetviz-and-autoviz-in-python/">https://analyticsindiamag.com/tips-for-automating-eda-using-pandas-profiling-sweetviz-and-autoviz-in-python/</a></p><p>2. Better, Faster, Stronger Python Exploratory Data Analysis (EDA) — <a href="https://towardsdatascience.com/better-faster-stronger-python-exploratory-data-analysis-eda-e2a733890a64">https://towardsdatascience.com/better-faster-stronger-python-exploratory-data-analysis-eda-e2a733890a64</a></p><p>3. Powerful EDA (Exploratory Data Analysis) in just two lines of code using Sweetviz — <a href="https://towardsdatascience.com/powerful-eda-exploratory-data-analysis-in-just-two-lines-of-code-using-sweetviz-6c943d32f34">https://towardsdatascience.com/powerful-eda-exploratory-data-analysis-in-just-two-lines-of-code-using-sweetviz-6c943d32f34</a></p><p>4. Sweetviz: Automated EDA in Python — <a href="https://towardsdatascience.com/sweetviz-automated-eda-in-python-a97e4cabacde">https://towardsdatascience.com/sweetviz-automated-eda-in-python-a97e4cabacde</a></p><p>5. AutoViz: A New Tool for Automated Visualization — <a href="https://towardsdatascience.com/autoviz-a-new-tool-for-automated-visualization-ec9c1744a6ad">https://towardsdatascience.com/autoviz-a-new-tool-for-automated-visualization-ec9c1744a6ad</a></p><p>6. Webinar: AutoViz and AutoViML: Automated Visualization and Machine Learning — <a href="https://www.youtube.com/watch?v=QZhq8g9W-pQ">https://www.youtube.com/watch?v=QZhq8g9W-pQ</a></p><p>7. You Are Plotting the Wrong Things — <a href="https://towardsdatascience.com/youre-plotting-the-wrong-things-3914402a3653">https://towardsdatascience.com/youre-plotting-the-wrong-things-3914402a3653</a></p><p>8. Insights to Mechanisms of Action Prediction Competition -</p><p>9. Drugs classification: Mechanisms of Action — <a href="https://www.kaggle.com/amiiiney/drugs-classification-mechanisms-of-action">https://www.kaggle.com/amiiiney/drugs-classification-mechanisms-of-action</a></p><p>10. Mechanisms of Action: EDA and Feature Importance Findings — <a href="https://www.kaggle.com/c/lish-moa/discussion/190647">https://www.kaggle.com/c/lish-moa/discussion/190647</a></p><p>11. Feature Importance in the Age of Explainable AI/ML — <a href="https://medium.com/sbc-group-blog/feature-importance-in-the-age-of-explainable-ai-ml-242327a70017">https://medium.com/sbc-group-blog/feature-importance-in-the-age-of-explainable-ai-ml-242327a70017</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1ccd159ab07" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sbc-group-blog/pros-and-cons-of-rapid-eda-tools-e1ccd159ab07">PROs and CONs of Rapid EDA Tools</a> was originally published in <a href="https://medium.com/sbc-group-blog">SBC Group Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pythonic Audio Feature Extraction in the Age of Parallel and Distributed Computing]]></title>
            <link>https://medium.com/@gvyshnya/pythonic-audio-feature-extraction-in-the-age-of-parallel-and-distributed-computing-793b27641b6d?source=rss-11881363dca------2</link>
            <guid isPermaLink="false">https://medium.com/p/793b27641b6d</guid>
            <dc:creator><![CDATA[George Vyshnya]]></dc:creator>
            <pubDate>Wed, 02 Sep 2020 20:01:46 GMT</pubDate>
            <atom:updated>2020-09-02T20:01:46.700Z</atom:updated>
            <content:encoded><![CDATA[<p><em>“Redesigning your application to run multithreaded on a multicore machine is a little like learning to swim by jumping into the deep end.” <br>Herb Sutter, chair of the ISO C++ standards committee, Microsoft.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*pYC2rjrn2oUwVCZqgWLRMA.jpeg" /></figure><h3>Preface: Curse of Speed Limitations</h3><blockquote>Python has been the standard de facto for the majority of industrial ML/AI solutions. Its extreme programmer-friendliness, along with the convenience and wide range of libraries for data analytics, data processing, data engineering, system integration and ML made it the language of choice of many experts, vendors and corporate Data Science teams.</blockquote><blockquote>At the same time, it isn’t the fastest programming language around (although doing much better than R, its eternal AI/ML rival). Some of Python’s speed limitations are due to its default implementation, cPython, being single-threaded. That is, cPython doesn’t use more than one hardware thread at a time.</blockquote><p>While you can use <em>threading</em> module built into Python to speed things up, threading only gives you concurrency, not parallelism. It’s good for running multiple tasks that aren’t CPU-dependent but does nothing to speed up multiple tasks that each require a full CPU.</p><blockquote>Python does include a native way to run a Python workload across multiple CPUs. The <em>multiprocessing</em> module spins up multiple copies of the Python interpreter, each on a separate core, and provides primitives for splitting tasks across cores. For years and years, multiprocessing outperformed a bunch of third-party Python libraries invented to facilitate the same. Python’s core maintenance team was good at keeping the focus on the operability and usability of this part of the native built-in framework.</blockquote><blockquote>However, the advent of distributed multi-node computation technologies had an unexpected impact on Python-based multiprocessing technologies. A number of new frameworks arrived that started to be on a par with the standard multiprocessing module, even if run in a single-node configuration.</blockquote><blockquote>Among such newcomers, you can point our two outstanding leaders, namely</blockquote><blockquote><a href="https://ray.io/">- Ray</a></blockquote><blockquote><a href="https://dask.org/">- Dask</a></blockquote><p>In the sections below, we will compare the multiprocessing capabilities and performance/speed of three implementation of audio feature extractions from the bird call tracks.</p><p><strong>Note</strong>: the full source code of the audio feature extraction utilities is available in a <a href="https://github.com/gvyshnya/BirdCallClassification">Github repo</a>.</p><h3>Bird Calls: Audio Feature Extraction Flow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*Rm_V_PhbXsrep9jcSntESw.jpeg" /></figure><p>As a part of the effort to classify bird calls (songs) in <a href="https://www.kaggle.com/c/birdsong-recognition"><em>Cornell Birdcall Identification</em> competition</a>, there is a need to extract audio features from the digital audio records of the bird songs.</p><blockquote>In this project, librosa library is used for audio feature extraction. Since librosa-based audio feature computations are CPU-intensive, parallelizing such computations is the key architectural pattern to build a high-performance data transformation toolset to process the audio files.</blockquote><p>From the conceptual standpoint, the audio feature extraction flow was implemented as follows</p><ol><li>The training set was split by the species</li><li>Every per-species batch of audio files with the bird songs have been processed with librosa as follows</li></ol><p>a. Features extracted/calculated per the list below</p><p>b. Every feature that librosa extracts as np.array is further processed to calculate mean across the numeric values in np.array (actually it is everything except BPM variable)</p><p>c. The final results saved as a pandas dataframe</p><p>d. The dataframes serialized as CSV files for future use by ML pipeline tools down the road</p><p>Activities per step 2 have been parallelized using three technologies compared in this research. When working with Dask and Ray, the output in a Pandas dataframe was consciously preferred over the native Dask and Ray counterparts of Pandsas to maximize the reuse of audio feature extraction code across all of the solutions.</p><p>Data files for just three species (<em>‘American Avocet’, ‘American Bittern’, and ‘American Crow’</em>) have been used in order to run the calculation within the reasonable time frame (as well as not to hit the Kaggle timeout issues when running the experiment on Kaggle).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*_LPNKMkjNW7JLI6xYe3y0A.jpeg" /></figure><p>The list of audio features extracted with librosa is below</p><ol><li>Spectrogram (decibel-scaled)</li><li>Mel Spectrograms (decibel-scaled)</li><li>Zero-crossing rate</li><li>Harmonics (harmonics are characteristics that represent the sound color)</li><li>Perceptual shock wave (it represents the sound rhythm and emotion of a soundtrack)</li><li>Spectral Centroid (it indicates where the ”center of mass” for a sound is located and is calculated as the weighted mean of the frequencies present in the sound)</li><li>Chroma Frequencies (Chroma features). These are an interesting and powerful representation for music audio in which the entire spectrum is projected onto 12 bins representing the 12 distinct semitones (or chromas) of the musical octave.</li><li>Tempo BPM (beats per minute). Dynamic programming beat tracker.</li><li>Spectral Rolloff. This is a measure of the shape of the signal. It represents the frequency below which a specified percentage of the total spectral energy (e.g. 85 %) lies.</li><li>Spectral flux.</li><li>Spectral Bandwidth. The spectral bandwidth is defined as the width of the band of light at one-half the peak maximum (or full width at half maximum [FWHM]) and is represented by the two verticalred lines and λSB on the wavelength axis (in fact, we measure three spectral bandwidth features in this experiment)</li><li>MFCC features (20 MFCC coefficients, 20 MFCC delta coefficients, and 20 MFCC accelerate coefficients)</li></ol><p>Total of 87 audio features has been extracted.</p><p>The principal feature extraction flow is demonstrated per the code fragment below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f4af4e430c453c765b1fa002e967baf0/href">https://medium.com/media/f4af4e430c453c765b1fa002e967baf0/href</a></iframe><p><strong>Note</strong>: the audio feature extraction flow has been designed to calculate tabular features for each audio file provided in the training set. Such an approach will be helpful to try both DL and classical ML algorithms during the feature importance analysis and ML predictions. However, you can find alternative approaches to audio feature extraction where the entire np.arrays are saved as files for futher processing (you can see examples of such a flow in <a href="https://github.com/Cocoxili/DCASE2018Task2/blob/master/data_transform.py">https://github.com/Cocoxili/DCASE2018Task2/blob/master/data_transform.py</a> etc., for instance)</p><h3>Doing it Classical Way: Python MultiProcessing</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*IH7MlfVg11haqQ83X1lR8w.jpeg" /></figure><p>When using multiprocessing lib in Python, it goes its typical MP way, as presented below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/568167b84e59c5021e6c8be3368c5494/href">https://medium.com/media/568167b84e59c5021e6c8be3368c5494/href</a></iframe><h3>Doing it Madrigal Way: Ray</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*qRbKhdA-FqtGV6pZ35FXNQ.png" /></figure><p>Whenever good scientists have enough courage and motivation to put themselves at the risky frontier in the business realms, it could result in a good product to launch. Big Data/ML area knows several good examples of this sort. For example, when research team behind something that later got called Spark entered the market, it immediately opened a new page in the map-reduce technology “cookbook”.</p><p>Ray (ray.io) is also a child of prominent scientists who did not fail in business. Developed by a team of researchers at the University of California, Berkeley, Ray underpins a bunch of existing distributed machine learning libraries now. Such a scientific origin brings the madrigal fleur — whether you talk about sun rays or X-rays of a mystical physical science research, it brings you to the wave of the high-style music genre.</p><p>At the same time, Ray provides its core framework to build distributed parallelized applications of any sort (not just machine learning tasks alone).</p><p>Ray’s syntax is minimalistic, so you don’t have to refactor existing apps extensively to parallelize them.</p><p>The <em>@ray.remote</em> decorator distributes any function decorated this way across any available nodes in a Ray cluster.</p><p>You can optionally specify parameters for how many CPUs or GPUs to use. The results of each distributed function are returned as Python objects, so they’re easy to manage and store, and the amount of copying across or within nodes is kept to a minimum. This last feature comes in handy when dealing with NumPy arrays, for instance.</p><p>On top of the core multiprocessor and multi-computer application development foundation libraries, Ray offers a number of prominent features and frameworks</p><ul><li>Tune: Scalable Hyperparameter Tuning</li><li>RLlib: Scalable Reinforcement Learning</li><li>RaySGD: Distributed Training Wrappers</li></ul><p>Ray even includes its own built-in cluster manager (so called Ray Serve) , which can automatically spin up nodes as needed on local hardware or popular cloud computing platforms. It also provides the native support to distributed PyTorch, Tensorflow, and Keras, making its offensive into the areas of distributed Deep Learning (modern neural network-based AI applications).</p><p>Ray has the active user community around the trunk products. The community was active enough to contribute the additional features like</p><ul><li>Pandas on Ray</li><li>Distributed Scikit-learn / Joblib</li><li>Distributed multiprocessing.Pool</li></ul><p>Btw, the advent of <em>Distributed Scikit-learn / Joblib</em> is inspired by the competition with Dask (one more participant of this review — see below) as it helped to mature Ray Actor model (which was inspired by the ideas behind Dask Actor model).</p><p>Coming back to the test drive with the bird call recognition data, the essence of Ray-based parallelized audio feature extraction flow has been implemented per the code fragment below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c1d85aa7bf02de9b4095356fdf74a56/href">https://medium.com/media/9c1d85aa7bf02de9b4095356fdf74a56/href</a></iframe><p><strong>Note</strong>: Ray has not been ported to Windows for ages. As of this moment, Ray has been enabled to run under Windows subsystem for Linux as well as under Core Windows in alpha mode (see the latest comments in <a href="https://github.com/ray-project/ray/issues/631">https://github.com/ray-project/ray/issues/631</a>). So running the above-mentioned code under Windows 10 and Anaconda/Python 3.7 resulted in the error fired at import ray statement per the note below</p><blockquote><em>OSError: [WinError 126] The specified module could not be found</em></blockquote><h3>Doing it Swinger Way: Dask</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*QhPgKNfcxUvRhFUjyi4nhg.jpeg" /></figure><p>From the outside, Dask looks a lot like Ray. It, too, is a library for distributed parallel computing in Python, with its own task scheduling system, awareness of Python data frameworks like NumPy, and the ability to scale from one machine to many.</p><p>If looking for a musical metaphor for Dask, it resembles swing. Both in its style and its attitude. You do not have to study a lot to write your first Dask-based mulitiprocessing or distributed application — much like in composing, playing or dancing swing. As we remember, only a few of the pioneers and world starts of swing were ever formally trained in music.</p><p>Dask is composed of two parts</p><ul><li>Dynamic task scheduling optimized for computation. This is similar to Airflow, Luigi, Celery, or Make, but optimized for interactive computational workloads.</li><li>“Big Data” collections like parallel arrays, dataframes, and lists that extend common interfaces like NumPy, Pandas, or Python iterators to larger-than-memory or distributed environments. These parallel collections run on top of dynamic task schedulers.</li></ul><blockquote>Dask works in two basic ways. The first is by way of parallelized data structures — essentially, Dask’s own versions of NumPy arrays, lists, or Pandas DataFrames. Whenever you swap in the Dask versions of those constructions for their defaults, and Dask will automatically spread their execution across your cluster.</blockquote><p>Typically, it barely involves little more than changing the name of an import. However, in some case you may find yourself rewriting your code completely to make it work.</p><p>The second way is through Dask’s low-level parallelization mechanisms, including function decorators, that parcel out jobs across nodes and return results synchronously (“immediate” mode) or asynchronously (“lazy”). Both modes can be mixed as needed, too.</p><p>One key difference between Dask and Ray is the scheduling mechanism. Dask uses a centralized scheduler that handles all tasks for a cluster. Ray is, in turn, totally decentralized. It means each machine runs its own scheduler, so any issues with a scheduled task are handled at the level of the individual machine, not the whole cluster.</p><blockquote>When applying the capabilities of Dask to the test drive with the bird call recognition, we will use the second approach. The essence of the implementation was to use Dask’s low-level <em>delayed</em> interface.</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/412afa3a4a7ff06ecc64e564b59caead/href">https://medium.com/media/412afa3a4a7ff06ecc64e564b59caead/href</a></iframe><p><strong>Note</strong>: the above-mentioned code works well both under core Windows 10 and Windows Subsystem for Linux.</p><h3>Summary and Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*bWGGgD0ZBQkg8iFTOyU1Dg.jpeg" /></figure><p>When running the experiment on a local notebook (Intel Core i7–8750H, 2 CPU @ 2,20 Gh, 8 virtual CPUs over hyper-v, 16 GB RAM, Windows 10, Python 3.7 / Anaconda), the comparative performance has been measured as follows</p><ul><li>Classical multiprocessing with parallelizing across 6 CPUs — 6 min 17 s</li><li>Parallelized processing with Dask — 7 min 21 s</li></ul><p>It was not possible to run a Ray-based feature extraction workflow on a Windows platform due to the limitations explained in <a href="https://github.com/ray-project/ray/issues/631">https://github.com/ray-project/ray/issues/631</a></p><p>When running the experiment on Kaggle kernels, the indicative times of the parallelized audio features extraction were displayed as follows</p><ul><li>Classical multiprocessing with parallelizing across 6 CPUs (<a href="https://www.kaggle.com/gvyshnya/parallel-audio-feature-extraction-with-mp/">https://www.kaggle.com/gvyshnya/parallel-audio-feature-extraction-with-mp/</a> ) — 11 min 14 s</li><li>Parallelized processing with Dask (<a href="https://www.kaggle.com/gvyshnya/parallel-audio-feature-extraction-with-dask/">https://www.kaggle.com/gvyshnya/parallel-audio-feature-extraction-with-dask/</a> ) — 12 min 55 s</li><li>Parallelized processing with Ray (<a href="https://www.kaggle.com/gvyshnya/parallel-audio-extraction-with-ray/">https://www.kaggle.com/gvyshnya/parallel-audio-extraction-with-ray/</a> ) — 30 min 18 s</li></ul><p>So the classical <em>multiprocessing</em> performed a little better then <em>Dask</em> on both the local notebook hardware and Kaggle Kernels.</p><p>Ray did not perform well on Kaggle Kernels (probably, due to the limitations in its setup in the docker image for Kaggle Python notebooks), despite the promise it shared per other computational experiments (see <a href="https://towardsdatascience.com/10x-faster-parallel-python-without-python-multiprocessing-e5017c93cce1">https://towardsdatascience.com/10x-faster-parallel-python-without-python-multiprocessing-e5017c93cce1</a> ).</p><p><strong>Note:</strong> Dask in its default settings has spinned 4 parallel processes when running in Kaggle. When running on the local computer, it was able to run 8 processes in parallel.</p><h3>Other Python-based Modern Parallel Computing Frameworks</h3><p>As of summer 2020, there is a good number of other excellent Python-based distributed computing and parallel computing frameworks that have not been covered by the comparative study described here. These are</p><ul><li><em>Dispy </em>— a generic-purpose framework for parallel computing in Python</li><li><em>Pandaral·lel</em> — a specific Python framework to enable distributed and parallelized computations over Pandas dataframes</li><li><em>Ipyparallel </em>— a framework to run Jupyter notebooks over distributed clusters</li></ul><p>If you have a chance, I do recommend to play around them.</p><h3>References</h3><p>“Redesigning your application to run multithreaded on a multicore machine is a little like learning to swim by jumping into the deep end.” Herb Sutter, chair of the ISO C++ standards committee, at Microsoft, said once. We have been finding ourselves to prove this point whenever we design parallel algorithms.</p><p>The links below will help you navigate within the latest trends in Pythonic parallel computing</p><ul><li>Parallel Processing in Python — A Practical Guide with Examples — <a href="https://www.machinelearningplus.com/python/parallel-processing-python/">https://www.machinelearningplus.com/python/parallel-processing-python/</a></li><li>Parallel Processing in Python Using DASK — <a href="https://medium.com/swlh/parallel-processing-in-python-using-dask-a9a01739902a">https://medium.com/swlh/parallel-processing-in-python-using-dask-a9a01739902a</a></li><li>How to Run Parallel Data Analysis in Python using Dask Dataframes — <a href="https://towardsdatascience.com/trying-out-dask-dataframes-in-python-for-fast-data-analysis-in-parallel-aa960c18a915">https://towardsdatascience.com/trying-out-dask-dataframes-in-python-for-fast-data-analysis-in-parallel-aa960c18a915</a></li><li>10x Faster Parallel Python Without Python Multiprocessing — <a href="https://towardsdatascience.com/10x-faster-parallel-python-without-python-multiprocessing-e5017c93cce1">https://towardsdatascience.com/10x-faster-parallel-python-without-python-multiprocessing-e5017c93cce1</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=793b27641b6d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Feature Importance in the Age of Explainable AI/ML]]></title>
            <link>https://medium.com/sbc-group-blog/feature-importance-in-the-age-of-explainable-ai-ml-242327a70017?source=rss-11881363dca------2</link>
            <guid isPermaLink="false">https://medium.com/p/242327a70017</guid>
            <dc:creator><![CDATA[George Vyshnya]]></dc:creator>
            <pubDate>Sun, 12 Jul 2020 16:49:51 GMT</pubDate>
            <atom:updated>2020-07-12T17:26:10.217Z</atom:updated>
            <content:encoded><![CDATA[<p>“By far, the greatest danger of Artificial Intelligence is that people conclude too early that they understand it.” — Eliezer Yudkowsky</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PxiDmPuQxgRUe35UHDn5fA.jpeg" /><figcaption>Explain or Not to Explain? (A photo is courtesy by Andrea Piacquadio from Pexels)</figcaption></figure><h3>Why to Explain AI/ML?</h3><p>In the recent years, we see increasing demand in interpretable AI/ML. Human decision-makers would like to trust their AI-based decision support on the ground of rationale rather than via religious belief in what AI system calculates/suggests/forecasts.</p><p>It was quite easy in good old days when the stage was preoccupied by the easily interpretable ML algorithms like (Linear regression, polynomial regression, logistic regression, CART-based decision trees etc.).</p><p>However, such algorithms lacked accuracy in many real-world business forecasting and decision support scenarios. It resulted in the advent of highly accurate and complicated algorithms (starting from Random Forests through Gradient Boosting Machine-like algorithms up to the wide spectrum of Neural Networks of the second generation). However, the accuracy came at a price. There was no more easy way to interpret the decision-making flow of such AI/ML algorithms in a human-friendly and rational way.</p><p>One of the early attempts to address the challenge was adding the supplementary capability to calculate the feature importance scores by some of the modern algorithms (this is featured, for instance, by Random Forest, GBM, and lightgbm).</p><p>However, the feature importance scores are sometimes confusing / misleading due to the fact they are calculated separately from the ML model training itself.</p><p>Such a collision gave a birth to several analytical algorithms to calculate the feature importance / do the feature selection for ML models. As opposed to the classical statistical (filtering) approaches (where feature importance is determined on a basis of a certain statistical metric, whether it is a Pierson correlation or Chi Square), such techniques embrace a series of model training experiments under certain feature space tweaks. In such a way, they relatively score the importance of each feature for a specific model to train.</p><p>In this post, we are going to review such feature selection / feature importance detection methods. They all will be useful in the strive to build the industrial culture of interpretable AI/ML.</p><p>With this direction, we are on a par with the industry giant like Google (who recently launched the services of Explainable AI — see <a href="https://cloud.google.com/explainable-ai">https://cloud.google.com/explainable-ai</a>).</p><p><strong>Note</strong>: in addition to building comprehensive interpretable ML models, the relevant feature selection will also help you to handle other ML challenges as follows</p><ul><li>Dropping the garbage (non-informative) features from your modelling pipeline</li><li>Tackling the curse of dimensionality as well as minimizing the impact of the model overfitting</li></ul><h3>What’s In Your Toolbox?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zTYCeNawOdcpwrdC175sfw.jpeg" /><figcaption>Photo by Andrea Piacquadio from Pexels</figcaption></figure><p>In real machine-learning problems, tackling the curse of dimensionality as well as increasing the model interpretability are translated to solid feature selection techniques.</p><p>Apart from the filtering-based methods (correlation- or chi-square-based) or PCA (the latter is mostly applicable to linear regression problems), there is a set of analytical (computational) feature selection methods.</p><p>In the subsections below, we will review the available analytical feature selection options, along with the technical implementation details for them in Python.</p><h3>Wrapper-based methods</h3><p>Wrapper-based methods treat the selection of a set of features as a search problem.</p><p>RFE and its implementation in sckit-learn can be referred to as one of the good options to go on with it, for example.</p><p>The utility function to benefit from RFE feature importance score calculations is provided below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31d2bda73e436d94cc3e0e8c1e5080de/href">https://medium.com/media/31d2bda73e436d94cc3e0e8c1e5080de/href</a></iframe><p>Other options in the wrapper-based feature selection domain are Backward Elimination, Forward Selection, and Bidirectional Elimination.</p><h3>Embedded methods</h3><p>Embedded methods are applicable to ML algorithms that have built-in feature selection methods. For instance, Lasso, Random Forest, and lightgbm have them.</p><p>From the technical standpoint, feature selection with the embedded methods relies on scikit-learn’s SelectFromModel. You can see it in action with the demo code snippet below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a64ca97aaa892b949ddad3cc6cb482b2/href">https://medium.com/media/a64ca97aaa892b949ddad3cc6cb482b2/href</a></iframe><h3>Permutation method</h3><p>Put simply, this method changes the data in a column and then tests how much that affects the model accuracy. If changing the data in a column drops the accuracy, then that column is assumed to be important.</p><p>You can benefit from the out-of-the box scikit-learn utility to facilitate permutable feature importance calculation. I wrapped it up in a utility function below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a1561fd2a3f58c0ac4788910b95b41c/href">https://medium.com/media/8a1561fd2a3f58c0ac4788910b95b41c/href</a></iframe><h3>Drop-Column Method</h3><p>This method is focused on measuring performance of a ML model on a full-feature dataset of predictors vs. the set of smaller datasets (each one of them to drop exactly one feature from the original fully featured dataset). In such a way, the difference between the performance of the model on the original dataset and every dataset with a dropped feature will be a metric of the dropped feature importance for the model performance.</p><p>I could not find the stable Pythonic implementation of such a feature selection / feature importance measurement therefore I ended up with the custom implementation below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/90b3e1a366e2bc1c982870006d2668c2/href">https://medium.com/media/90b3e1a366e2bc1c982870006d2668c2/href</a></iframe><h3>Several Practical Considerations</h3><p>When deciding which feature selection method is the best one for your specific problem, you should keep in mind the points below</p><ul><li>there is no a single silver bullet-proof method of feature selection that worsks well for each and every project — typically, you will always have to undertake several feature selection experiments (using different methods) to figure out which one leads to the ML model with the highest performance metric score</li><li>all methods except filter-based ones have its computational time tall, and it may take too much time to go through them appropriately for large datasets</li><li>embedded methods sometimes introduce the confusion (or even a misinterpretation on the feature importance) as the feature importance scores are often calculated separately from the model training</li></ul><h3>Getting Your Hands Dirty: Selecting Features for Advanced House Pricing</h3><p>We will illustrate the importance of feature importance/appropriate feature selection in real-world ML projects on the show-case example of the Advance House Price Prediction problem.</p><p>You can review its code fully in the <a href="https://github.com/gvyshnya/AdvancedHousePricingProblem/blob/master/Advanced%20House%20Prediction%20-%20Feature%20Selection%20Comparison.ipynb">notebook</a>.</p><p>The major ML experiment setup/plan implemented in the notebook is as follows</p><ul><li>do the feature engineering and data preprocessing as invented <a href="https://www.kaggle.com/datafan07/beginner-eda-with-feature-eng-and-blending-models">here</a></li><li>calculate feature importance scores with different analytical methods (model-calculated feature important coefficients, embedded feature selection from the model, wrapper-based feature importance score with RFE, permutable feature importance scoring, and drop-column feature importance scoring)</li><li>train a bunch of Random Forest models (both using the full set of predictors in the training set as well as the top features selected by each of the feature importance scoring algorithms above)</li><li>concatenate the model training/testing scores in a single Pandas dataframe</li><li>compare performance of the models trained, to choose the one to indicate the least CV test score (which is in fact average RMSE on the hold-out folds in n-fold CV rounds)</li></ul><h3>Results Discussion</h3><p>We will find that</p><ul><li>almost every model we trained (except for the set of models to use the tiny subset of features selected via embedded feature selection method)</li><li>feature importance scores calculated by RF algorithm directly are really misleading, and they do not reflect the actual feature importance for the model trained</li><li>the reason why embedded feature selection led to a poor result (see the chart above) can be explained by the fact RF needs reasonable variance in the feature space (to train diverse set of poor predictors — individual decision tree model estimators — to represent the regression problem complexity in a proper way)</li><li>among the bunch of models we trained, the best performance on the CV testing sets (withe testing score of 0.127509) was demonstrated by the model that used top 50 RFE features and n_trees = 800</li><li>in this case, the appropriate feature selection not only improved the interpretability of our model but also added the edge in its forecasting performance (vs. the set of RF models that used the entire set of features in training)</li></ul><h3>Summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ETHFM36LBvs7xdCFL5zVjw.png" /></figure><p>“What is vital is to make anything about AI explainable, fair, secure and with lineage, meaning that anyone could see very simply see how any application of AI developed and why.”</p><p>Ginni Rometty, IBM’s CEO, made that statement during her keynote address at CES on January 9, 2019. The background of asserting the need for explainable AI is that keeping it as a sealed black box makes it impossible to check and fix biases or other problems in the programming.</p><p>As we can conclude from the case study above, the solid feature importance scoring with analytical methods (as opposed to statistical and embedded methods) can draw the solid intelligence on which features are really important for your ML model. If applied in a wise manner, they will lead to both the explainable ML solutions and better performance of your ML models.</p><h3>References</h3><p>You can refer to the blog posts below if you like to undertake the deeper dive into industrial feature selection/feature importance calculation techniques</p><ul><li>Feature Selection with pandas — <a href="https://towardsdatascience.com/feature-selection-with-pandas-e3690ad8504b">https://towardsdatascience.com/feature-selection-with-pandas-e3690ad8504b</a></li><li>5 feature selection methods every data scientist need to know — <a href="https://towardsdatascience.com/the-5-feature-selection-algorithms-every-data-scientist-need-to-know-3a6b566efd2">https://towardsdatascience.com/the-5-feature-selection-algorithms-every-data-scientist-need-to-know-3a6b566efd2</a></li><li>Feature Importance May Be Lying to You — <a href="https://towardsdatascience.com/feature-importance-may-be-lying-to-you-3247cafa7ee7">https://towardsdatascience.com/feature-importance-may-be-lying-to-you-3247cafa7ee7</a></li><li>Beware Default Random Forest Importance — <a href="https://explained.ai/rf-importance/">https://explained.ai/rf-importance/</a></li><li><a href="http://www.feat.engineering/">Feature Engineering and Selection: A Practical Approach for Predictive Models</a> by Max Kuhn and Kjell Johnson</li><li><a href="https://www.oreilly.com/library/view/feature-engineering-for/9781491953235/">Feature Engineering for Machine Learning</a> by Alice Zheng and Amanda Casari</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=242327a70017" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sbc-group-blog/feature-importance-in-the-age-of-explainable-ai-ml-242327a70017">Feature Importance in the Age of Explainable AI/ML</a> was originally published in <a href="https://medium.com/sbc-group-blog">SBC Group Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>