<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dr Anton Lokhmotov on Medium]]></title>
        <description><![CDATA[Stories by Dr Anton Lokhmotov on Medium]]></description>
        <link>https://medium.com/@lokhmotov?source=rss-c2c73b48760f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*k6Itl-FiMsb11Qj30QyPKA.jpeg</url>
            <title>Stories by Dr Anton Lokhmotov on Medium</title>
            <link>https://medium.com/@lokhmotov?source=rss-c2c73b48760f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 02:44:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lokhmotov/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Accelerating AI, Part 1]]></title>
            <link>https://medium.com/@lokhmotov/accelerating-ai-part-1-dd392169f7ec?source=rss-c2c73b48760f------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd392169f7ec</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[assembly]]></category>
            <dc:creator><![CDATA[Dr Anton Lokhmotov]]></dc:creator>
            <pubDate>Tue, 10 Oct 2023 22:56:47 GMT</pubDate>
            <atom:updated>2023-10-11T17:12:31.985Z</atom:updated>
            <content:encoded><![CDATA[<h3>Accelerating AI, Part 1: “You can always speed things up”</h3><p>I had my first crush on Computer Engineering over 20 years ago, during an internship in Cambridge (UK). I reported directly to <a href="https://en.wikipedia.org/wiki/Sophie_Wilson">Sophie Wilson</a>, chief architect of <a href="https://en.wikipedia.org/wiki/Broadcom_Corporation">Broadcom</a>’s <a href="https://old.hotchips.org/wp-content/uploads/hc_archives/hc14/3_Tue/23_wilson.pdf">FirePath</a> processor and designer of the original <a href="https://en.wikipedia.org/wiki/ARM_architecture_family">ARM instruction set architecture</a>.</p><p>What struck me back then was that over 90% of the FirePath software had to be written in <a href="https://en.wikipedia.org/wiki/Assembly_language">assembly</a>, practically the most tedious and error-prone way of programming a machine (except for in binary code). While I was familiar with assembly as a compiler student, I just did not expect to find it still actively in use in the early 2000s!</p><p>The FirePath processor was designed for <a href="https://en.wikipedia.org/wiki/ADSL">ADSL</a> protocols, i.e. faster data transmission over telephone lines, as a step-up from <a href="https://en.wikipedia.org/wiki/Dial-up_Internet_access">dial-up</a> of the 1990s. FirePath had smaller area and consumed less power than many <a href="https://en.wikipedia.org/wiki/Application-specific_integrated_circuit">fixed-function</a> competitors. Importantly, its programmable design provided the agility to track emerging communication standards. No wonder, Broadcom snatched the FirePath technology for $640M, a huge amount back in 2000.</p><p>Described as ARM-on-steroids, the FirePath architecture enhanced an elegant <a href="https://en.wikipedia.org/wiki/Reduced_instruction_set_computer">RISC</a> design with <a href="https://en.wikipedia.org/wiki/Single_instruction,_multiple_data">SIMD</a> processing (5 years before <a href="https://developer.arm.com/Architectures/Neon">ARM NEON</a>), <a href="https://en.wikipedia.org/wiki/Very_long_instruction_word">LIW</a> scheduling, domain-specific instructions, and other clever features. Computer Architecture afficionados among the readers can savour the details in Sophie Wilson’s <a href="https://antville.org/static/sites/tiny/files/Wilson-Firepath1.pdf">2001</a> and <a href="https://old.hotchips.org/wp-content/uploads/hc_archives/hc14/3_Tue/23_wilson.pdf">2002</a> presentations.</p><p>I will just reproduce <em>the conclusions of the 2001 presentation</em> with my running commentary<strong>:</strong></p><ul><li><em>You can always speed things up </em>— there is always room for improvement from one generation to the next; still, the biggest leap over competition comes from a highly innovative design.</li><li><em>The machine architecture has a big effect on what you can do —</em> the design matters a lot.</li><li><em>The machine microarchitecture also has a big effect on what you can do </em>— the implementation also matters a lot.</li><li><em>You need to enlist your programmers, too</em> — well, this is just a perfect example of <a href="https://en.wikipedia.org/wiki/English_understatement">English understatement</a>: for many years (decades?), handsomely-paid wizard coders had to write most of the FirePath sofware in assembly, because compiler-generated code could easily be 10 times or more slower; in 2005, <a href="https://www.graphcore.ai/simon-knowles">Simon Knowles</a>, currently CTO of Graphcore and also a major contributor to FirePath, estimated its hardware costs at 40 person-years and software costs at 60 person-years.</li></ul><p>So while “you can always speed things up” with innovative hardware, you should never forget about software, which is only becoming more sophisticated and challenging. Stay tuned!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uXMpAoUG_eQ_xw_v1P-mxg.jpeg" /><figcaption>John Redford, Anton Lokhmotov, Sophie Wilson. August 2003.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd392169f7ec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying MLPerf Inference]]></title>
            <link>https://medium.com/data-science/demystifying-mlperf-inference-956e41790df4?source=rss-c2c73b48760f------2</link>
            <guid isPermaLink="false">https://medium.com/p/956e41790df4</guid>
            <category><![CDATA[trust]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[benchmarking]]></category>
            <category><![CDATA[mlperf]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Dr Anton Lokhmotov]]></dc:creator>
            <pubDate>Tue, 05 May 2020 15:33:03 GMT</pubDate>
            <atom:updated>2020-05-06T08:45:20.456Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*Wx4q0dXiTENdpqwrQu5_AQ.png" /></figure><p><a href="http://mlperf.org/"><strong>MLPerf</strong></a> is a broad community focused on benchmarking Machine Learning (ML) with over 70 participating organizations (and counting). <em>Benchmarking </em>means <em>evaluating the “performance”</em> (quality, speed, energy efficiency, etc.) of computer systems using typical <em>workloads </em>(tasks, datasets, models). Unlike most other ML benchmarks, <strong>MLPerf</strong> encompasses both <strong>Training</strong> and <strong>Inference</strong> (deployment) stages of using ML in practice.</p><p>The second <strong>MLPerf Training</strong> round (v0.6, with the results <a href="https://mlperf.org/press#mlperf-training-v0.6-results">announced</a> on 10 July 2019) received submissions from Intel, NVIDIA, Google, Alibaba and Fujitsu. The first <a href="https://arxiv.org/abs/1911.02549"><strong>MLPerf Inference</strong></a> round (v0.5, with the results <a href="https://mlperf.org/press#mlperf-inference-v0.5-results">announced</a> on 6 November 2019) received submissions from 14 organizations, including from the above companies, Habana Labs (<a href="https://newsroom.intel.com/news-releases/intel-ai-acquisition">acquired</a> by Intel in December 2019) and <a href="https://dividiti.blogspot.com/2019/11/dividiti-accelerate-omni-benchmarking-for-mlperf.html">dividiti</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*G42KTn7LwyCZ-zdPFgfWoQ.jpeg" /><figcaption>MLPerf Inference v0.5: breaking down the 595 accepted results by the submitter.</figcaption></figure><p>Thus, <strong>MLPerf</strong> is quickly becoming <em>the</em> industry benchmark for ML systems and ideal forum for announcing new products with benchmarking results that analysts, investors and buyers will <em>trust</em>.</p><p>As one of the top <a href="https://github.com/mlperf/inference/graphs/contributors">contributors</a> to <strong>MLPerf Inference</strong>, I often answer questions from newcomers. While an excellent <a href="https://arxiv.org/abs/1911.02549">whitepaper</a> is available that explains the rationale and methodology of the benchmark suite, the questions I get from potential submitters (hardware designers, cloud providers, etc.) and users (buyers, investors, etc.) are of a more practical nature. In this post, I begin to answer some commonly asked questions.</p><p>In what follows, in questions, “I” means you, the reader, and “we” means “your organization”; in answers, “I” means me, the author, and “we” means “the <strong>MLPerf </strong>community” (but only when I believe I can speak for the community as a whole).</p><h3>Why should we consider participating in MLPerf?</h3><p>One word — <em>trust</em>.</p><p>With many options becoming available for accelerating ML in the cloud and at the edge, discerning buyers are no longer satisfied with solely relying on vendors’ sales pitches and marketing materials to make purchasing decisions. Instead, buyers demand vendors to provide a great deal of detail such as the exact workloads and optimization options used, performance and accuracy trade-offs, power consumption, toolchain maturity, and so on. (For example, see a <a href="https://www.youtube.com/watch?v=1ldgVZ64hEI">motivating presentation</a> by General Motors at the <a href="https://www.edge-ai-vision.com/the-summit/may-2017-embedded-vision-summit-replay/">Embedded Vision Summit 2017</a>.)</p><p><strong>MLPerf </strong>aims to enable meaningful and fair comparisons between systems by agreeing on a common rigorous methodology. In particular, all <strong>Inference</strong> submissions must follow strict <a href="https://github.com/mlperf/inference_policies/blob/master/inference_rules.adoc">rules</a> regarding <a href="https://github.com/mlperf/inference_policies/blob/master/inference_rules.adoc#8-model">permitted model transformations</a>, <a href="https://github.com/mlperf/inference_policies/blob/master/inference_rules.adoc#data-sets">data pre- and post-processing</a>, and so on. Importantly, all <strong>Inference</strong> submissions must use the so-called <a href="https://github.com/mlperf/inference_policies/blob/master/inference_rules.adoc#load-generator">LoadGen</a> API for generating execution traces of workloads (ML models) on Systems-Under-Tests (software + hardware) in one or more <a href="https://github.com/mlperf/inference_policies/blob/master/inference_rules.adoc#scenarios">usage scenarios</a> (single-stream, multi-stream, server, offline).</p><p>Now comes the most important bit for generating trust. All <strong>MLPerf</strong> submissions get <a href="https://github.com/mlperf/policies/blob/master/submission_rules.adoc#review">reviewed</a> by other submitters, who may potentially include your direct competitors. Therefore, if your submission stands to scrutiny of the reviewers, your result will be viewed as the most trustworthy. That’s golden!</p><p>For example, the three hardware startups that submitted to <strong>MLPerf Inference v0.5</strong> have already reaped the benefits of hard-earned trust:</p><ul><li><a href="https://www.furiosa.ai/">Furiosa</a> <a href="http://www.theinvestor.co.kr/view.php?ud=20191107000614">got</a> a Series A investment of $7 million.</li><li><a href="https://hailo.ai/#">Hailo</a> <a href="https://techcrunch.com/2020/03/05/hailo-raises-60m-series-b-for-its-ai-chips">got</a> a Series B investment of $60 million.</li><li>Finally, <a href="https://habana.ai/">Habana Labs</a> got <a href="https://newsroom.intel.com/news-releases/intel-ai-acquisition/">acquired</a> by Intel for $2 billion.</li></ul><h3>Is MLPerf only for hardware vendors?</h3><p>As <a href="https://www.linleygroup.com/newsletters/newsletter_detail.php?num=6097">Linley Newsletter</a> summarized it, <strong>MLPerf Inference v0.5</strong> submissions came from “cloud-service providers, server vendors, chip vendors, and professional-services companies”. So the answer would appear to be “largely yes”.</p><p>There is no limit, however, to what fair benchmarking can help showcase: professional services (e.g. <a href="http://bit.ly/benchmarking-product">benchmarking-as-a-product</a>), intellectual property (e.g. state-of-the-art techniques), skills (e.g. model design and optimization, retraining), tools (e.g. graph compilers), and so on.</p><h3>OK, how do we run MLPerf?</h3><p><strong>MLPerf</strong> is not a benchmark that you can simply download and run, at least not today. (The <strong>MLPerf Mobile</strong> working group is making great strides towards developing an <a href="https://github.com/mlperf/mobile_app">Android app</a> for user-friendly benchmarking of smartphones and tablets, but the ML inference landscape is clearly much broader than mobile.)</p><p>So to answer this question, we should first consider how the <strong>MLPerf</strong> benchmark definition process works. The <strong>MLPerf </strong>(<strong>Training</strong> or <strong>Inference</strong>)<strong> </strong>working groups agree on a set of representative <em>tasks </em>such as Image Classification and Object Detection, and one or more representative <em>workloads</em> (models) for each task. One or more persons then volunteer to create <em>a reference implementation</em> of each workload. This helps <a href="https://en.wikipedia.org/wiki/Crystallization">crystallize</a> the requirements on the model (e.g. ideally, there should only be one source of pretrained weights for inference), validation dataset, performance and accuracy metrics, and so on.</p><p>The reference implementation, however, is not intended to be the fastest. In fact, the submitter is expected to <em>port </em>the workload to their target hardware. Typically, they need to create <em>a new implementation</em> blending the <strong>MLPerf </strong>LoadGen API for issuing inference queries and a hardware-specific API for processing the queries. They should still mimic the preprocessing and postprocessing behaviour of the reference implementation to make their submission compliant.</p><p>At the heart of accelerating inference, of course, lies the hardware-specific API. For example, for <strong>MLPerf Inference v0.5</strong>, NVIDIA created new workload implementations using <a href="https://developer.nvidia.com/tensorrt">TensorRT</a>, Intel — using <a href="https://01.org/openvinotoolkit">OpenVINO</a>, dividiti — using <a href="https://github.com/ARM-software/armnn-mlperf">ArmNN</a>. According to the <a href="https://github.com/mlperf/inference_policies/blob/master/inference_rules.adoc#benchmark-implementations-must-be-shared">rules</a>, the accelerated implementations must be shared under an open-source license allowing benchmarking.</p><p>Of course, creating new <strong>MLPerf</strong>-compliant implementations requires significant effort on behalf of submitters, but it’s necessary for achieving credibility. The good news is that the effort should be amortized across several submission rounds. It can also inspire a wider <a href="http://bit.ly/ob-ic">community-driven benchmarking</a> across other platforms, workloads, datasets, and so on.</p><h3>Is <strong>MLPerf </strong>Inference only about Computer Vision workloads?</h3><p>Not at all! It is true that out of 5 workloads used for the <strong>v0.5</strong> round, 4 were for the <a href="https://en.wikipedia.org/wiki/Computer_vision">Computer Vision</a> tasks of Image Classification and Object Detection. For the <strong>v0.7</strong> round, however, new tasks are being introduced such as <a href="https://github.com/mlperf/inference/tree/master/v0.7/language">Natural Language Processing</a>, <a href="https://github.com/mlperf/inference/tree/master/v0.7/speech_recognition/">Speech Recognition</a> and <a href="https://github.com/mlperf/inference/tree/master/v0.5/recommendation">Recommendation</a>. The complexity of inference has been dramatically increased as well: from 25 million parameters in ResNet50 to 25 billion parameters in <a href="https://ai.facebook.com/blog/dlrm-an-advanced-open-source-deep-learning-recommendation-model">DLRM</a>.</p><h3>What if my customer’s favourite workload is not part of MLPerf Inference?</h3><p>If you are a potential submitter, you might have already performed some benchmarking for your customer using their favourite workload. Even if this workload is not part of <strong>MLPerf Inference</strong>, you might be able to reuse the effort you put into optimizing this workload and showcase your solution under two conditions:</p><ol><li>The workload must still perform one of the tasks defined by <strong>MLPerf Inference</strong>. For example, you can use another <a href="http://bit.ly/ob-od">Object Detection</a> model.</li><li>You must submit to the <a href="https://github.com/mlperf/inference_policies/blob/master/inference_rules.adoc#open-division">Open Division</a> of <strong>MLPerf Inference</strong>. While the Open Division relaxes some rules (e.g. allows to select an arbitrary model or to retrain one from the <a href="https://github.com/mlperf/inference_policies/blob/master/inference_rules.adoc#closed-division">Closed Division</a>), your submission must still use the <a href="https://github.com/mlperf/inference_policies/blob/master/inference_rules.adoc#load-generator">LoadGen API</a> and conform to the <a href="https://github.com/mlperf/policies/blob/master/submission_rules.adoc#submission">submission format</a>.</li></ol><h3>What if my favourite workload is not part of MLPerf Inference?</h3><p>If you have your favourite workload (indeed, perhaps the only workload you truly care about!), you also might want to use the rigorous <strong>MLPerf </strong>methodology to evaluate your potential suppliers. In theory, you would just have to nudge them to produce <strong>MLPerf</strong>-compliant results and watch the competition unfold. In practice, they might not budge, especially if they are resource-constrained or if your request is not deemed to be of sufficiently high priority. In this case, you can still evaluate them yourself or ask a credible third-party to help.</p><p>Other noteworthy cases of stepping outside of the formal <strong>MLPerf </strong>framework include non-public competitive analysis (e.g. to justify internal R&amp;D expenditure) or public marketing (e.g. to support your product release out-of-cycle with <strong>MLPerf </strong>submission rounds). In the latter case, make sure to observe the <a href="https://github.com/mlperf/policies/blob/master/TERMS%20OF%20USE.md"><strong>MLPerf </strong>terms of use</a>. Again, think about how to boost the credibility of your claims <strong>MLPerf-</strong>style, with full transparency and objectivity.</p><p>In this post, I’ve attempted to dispel an aura of mystery surrounding <strong>MLPerf Inference</strong> for potential submitters and users of this benchmark. But as I’ve barely scratched the surface, please feel free to contact <a href="https://www.linkedin.com/in/lokhmotov/">me</a> with further questions, and I’ll try to answer them in future posts.</p><p>One question that repeatedly comes up is that of effort and cost of rigorous benchmarking. <a href="http://bit.ly/benchmarking-product">Drawing parallels with software engineering</a>, it can indeed cost several times more than ad-hoc approaches. Workflow automation will be key to making rigorous ML benchmarking (across diverse hardware, software, workloads, datasets, etc.) affordable for everyone, while not compromising on credibility and trust.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=956e41790df4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/demystifying-mlperf-inference-956e41790df4">Demystifying MLPerf Inference</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Benchmarking as a product: a case for Machine Learning]]></title>
            <link>https://medium.com/swlh/benchmarking-as-a-product-a-case-for-machine-learning-b9648c61f8b6?source=rss-c2c73b48760f------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9648c61f8b6</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[mlperf]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[benchmarking]]></category>
            <dc:creator><![CDATA[Dr Anton Lokhmotov]]></dc:creator>
            <pubDate>Tue, 07 Apr 2020 13:35:04 GMT</pubDate>
            <atom:updated>2020-04-09T07:29:12.095Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7N2P0rqmbADag3r9bzENeA.png" /><figcaption>An interactive <a href="http://cknowledge.org/">Collective Knowledge</a> dashboard for the <a href="http://cknowledge.org/dashboard/mlperf.inference">MLPerf Inference v0.5</a> results.</figcaption></figure><p>Every professional programmer is familiar with <a href="https://en.wikipedia.org/wiki/Fred_Brooks">Fred Brooks</a>’s timeless classic “<a href="https://en.wikipedia.org/wiki/The_Mythical_Man-Month">The Mythical Man-Month</a>”. In Fig 1.1, Brooks distinguishes between two types of programming objects:</p><ul><li><em>a program:</em> “complete in itself, ready to be run by the author on the system on which it was developed”; “the thing commonly produced in garages”; “the object the individual programmer uses in estimating productivity”.</li><li><em>a programming product</em>: “can be run, tested, repaired, and extended by anybody”; “usable in many operating environments, for many sets of data”; “written in a generalized fashion”; “must be thoroughly tested”; “requires thorough documentation”.</li></ul><p>Furthermore, Brooks estimates that “a programming product <strong>costs at least three times</strong> as much as a debugged program with the same function”.</p><p>It turns out, we can make a similar distinction for objects used for performance evaluation, colloquially known as benchmarking. <a href="https://www.linkedin.com/in/ed-plowman-b3738b1/">Ed Plowman</a>, former Director of Performance Analysis Strategy at <a href="http://arm.com/">Arm</a>, <a href="https://github.com/ctuning/ck/wiki/Demo-ARM-TechCon&#39;16">gave</a> the following definitions:</p><ul><li><em>a benchmark:</em> an abusive term for questionably constructed software, e.g. “This piece of software is a benchmark”;</li><li><em>to benchmark:</em> to create a meaningless set of measurements, e.g. “We benchmarked the latest device”.</li></ul><p>It is no surprise then to encounter the following <a href="https://www.cs.kent.ac.uk/pubs/2011/3174/content.pdf">attitude</a> to using questionably constructed software coupled with questionable methodology:</p><blockquote>“In industry, we always ignore the evaluation in academic papers. It is always irrelevant and often wrong.” — Head of a major industrial lab, 2011</blockquote><p>So if <em>a benchmark</em> is similar to <em>a program</em>, “the thing commonly produced in garages” (or research labs), what can be said about a <em>benchmarking product</em>?</p><p>Consider several purposes of benchmarking <strong>Z</strong>, where <strong>Z</strong> is<strong> </strong>algorithm/model/technique/software/hardware/etc.</p><ul><li>Internal R&amp;D and competitive analysis: <em>Is</em> my <strong>Z</strong> better than competing ones? Under which conditions? At what cost?</li><li>External marketing and sales: I’ve got a proof that my <strong>Z </strong><em>is</em> better than competing ones.</li><li>Purchasing decisions: Which <strong>Z</strong> is the <em>best</em> for my needs given <em>my </em>requirements on performance/quality/cost/etc.?</li></ul><p>Benchmarking for internal purposes often resembles Brooks’s definition of <em>a program</em>: It’s done by a closed circle of people (e.g. the authors of a paper), in a few environments (e.g. on a couple of platforms), under particular conditions (e.g. using the peak performance settings), etc.</p><p>Sometimes, this approach suffices, in particular, in an academic setting where the primary goal is still to publish rather than to produce useful artifacts. (Otherwise, it would not be the <a href="https://medium.com/atlas-ml/state-of-deep-learning-h2-2018-review-cc3e490f1679">case</a> that only one in seven Machine Learning papers comes with <em>any </em>means to reproduce them.)</p><p>Using the same approach for external purposes, however, usually fails to convince the skeptics and to inform the curious. For example, with <a href="https://www.eetimes.com/chip-startups-for-ai-in-edge-and-endpoint-applications/">scores of startups</a> and established vendors developing hardware for Machine Learning from the data centre to the endpoint, how can a discerning buyer get a solid grasp of this landscape and make the best purchasing decision for their needs?</p><p>Following Brooks and Plowman, we offer a new definition:</p><ul><li><em>a benchmarking product</em>, a set of <em>conclusive </em>results involving <em>meaningful </em>workloads with <em>easy means</em> to reproduce and/or <em>verified </em>by a credible<strong> </strong>third party.</li></ul><p>If it sounds like an unattainable ideal, the <a href="https://mlperf.org/"><strong>MLPerf</strong></a><strong> </strong>community is making steady progress towards a framework for creating and evaluating such <em>benchmarking products</em> for Machine Learning <a href="https://arxiv.org/abs/1911.02549">Inference</a> and <a href="https://arxiv.org/abs/1910.01500">Training</a>. This is not to say that every <strong>MLPerf</strong> submission fully meets this definition today. But <strong>MLPerf </strong>does give a boost to the credibility, especially due to the strict <a href="https://github.com/mlperf/policies/blob/master/submission_rules.adoc">submission rules</a> and <a href="https://github.com/mlperf/policies/blob/master/submission_rules.adoc#6-review">peer-review requirements</a>.</p><p>Similar to <em>a programming product</em> requiring more effort and thus costing more than <em>a program</em>, <em>a benchmarking product</em> costs more than <em>a benchmark</em>. One should, however, consider a potential return on investment (RoI). For example, the three hardware startups that submitted to <a href="https://github.com/mlperf/inference_results_v0.5"><strong>MLPerf Inference v0.5</strong></a> have all received an excellent RoI:</p><ul><li><a href="https://www.furiosa.ai/">Furiosa</a> <a href="http://www.theinvestor.co.kr/view.php?ud=20191107000614">raised</a> a Series A round of $7 million.</li><li><a href="https://hailo.ai/">Hailo</a> <a href="https://techcrunch.com/2020/03/05/hailo-raises-60m-series-b-for-its-ai-chips">raised</a> a Series B round of $60 million.</li><li><a href="https://habana.ai/">Habana Labs</a> <a href="https://newsroom.intel.com/news-releases/intel-ai-acquisition/">got acquired</a> by Intel for $2 billion.</li></ul><p>Is <em>benchmarking-as-a-product</em> only relevant for hardware companies? Not at all! There is no limit to what fair and credible benchmarking can help showcase: professional services (e.g. <a href="https://dividiti.com/services.html#vendors">helping a vendor</a> create <em>a benchmarking product</em>), intellectual property (e.g. state-of-the-art techniques), skills (e.g. in model design, optimization, retraining), tools (e.g. compilers), and so on.</p><p>Over time, any <em>product </em>will need to be accompanied by <em>a benchmarking product</em> to be taken seriously by buyers and investors. Perhaps this time has already come?..</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9648c61f8b6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/benchmarking-as-a-product-a-case-for-machine-learning-b9648c61f8b6">Benchmarking as a product: a case for Machine Learning</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Omni-benchmarking Image Classification (and much more)]]></title>
            <link>https://medium.com/data-science/omni-benchmarking-image-classification-and-much-more-5e0d08bb2a7?source=rss-c2c73b48760f------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e0d08bb2a7</guid>
            <category><![CDATA[benchmarking]]></category>
            <category><![CDATA[efficiency]]></category>
            <category><![CDATA[reproducibility]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[image-classification]]></category>
            <dc:creator><![CDATA[Dr Anton Lokhmotov]]></dc:creator>
            <pubDate>Tue, 24 Sep 2019 17:46:40 GMT</pubDate>
            <atom:updated>2019-09-25T08:14:08.858Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tVz01yW2a0KxfJq9dLK34w.png" /><figcaption>Performance/accuracy exploration of image classification models on hardware ranging from IoT-class to server-class platforms from the submissions to the 1st <a href="http://cknowledge.org/request-cfp-asplos2018.html">ACM ReQuEST tournament at ASPLOS’18</a>.</figcaption></figure><p>Image classification is a popular playground for investigating efficiency/quality trade-offs of ML models. For example, the <a href="https://paperswithcode.com/sota/image-classification-on-imagenet">state-of-the-art</a> <a href="https://arxiv.org/abs/1906.06423">FixResNeXt-101</a> model achieves the Top-1 accuracy of 86.4% on the <a href="http://image-net.org/">ImageNet</a> validation set but with 829M parameters is only suitable for <a href="https://github.com/facebookresearch/FixRes#cluster-settings">cluster use</a>. As another example, the Top-1 accuracy of the <a href="https://arxiv.org/abs/1801.04381">MobileNets-v2</a> model family suitable for mobile devices <a href="https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet#imagenet--checkpoints">ranges</a> from 75.0% with 6.1M parameters to 45.5% with 1.7M parameters.</p><p>Note that the distinction between “cluster” and “mobile” is purely <em>qualitative</em>. In terms of power consumption, the difference can easily be a thousand times or a million times. Performing a <a href="https://www.sigarch.org/artifact-evaluation-for-reproducible-quantitative-research/"><em>quantitative</em></a> evaluation, however, can be quite tricky.</p><p>Answer yourself honestly, when was the last time you were able to take an exciting new paper for a spin within minutes or even hours? We mean, <em>for real</em>: downloading all artifacts (code, data, etc.), installing all dependencies (e.g. TensorFlow) and dependencies of dependencies (e.g. Java and Bazel), porting to a new platform (Android, anyone?), testing with a new dataset, and so on?</p><p>Never happened to us. Not in a million years. At best, we could do that within days. More often, within weeks. And that even assuming that the paper did come with artifacts. (Disappointingly, only <a href="https://medium.com/atlas-ml/state-of-deep-learning-h2-2018-review-cc3e490f1679">1 ML paper in 7</a> does.)</p><p>Now, imagine opening a paper describing a novel approach to image classification on mobile devices which conveniently comes with an artifact. Imagine you can install and customize the artifact within minutes, while you are reading the introduction, and reproduce the experiments on your mobile phone, while you are getting over to the conclusion? With one tap on a plot, share your results with the community and instantly see any anomalies and outliers, which may warrant further investigation.</p><p>Yes, this may sound like a dream. Yet, we believe that rapid artifact and knowledge sharing are crucial to accelerating ML/systems research and development. Today we describe a <em>small</em> step towards this dream we made with the community.</p><h3>ACM ReQueEST: Reproducible Quality/Efficient Systems Tournaments</h3><p>In 2018, we <a href="https://portalparts.acm.org/3230000/3229762/fm/frontmatter.pdf">co-organized</a> the <a href="http://cknowledge.org/request">1st ACM ReQuEST tournament</a> at <a href="https://asplos-conference.org/">ASPLOS</a>, we <a href="https://towardsdatascience.com/acm-request-1st-open-and-reproducible-tournament-to-co-design-pareto-efficient-deep-learning-ea8e5a13d777">invited the community</a> to submit papers with complete artifacts (code, data, scripts, etc.). <a href="https://portalparts.acm.org/3230000/3229762/fm/frontmatter.pdf">Our goal</a> was to put every<em> </em>submitted artifact through the well-established <a href="http://ctuning.org/ae/reviewing.html">Artifact Evaluation</a> process. In other words, we focused on <a href="https://www.sigarch.org/artifact-evaluation-for-reproducible-quantitative-research/">practical quantitative reproducibility</a> rather than on the sheer novelty of submissions.</p><p>We were quite surprised by the diversity of submissions covering all levels of the ML/software/hardware stack (which an analyst <a href="https://twitter.com/jameskobielus/status/994175022586613760">called</a> <em>omni-benchmarking</em>):</p><ul><li><strong>Models:</strong> MobileNets, ResNet-18, ResNet-50, Inception-v3, VGG16, AlexNet.</li><li><strong>Frameworks and libraries:</strong> TensorFlow, Caffe, MXNet, Keras, <a href="https://github.com/ARM-software/ComputeLibrary/">Arm Compute Library</a>, cuDNN, <a href="https://tvm.ai/">TVM</a>, NNVM.</li><li><strong>Platforms:</strong> Xilinx Pynq-Z1 FPGA, Arm Cortex CPUs and Arm Mali GPGPUs (Linaro HiKey960 and T-Firefly RK3399), a farm of Raspberry Pi devices, NVIDIA Jetson TX1 and TX2, and Intel Xeon servers in the Amazon, Microsoft and Google clouds.</li><li><strong>Data types:</strong> 8-bit integer, 16-bit floating-point (half), 32-bit floating-point (float).</li></ul><p>For every<em> </em>artifact, we created an automated, customizable and reproducible <a href="https://github.com/ctuning/ck">Collective Knowledge</a> workflow to unify evaluation of accuracy, latency (seconds per image), throughput (images per second), peak power consumption (Watts), price and other metrics. We then published the unified workflows on <a href="https://github.com/ctuning/ck-request-asplos18-results">GitHub</a> and added snapshots to the <a href="https://dl.acm.org/citation.cfm?doid=3229762">ACM Digital Library</a>.</p><p>Rather than declaring a single winner or <a href="https://en.wikipedia.org/wiki/Name_and_shame">naming-and-shaming</a>, we decided to plot all the results on a public <a href="http://cknowledge.org/dashboard/request.asplos18">interactive dashboard</a>. Using the dashboard, you can apply your own criteria to explore the solution space and look for <a href="https://en.wikipedia.org/wiki/Pareto_efficiency">Pareto-optimal</a> solutions (e.g. to find the most energy-efficient solution achieving the required accuracy). Basically, it’s up to you how to interpret the results through your own preferences and requirements!</p><h3>Vision</h3><p>From a purely benchmarking perspective, <a href="http://cknowledge.org/request">ReQuEST</a> can be viewed as a precursor to the open division of <a href="https://www.mlperf.org/">MLPerf</a>, where submitters have no restrictions on workloads and systems they use, as long as their submissions are reproducible.</p><p>Our long-term goal, however, is to create <a href="https://reuseresearch.com/components">open repositories of reusable ML/SW/HW components</a>. This would allow researchers to build upon such components in their reproducible and interactive papers while eventually solving the thorny issue of benchmarking techniques across the latest software and hardware.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*WNzMpIH32BFqhXAPq6MFqw.png" /></figure><p>We envision that eventually the community will crowdsource benchmarking, co-design and optimization of complete ML/SW/HW stacks (e.g. see <a href="http://cknowledge.org/android-apps.html">our Android crowd-benchmarking demo</a>), which in turn will dramatically reduce effort and time-to-market required for real-world products.</p><p><a href="http://cknowledge.org/contacts.html">Please get in touch</a> if you would like to know more and join <a href="http://cknowledge.org/partners.html">our growing community</a>!</p><h3>Authors</h3><p><a href="https://towardsdatascience.com/@dividiti?source=follow_footer--------------------------follow_footer-">Anton Lokhmotov</a> is co-founder and CEO at <a href="http://dividiti.com/">dividiti</a>. Researcher, engineer and entrepreneur, passionate about optimally using valuable resources such as computer systems… and human talent.</p><p><a href="https://www.linkedin.com/in/grigorifursin/">Grigori Fursin</a> is co-founder and CTO at <a href="http://codereef.ai/">CodeReef.ai</a>. Open-source contributor and reproducibility expert, cleaning up the mess in AI innovation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e0d08bb2a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/omni-benchmarking-image-classification-and-much-more-5e0d08bb2a7">Omni-benchmarking Image Classification (and much more)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Omni-benchmarking Object Detection]]></title>
            <link>https://medium.com/data-science/omni-benchmarking-object-detection-b390cc4114cd?source=rss-c2c73b48760f------2</link>
            <guid isPermaLink="false">https://medium.com/p/b390cc4114cd</guid>
            <category><![CDATA[benchmarking]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[reproducibility]]></category>
            <dc:creator><![CDATA[Dr Anton Lokhmotov]]></dc:creator>
            <pubDate>Fri, 20 Sep 2019 17:08:00 GMT</pubDate>
            <atom:updated>2019-09-25T08:32:10.777Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7mx2SUMO8Bcs3pkJ8fUZbw.png" /><figcaption>Performance/accuracy exploration with 12 object detection models, 5 TensorFlow backends and 6 batch sizes.</figcaption></figure><p>Investigating efficiency/quality trade-offs is of great interest to the ML/systems community. In a <a href="https://arxiv.org/abs/1611.10012">comprehensive CVPR’17 paper</a>, Google researchers focused on exploring speed/accuracy trade-offs of state-of-the-art convolutional approaches to object detection. Rather helpfully, the researchers also released a couple dozen trained models as <a href="https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md">part of the TensorFlow model zoo</a>. These models are quite easy to run and evaluate, as they have standardized input and output tensors.</p><p>While we have found the object detection zoo a great starting point for our system design and optimization work, we could not help wondering about several aspects that we could not extract from the README file of the zoo or the paper.</p><p>First, the README provides execution timings on <em>an</em> NVIDIA TITAN X card. However, over the years NVIDIA have released <em>several versions</em> of TITAN X. While we can guess from the paper timing that that card was based on the Maxwell architecture, there would also be all sorts of other guesswork involved about the GPU driver, CUDA version, TensorFlow version, batch sizes used, etc. Most importantly, <em>we are primarily interested in the performance on our own hardware with the latest software today</em>, not in (ML/SW/HW) archaeology! This includes running on the CPU with MKL and the GPU with TensorRT for comparison.</p><p>Second, the README provides the accuracy over a subset of the COCO 2014 validation set (“minival”), while <em>we are interested in the accuracy over the COCO 2017 validation set</em> (used, for example, in MLPerf). Moreover, the “lowproposals” variants of some of the most accurate models are 1.3–3.4 faster than the baseline ones. However, as no accuracy is provided, it’s not clear what the trade-offs are.</p><p>We realize that the same questions are of great interest to a broader ML/systems community, and we want to help. Rather than to only satisfy our own curiosity and be done with it, we are most keen to make our approach to be <em>reproducible and extensible</em> by others, so that the community can <em>crowdsource</em> evaluation across other platforms, models, datasets, batch size, etc. That’s what we call <em>omni-benchmarking</em>.</p><p>Plotted above are the results from our performance/accuracy exploration with 12 object detection models, 5 TensorFlow backends and 6 batch sizes. This post is about how we obtained them.</p><h3>Model selection</h3><p>We realized we would not have time to test all models in the zoo, so we selected 11 diverse models ranging from the fast SSD-MobileNet to the accurate Faster-RCNN-NAS. We selected 5 Faster-RCNN based models: the 4 available “lowproposals” variants, and Faster-RCNN-Inception-v2. We also selected 6 SSD based models: SSD-ResNet50-FPN, SSD-Inception-v2, 3 variants of SSD-MobileNet-v1 (non-quantized, quantized, FPN), and SSDLite-MobileNet-v2 (an even lighter version of SSD suitable for mobile devices). With this selection, the mAP accuracy of the zoo models ranges between 18% and 43% and the execution time between 30 ms and 540 ms, according to the README. Finally, we added a <a href="https://github.com/YunYang1994/tensorflow-yolov3">reimplementation of YOLO-v3</a>.</p><h3>Customizability</h3><p>To evaluate models in a uniform way, we designed a modular and customizable Python application: the user can plug in different datasets and models (possibly with custom pre- and post-processing), and select one of the supported backends (CPU, CUDA, TensorRT).</p><p>The applications provides “hooks” for supporting model-specific functionality. We identified four such hooks:</p><ul><li>Preprocess: load an image to a tensor and prepare the tensor for inference (convert, reshape, resize).</li><li>Load Model: load a frozen model graph (convert to TensorRT if required), and prepare a session for inference.</li><li>Get Tensor: get the input/output tensor names for the given model.</li><li>Postprocess: convert the output from the graph into detections (bounding boxes and objects).</li></ul><p>For the standardized models from the zoo, we implemented some functions that get used by default. For the YOLO-v3 model, however, we created custom functions with a <a href="https://github.com/ctuning/ck-object-detection/tree/master/package/model-tf-yolo-v3-coco">specific interface</a>. We also implemented batch processing, which involved resizing images to a model-specific size.</p><h3>Design Space of Experiments</h3><p>We are interested in measuring both accuracy (<a href="https://medium.com/@jonathan_hui/map-mean-average-precision-for-object-detection-45c121a31173">mAP and recall</a> on the COCO 2017 validation dataset of 5,000 images), and performance (execution time or images per second) of the examined models.</p><p>We evaluate performance along <em>two</em> dimensions:</p><ul><li><em>Backend configuration of TensorFlow</em>:</li></ul><ol><li>Prebuilt for the CPU (installed via pip), which complains that it does not support AVX2 FMA instructions.</li><li>Built from sources for the CPU with AVX2 support.</li><li>Built from sources for the GPU with CUDA.</li><li>Built from sources for the GPU with TensorRT configured to build an optimized graph statically (i.e. without knowing the image size).</li><li>Built from sources for the GPU with TensorRT configured to build an optimized graph dynamically (i.e. once the first image is given).</li></ol><ul><li><em>Batch size</em>: 1 (no batching), 2, 4, 8, 16 and 32 images per batch.</li></ul><h3>Reproducibility</h3><p>We used automated, portable and customizable <a href="https://cknowledge.org/">Collective Knowledge</a> (CK) workflows to make our experiments reproducible and extensible.</p><p>First, CK installs all the prerequisites: TensorFlow with different backends (see above), the models and the dataset, while resolving all the dependencies (e.g. Java and Bazel when building TensorFlow from sources). Since CK takes care of all the prerequisites and tracks their versions, it is easy to port the whole experimental setup to different machines.</p><p>Second, CK configures each experiment individually: the model, the backend configuration and the batch size. As CK stores experiment configuration (prerequisites, values of environment variables, etc.) alongside experimental results in a unified JSON format, we can conveniently analyze and visualize the experimental results using <a href="https://jupyter.org/">Jupyter</a>, <a href="https://pandas.pydata.org/">pandas</a> and other popular tools.</p><p>Third, we script CK to perform a Cartesian product of all the experimental parameters. In our study, this resulted in 360 performance experiments (12 models, 5 backends, 6 batch sizes) and 24 accuracy experiments (12 models, 1 CUDA backend, with model-internal resizing and with external resizing required for batching).</p><h3>Ease of Use</h3><p>We created a <a href="https://github.com/ctuning/ck-object-detection/blob/master/docker/object-detection-tf-py.tensorrt.ubuntu-18.04/README.md">Docker container</a> encapsulating all the installation steps (as following them from a README can be quite tedious even with CK handling the whole experimental setup). While containerization introduced some overheads, we found them acceptable for our purposes (as we will report in a future post).</p><p>Importantly, using CK and Docker together allows anyone to start running experiments in just a few steps, making them truly portable and reproducible.</p><h3>Results</h3><p>Below, we present some of our results in three parts:</p><ul><li>First, we show the <strong>accuracy</strong> of the examined models on the COCO 2017 validation dataset.</li><li>Second, we analyze the <strong>performance</strong> using different backends and batch sizes.</li><li>Third, we show a comprehensive <strong>design space exploration</strong> graph with the best performance/accuracy trade-offs lying on the <a href="https://en.wikipedia.org/wiki/Pareto_efficiency#Pareto_frontier">Pareto frontier</a>.</li></ul><p>(The full results are available in a <a href="https://nbviewer.jupyter.org/urls/dl.dropbox.com/s/5yqb6fy1nbywi7x/medium-object-detection.20190923.ipynb">Jupyter notebook</a>.)</p><p>Our experimental platform is an <a href="http://h20195.www2.hp.com/v2/default.aspx?cc=ie&amp;lc=en&amp;oid=7528701">HP Z640 workstation</a> with an <a href="https://ark.intel.com/products/81705/Intel-Xeon-Processor-E5-2650-v3-25M-Cache-2_30-GHz">Intel Xeon CPU E5–2650 v3</a> CPU (launched in Q3&#39;14), and an NVIDIA GeForce GTX1080 GPU (launched in Q2’16). <a href="https://hub.docker.com/r/ctuning/object-detection-tf-py.tensorrt.ubuntu-18.04">Our Docker image</a> is based on <a href="https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-07.html">the TensorRT 19.07 image</a> from NVIDIA with <a href="https://developer.nvidia.com/cuda-zone">CUDA</a> 10.1 and <a href="https://developer.nvidia.com/tensorrt">TensorRT</a> 5.1.5, and TensorFlow 1.14.</p><h4>Accuracy</h4><p>First, we show the overall mAP, as well as the mAP for large, medium and small objects, assuming only internal resizing of input images (more on this later).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O6w2qPfGWBuEMmMRGNvBuQ.png" /></figure><p>The most accurate model is Faster-RCNN-NAS, reaching the overall mAP of 44%. Usually a model with a good overall mAP performs consistently well across all the three object categories. There are, however, some exceptions: SSD-Inception-v2 has the 2nd best score on large objects, but performs rather poorly on medium and small objects; on the contrary, YOLO-v3 has the 2nd worst score on large objects, but is on the 4th place on small objects and performs OK on medium objects.</p><p>Performing poorly on small objects is a well-known problem of SSD-based models. However, the SSD-ResNet50 and SSD-MobileNet-v1 models with the <a href="https://arxiv.org/abs/1612.03144">Feature Pyramid Network</a> (FPN) feature are on the 2nd and 3rd place on small objects (and on the 2nd and 4th place overall).</p><p>As images in the COCO dataset come in different shapes and sizes, they need to be resized to the model input size. All models from the zoo have resizing embedded within the graph. To enable batch processing, however, images must be resized outside the graph (since all images in a batch must have the same size). This may damage the accuracy, as can be seen below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zZ2sJfZrr_ecu-rh77f8WQ.png" /></figure><p>The SSD, YOLO and Faster-RCNN-NAS models all include a <em>fixed_shape</em> resizing layer. Using the size of this layer for external resizing has almost no impact on the accuracy (in some cases, the mAP improves a bit).</p><p>The other Faster-RCNN-based models, however, include a <em>keep_aspect_ratio</em> resizing layer. Forcing a “fixed shape” external resizing for these models adversely affects the accuracy (in some cases, the mAP drops by more than 10%). (We would love to hear from you if you know how to solve this issue!)</p><h4>Performance</h4><p>We report the performance as the number of images (or frames) processed per second (FPS). We run each experiment 10 times. In bar plots, the bars show the FPS averaged over 10 runs, while the whiskers show the standard deviation. (Tight whiskers indicate that the performance is stable.)</p><p>First, we present the performance of 12 models and 6 batch sizes with the CUDA backend:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ebgJn0uhjpWeGsTKl_xr6A.png" /></figure><p>It comes as no surprise that the most accurate model, Faster-RCNN-NAS, is also the slowest at 3 FPS. The fastest model, quantized SSD-MobileNet used in MLPerf Inference, is 15–25 times faster than Faster-RCNN-NAS depending on the batch size. This suggests that our Pascal-based GPU is roughly two times faster than the Maxwell-based GPU that was used to obtain performance figures available in the README. However, what if the performance advantage is not primarily down to architecture improvements?</p><p>Generally, the larger the batch size, the higher the performance. For some models, however, batching can be detrimental to performance. For example, Faster-RCNN-NAS on the CPU is the fastest with the batch size of 1, which is probably due to high memory requirements:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_ViKJPNRilV5vDtYP6x1iw.png" /></figure><p>In fact, this model cannot even run on the GPU with the batch sizes of 16 and 32. (This GPU has only 8 GB RAM.) Moreover, the TensorRT-Dynamic backend can only run with the batch size of 1.</p><p>The non-quantized SSD-MobileNet model runs faster on the CPU:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*29-QNyG2sZTrpOx2tl3mEw.png" /></figure><p>The quantized SSD-MobileNet model, however, runs faster on the GPU (and almost 1.5x times faster than the non-quantized one):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*17bhYbJfCzkYFzO-hYRuBA.png" /></figure><p>Both models, included in MLPerf Inference v0.5, are executed in 32-bit floating-point. The quantized model probably benefits from a better data organization in GPU memory.</p><p>For the examined models, we observed almost no gains of using TensorRT in the static mode over CUDA. We believe it’s because TensorRT in the static mode often falls back to CUDA, as it does not know tensor shapes. <a href="https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#static-dynamic-mode">TensorRT in the dynamic mode</a>, however, creates an optimized graph at runtime once the first image is given. In the plot below, red/blue pairs of bars represent CUDA/TensorRT-Dynamic performance for progressively increasing batch sizes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w1bFUhbq6n_FWKF7kWMY3A.png" /></figure><p>Generally, TensorRT is faster than CUDA but YOLO-v3 is an exception. We conjecture it’s because it’s a newer model, which NVIDIA haven’t yet optimized TensorRT for. We also note that TensorRT-Dynamic almost always fails for the slowest models (returns Out-Of-Memory errors).</p><h4>Exploration</h4><p>Finally, we present the performance and accuracy on a single plot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7mx2SUMO8Bcs3pkJ8fUZbw.png" /></figure><p>Different colours represent different models. Different markers represent different backends. The size of a marker represents the batch size from 1 (the smallest) to 32 (the largest).</p><p>Note how the accuracy drops for most Faster-RCNN-based models due to external resizing for batching. We can plot the same data points with the “ideal” accuracy taken with internal resizing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aYwlhKZnDpZh_TSvWY0DVA.png" /></figure><p>The black dots show the Pareto frontier: the most accurate and slowest model is Faster-RCNN-NAS is in the top left corner; the fastest is quantized SSD-MobileNet-v1 in the bottom right corner (it dominates non-quantized SSD-MobileNet-v1 both in terms of performance and accuracy).</p><p>The picture changes if we consider the mAP for small objects, where the SSD-based models with FPN are almost as good as Faster-RCNN-NAS, but the other SSD-based models perform abysmally with the mAP below 4%:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o6jW71mdyeBKJxmLosjPag.png" /></figure><p>For large objects, SSD-Inception-v2 and SSDLite-MobileNet-v2 get onto the Pareto frontier:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tdc1jYsWVreIryIeY7ynGg.png" /></figure><h3>Conclusion</h3><p>We have presented our results of <em>omni-benchmarking</em> 12 popular object detection models using 5 TensorFlow backends and 6 batch sizes for small, medium and large objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7qwd_BkAiZ89sq7GDiYbnw.png" /></figure><p>It is clear that there is no one-fits-all optimal configuration. Even the CPU is sometimes faster than the GPU (for SSDLite-MobileNet-v2 and non-quantized SSD-MobileNet-v1). A larger batch size does not always guarantee better performance (1 is optimal for Faster-RCNN-NAS). Finally, optimal model selection may depend on the chosen accuracy metric and real-time constraints.</p><p>That’s why we believe that omni-benchmarking is proving really valuable for designing efficient and reliable computer systems! Having already <a href="http://bit.ly/ob-ic">omni-benchmarked image classification</a> for the 1st <a href="http://cknowledge.org/">ACM ReQuEST</a> tournament, we are open to collaboration to omni-benchmark any other ML tasks.</p><p><a href="http://cknowledge.org/contacts.html">Please get in touch</a> if you would like to know more and join <a href="http://cknowledge.org/partners.html">our growing community</a>!</p><h3>Authors</h3><p>Emanuele Vitali is a PhD candidate at Politecnico di Milano. His research interests include hardware architecture and application autotuning on heterogeneous systems.</p><p><a href="https://www.linkedin.com/in/lokhmotov/">Dr Anton Lokhmotov</a> is co-founder of <a href="http://dividiti.com/">dividiti</a>. He is hell-bent on optimal use of valuable compute resources… and human talent.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b390cc4114cd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/omni-benchmarking-object-detection-b390cc4114cd">Omni-benchmarking Object Detection</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Breathing life into AI/ML marketplaces]]></title>
            <link>https://medium.com/@lokhmotov/breathing-life-into-ai-ml-marketplaces-9cb5d113130c?source=rss-c2c73b48760f------2</link>
            <guid isPermaLink="false">https://medium.com/p/9cb5d113130c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[benchmarking]]></category>
            <category><![CDATA[marketplaces]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Dr Anton Lokhmotov]]></dc:creator>
            <pubDate>Fri, 23 Aug 2019 14:53:20 GMT</pubDate>
            <atom:updated>2019-09-11T15:00:02.276Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tVz01yW2a0KxfJq9dLK34w.png" /><figcaption>ACM ReQuEST @ ASPLOS’18: an interactive dashboard with all the submissions</figcaption></figure><p>We have grown to like online marketplaces such as Amazon and eBay, where the variety of goods, competitive pricing and honest customer feedback all make for gratifying shopping experience. Given the growing interest in Artificial Intelligence (AI) and Machine Learning (ML) technologies, it is only natural to extend the marketplace concept to products and services in this space.</p><p>In fact, AI/ML marketplaces already exist. For example, Amazon Web Services (AWS) provide an <a href="https://aws.amazon.com/marketplace/solutions/machinelearning">ML &amp; AI marketplace</a> to help their customers subscribe and securely deploy stable containerized algorithms and models for popular categories including Computer Vision, Natural Language Processing and Speech Recognition. For example, at the time of this writing, the marketplace <a href="https://aws.amazon.com/marketplace/search/results?page=1&amp;filters=fulfillment_options&amp;fulfillment_options=SAGEMAKER">contains</a> 243 free, free-trial and paid packages from 43 vendors. In addition, vendors provide information to help customers guide their choice of model via the product details such as descriptions, usage instructions and pricing per hour for each instance.</p><p>Even with the available information, customers may find it daunting to make the right selection from the marketplace. For example, searching for ResNet50, a popular image classification model, returns several choices:</p><p>- “GluonCV ResNet50 Classifier” by AWS reaches ”the state-of-the-art-performance with accuracy of 79.15% vs. 75.3% in the original paper”.</p><p>- “MXNet ResNet50 Inference” by Intel is “accelerated with MKL-DNN” and is “faster and higher speed”.</p><p>- “Xilinx ML Suite” uses f1* FPGA instances.</p><p>With the lack of quantitative data available for comparison and the lack of feedback, customers may wonder which choice will meet their particular service quality targets under the given budget constraints? For example, would using FPGA instances be faster and/or cheaper than using CPU instances? Add to this possible questions around retraining via <a href="https://aws.amazon.com/sagemaker/">AWS SageMaker</a> or deployment via <a href="https://aws.amazon.com/sagemaker/neo/">AWS SageMake Neo</a>, and it is clear that further steps are needed to help customers make informed decisions.</p><p>These issues are definitely not unique to AWS or other ML marketplaces. We have been grappling with similar issues in various industrial and academic contexts for many years. If you consider the diversity and ever changing nature of ML models, datasets, frameworks, software environments, hardware platforms and so on, it is hard not to despair and see any light at the end of the tunnel!</p><h3>Community-driven omni-benchmarking to the rescue!</h3><p>We believe that the above issues can be gradually solved with help from the community. The idea is simple: make benchmarking so easy that it can be crowdsourced!</p><p>To test this idea, we co-organized the <a href="http://cknowledge.org/request">1st ACM ReQuEST tournament at ASPLOS</a> in 2018, where we invited the community to submit complete implementations (code, data, scripts, etc.) for the popular <a href="http://www.image-net.org/">ImageNet object classification challenge</a>. <a href="https://portalparts.acm.org/3230000/3229762/fm/frontmatter.pdf">Our goal</a> was to put every<em> </em>submitted implementation (artifact) through the <a href="http://ctuning.org/ae/reviewing.html">established Artifact Evaluation process</a>. We also created an automated, customizable and reproducible <a href="https://github.com/ctuning/ck">Collective Knowledge</a> workflow for every<em> </em>artifact to unify evaluation of accuracy, latency (seconds per image), throughput (images per second), peak power consumption (Watts), price and other metrics. We published the unified workflows on <a href="https://github.com/ctuning/ck-request-asplos18-results">GitHub</a> and added snapshots to the <a href="https://dl.acm.org/citation.cfm?doid=3229762">ACM Digital Library</a>. Importantly, rather than declaring a single winner or <a href="https://en.wikipedia.org/wiki/Name_and_shame">naming-and-shaming</a>, we decided to plot all the results on a public <a href="http://cknowledge.org/dashboard/request.asplos18">interactive dashboard</a>. Using the dashboard, anyone can apply their own criteria to explore the solution space and look for <a href="https://en.wikipedia.org/wiki/Pareto_efficiency">Pareto-optimal</a> solutions (e.g. to find the most energy efficient solution achieving the required accuracy).</p><p>Working with a <a href="https://www.linkedin.com/in/gauravkaul/">Senior Consultant at AWS</a>, we easily deployed two CK workflows in the AWS cloud: <a href="https://doi.org/10.1145/3229762.3229763">one from Intel</a> showing ~50x speedups of Intel-optimized Caffe over vanilla Caffe with negligible accuracy loss for 8-bit inference on an <a href="https://en.wikichip.org/wiki/intel/xeon_platinum/8124m">Intel Skylake CPU</a>, and <a href="https://doi.org/10.1145/3229762.3229766">another one based on TVM</a> on a Xilinx FPGA. In addition, a proof-of-concept integration of CK with <a href="https://aws.amazon.com/sagemaker/">AWS SageMaker</a> was developed and <a href="https://conferences.oreilly.com/artificial-intelligence/ai-eu-2018/public/schedule/detail/71549">presented</a> at the <a href="https://conferences.oreilly.com/artificial-intelligence/ai-eu-2018">O’Reilly AI conference in London</a>.</p><p>CK workflows naturally help vendors deploy optimized packages in the cloud, but it’s only the beginning. Importantly, running different CK-instrumented packages on different cloud instances generates a wealth of experimental performance data. Finally, predictive models and interactive visualizations distilled from the performance data can completely transform the marketplace experience for customers!</p><h3>What’s next?</h3><p>We are only at the beginning of a long-term journey, so we invite you to join our community effort to help bring order to the ML chaos!</p><p>For example, we are working on applying the above ReQuEST methodology and modular CK infrastructure to automate and unify the <a href="http://mlperf.org/">MLPerf benchmark</a>. Due to the broad community interest surrounding ML, we hope to reach a critical mass to create <a href="https://reuseresearch.com/components">open repositories of reusable ML/SW/HW components</a>. This would allow researchers to build upon such components in their reproducible and interactive papers while eventually solving the thorny issue of benchmarking techniques across the latest software and hardware. With over 30,000 ML papers <a href="https://medium.com/atlas-ml/state-of-deep-learning-h2-2018-review-cc3e490f1679">predicted</a> to be published in 2019 but only 1 in 7 to come with any means to reproduce them, the time is ripe to disrupt the status quo!</p><p>Finally, our approach is not limited to ML or traditional computing systems: we have successfully applied it in several <a href="http://cknowledge.org/quantum">reproducible competitions on quantum computing</a> while sharing code and data as reusable CK components.</p><p>Please <a href="http://cknowledge.org/contacts.html">get in touch</a> if you would like to know more and join our growing community!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9cb5d113130c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>