<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Punchh Technology Blog on Medium]]></title>
        <description><![CDATA[Stories by Punchh Technology Blog on Medium]]></description>
        <link>https://medium.com/@punchhtech?source=rss-cf87ad15791e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*48bxNmdfkRf0RcrYLgkCSQ.png</url>
            <title>Stories by Punchh Technology Blog on Medium</title>
            <link>https://medium.com/@punchhtech?source=rss-cf87ad15791e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 04:42:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@punchhtech/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Scaling new heights with Super Bowl-2021]]></title>
            <link>https://punchhtech.medium.com/scaling-new-heights-with-super-bowl-2021-c7c9ea33338?source=rss-cf87ad15791e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7c9ea33338</guid>
            <category><![CDATA[2021-super-bowl-sunday]]></category>
            <category><![CDATA[super-bowl]]></category>
            <dc:creator><![CDATA[Punchh Technology Blog]]></dc:creator>
            <pubDate>Mon, 08 Feb 2021 18:07:36 GMT</pubDate>
            <atom:updated>2021-02-09T17:40:55.749Z</atom:updated>
            <content:encoded><![CDATA[<h3>A busy Super Bowl Sunday despite COVID-19</h3><p>Authors: Sweta Gupta, Saloni Jain and Shruti Mishra(Feb 08, 2021)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/0*C3bdaD_H-1zFeLZ-" /><figcaption><a href="https://granthshala.com/a-collection-excellent-super-bowl-lv-food-deals/">Image Source</a></figcaption></figure><p>This year’s Super Bowl event saw Punchh platform register an increasing user engagement with brands. Looking at the same brand sales, there has been an uptick of 19.27% increase in transaction volume YoY compared to the Super Bowl Sunday in 2020.</p><p>What’s different this year though is the increased (but not surprising) engagement in online channel. Between 2019 and 2020, the increase in online transaction volume (looking at same brand sales) on Super Bowl events is 10.06%. In comparison, the increase in online transaction volume between 2020 and 2021 is 21.23%</p><p>In terms of no.of campaigns sent out in previous years compared to this year, it is an increase of 13.83%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*abXXHtcU9I3qooAH6PtSxw.png" /></figure><p>Another interesting insight is the shift in the communication channels that brands are engaging with users. While the no.of emails sent for 2021 Super Bowl has increased 133% YoY (incl. existing and newly joined brands on the platform), the no.of push notifications (sent through the APP developed by Punchh for the brands) has increase 250% YoY!</p><p>Similarly, the number of users reached out increased 85% YoY. Whereas the number of offers redeemed increased by 45%.</p><h3>About the Authors</h3><p><strong>Sweta Gupta </strong>is a Senior Data Analyst at Punchh.</p><p><strong>Saloni Jain and Shruti Mishra </strong>are a Data Analysts at Punchh .</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7c9ea33338" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Punchh Adopts DocumentDB with a Two-Fold Increase in Database Workload]]></title>
            <link>https://punchhtech.medium.com/punchh-adopts-documentdb-with-a-two-fold-increase-in-database-workload-a3fb1e2e85d7?source=rss-cf87ad15791e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3fb1e2e85d7</guid>
            <category><![CDATA[punchh]]></category>
            <category><![CDATA[production]]></category>
            <category><![CDATA[aws-documentdb]]></category>
            <dc:creator><![CDATA[Punchh Technology Blog]]></dc:creator>
            <pubDate>Sat, 28 Nov 2020 15:52:55 GMT</pubDate>
            <atom:updated>2020-11-28T15:55:40.195Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T0iVu4iNFz_Dzd-ckqlGUQ.png" /><figcaption>From punchh.com</figcaption></figure><p>In Punchh, we have chosen AWS’s Document DB technology to combat 2X traffic while reducing our production DB spending by 20%.</p><p>Bhanu Mohanty, our Chief Architect says, “AWS provided us with first-hand insight on the database services that were available to address the challenges we were facing. As an AWS Enterprise Support customer, we also had 24x7 technical support in finetuning the performance of our platform with Amazon DocumentDB.”</p><p>The full case study is below.</p><p><a href="https://aws.amazon.com/solutions/case-studies/punchh1/">Punchh Case Study</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3fb1e2e85d7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Punchh’s automation framework — Enhancing scale and quality of mobile app testing]]></title>
            <link>https://punchhtech.medium.com/punchhs-automation-framework-enhancing-scale-and-quality-of-mobile-app-testing-86c59658993?source=rss-cf87ad15791e------2</link>
            <guid isPermaLink="false">https://medium.com/p/86c59658993</guid>
            <category><![CDATA[qa]]></category>
            <category><![CDATA[mobile-app-development]]></category>
            <category><![CDATA[automation-testing]]></category>
            <dc:creator><![CDATA[Punchh Technology Blog]]></dc:creator>
            <pubDate>Mon, 06 Jul 2020 10:16:03 GMT</pubDate>
            <atom:updated>2020-07-06T10:16:03.034Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Punchh’s automation framework — Enhancing scale and quality of mobile app testing</strong></h3><p>Authors: Naveen Jangid, Asharam Yadav</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ktONOAPy2UfjPmRA" /></figure><p><strong>Introduction</strong></p><p>Automation testing is the process of executing test cases using automation testing tools and frameworks, and generating automated test reports which compare actual results with expected results. There are various test automation tools such as Appium, Calabash, Selenium, Cypress etc. and by using these tools one can build a framework to suit one’s scenarios and requirements. Automation testing improves test coverage, enhances testing speed, early detection improves quality of testing and scores high on re-usability, simultaneity (on multiple devices and mobile applications) of the testing process.</p><p>At Punchh, we have 150+ loyalty based mobile apps currently — and we are building mobile applications for new clients, while working on app updates for existing applications for feature enrichment. Moreover each mobile app has at least 70+ screens. Our mobile app QA team had been doing manual testing rigorously which has worked well, but with increasing scale, we found it more efficient and effective to automate a large part of the testing process. Automation testing enables quick turn around for the majority of the known test cases, providing enough time for critical testing and risk coverage. This article will give an overview of what we have done, and what we have achieved from our adoption of automation testing for mobile apps which touch user volumes of greater than 10% of the USA population per day.</p><p><strong>Benefits that we achieved</strong></p><p><strong><em>Efficiency</em></strong></p><p>Overall, we have been able to save time, increase the velocity of the testing process and enhance the quality. Through detailed automation reports, we have more effective QA feedback. Automation enables easy sanity testing, which is a positive round of testing for developers before handing over to QA that has generated value in the long run in the app delivery process.</p><p>The QA analysts are able to focus more on custom features, or those which need negative testing rather than the central features of our app which can be covered through the automation framework. Also, feedback is faster now — developers are aware of the issues for 60% of the scope through automated detection, and thus accelerated results and reduced costs. Automation has also aided in the regression testing process.</p><p><strong><em>Flexibility</em></strong></p><p>Automation has brought in huge efficiencies through reusability, flexibility and concurrent execution. Automation provides reusability — once scripted, test suites can be executed any number of times in every release, and can run for each device.</p><p>Automation provides us the flexibility to run tests at any time — By integrating the automation framework with Bitrise CI tool, execution can be triggered automatically (At a specific time or at the time of build release). Also, parallel device execution concurrently is possible through automation — Execution of test cases on multiple devices at the same time reduces QA time significantly.</p><p><strong>How we went about this</strong></p><p><strong><em>Tools &amp; Technologies</em></strong></p><p>We used the following tools and technologies primarily to develop the automation framework :</p><blockquote>Automation Tool and Framework — Appium with Java</blockquote><blockquote>Test Framework — testNG</blockquote><blockquote>CI tool — Bitrise</blockquote><ul><li><strong><em>Why we choose Appium</em></strong></li></ul><p>Appium supports multiple frameworks and programming languages. Being an open source framework, it enables development of the framework as per our requirements. It supports native, web and hybrid applications, and also allows cross-platform application testing (across Android and iOS).</p><p>Also, it doesn’t need app source code for running the tests. And doesn’t require modifications in the application in order to automate it.</p><p><strong><em>Process</em></strong></p><p>Our list of test cases (around 600 in TestCollab) were translated to automation test scripts. Around 150 end to end automation test cases (all based on our major module — loyalty) cover almost 400 manual test cases of TestCollab, related to sanity coverage, device coverage and regression — These are mostly functional ones, with scope of UI testing limited to checking presence of element (like verbiage and alerts). In one automation test class, we have combined several manual test cases and also several permutations and combinations of user inputs (Appendix A shows an example).</p><p>When any mobile app comes to QA with even a small change, both regression testing and device coverage testing has to be done. Automation testing helps with functional testing, regression testing and device coverage. Not just mobile app testing, but replication of server side dashboard operations such as barcode generation, fetching Whitelabel data from our server side is done through the automation scripts. We run these for both new app launches and big app updates.</p><p><strong><em>Execution aspects</em></strong></p><p>Test framework <strong>configurations</strong> are essential to trigger the tests appropriately — Among the configurations, mobile app related (app name,package, bundle ID) and device related (device name, OS, platform) are primary. Having a diverse set of device configurations (minimum 3 platforms of iOS and 5 platforms for Android, various device types) ensures we cover a large spectrum of mobile devices.</p><p>We use separate classes (loosely coupled) for each testing module — in the testng.xml, all the modules which are to be executed are listed. Automation cases are managed on the basis of <strong>Tags </strong>(Annotations) — the tags used are Sanity, Smoke and Functional. Whenever a tag is chosen, only test cases pertaining to the tag are executed. (Code snippets are shared in the appendix A below for illustration — For each class, the corresponding tags are mentioned)</p><ol><li>Sanity Tag — Sanity testing is a subset of regression testing. It is performed to ensure that the code changes that are made are working properly, at a broad level. Both functional and UI cases are included in the sanity tag.</li><li>Smoke Tag — We include the critical cases that check the stability of the build, and these act as a gate to proceed for further testing.</li><li>Functional Tag — Includes only functional test cases.</li></ol><p>By using tags we are able to trigger the appropriate set of test cases through automation, in phases of testing like Build Acceptance, Functional Testing, Regression and Retesting.</p><p>As the tests run, the screens on the simulator keep moving correspondingly with a screenshot of the automated step and when errors are found, details are shown (for ready reference as shown in Appendix B.4).</p><p><strong><em>LifeCycle of automation testing</em></strong></p><p>The implementation of Automation Life Cycle executes in parallel with the Software Development Life Cycle(SDLC) process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5TtK4IyMKI386iQ3" /></figure><p><strong>Appendix A</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Sqkbv3D9939KFWuh" /></figure><p><strong>Appendix B</strong></p><p><strong>B.1</strong> Execution report screenshots (on the console)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ATjv9VhjE9D53kUS" /></figure><p><strong>B.2</strong> Pass test case steps screenshot</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JNPOjHA0QTJD-GUK" /></figure><p><strong>B.3</strong> Failure test case steps screenshot</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hh5eKPYbRDc99GDy" /></figure><p><strong>B.4</strong> App screenshot is also automatically captured in case of test case failure and adds the cause of failure at the bottom of screenshot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Oz-fw9atkAwVNey0" /></figure><h3>About the Authors</h3><p><strong>Naveen Jangid</strong> leads the automation testing initiative for the mobile delivery team at Punchh.</p><p><strong>Asharam Yadav</strong> leads the mobile QA team at Punchh.</p><p>The authors would like to thank Praveen Pandey (Head of Delivery), Narendra Verma (Engineering Manager, Delivery) and Arun Ginjala (Chief of Staff, Engineering) for their guidance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86c59658993" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Punchh’s Campaign Engine is at Scale]]></title>
            <link>https://punchhtech.medium.com/punchhs-campaign-engine-is-at-scale-c7e274b033d5?source=rss-cf87ad15791e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7e274b033d5</guid>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[omnichannel]]></category>
            <category><![CDATA[restaurant]]></category>
            <category><![CDATA[cinco-de-mayo]]></category>
            <category><![CDATA[loyalty-marketing]]></category>
            <dc:creator><![CDATA[Punchh Technology Blog]]></dc:creator>
            <pubDate>Wed, 06 May 2020 19:19:29 GMT</pubDate>
            <atom:updated>2020-05-20T10:09:34.920Z</atom:updated>
            <content:encoded><![CDATA[<h3>A busy Cinco de Mayo despite COVID-19</h3><p>Authors: Arun Ginjala, Shubham Goyal, Amit Suroliya (May 06, 2020)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6ztgljO9QjNUZcTF" /><figcaption><a href="https://www.shopadvisor.com/cinco-de-mayo-overcoming-the-odds-in-your-marketing-campaign/">Image source</a></figcaption></figure><p><strong>Mass Campaigns: Reliability and Performance at Scale</strong></p><p>The number of business clients whom Punchh serves and the quantum of individual customers targeted using our platform has grown exponentially over the past two years. In 2019, using our campaign engine, our clients have targeted 100 million+ consumers and 5 billion+ times. On Cinco De Mayo 2020, 42 million+ consumers (13% of the US population) were targeted using our mass campaign engine. The Punchh platform has thus enabled our clients to engage with their customers and increase their sales on this day amidst COVID-19, emblematic of the triumphant spirit that Cinco de Mayo stands for.</p><p>One of the major objectives of the engineering team at Punchh is to scale our application and infrastructure, without slowing down campaign delivery during peak loads. Campaigns are used by Punchh’s clients to target their customers via emails, push notifications, or in-app messages and inform them about offers. The term ‘mass campaign’ refers to a large scale. In addition, Punchh’s ongoing campaigns are triggered only for customers on special occasions: birthday, anniversary, membership milestones, to name a few.</p><p>To give a sense of scale: in April 2020, the Punchh platform has delivered nearly 5.5K mass campaigns targeting 920+ million consumers, which amounts to 430 million emails and 110+ million push notifications. The Cinco de Mayo saw further increases in loads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wbqeIb9meP7xJWrp" /><figcaption>Cinco de Mayo day benchmarking with an Avg day in April</figcaption></figure><p><strong>Performance at Scale</strong></p><p>70% of the mass campaigns in April were completed under 10 minutes, 86% under 30 mins, and 94% under 1 hour. Even on Cinco De Mayo, with an elevated load of 43 million consumers (a 200% increase in guests targeted over the average daily load in 2019), 286 mass campaigns were successfully executed with 44% of them completed under 10 minutes, 78% under 30 mins, and 98% under 1 hour. In addition, what was even more challenging was that most of the mass campaigns had to be delivered at almost the same time, within several hours’ time span.</p><p>On our multi-tenant stack which serves our customers, we have handled massive peak throughputs of up to 6 million+ per hour reliably. We have achieved this through:</p><ul><li>Augmented monitoring and reporting to ensure that during elevated volumes, API response times are stable with either no timeouts or smart handling of timeouts — the number of campaign workers and containers are dynamically scaled up when needed.</li><li>Setting up systems and processes with real-time alerts, anomaly detection, health monitoring systems, and on-call SRE teams, to ensure campaign delivery reliability.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SqYGsYgYD6r_GEl7" /></figure><p><strong>Steps that are Taken to Fortify and Future-proof Our System</strong></p><ul><li>Instituted a process to initiate multiple tests (or drills) before large-scale events with simulated loads to stress test systems where necessary.</li><li>Complex segment (permutations and combinations of segments using set operations on arrays) evaluation had been modified to scale to evaluate larger segments using Redis — this had worked well but introduced new problems relating to Redis scalability. We had then moved to the next stage of this using Go, which will allow more segments to be evaluated in parallel and quickly, with less moving parts.</li><li>Segment architecture is being overhauled to scale to perform more dynamic segmentation and on larger sets of data. We continue to evolve the underlying storage layer to enable us to do this at scale that are alternatives to our Snowflake warehouse.</li><li>Campaign services are also being re-architected with database sharding, campaign batch processing, usage of NoSQL database, and regular archival of old data, through which we believe we could multiply our campaign throughput two-fold.</li></ul><p>From a product perspective, we are driving deeper campaign personalization with AI products that help our customers create data-driven segments along with a prediction of the most appropriate omnichannel offers across all the engagement touchpoints. Furthermore, Punchh is building a data lake — a centralized data repository of all transactions, application logs, user mobile data, etc.</p><h3>About the Authors</h3><p><strong>Arun Ginjala</strong> is the Chief of Staff of the engineering team at Punchh. He is assisting the CTO and the engineering leaders on tracking all programs and operations, and steering strategic initiatives.</p><p><strong>Amit Suroliya</strong> leads the production support team at Punchh. His team ensures reliable campaign delivery of 24/7 and takes care of all production issues.</p><p><strong>Shubham Goyal</strong> is the Senior Analyst of the Business Intelligence team. He interacts closely with internal stakeholders to understand the reporting needs and then to collect requirements, design, and build BI solutions.</p><blockquote>The authors would like to thank Aditya Sanghi (Co-founder and CTO) and Xin Heng (VP, Data) for their guidance and support.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7e274b033d5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[COVID-19 Impact Analysis Series Article Six: Which cuisines are capturing higher order values?]]></title>
            <link>https://punchhtech.medium.com/covid-19-impact-analysis-series-article-six-which-cuisines-are-capturing-higher-order-values-7d847a6c3488?source=rss-cf87ad15791e------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d847a6c3488</guid>
            <category><![CDATA[loyalty-marketing]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[pizza]]></category>
            <category><![CDATA[coronavirus]]></category>
            <category><![CDATA[restaurant-analytics]]></category>
            <dc:creator><![CDATA[Punchh Technology Blog]]></dc:creator>
            <pubDate>Fri, 24 Apr 2020 08:32:26 GMT</pubDate>
            <atom:updated>2020-04-24T10:09:08.122Z</atom:updated>
            <content:encoded><![CDATA[<h3>Author: Shubham Goyal (April 24, 2020)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Nbjutpnr1psW0-lO" /></figure><p><strong>THIS COVID-19 Impact Analysis Series designed by Punchh’s data team aims at sharing the real-time insight we have discovered when we study our consumer behavior data set.</strong></p><p><strong>In this sixth article of the series, we highlighted how some cuisine types are able to offset the steep drop in transactions by capturing higher value in each order (in other words, higher Average Order Value a.k.a AOV)</strong></p><p>Due to the stay home directives, people commuting daily to the work has declined significantly. As a result, transaction volume and sales have declined by up to 50% within 6 weeks, however, AOV has increased a little over 10% last week compared to the first week of the pandemic, albeit there was a decline by ~5% in the first two weeks of the pandemic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XnidBaJa3OlXAgrK" /><figcaption><strong>Weekly % change in transactions, sales &amp; AOV from the first week of the pandemic across industry</strong></figcaption></figure><p><strong>So, what is the AOV trend over weeks for different cuisine types?</strong></p><p>To answer this we have compared the AOV trend across the top 5 cuisines based on the total transactions in each cuisine. Together they have contributed &gt;80% to total transactions during the last two weeks in our data set.</p><p>The chart indicates that the Pizza category has gained the most followed by beverages and Mexican. For both the categories, most of the brands are seeing AOV growth of 4–5% weekly after the first half of March.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LZHGXYYWlq8KH5V8" /><figcaption><strong>Weekly % change in AOV from the first week of the pandemic by cuisine type</strong></figcaption></figure><p><strong>Ok, so why are Pizza and Beverage outperforming other cuisine types?</strong></p><p>To understand this we have examined the distribution of transactions by AOV. One common trend that comes out from the chart below is that the % of transactions of &lt;$10 has decreased across cuisine types. On the other hand, Pizza and Beverage were able to capture a higher proportion of transactions coming from higher AOV transactions. Pizza has gained 8% in the proportion of transactions above $30 in a matter of 4 weeks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QJY0IpLOhrb2d9xr" /><figcaption><strong>Distribution of transactions by AOV by cuisine type</strong></figcaption></figure><p><strong>Wait, how does it reflect in trends of transactions and sales?</strong></p><p>So coming back to the sales and transactions trend, we have plotted the chart for all the pizza brands we have in our data set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PmXjySn6Wl4M07oa" /><figcaption><strong>Weekly % change in transactions, sales from the first week of the pandemic for pizza cuisine</strong></figcaption></figure><p>Due to a bigger shift in their proportion of business coming from higher AOV ( &gt;$30) even the sales started showing an upward trend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wVUr0slJAyYQ6iyW" /></figure><p><strong>KEY TAKEAWAYS</strong></p><ul><li>Considering the shift in consumption behavior of certain cuisine types, brands should focus on creating more bundled offers. This can help offset drop in visits and the resultant drop in sales to a certain extent.</li><li>For brands with mixed menu items (from several cuisines), the focus should be on creating bundled offers with items from one of the 3 cuisine types — Pizza, Mexican and Beverage</li></ul><p><strong>ABOUT THE DATA</strong></p><p>The underlying data for this analysis is from 20K+ restaurant stores across the country covering 100+ brands for the period of Mar 09 — Apr 19.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d847a6c3488" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[COVID-19 Impact Analysis Series Article Five: Shift in Consumer Food Habits]]></title>
            <link>https://punchhtech.medium.com/covid-19-impact-analysis-series-article-five-shift-in-consumer-food-habits-b8b422319796?source=rss-cf87ad15791e------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8b422319796</guid>
            <category><![CDATA[trends]]></category>
            <category><![CDATA[loyalty-program]]></category>
            <category><![CDATA[consumer-food-habits]]></category>
            <category><![CDATA[restaurant-business]]></category>
            <category><![CDATA[covid19]]></category>
            <dc:creator><![CDATA[Punchh Technology Blog]]></dc:creator>
            <pubDate>Wed, 15 Apr 2020 06:18:50 GMT</pubDate>
            <atom:updated>2020-04-15T06:18:50.493Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Which type of food are people consuming the most?</strong></p><p>Author: Bhavika Sharma [Analyst], Phylis Savari [Data Eng.](April 14, 2020)</p><p><strong>THIS COVID-19 impact analysis series designed by Punchh’s data team aims at sharing the real-time insight we have discovered when we study our consumer behavior data set. In this fifth article, we analyzed how different food categories are being consumed, and provided a holistic view of the shift in consumer food habits. In addition, we show how different cuisine types are faring post the outbreak.</strong></p><p>Over the past few weeks, there has been a significant shift in the way consumers are procuring (delivery orders, drive-thru, to-go orders) and consuming food. In one of our previous <a href="https://medium.com/@punchhtech/covid-19-impact-analysis-series-article-one-behavioral-changes-in-food-preferences-2a66be52db79">articles</a>, when the COVID-19 outbreak became apparent in Washington and California states at the beginning of March, we showed how consumers’ preference towards a specific item -Salads- changed pre- and post-outbreak periods. With the outbreak spreading to the rest of the country, we discovered that few food categories did better than others.</p><p><strong>Meat is the least impacted food category, while dessert is one of the most affected</strong></p><p>One major observation that came out from the study was that meat (incl. pork, beef, and chicken) seems to be the least impacted amongst all the food categories in terms of a number of orders made. Once the overall orders for restaurants started declining after President Trump declared National Emergency <strong>(13th March), </strong>meat orders did not slump as much (-23%) as the total orders (-27%) and other food categories. The below graph highlights this trend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Klj3t9RNnjU9mwuF1cf2gA.png" /><figcaption>Graphs comparing items ordered for a specific category and all food items. Data is scaled for privacy reasons</figcaption></figure><p>Among meat-based orders, pork items, interestingly, saw the least decline (-18%).</p><p>On the other hand, desserts and alcoholic beverages as food item categories suffered the biggest declines (-32% and -45% respectively). The below graph highlights this trend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gl9yojRc8nE_RoKOeu8prg.png" /><figcaption>Data is scaled for privacy reasons</figcaption></figure><p>The percentage declines for the above four categories are summarised in the below table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*Z3jf40Bk9hV8QRG1tD1FNg.png" /><figcaption>Time period considered is Feb-01 to Feb-29 vs. Mar-01 to Mar-29</figcaption></figure><p><strong>Mexican and Burgers are the least impacted cuisines, while Asian cuisine is the most affected</strong></p><p>In terms of cuisine types, Mexican, Burger, and Italian are doing better (-20%, -21% and -22%) while Asian cuisine type is the most affected (-42%). Below chart depicts these trends</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YIcAGNuk_Z7zV7SLcsLRvg.png" /><figcaption>The time period considered is Feb01 to Feb29 vs. Mar01 to Mar29</figcaption></figure><p>While desserts as a food category declined more than total orders, ice cream as a cuisine type did better than the total orders.</p><p><strong>KEY TAKEAWAYS:</strong></p><ul><li>The trends prove that meat is a key ingredient in the American diet. In specific, pork is being heavily relied upon. On the other hand, the venue for the purchase of alcoholic beverages has moved away from restaurants, possibly because of a decline in dine-in orders. Hence restaurants need to plan the availability of their inventory accordingly</li><li>Given that Mexican and Italian cuisines are being consumed comparatively more, brands with mixed menu might benefit from highlighting these options for consumers</li></ul><p><strong>ABOUT THE DATA</strong></p><p>The underlying data for this analysis is from 10K+ restaurant stores across the country covering 60 brands for the period of 2020-Feb 01–2020-Apr 10.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8b422319796" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Seamless migration of machine learning codebase from the local or EC2 to Databricks PySpark…]]></title>
            <link>https://punchhtech.medium.com/seamless-migration-of-machine-learning-codebase-from-the-local-or-ec2-to-databricks-pyspark-eba5ae7270d6?source=rss-cf87ad15791e------2</link>
            <guid isPermaLink="false">https://medium.com/p/eba5ae7270d6</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[pyspark]]></category>
            <dc:creator><![CDATA[Punchh Technology Blog]]></dc:creator>
            <pubDate>Fri, 10 Apr 2020 06:44:21 GMT</pubDate>
            <atom:updated>2020-04-10T17:45:12.016Z</atom:updated>
            <content:encoded><![CDATA[<h3>Seamless migration of machine learning codebase from the local or EC2 to Databricks PySpark Clusters(Part IV: Job clusters API)</h3><p>Author: Tian Lan</p><p>The job cluster is created ad-hoc when you run a job and the cluster is terminated when the job is complete. You <em>cannot</em> restart a job cluster. While the all-purpose cluster is great for development, testing, and coding collaboration, the one-time job cluster is significantly cheaper to run than an all-purpose cluster. Therefore the job cluster is especially suitable for production jobs.</p><p>To review the previous sections of this topic, here is the Table of Contents for the related blogs:</p><ol><li>Write the Spark User-Defined Function and codebase adaptor (<a href="https://medium.com/@punchhtech/how-to-seamlessly-migrate-a-machine-learning-codebase-and-cicd-pipeline-running-python-code-e-g-ecc16827d6d0"><strong>Part I link</strong></a>)</li><li>Manage Databricks all-purpose clusters via CLI and API (<a href="https://medium.com/@punchhtech/how-to-seamlessly-migrate-a-machine-learning-codebase-running-python-code-e-g-d35e6f7f5bfa"><strong>Part II link</strong></a>)</li><li>Build, synchronize and submit job scripts at local (<a href="https://medium.com/@punchhtech/seamless-migration-of-machine-learning-codebase-from-the-local-or-ec2-to-databricks-pyspark-d213ac37db4b"><strong>Part III link</strong></a>)</li><li>Manage Databricks job clusters exclusively via CLI and API (<strong>Part IV</strong>)</li></ol><p>Compared with the all-purpose cluster, managing a job cluster and submitting jobs is actually a bit simpler. You do not need to worry about the library conflict issue from the previously installed packages because the cluster is new, so you do not need to spend time detecting or uninstalling any package. You can embed your library installation request inside the job submission. As for job creation, there is no difference at all.</p><p><strong>Package the Codebase and Copy to DBFS</strong></p><p>So the first step is packaging the machine learning codebase as we did before, then we just push them to DBFS</p><p><strong><em>update_wheel.sh</em></strong></p><pre>whl_file={modulename}-${version}-py3-none-any.whl<br>rm -rf dist<br>python setup.py sdist bdist_wheel<br>dbfs cp --overwrite ./dist/${whl_file} ${dbfs_dir}</pre><p><strong>New Cluster Setup</strong></p><p>We need to have a configure so that Databricks knows what kind of job cluster you want to use and a few essential settings including AWS attributes. You also need to specify what packages you need to install (such as PyPI packages and wheel file you just uploaded). Here I wrote a <em>dblibrary</em> module to manage it, essentially it just returns all required libraries in a JSON format following the Databricks API convention. That’s it, the rest of the code is the same as <em>executenotebook_allpurpose_cluster.sh</em></p><p><strong><em>executenotebook_job_cluster.py</em></strong></p><pre><em>#!/usr/bin/python3<br></em>import json<br>import requests<br>import os<br>import time<br>from spark_runner import dblibrary<br>from spark_common.utils import get_security_value<br></pre><pre># the bold highlight is the Json configure for Job cluster<br><strong>CFG = {<br>    &quot;new_cluster&quot;: {<br>        &quot;spark_version&quot;: &quot;6.4.x-scala2.11&quot;,<br>        &quot;node_type_id&quot;: &quot;i3.xlarge&quot;,<br>        &quot;aws_attributes&quot;: {<br>            &quot;availability&quot;: &quot;SPOT_WITH_FALLBACK&quot;,<br>            &quot;instance_profile_arn&quot;: &quot;&quot;<br>        },<br>        &quot;num_workers&quot;: 2<br>    },<br>    &quot;spark_conf&quot;: {<br>        &quot;spark.sql.crossJoin.enabled&quot;: &quot;true&quot;,<br>        &quot;spark.databricks.service.server.enabled&quot;: &quot;true&quot;,<br>    },<br>    &quot;libraries&quot;: dblibrary.CFG[&quot;libraries&quot;]<br>}</strong><br><br><br>def main(user=&quot;&quot;,<br>         token_file=&quot;&quot;,<br>         node_type=None,<br>         num_workers=None,<br>         workspacepath=&quot;&quot;,<br>         outfilepath=&quot;&quot;):<br><br>    if node_type is not None:<br>        CFG[&quot;node_type_id&quot;] = node_type<br>    if num_workers is not None:<br>        CFG[&quot;num_workers&quot;] = num_workers<br>    CFG[&quot;new_cluster&quot;][&quot;aws_attributes&quot;][&quot;instance_profile_arn&quot;] = get_security_value(token_file, user, &quot;iam_role&quot;)<br>    server = get_security_value(token_file, user, &quot;server&quot;)<br><br>    <em># Generate array from walking local path<br>    </em>localpath = &quot;%s/run&quot; % os.path.dirname(os.path.realpath(__file__))<br><br>    token = json.load(open(token_file, &quot;r&quot;))[user][&quot;token&quot;]<br>    notebooks = []<br>    for path, subdirs, files in os.walk(localpath):<br>        for name in files:<br>            fullpath = path + &#39;/&#39; + name<br>            <em># removes localpath to repo but keeps workspace path<br>            </em>fullworkspacepath = workspacepath + path.replace(localpath, &#39;&#39;)<br><br>            name, file_extension = os.path.splitext(fullpath)<br>            if file_extension.lower() in [&#39;.scala&#39;, &#39;.sql&#39;, &#39;.r&#39;, &#39;.py&#39;]:<br>                row = [fullpath, fullworkspacepath, 1]<br>                notebooks.append(row)<br><br>    <em># run each element in list<br>    </em>for notebook in notebooks:<br>        nameonly = os.path.basename(notebook[0])<br>        workspacepath = notebook[1]<br><br>        name, file_extension = os.path.splitext(nameonly)<br><br>        <em># workpath removes extension<br>        </em>fullworkspacepath = workspacepath + &#39;/&#39; + name<br><br>        print(&#39;Running job for:&#39; + fullworkspacepath)<br>        timeout = 3600<br>        values = {&#39;run_name&#39;: name, &#39;timeout_seconds&#39;: timeout, &#39;notebook_task&#39;: {&#39;notebook_path&#39;: fullworkspacepath}}<br>        CFG.update(values)<br>        resp = requests.post(server + &#39;/api/2.0/jobs/runs/submit&#39;,<br>                             data=json.dumps(CFG),<br>                             auth=(&quot;token&quot;, token)<br>                             )<br>        print(server + &#39;/api/2.0/jobs/runs/submit&#39;)<br>        print(CFG)<br>        runjson = resp.text<br>        print(&quot;runjson:&quot; + runjson)<br>        d = json.loads(runjson)<br>        runid = d[&#39;run_id&#39;]<br><br>        i=0<br>        waiting = True<br>        while waiting:<br>            time.sleep(10)<br>            jobresp = requests.get(server + &#39;/api/2.0/jobs/runs/get?run_id=&#39;+str(runid),<br>                                   data=json.dumps(values),<br>                                   auth=(&quot;token&quot;, token)<br>                                   )<br>            jobjson = jobresp.text<br>            print(&quot;jobjson:&quot; + jobjson)<br>            j = json.loads(jobjson)<br>            current_state = j[&#39;state&#39;][&#39;life_cycle_state&#39;]<br>            runid = j[&#39;run_id&#39;]<br>            if current_state in [&#39;TERMINATED&#39;, &#39;INTERNAL_ERROR&#39;, &#39;SKIPPED&#39;] or i &gt;= timeout/10:<br>                break<br>            i=i+1<br><br>        if outfilepath != &#39;&#39;:<br>            file = open(outfilepath + &#39;/&#39; +  str(runid) + &#39;.json&#39;, &#39;w&#39;)<br>            file.write(json.dumps(j))<br>            file.close()<br><br><br>if __name__ == &#39;__main__&#39;:<br>    import argparse<br><br>    parser = argparse.ArgumentParser()<br>    parser.add_argument(&quot;--token_file&quot;)<br>    parser.add_argument(&quot;--user&quot;)<br>    parser.add_argument(&quot;--node_type&quot;, help=&quot;node type, such as i3.xlarge&quot;)<br>    parser.add_argument(&quot;--num_workers&quot;, help= &quot;num of workers&quot;, type=int)<br>    parser.add_argument(&quot;--workspacepath&quot;, help=&quot;workspacepath&quot;)<br>    parser.add_argument(&quot;--outfilepath&quot;, default=&quot;%s/joblogs&quot; % os.path.dirname(os.path.realpath(__file__)), help=&quot;outfilepath for writing response&quot;)<br>    args = parser.parse_args()<br>    args = vars(args)  <em># convert to dict<br><br>    </em>main(**args)</pre><p>The following commands finish the entire workflow from job creation, cluster management, and job submission. Here <em>run_job_cluster.sh </em>calls the <em>executenotebook_job_cluster.py, </em>and<em> </em>the rest have been discussed in previous blogs already. Again, we see that a user only needs to interact with <em>create_notebooks.sh</em> to create and register jobs as he/she should always do. With all other components entirely hidden, the user only needs to call the ML codebase at local and gets the job done at the remote cluster.</p><pre>execute-adhoc-jobs:<br>        bash sh/create_notebooks.sh<br>        bash sh/workspace.sh<br>        bash sh/update_wheel.sh<br>        bash sh/run_job_cluster.sh<br>        bash sh/clear_localnotebooks.sh</pre><p>And you should get the following response log (I have masked sensitive information). At the moment when the job finished, the cluster will be immediately terminated.</p><pre><strong>jobjson:{&quot;job_id&quot;:4318,&quot;run_id&quot;:11641,&quot;number_in_job&quot;:1,&quot;state&quot;:{&quot;life_cycle_state&quot;:&quot;PENDING&quot;,&quot;state_message&quot;:&quot;Waiting for cluster&quot;},&quot;task&quot;:{&quot;notebook_task&quot;:{&quot;notebook_path&quot;:&quot;XXXXXX&quot;}},&quot;cluster_spec&quot;:{&quot;new_cluster&quot;:{&quot;spark_version&quot;:&quot;6.4.x-scala2.11&quot;,&quot;aws_attributes&quot;:{&quot;zone_id&quot;:&quot;us-east-1e&quot;,&quot;availability&quot;:&quot;SPOT_WITH_FALLBACK&quot;,&quot;instance_profile_arn&quot;:&quot;XXXXXXXXXX&quot;},&quot;node_type_id&quot;:&quot;i3.xlarge&quot;,&quot;enable_elastic_disk&quot;:false,&quot;num_workers&quot;:2},&quot;libraries&quot;:[{&quot;pypi&quot;:{&quot;package&quot;:&quot;scikit-learn==0.22.1&quot;}},{&quot;pypi&quot;:{&quot;package&quot;:&quot;s3fs&quot;}},{&quot;pypi&quot;:{&quot;package&quot;:&quot;pymysql&quot;}},{&quot;pypi&quot;:{&quot;package&quot;:&quot;scipy&quot;}},{&quot;pypi&quot;:{&quot;package&quot;:&quot;pandas&quot;}},.......]},&quot;cluster_instance&quot;:{&quot;cluster_id&quot;:&quot;XXXXX&quot;},&quot;start_time&quot;:1586500427510,&quot;setup_duration&quot;:0,&quot;execution_duration&quot;:0,&quot;cleanup_duration&quot;:0,&quot;creator_user_name&quot;:&quot;XXXXXX&quot;,&quot;run_name&quot;:&quot;XXXXXXXXXX&quot;,&quot;run_page_url&quot;:&quot;XXXXXX&quot;,&quot;run_type&quot;:&quot;SUBMIT_RUN&quot;}</strong></pre><pre><strong>......<br>......</strong></pre><pre><strong>jobjson:{&quot;job_id&quot;:4318,&quot;run_id&quot;:11641,&quot;number_in_job&quot;:1,&quot;state&quot;:{&quot;life_cycle_state&quot;:&quot;RUNNING&quot;,&quot;state_message&quot;:&quot;In run&quot;}</strong></pre><pre><strong>......</strong></pre><pre><strong>jobjson:{&quot;job_id&quot;:4318,&quot;run_id&quot;:11641,&quot;number_in_job&quot;:1,&quot;state&quot;:{&quot;life_cycle_state&quot;:&quot;TERMINATED&quot;,&quot;result_state&quot;:&quot;SUCCESS&quot;,&quot;state_message&quot;:&quot;&quot;}</strong></pre><h3>About Punchh</h3><p>Headquartered in San Mateo, CA, Punchh is the world leader in innovative digital marketing products for brick and mortar retailers, combining AI and machine learning technologies, mobile-first expertise, and Omni-Channel communications designed to dramatically increase lifetime customer value. Leading global chains in the restaurant, health and beauty sectors rely on Punchh to grow revenue by building customer relationships at every stage, from anonymous, to known, to brand loyalists, including more than 100 different chains representing more than $12 billion in annual spend.</p><h3>About the Author</h3><p><strong>Dr. Tian Lan </strong>is Tech Lead of A.I. at Punchh, where he leads the development of large-scale and distributed machine learning for recommender systems and personalized marketing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eba5ae7270d6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Seamless migration of machine learning codebase from the local or EC2 to Databricks PySpark…]]></title>
            <link>https://punchhtech.medium.com/seamless-migration-of-machine-learning-codebase-from-the-local-or-ec2-to-databricks-pyspark-d213ac37db4b?source=rss-cf87ad15791e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d213ac37db4b</guid>
            <category><![CDATA[api]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Punchh Technology Blog]]></dc:creator>
            <pubDate>Thu, 09 Apr 2020 23:35:04 GMT</pubDate>
            <atom:updated>2020-04-10T17:28:46.501Z</atom:updated>
            <content:encoded><![CDATA[<h3>Seamless migration of machine learning codebase from the local or EC2 to Databricks PySpark Clusters(Part III: Job management for all-purpose clusters via CLI and API)</h3><p>Author: Tian Lan</p><p>After you transformed your codebase to a PySpark adaptable and prepared an all-purpose cluster, it is time to create and submit your job for development and testing.</p><p>To review the previous sections of this topic, here is the Table of Contents for the related blogs:</p><ol><li>Write the Spark User-Defined Function and codebase adaptor (<a href="https://medium.com/@punchhtech/how-to-seamlessly-migrate-a-machine-learning-codebase-and-cicd-pipeline-running-python-code-e-g-ecc16827d6d0"><strong>Part I link</strong></a>)</li><li>Manage Databricks all-purpose clusters via CLI and API (<a href="https://medium.com/@punchhtech/how-to-seamlessly-migrate-a-machine-learning-codebase-running-python-code-e-g-d35e6f7f5bfa"><strong>Part II link</strong></a>)</li><li>Build, synchronize and submit job scripts at local (<strong>Part III</strong>)</li><li>Manage Databricks job clusters exclusively via CLI and API (<a href="https://medium.com/@punchhtech/seamless-migration-of-machine-learning-codebase-from-the-local-or-ec2-to-databricks-pyspark-eba5ae7270d6"><strong>Part IV link</strong></a>)</li></ol><p><strong>Create a Spark Notebook locally</strong></p><p>If you are familiar with Databricks UI, you may notice that their development environment is heavily relying on the notebook-style coding. A notebook is a fantastic development interface because it is highly interactive, however, if you are talking about submitting a production job, this could be quite inconvenient. For example, it is difficult for a notebook to accept runtime arguments like a Python script could do otherwise. You can easily write a Python script to accept runtime arguments like this:</p><p><strong><em>create_job.py</em></strong></p><pre>from spark_mypythonML.spark_data_gen import main</pre><pre>main(business_id=business_id,<br>     date_range=date_range,<br>     out_dir=out_dir)</pre><pre>if __name__ == &#39;__main__&#39;:<br>    import argparse<br><br>    parser = argparse.ArgumentParser()<br>    parser.add_argument(&quot;--business_id&quot;)<br>    parser.add_argument(&quot;--date_range&quot;)<br>    parser.add_argument(&quot;--out_dir&quot;)<br>    args = parser.parse_args()<br>    args = vars(args)<em><br>    </em>main(**args)</pre><pre># you can call the job like<br># python create_job.py --business_id=12345 --date_range=&quot;2019-01-01,2020-01-01&quot; --out_dir XXXXX</pre><p>When you are creating the job for submission, in this case, a notebook, it would be a huge inconvenience if you cannot easily assign values to the variables, for example, if you have 1000 businesses to run, you may not want to create 1000 copies of the above codes and manually change to different business_id.</p><p>To make the notebook accept a realtime argument, I used the following trick to mimic the argument parser. First, I have a notebook job template that specifies the placeholders for the runtime variables:</p><p><strong><em>run_template_data_gen.py</em></strong></p><pre>from spark_mypythonML.spark_data_gen import main</pre><pre>business_id=BUSINESS_ID<br>date_range=&quot;DATE_RANGE&quot;<br>out_dir=&quot;OUT_DIR&quot;</pre><pre>main(business_id=int(business_id),<br>     date_range=date_range,<br>     out_dir=out_dir)</pre><p>Then, when you create the job, you have a companion code to replace the template notebook with your desired values at the runtime:</p><p><strong><em>create_job_from_template.py</em></strong></p><pre>import os<br>import argparse</pre><pre>MOUNT_NAME = &quot;XXXXXXX&quot;</pre><pre>parser = argparse.ArgumentParser()<br>parser.add_argument(&quot;--business_id&quot;)<br>parser.add_argument(&quot;--date_range&quot;)<br>parser.add_argument(&quot;--out_dir&quot;, default=&quot;/mnt/%s/spark_features&quot; % MOUNT_NAME)</pre><pre>args = parser.parse_args()</pre><pre>template_dir = &quot;%s/run_templates&quot;  % os.path.dirname(os.path.realpath(__file__))<br>run_dir = &quot;%s/run&quot; % os.path.abspath(os.path.join(__file__ ,&quot;../..&quot;))</pre><pre>inputfile = &quot;%s/run_template_data_gen.py&quot; % template_dir<br>outputfile = &quot;%s/%s&quot; % (run_dir, args.output_filename)</pre><pre>with open(inputfile) as f:<br>    file_str = f.read()</pre><pre>file_str_replaced = file_str.replace(&quot;BUSINESS_ID&quot;, args.business_id).\<br>    replace(&quot;DATE_RANGE&quot;, args.date_range).\<br>    replace(&quot;OUT_DIR&quot;, args.out_dir)</pre><pre>with open(outputfile, &quot;w&quot;) as f:<br>    f.write(file_str_replaced)</pre><p>Now, on the surface, creating a job script looks exactly the same as the old <em>create_job.py.</em> In the end, the following script <em>create_notebooks.sh</em> is the only portion that an end-user needs to actively take care of, for example, he/she can add as many jobs as he/she likes.</p><p><strong><em>create_notebooks.sh</em></strong></p><pre>python spark_runner/create/<strong>create_job_from_template.py</strong> --output_filename spark_data_gen_12345.py --business_id=12345 --date_range=&quot;2019-01-01,2020-01-01&quot;</pre><p><strong>Synchronize Spark Notebook with the Remote Cluster</strong></p><p>A Databricks Workspace is an environment for accessing all of your Databricks assets. The Workspace organizes objects (<a href="https://docs.databricks.com/workspace/workspace-assets.html#ws-notebooks">notebooks</a>, <a href="https://docs.databricks.com/workspace/workspace-assets.html#ws-libraries">libraries</a>, and <a href="https://docs.databricks.com/applications/mlflow/tracking.html#mlflow-experiments">experiments</a>) into <a href="https://docs.databricks.com/workspace/workspace-objects.html#folders">folders</a>, and provides access to data and computational resources. In the following script, Databricks Workspace CLI is utilized to push all your local notebooks to the cluster. You can see that I also pay attention to the local logging because this is a good practice to always bookkeeping the job creation pipeline.</p><p><strong><em>workspace.sh</em></strong></p><pre>WORKSPACEPATH=/Users/${account}/workspace/PROJECT_NAME</pre><pre>local_timestamp=$(date &quot;+DATE: %Y-%m-%d%nTIME: %H:%M:%S&quot;)<br>echo &quot;----------------------------------------------------------------------------------&quot; | tee -a ./spark_runner/run/run.log</pre><pre>echo ${local_timestamp} | tee -a ./spark_runner/run/run.log<br>echo &quot;Push local run files to remote workspace notebooks&quot;  2&gt;&amp;1 | tee -a ./spark_runner/run/run.log</pre><pre>databricks workspace import_dir -o ./spark_runner/run ${WORKSPACEPATH} 2&gt;&amp;1 | tee -a ./spark_runner/run/run.log</pre><p>If you execute the above two scripts</p><pre>create_notebooks.sh<br>workspace.sh</pre><p>You will record the following log</p><pre><strong>bash create_notebooks.sh<br>bash workspace.sh<br>----------------------------------------------------------------------------------<br>DATE: 2020-04-09 TIME: 15:51:04<br>Push local run files to remote workspace notebooks<br>./spark_runner/run/run.log does not have a valid extension of .scala, .py, .sql, .SQL, .r, .R, .ipynb, .html. Skip this file and continue.<br>./spark_runner/run/spark_data_gen_12345.py -&gt; YOUR_DBRICKS_ACCOUNT/workspace/PROJECT_NAME/spark_data_gen_12345</strong></pre><p>and if you look at the Databricks UI as a monitor, you will convince yourself that the notebooks are indeed created inside its Workspace:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8LVG253Il8QkZ8mchjjalw.png" /></figure><p><strong>Submit and log Job via Job API</strong></p><p>At this point, you can, of course, run the job as a notebook directly from the Databricks UI, but this is not our goal. We want to run the job from the local and now Job API comes into play. The following script is handling this task. Essentially, there are three steps (1) go through the local job folder to find all the notebook jobs we just created in the previous step and replace the local directory with the remote Workspace directory as the target job directory. (2) submit job requests via API (3) get the job running status every few seconds via API. By the way, to make your job secure, it is a good practice not to expose your credentials to the program.</p><p><strong>executenotebook_allpurpose_cluster.py</strong></p><pre><em>#!/usr/bin/python3<br></em>import json<br>import requests<br>import os<br>import time<br>from spark_common.utils import get_security_value, get_cluster_id<strong><br></strong><br><br>def main(user=&quot;&quot;,<br>         token_file=&quot;&quot;,<br>         cluster_name=&quot;&quot;,<br>         workspacepath=&quot;&quot;,<br>         outfilepath=&quot;&quot;):<br><br>    <em># Generate array from walking local path<br>    </em>localpath = &quot;%s/run&quot; % os.path.dirname(os.path.realpath(__file__))<br><br>    token = get_security_value(token_file, user, &quot;token&quot;)<br>    server = get_security_value(token_file, user, &quot;server&quot;)<br>    clusterid = get_cluster_id(token_file, user, cluster_name)</pre><pre>    notebooks = []<br>    for path, subdirs, files in os.walk(localpath):<br>        for name in files:<br>            fullpath = path + &#39;/&#39; + name<br>            <em># removes localpath to repo but keeps workspace path<br>            </em>fullworkspacepath = workspacepath + path.replace(localpath, &#39;&#39;)<br><br>            name, file_extension = os.path.splitext(fullpath)<br>            if file_extension.lower() in [&#39;.scala&#39;, &#39;.sql&#39;, &#39;.r&#39;, &#39;.py&#39;]:<br>                row = [fullpath, fullworkspacepath, 1]<br>                notebooks.append(row)<br><br>    <em># run each element in list<br>    </em>for notebook in notebooks:<br>        nameonly = os.path.basename(notebook[0])<br>        workspacepath = notebook[1]<br><br>        name, file_extension = os.path.splitext(nameonly)<br><br>        <em># workpath removes extension<br>        </em>fullworkspacepath = workspacepath + &#39;/&#39; + name<br><br>        print(&#39;Running job for:&#39; + fullworkspacepath)<br>        timeout = 3600<br>        values = {&#39;run_name&#39;: name, &#39;existing_cluster_id&#39;: clusterid, &#39;timeout_seconds&#39;: timeout, &#39;notebook_task&#39;: {&#39;notebook_path&#39;: fullworkspacepath}}<br><br>        resp = requests.post(server + &#39;/api/2.0/jobs/runs/submit&#39;,<br>                             data=json.dumps(values),<br>                             auth=(&quot;token&quot;, token)<br>                             )<br>        print(server + &#39;/api/2.0/jobs/runs/submit&#39;)<br>        print(values)<br>        runjson = resp.text<br>        print(&quot;runjson:&quot; + runjson)<br>        d = json.loads(runjson)<br>        runid = d[&#39;run_id&#39;]<br><br>        i=0<br>        waiting = True<br>        while waiting:<br>            time.sleep(10)<br>            jobresp = requests.get(server + &#39;/api/2.0/jobs/runs/get?run_id=&#39;+str(runid),<br>                                   data=json.dumps(values),<br>                                   auth=(&quot;token&quot;, token)<br>                                   )<br>            jobjson = jobresp.text<br>            print(&quot;jobjson:&quot; + jobjson)<br>            j = json.loads(jobjson)<br>            current_state = j[&#39;state&#39;][&#39;life_cycle_state&#39;]<br>            runid = j[&#39;run_id&#39;]<br>            if current_state in [&#39;TERMINATED&#39;, &#39;INTERNAL_ERROR&#39;, &#39;SKIPPED&#39;] or i &gt;= timeout/10:<br>                break<br>            i=i+1<br><br>        if outfilepath != &#39;&#39;:<br>            file = open(outfilepath + &#39;/&#39; +  str(runid) + &#39;.json&#39;, &#39;w&#39;)<br>            file.write(json.dumps(j))<br>            file.close()<br><br><br>if __name__ == &#39;__main__&#39;:<br>    import argparse<br><br>    parser = argparse.ArgumentParser()<br>    parser.add_argument(&quot;--token_file&quot;)<br>    parser.add_argument(&quot;--user&quot;)<br>    parser.add_argument(&quot;--cluster_name&quot;, help=&quot;cluster alias name&quot;)<br>    parser.add_argument(&quot;--workspacepath&quot;, help=&quot;workspacepath)<br>    parser.add_argument(&quot;--outfilepath&quot;, default=&quot;%s/joblogs&quot; % os.path.dirname(os.path.realpath(__file__)), help=&quot;outfilepath for writing response&quot;)<br>    args = parser.parse_args()<br>    args = vars(args)<em><br><br>    </em>main(**args)</pre><p>if you run it, you should get the following response (I have masked a few sensitive information) in your log every 10 seconds, you can also monitor the job status and some interesting Spark metrics via Databricks UI with the job id returned by the API. By the end of the log, you may notice that I have a <em>clear_localnotebooks.sh</em>, this is a simple script to remove all local notebooks and log this deletion so that next time no conflicting or redundant jobs would still exist.</p><pre><strong>runjson:{&quot;run_id&quot;:11625}</strong></pre><pre><strong>jobjson:{&quot;job_id&quot;:4317,&quot;run_id&quot;:11625,&quot;number_in_job&quot;:1,&quot;state&quot;:{&quot;life_cycle_state&quot;:&quot;RUNNING&quot;,&quot;state_message&quot;:&quot;In run&quot;},&quot;task&quot;:{&quot;notebook_task&quot;:{&quot;notebook_path&quot;:&quot;XXXXX&quot;}},&quot;cluster_spec&quot;:{&quot;existing_cluster_id&quot;:&quot;XXXXX&quot;},&quot;cluster_instance&quot;:{&quot;cluster_id&quot;:&quot;XXXXX&quot;,&quot;spark_context_id&quot;:&quot;XXXXX&quot;},&quot;start_time&quot;:1586474824371,&quot;setup_duration&quot;:2000,&quot;execution_duration&quot;:0,.....&quot;run_type&quot;:&quot;SUBMIT_RUN&quot;}</strong></pre><pre><strong>...</strong></pre><pre><strong>jobjson:{&quot;job_id&quot;:4317,&quot;run_id&quot;:11625,&quot;number_in_job&quot;:1,&quot;state&quot;:{&quot;life_cycle_state&quot;:&quot;TERMINATED&quot;,&quot;result_state&quot;:&quot;SUCCESS&quot;,&quot;state_message&quot;:&quot;&quot;},......,&quot;run_type&quot;:&quot;SUBMIT_RUN&quot;}</strong></pre><pre><strong>bash clear_localnotebooks.sh</strong></pre><p>Now the machine learning jobs available from your old codebase can be scripted, submitted, run and logged using a similar interface as your original codebase did. You have a very similar script to create your job, then you can have a simple bash script to manage your job at remote clusters.</p><p><strong>Summary</strong></p><p>The following workflow chart summarizes everything: essentially, we have a UDF layer to transform the codebase to adapt to the PySpark environment, we have a job manager to emulate the job creation as your old codebase did and finally we use CLI and API to manage the remote jobs and clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u3oq3RC_aEqmQtOHpTTtxQ.png" /></figure><p>All these tasks can be simply achieved by the following 6 lines of bash scripts, and more importantly, a user only needs to interact with <em>create_notebooks.sh</em> to create and register jobs as he/she should always do. With all other components entirely hidden, the user only needs to call the ML codebase at local and gets the job done at the remote cluster.</p><pre>configure-allpurpose-databricks:<br>        bash sh/uninstall.sh<br>        bash sh/install.sh<br>create-notebooks:<br>        bash sh/create_notebooks.sh<br>        bash sh/workspace.sh<br>execute-allpurpose-jobs:<br>        bash sh/execute_notebooks.sh<br>        bash sh/clear_localnotebooks.sh</pre><p>Next <a href="https://medium.com/@punchhtech/seamless-migration-of-machine-learning-codebase-from-the-local-or-ec2-to-databricks-pyspark-eba5ae7270d6"><strong>blog</strong></a>, I am going to talk about how to manage a job cluster and submit jobs right there. A Job cluster is a much cheaper setting than an all-purpose cluster and hence is considered to be the best candidate for production jobs. If you understand the infrastructure I discussed in this and the previous sections, it should be pretty straightforward to combine them together and come up with a good strategy.</p><h3>About Punchh</h3><p>Headquartered in San Mateo, CA, Punchh is the world leader in innovative digital marketing products for brick and mortar retailers, combining AI and machine learning technologies, mobile-first expertise, and Omni-Channel communications designed to dramatically increase lifetime customer value. Leading global chains in the restaurant, health and beauty sectors rely on Punchh to grow revenue by building customer relationships at every stage, from anonymous, to known, to brand loyalists, including more than 100 different chains representing more than $12 billion in annual spend.</p><h3>About the Author</h3><p><strong>Dr. Tian Lan </strong>is Tech Lead of A.I. at Punchh, where he leads the development of large-scale and distributed machine learning for recommender systems and personalized marketing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d213ac37db4b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[COVID-19 Impact Analysis Series Article Four: Delivery orders during Breakfast]]></title>
            <link>https://punchhtech.medium.com/covid-19-impact-analysis-series-article-four-delivery-orders-during-breakfast-736a658d3413?source=rss-cf87ad15791e------2</link>
            <guid isPermaLink="false">https://medium.com/p/736a658d3413</guid>
            <category><![CDATA[food-delivery]]></category>
            <category><![CDATA[consumer-behavior]]></category>
            <category><![CDATA[breakfast]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[restaurant]]></category>
            <dc:creator><![CDATA[Punchh Technology Blog]]></dc:creator>
            <pubDate>Sat, 04 Apr 2020 07:43:43 GMT</pubDate>
            <atom:updated>2020-04-24T08:36:21.201Z</atom:updated>
            <content:encoded><![CDATA[<h3>COVID-19 Impact Analysis Series Article Four: D<strong>elivery Orders and Breakfast</strong></h3><h3>Author: Shubham Goyal (April 03, 2020)</h3><p>Breakfast orders are in the decline</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*YziUOsUqYCl8s18h" /></figure><p><strong>THIS COVID-19 impact Analysis Series designed by Punchh’s data team aims at sharing the real-time insight we have discovered when we study our consumer behavior data set. In our previous </strong><a href="https://medium.com/@punchhtech/covid-19-impact-analysis-series-article-three-ordering-and-delivering-9a499171068f"><strong>article</strong></a><strong>, we provided our insight into the increase in the off-prem orders as % of total orders. In this fourth article, we have zoomed in on two categories within off-prem orders: delivery and takeout.</strong></p><p>As of April 02, the governors of more than 40 states have imposed a <strong>stay-at-home</strong> directive, essentially putting 80% of Americans under lockdown. Under these circumstances, the off-prem orders as % of total orders have increased by 20% in a matter of three weeks. The graph below shows the shift in this trend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RQmiplLRYPKtrHFS" /><figcaption>Increase in % share of off-prem orders among total orders</figcaption></figure><p><strong><em>(dine-in orders are those where customers consume the orders within the premises of the restaurant. off-prem orders are those where customers consume the orders elsewhere)</em></strong></p><p>Among the off-prem orders, which are either “delivery” or “takeout”, we found that <strong>delivery orders</strong> as % of total off-prem orders have doubled over a period of 3 weeks, going from 6% to 15%. The graph below shows the trend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DZLh9R1YzYJR35-K" /><figcaption>Increase in delivery orders as % of off-prem orders</figcaption></figure><p><strong><em>(takeout orders include to-go orders, drive-through, curbside pickups)</em></strong></p><p>With this context in mind, we have further explored whether the way people are placing delivery orders in different parts of the day has shifted. For example, the morning orders may be going down because consumers would increase their tendency of preparing their own breakfast at home. Our analysis confirms our hypothesis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ez7S0tiSFhjO1k79" /><figcaption>(Sum of percentages for each period (horizontally) will add up to 100%)</figcaption></figure><p>The data shows that the share of delivery orders placed during breakfast has decreased the most (-6 %-point). This slack is picked up during lunch hours and further into the afternoon and dinner time. However, consumers have started to prepare their own late-night snack food, or consumer behavior might have been affected by the new store hours.</p><p><strong>KEY TAKEAWAYS:</strong></p><ul><li>The majority of the consumers are still coming to the stores to pick up their meals. They are consuming it elsewhere, most likely at home.</li><li>The trend of having orders delivered is increasing. In specific, there is a significant shift from the delivery orders placed for breakfast to the delivery orders placed for lunch. Hence, brands should be cognisant of the fact that delivery traffic tends to concentrate during lunch and dinner times. They should plan their operations accordingly.</li></ul><p><strong>ABOUT THE DATA</strong></p><p>The underlying data for this analysis is from 10K+ restaurant stores across the country covering 12 brands for the period of Feb 01 — Mar 31.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=736a658d3413" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to seamlessly migrate a machine learning codebase running Python code (e.g.]]></title>
            <link>https://punchhtech.medium.com/how-to-seamlessly-migrate-a-machine-learning-codebase-running-python-code-e-g-d35e6f7f5bfa?source=rss-cf87ad15791e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d35e6f7f5bfa</guid>
            <category><![CDATA[command-line]]></category>
            <category><![CDATA[migration]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Punchh Technology Blog]]></dc:creator>
            <pubDate>Sun, 29 Mar 2020 04:21:43 GMT</pubDate>
            <atom:updated>2020-04-10T17:22:09.651Z</atom:updated>
            <content:encoded><![CDATA[<h3>Seamless migration of machine learning codebase from the local or EC2 to Databricks PySpark Clusters(Part II: Manage all-purpose clusters via CLI and API)</h3><p>Author: Tian Lan</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*byhfoFos2hvlAx1YF-XFDA.png" /></figure><p>Following the previous <a href="https://medium.com/@punchhtech/how-to-seamlessly-migrate-a-machine-learning-codebase-and-cicd-pipeline-running-python-code-e-g-ecc16827d6d0"><strong>blog (Part I)</strong></a><em>, </em>in this article,<strong> </strong>we are going to show how we configure and start the remote Databricks cluster from the local, and how we upload and install our machine learning codebase discussed in the previous blog to Databricks cluster. This is an essential step to fulfill our goal that “<em>the user interface looks like still running on your old local or EC2 environment.</em>” so that we can “<em>migrate the monster codebase in a seamless way</em>”.</p><p>According to Databricks Documentation: “You use all-purpose clusters to analyze data collaboratively using interactive notebooks. You use job clusters to run fast and robust automated jobs”. Essentially, you can manually terminate and restart an all-purpose cluster. Multiple users can also share such clusters to do collaborative interactive analysis. The job cluster is created ad-hoc when you run a job and the cluster is terminated when the job is complete. You <em>cannot</em> restart a job cluster.</p><p>In Part II and <a href="https://medium.com/@punchhtech/seamless-migration-of-machine-learning-codebase-from-the-local-or-ec2-to-databricks-pyspark-d213ac37db4b"><strong>Part III</strong></a>, I will present the solution to managing the all-purpose clusters; for the job clusters, the solution is similar and will be discussed in <a href="https://medium.com/@punchhtech/seamless-migration-of-machine-learning-codebase-from-the-local-or-ec2-to-databricks-pyspark-eba5ae7270d6"><strong>Part IV</strong></a>.</p><p>To review all the sections of this topic, here is the Table of Contents for the related blogs:</p><ol><li>Write the Spark User-Defined Function and codebase adaptor (<a href="https://medium.com/@punchhtech/how-to-seamlessly-migrate-a-machine-learning-codebase-and-cicd-pipeline-running-python-code-e-g-ecc16827d6d0"><strong>Part I link</strong></a>)</li><li>Manage Databricks all-purpose clusters via CLI and API (<strong>Part II</strong>)</li><li>Build, synchronize and submit job scripts at local (<a href="https://medium.com/@punchhtech/seamless-migration-of-machine-learning-codebase-from-the-local-or-ec2-to-databricks-pyspark-d213ac37db4b"><strong>Part III link</strong></a>)</li><li>Manage Databricks job clusters exclusively via CLI and API (<a href="https://medium.com/@punchhtech/seamless-migration-of-machine-learning-codebase-from-the-local-or-ec2-to-databricks-pyspark-eba5ae7270d6"><strong>Part IV link</strong></a>)</li></ol><h3>Part II: All-purpose cluster: uninstall, install and manage the machine learning codebase from the local (Laptop, AWS EC2, etc.)</h3><p><strong>Databricks CLI</strong></p><p>According to Databricks documentation, the Databricks command-line interface (CLI) provides an easy-to-use interface to the Databricks platform. The CLI is built on top of the Databricks <a href="https://docs.databricks.com/dev-tools/api/latest/index.html#rest-api-v2">REST API 2.0</a> and is organized into command groups based on the <a href="https://docs.databricks.com/dev-tools/api/latest/workspace.html#workspace-api">Workspace API</a>, <a href="https://docs.databricks.com/dev-tools/api/latest/clusters.html#clusters-api">Clusters API</a>, <a href="https://docs.databricks.com/dev-tools/api/latest/dbfs.html#dbfs-api">DBFS API</a>, <a href="https://docs.databricks.com/dev-tools/api/latest/groups.html#groups-api">Groups API</a>, <a href="https://docs.databricks.com/dev-tools/api/latest/jobs.html#jobs-api">Jobs API</a>, <a href="https://docs.databricks.com/dev-tools/api/latest/libraries.html#libraries-api">Libraries API</a>, and <a href="https://docs.databricks.com/dev-tools/api/latest/secrets.html#secrets-api">Secrets API</a>.</p><p><strong>Package the Codebase</strong></p><p>We choose wheel to package the codebase. A wheel is a ZIP-format archive with a specially formatted filename and the .whl extension. It is designed to contain all the files for a PEP 376 compatible install in a way that is very close to the on-disk format. Many packages will be properly installed with only the “Unpack” step (simply extracting the file onto sys.path), and the unpacked archive preserves enough information to “Spread” (copy data and scripts to their final locations) at any later time.</p><p>Remember we have two codebases that need to be packaged for the installation later. One is the original machine learning codebase, the other is the PySpark adaptor codebase that we have discussed in the previous <a href="https://medium.com/@punchhtech/how-to-seamlessly-migrate-a-machine-learning-codebase-and-cicd-pipeline-running-python-code-e-g-ecc16827d6d0"><strong><em>blog</em></strong></a>.</p><p><strong>Uninstall the Old ML Codebase</strong></p><p>If it is an existing all-purpose cluster, before you install a new version of ML codebase and the corresponding PySpark codebase(we will show how to install them from the local shortly) to the Databricks cluster, you should uninstall the old ones to avoid possible conflicts.</p><p>Firstly you should start your terminated cluster if it is not and make sure it is running. We are using Databricks CLI for clusters:</p><p><strong><em>uninstall.sh</em></strong></p><pre># if the cluster is terminated, start it<br>status=$(databricks clusters get --cluster-id ${CLUSTERID} | jq &#39;.state&#39;)</pre><pre>if [[ &quot;${status}&quot; = *&quot;TERMINATED&quot;* ]]<br>then<br>  echo &quot;The cluster is not running yet, start ...&quot;<br>  databricks clusters start --cluster-id ${CLUSTERID}<br>fi</pre><pre># wait till cluster turns to &quot;Running&quot;<br>waittime=0<br>while [[ &quot;${status}&quot; != *&quot;RUNNING&quot;* ]]<br>do<br>  echo &quot;Waiting for starting the cluster...$waittime seconds&quot;<br>  sleep 10<br>  status=$(databricks clusters get --cluster-id ${CLUSTERID} | jq &#39;.state&#39;)<br>  waittime=$((waittime+10))<br>done</pre><p>Next, you need to check if your old codebase has been installed already. If so, you should uninstall it. We are using Databricks CLI for libraries:</p><pre># check if the target lib is already &quot;Intalled&quot; or &quot;Installing&quot;, uninstall it<br>libs=$(databricks libraries cluster-status --cluster-id ${CLUSTERID} | jq &#39;.library_statuses&#39;)</pre><pre>need_restart=0<br>for row in $(echo &quot;${libs}&quot; | jq -r &#39;.[] | <a href="http://twitter.com/base64">@base64</a>&#39;); do<br>    _jq() {<br>     echo ${row} | base64 --decode | jq -r ${1}<br>    }</pre><pre>if [[ $(_jq &#39;.library.whl&#39;) = *&quot;${LIBNAME}&quot;* ]]<br>    then<br>        echo &quot;Uninstall:&quot; $(_jq &#39;.library&#39;)<br>        databricks libraries uninstall --cluster-id ${CLUSTERID} --whl $(_jq &#39;.library.whl&#39;)<br>        echo &quot;Uninstall finished&quot;<br>        need_restart=1<br>    fi</pre><pre>done</pre><p>Finally, we restart the cluster to make the uninstallation take effect:</p><pre>if [ ${need_restart} -eq 1 ]<br>then<br>    echo &quot;Restart cluster to make uninstallation effective...&quot;<br>    databricks clusters restart --cluster-id ${CLUSTERID}</pre><pre>status=$(databricks clusters get --cluster-id ${CLUSTERID} | jq &#39;.state&#39;)</pre><pre>waittime=0<br>    while [[ &quot;${status}&quot; != *&quot;RUNNING&quot;* ]]<br>    do<br>      echo &quot;Waiting for starting the cluster...$waittime seconds&quot;<br>      sleep 10<br>      status=$(databricks clusters get --cluster-id ${CLUSTERID} | jq &#39;.state&#39;)<br>      waittime=$((waittime+10))<br>    done<br>fi</pre><pre>echo &quot;Uninstall All Done&quot;</pre><p>If you run it, you can see the workflow like this</p><pre><strong>bash sh/uninstall.sh<br>The cluster is not running yet, start ...<br>Waiting for starting the cluster...0 seconds<br>Waiting for starting the cluster...10 seconds<br>Waiting for starting the cluster...20 seconds<br>Waiting for starting the cluster...30 seconds<br>Waiting for starting the cluster...40 seconds<br>Waiting for starting the cluster...50 seconds<br>Waiting for starting the cluster...60 seconds<br>Waiting for starting the cluster...70 seconds<br>Waiting for starting the cluster...80 seconds<br>Waiting for starting the cluster...90 seconds<br>Waiting for starting the cluster...100 seconds<br>Waiting for starting the cluster...110 seconds<br>Uninstall: { &quot;whl&quot;: &quot;dbfs:/FileStore/users/tianlan/XXXXXXXX.whl&quot; }<br>WARNING: Uninstalling libraries requires a cluster restart.<br>databricks clusters restart --cluster-id XXXXXXXXXX<br>Uninstall finished<br>Restart cluster to make uninstallation effective...<br>Waiting for starting the cluster...0 seconds<br>Waiting for starting the cluster...10 seconds<br>Waiting for starting the cluster...20 seconds<br>Waiting for starting the cluster...30 seconds<br>Waiting for starting the cluster...40 seconds<br>Uninstall All Done</strong></pre><p><strong>Install the New ML Codebase</strong></p><p>Installation of a new codebase is surprisingly simple with the help of Databricks CLI. As we packaged our code to the wheel file, we just need to upload it to the DBFS in the server and install it on the cluster</p><p><strong><em>install.sh</em></strong></p><pre>whl_file={modulename}-${version}-py3-none-any.whl</pre><pre>rm -rf dist<br>python setup.py sdist bdist_wheel</pre><pre>dbfs cp --overwrite ./dist/${whl_file} ${dbfs_dir}<br>echo &quot;databricks libraries install --cluster-id ${CLUSTERID} --whl ${dbfs_dir}/${whl_file}&quot;<br>databricks libraries install --cluster-id ${CLUSTERID} --whl ${dbfs_dir}/${whl_file}</pre><p><strong>Local Command</strong></p><p>This is as simple as calling two lines of codes</p><pre>bash uninstall.sh<br>bash install.sh</pre><p>You can see that we have achieved <strong>Milestone #2</strong>. At this point, we have 100% control over the clusters from your local system (laptop, EC2, etc.). You do not need any Databricks UI, but if you are interested, you can use UI as a monitor to track each status (.e.g, start, uninstallation, restart, upload and installation) along this process.</p><p>Next <a href="https://medium.com/@punchhtech/seamless-migration-of-machine-learning-codebase-from-the-local-or-ec2-to-databricks-pyspark-d213ac37db4b"><strong>blog</strong></a>, I am going to talk about how to build the local job scripts and synchronize with the cluster and submit the job there. My goal is the task creation, submission, and logging from your local to remote cluster looks as if all are taking place on your local alone.</p><h3>About Punchh</h3><p>Headquartered in San Mateo, CA, Punchh is the world leader in innovative digital marketing products for brick and mortar retailers, combining AI and machine learning technologies, mobile-first expertise, and Omni-Channel communications designed to dramatically increase lifetime customer value. Leading global chains in the restaurant, health and beauty sectors rely on Punchh to grow revenue by building customer relationships at every stage, from anonymous, to known, to brand loyalists, including more than 100 different chains representing more than $12 billion in annual spend.</p><h3>About the Author</h3><p><strong>Dr. Tian Lan </strong>is Tech Lead of A.I. at Punchh, where he leads the development of large-scale and distributed machine learning for recommender systems and personalized marketing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d35e6f7f5bfa" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>