<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lasantha Kularatne on Medium]]></title>
        <description><![CDATA[Stories by Lasantha Kularatne on Medium]]></description>
        <link>https://medium.com/@lkularatne?source=rss-4aa892ba9ae9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*kPL3KgISFE6ll2wX</url>
            <title>Stories by Lasantha Kularatne on Medium</title>
            <link>https://medium.com/@lkularatne?source=rss-4aa892ba9ae9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 03:34:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lkularatne/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Testing out Sony GM II and Yongnuo lenses in Sony a7 IV]]></title>
            <link>https://medium.com/@lkularatne/testing-out-sony-gm-ii-and-yongnuo-lenses-in-sony-a7-iv-4cfe404952df?source=rss-4aa892ba9ae9------2</link>
            <guid isPermaLink="false">https://medium.com/p/4cfe404952df</guid>
            <category><![CDATA[lenses]]></category>
            <category><![CDATA[cameras]]></category>
            <category><![CDATA[mirrorless]]></category>
            <category><![CDATA[photography]]></category>
            <category><![CDATA[slr]]></category>
            <dc:creator><![CDATA[Lasantha Kularatne]]></dc:creator>
            <pubDate>Tue, 13 Feb 2024 02:00:11 GMT</pubDate>
            <atom:updated>2025-08-25T21:58:36.101Z</atom:updated>
            <content:encoded><![CDATA[<p>I have been a Nikon user for a while but I had only APS-C SLR cameras (D7200 and D5000). Recently I did the plunge and switched to a full frame SLR — Sony a7 IV — which is a mirror less camera as well. The kit lens did not do a good job to the quality of the camera. So I bought two different lenses:</p><ol><li><strong>Sony f2.8/24–70mm GM II zoom lens</strong></li><li><strong>Yongnuo f1.8 85mm prime lens</strong></li></ol><p>The Sony lens is one of their best ones in their flagship G Master category and it’s pricey. The Yongnuo prime lens built up much hyped after a good review by <a href="https://www.dxomark.com/yongnuo-yn-85mm-f1-8-s-df-dsm-lens-review/">DXO Mark</a>. Although these are not macro lenses, my first test was doing macro type photography which is I am going to share here.</p><p>All these photos were captured in raw format and done minimal processing with DxO PhotoLab.</p><h3>Test Photos: Sony f2.8/24–70mm GM II zoom lens</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MXbkPQXidMvNyf-9CC_eSQ.jpeg" /><figcaption>Exposure: f2.8, 1/125 sec, ISO auto, RAW</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*twTJIHIzhCpvV1sXD7Zi-w.jpeg" /><figcaption>Exposure: f2.8, 1/125 sec, ISO auto, RAW</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6-xKvKCH7eAWAu_EhDJo2w.jpeg" /><figcaption>Exposure: f2.8, 1/125 sec, ISO auto, RAW</figcaption></figure><h3>Test Photos: Yongnuo f1.8 85mm prime lens</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k1-6fv1cDZqNXC-xX7nSfw.jpeg" /><figcaption>Exposure: f1.8, 1/125 sec, ISO auto, RAW</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fj3q8aKVWsrUwBEBqNbYTw.jpeg" /><figcaption>Exposure: f1.8, 1/125 sec, ISO auto, RAW</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jcVKHHipDI6HbbS72I7lgQ.jpeg" /><figcaption>Exposure: f1.8, 1/125 sec, ISO auto, RAW</figcaption></figure><h3>Verdict</h3><p>Both lenses are very sharp and depth of field is very good. Yongnuo being f1.8 introduces sharp blurring of the background. I’ve noticed that both lenses created very nice bokeh effect (sample photos not included here). These photos were taken indoor in limited light conditions without extra light sources or camera flash. But both the lenses and the camera did pretty well in this lighting condition.</p><p>As it was pointed out in numerous times, the Yongnuo lens creates noticeable chromatic aberration. Also the focus was not in the exact spot I wanted to be. But the Sony lens had very accurate focus and was very fast. It felt very smooth taking single or multi shot photos with the Sony lens.</p><p>Finally one word about pricing. The Yongnuo lens can be bought for less than $300 from their website as of February 2024, where as the Sony lens is much more expensive! So for the given price Yongnuo did well. But I really like taking photos with the Sony GM II lens.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4cfe404952df" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Error Budget Policy Adoption at Expedia Group]]></title>
            <link>https://medium.com/expedia-group-tech/error-budget-policy-adoption-at-expedia-group-7d80d41c4a8b?source=rss-4aa892ba9ae9------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d80d41c4a8b</guid>
            <category><![CDATA[reliability-engineering]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[service-level-objectives]]></category>
            <category><![CDATA[error-budget]]></category>
            <dc:creator><![CDATA[Lasantha Kularatne]]></dc:creator>
            <pubDate>Tue, 28 Jun 2022 14:02:20 GMT</pubDate>
            <atom:updated>2023-01-06T17:39:00.253Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="http://medium.com/expedia-group-tech/engineering/home">EXPEDIA GROUP TECHNOLOGY — ENGINEERING</a></h4><h4>Using data to set the standard for reliability</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4NDLe4oRhudBJBfe7retmw.jpeg" /><figcaption>Team meeting in a conference room in Expedia Group’s Rome office</figcaption></figure><p>As a travel platform, our goal at <a href="https://www.expediagroup.com/">Expedia Group</a>™ is to provide reliable experiences for prospective travelers. Although 100% reliability in a system is not practical, we want to be reliable enough to keep our customers happy. To achieve it, many teams are increasingly adopting service level objectives (SLOs) and error budgets across Expedia Group. But what should a team be doing if they are missing the SLOs and going out of budget? Maintaining the intended reliability level is not feasible unless the team adopts error budget policies based on a data-driven approach we propose here. We hope teams will adopt well-defined policies and be prepared to handle these situations.</p><h3>Goals</h3><p>The main goal is to eliminate bad customer experiences due to repeated SLO misses. Teams will be able to deliver new features while keeping up with the reliability expectations. “Eliminate Preventable Bad Experiences” is one of our objectives and key results (OKRs), and we are going to achieve it by operationalizing error budget policies. As a side effect, this will help stakeholders agree upon the product priorities without contention.</p><h3>Benefits</h3><p>Well-defined error budget policies will improve day-to-day reliability provided by teams. Critical problems that are immediately clear can be prevented (and fixed) by short change freezes. In contrast, error budgets can help prevent gradual degradation. Error budget policies give teams autonomy to manage their reliability while being able to communicate to the rest of the organization how reliable they have been.</p><h3>Out of scope</h3><p>These error budget policies are not meant to be punishments for SLO violations. At Expedia Group, we treat each and every failure as an opportunity to learn! These error budget policies encourage teams to prioritize reliability over product features when needed and take tolerable risks when there is enough budget left.</p><h3>Terminology</h3><ul><li><strong>Service Level Indicator</strong> (SLI) — A quantifiable measure of service reliability that tells you if things are working.</li></ul><figure><img alt="SLI Equation: SLI = Good Events / Valid Events * 100%" src="https://cdn-images-1.medium.com/max/1024/1*NGe0hmpk8RBcoyXQ3-x5NQ.png" /></figure><ul><li><strong>Service Level Objective</strong> (SLO) — A reliability target for an SLI that tells you if the users are happy or sad.</li></ul><figure><img alt="SLO = 99.9%" src="https://cdn-images-1.medium.com/max/1024/1*OnIT4CRQCRcNoih220Yl_w.png" /></figure><ul><li><strong>Service Level Agreement </strong>(SLA) — A contract that the service provider promises users on service reliability. Agreed reliability targets should be more relaxed than the internal SLOs to give enough room for unexpected outages.</li><li><strong>Error Budget</strong> — Either the number of requests out of the total requests or the amount of time for a given period, a system can afford to be unreliable before users become unhappy.</li></ul><figure><img alt="Error Budget Equation: Error Budget = 100% — SLO = 0.1%" src="https://cdn-images-1.medium.com/max/1024/1*ipQkuVZY0x5VkV9w5rjjnQ.png" /></figure><ul><li><strong>Error Budget Policy </strong>— Defines what you will do in case of SLO violations.</li><li><strong>Burn Rate</strong> — How fast, relative to the SLO, the service consumes the error budget.</li><li><strong>System Boundary</strong> — A point at which one or more components (a system) exposes functionality to its users.</li><li><strong>Capabilities</strong> — A set of functionalities provided by a system at the system boundary.</li><li><strong>User</strong> — The customer of a given system. This can be a human customer (travelers or internal users) or a system (machines).</li><li><strong>Site Reliability Engineering</strong> (SRE) — Engineering practices adopted by an organization in order to maintain the expected reliability of software systems.</li></ul><h3>Who should read this document?</h3><p>It is expected that you have a clear understanding about what capability, service level indicator (SLI), service level objective (SLO) and error budget mean. If you have error budgets set up for your system and are looking forward to implementing error budget policies or if you are someone who likes to practice operationalizing error budgets, you should follow this document.</p><figure><img alt="Diagram showing relationship of Capability, SLI, SLO &amp; Error Budget" src="https://cdn-images-1.medium.com/max/1024/1*B7qI20LIeRahpbnxuzTMGw.png" /><figcaption>Relationship of Capability, SLI, SLO &amp; Error Budget</figcaption></figure><p>This illustration shows how all these concepts are tied together. To emphasize the goal:</p><blockquote>We are going to keep the customers happy by maintaining the expected system reliability.</blockquote><h3>Prerequisites for effective monitoring</h3><p>For a given system, before adopting the proposed error budget policies, the following prerequisites must be met:</p><ol><li>System boundaries, capabilities &amp; users are defined.</li><li>System is set up to send proper metrics to the monitoring system (in our case it is <a href="https://www.datadoghq.com/">Datadog</a>).</li><li>SLIs &amp; SLOs are defined and measured in dashboards.</li><li>Error budgets are set.</li></ol><p>If these requirements are not met, a team should focus on implementing them before attempting to define the error budget policies.</p><h3>Types of errors</h3><p>Before laying out the error budget policies, a team must document the answer to this question:</p><blockquote>What are the errors that will consume the error budget, and which are tolerated?</blockquote><p>The following cases are examples of acceptable errors that <strong>will not</strong> consume the error budget when they occur:</p><ol><li>Company wide network outages.</li><li>Cloud vendor outages.</li><li>Known downtimes for maintenance.</li><li>Miscategorized errors when no real users were impacted.</li><li>Errors consumed by users out of scope for the SLOs (e.g., load tests, integration tests).</li><li>Outages caused by a service maintained by another team (internal dependencies).</li><li>Outages caused by a service maintained by a 3rd party vendor (external dependencies).</li></ol><p>Any other errors experienced by the users are taken out of your application’s error budget.</p><p>This is a general guidance. Individual teams will have the flexibility to make modifications to this list over time.</p><h3>Error budget window</h3><p>We want to encourage flexibility in choosing a time window for the error budget. While each team could create their own budget windows, collaboration would be easier if each team implemented a common budget window. As a rule-of-a-thumb we are proposing a rolling 30-day window. Fixed time windows, such as monthly, are discouraged because they will refresh the remaining budget to 100% at the beginning of each window while the customer happiness does not behave that way.</p><h3>Monitoring and alerting</h3><p>Monitoring and alerting should be set in place to notify the teams if their SLOs are not met, or they are on the way to consuming the error budget soon. It is recommended to have alerts based on both error budget consumption level and elevated burn rates.</p><h4>Alerting on error budget consumption</h4><p>Alerting should be set up based on total error budget consumed during a 30-day budget window.</p><figure><img alt="Graph for alerting based on error budget consumption" src="https://cdn-images-1.medium.com/max/1024/1*pLUG-SNygCmE3pY1C_BH-A.png" /><figcaption>Error budget burn up chart</figcaption></figure><figure><img alt="Alerting on Error Budget Consumption Table; 75–90% = warning, &gt; 90% = critical" src="https://cdn-images-1.medium.com/max/1024/1*HyWNgL_TMsI6P7irvE2XcA.png" /><figcaption>Proposed alerting thresholds and expected outcomes</figcaption></figure><h4>Alerting on elevated burn rate</h4><p>Burn rate is how fast the service consumes the error budget for a given SLO. It is calculated as the ratio between your observed error rate and the maximum tolerable error rate (i.e., error rate for the system to consume the entire budget during the budget window) during an alert window.</p><p><strong>Example:</strong></p><blockquote><em>Given an error budget window of 30 days and an SLO of 99%, you will have an error budget of 1% for that period. Let’s say an error rate of 2% was observed, then your burn rate is 2. Your application will consume the error budget in 15 days (within half of the budget window). If you observed an error of 10% then your burn rate is 10 and it will consume the error budget in 3 days!</em></blockquote><p>For alerting based on the burn rate, the following three factors should be considered:</p><ol><li><strong>Budget Window</strong> — The length of the error budget window. We recommend 30 days (720 hours).</li><li><strong>Alert Window</strong> — The length of the alert calculations window, such as 1 hour.</li><li><strong>Budget Consumption </strong>— The percentage of the error budget consumed during the alert calculation window.</li></ol><p>Then the observed burn rate is calculated as:</p><figure><img alt="Burn Rate Equation: Observed burn rate = (budget window / alert window) * (budget consumption / 100)" src="https://cdn-images-1.medium.com/max/1024/1*RI1E7pPKFylhFcHlTPszqA.png" /></figure><figure><img alt="Alerting on Elevated Burn Rate Graph" src="https://cdn-images-1.medium.com/max/1024/1*tSbTeaRMPjynE9tP53QyRQ.png" /><figcaption>Error budget burn up chart based on various burn rates</figcaption></figure><p><strong>Recommendation:</strong> For alerting based on burn rate, for a 30-day budget window, the following thresholds are proposed.</p><figure><img alt="Alerting on Elevated Burn Rate Table: Relates alert window, budget consumption, exhaustion time, burn rate, alert type, and escalation level" src="https://cdn-images-1.medium.com/max/1024/1*JNgKqOxw4QuVW8ZcjVxwjw.png" /></figure><h4>The proposed framework in summary</h4><p>The proposed approach for applying necessary error budget policies is:</p><ul><li>Monitor the error budget related to an SLO.</li><li>Set up alerts based on budget consumption level and elevated burn rates.</li><li>Classify the reliability state of the system based on alerts and error budget level.</li><li>Follow the policies defined for that particular reliability state.</li></ul><p>In the next blog post, “<a href="https://medium.com/expedia-group-tech/error-budget-policies-in-practice-4c98f56a28c1">Error Budget Policies in Practice</a>”, <a href="https://medium.com/@etrang_3293">Eric Trang</a> will explain the proposed framework in details with a sample implementation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d80d41c4a8b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/expedia-group-tech/error-budget-policy-adoption-at-expedia-group-7d80d41c4a8b">Error Budget Policy Adoption at Expedia Group</a> was originally published in <a href="https://medium.com/expedia-group-tech">Expedia Group Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predict It! An Engineer’s Journey into Data Science]]></title>
            <link>https://medium.com/expedia-group-tech/predict-it-an-engineers-journey-into-data-science-62f660f51877?source=rss-4aa892ba9ae9------2</link>
            <guid isPermaLink="false">https://medium.com/p/62f660f51877</guid>
            <category><![CDATA[prophet]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Lasantha Kularatne]]></dc:creator>
            <pubDate>Tue, 30 Mar 2021 14:01:29 GMT</pubDate>
            <atom:updated>2021-03-30T14:01:29.289Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/expedia-group-tech/data/home">EXPEDIA GROUP TECHNOLOGY — DATA</a></h4><h4>Using R, Fourier series, and a linear regression model to predict web site traffic</h4><p>I am going to tell you a story about one of my journeys. In the winter of 2019, my family went on a vacation to Denver, Colorado. From there, we drove to a popular ski resort up in the mountains. During that trip, I took these two photos:</p><figure><img alt="Photo of a road through a snowy, mountainous forest" src="https://cdn-images-1.medium.com/max/1000/1*adlNgUEv5aKi563MbitrxQ.jpeg" /><figcaption>The journey. Photo by the author.</figcaption></figure><figure><img alt="A photo of pristine snow on a mountainside" src="https://cdn-images-1.medium.com/max/1000/1*vIexKhp1HmnVbeRImKrsxA.jpeg" /><figcaption>The destination. Photo by the author.</figcaption></figure><p>The first photo shows the drive to the resort. The road was very difficult to drive on. There were piles of snow everywhere. You have to be an experienced driver to drive on that road. You would need a car with 4-wheel drive to ride it safely. But there would be enough help available if you got into any trouble. The second photo shows the view from the top of the mountain at the final destination. Once you got there, it was worth the trouble. The area was wide open for you to play and enjoy!</p><p>When I think about my journey into the data science, I feel the same way. I am not a data scientist — I am a software engineer. Being a software engineer should not hold you up from learning and applying data science techniques in your work. It should be beneficial to your work. For the drive to the ski resort I described, an experienced driver would get there safely. Like the 4-wheel drive car, you need proper tools to get it done. And be mindful to ask for help when you need it, as plenty is available. Once you get there, it can be beneficial in many ways to your work.</p><p>If you are new to data science, you might be wondering where to start. It is normal to be confused at the beginning. I was in my early days.</p><figure><img alt="An xkcd-style cartoon showing a complex decision tree for using data science" src="https://cdn-images-1.medium.com/max/782/1*KbuT3Eu9jmxorjx8_O2d4Q.png" /><figcaption>“<a href="https://hackernoon.com/what-on-earth-is-data-science-eb1237d8cb37">What on earth is data science?</a>” by <a href="https://twitter.com/quaesita">Cassie Kozyrkov</a> (used by permission)</figcaption></figure><p>But if you make an effort, <strong>you can</strong> get there. There are a lot of helpful resources available in the community. I want to share a story where I was able to put my data science knowledge to good use.</p><h4>Disclaimer</h4><p>While I solved a real-world problem using the techniques I describe, all the examples and datasets that you are about to see in this article and in the sample GitHub repository are simulated. These traffic patterns and metrics data do not represent the real behavior of any application in Expedia Group’s platform. These datasets are for educational purposes only.</p><p>Sample code and simulated datasets are available at <a href="https://github.com/lasanthak/traffic-forcast">https://github.com/lasanthak/traffic-forcast</a>.</p><h3>What to predict? And why?</h3><p>To get Vrbo applications and services ready for the peak season, my reliability engineering team needed to forecast how much traffic we should be able to handle without having scalability problems. We would test that by running weekly peak traffic tests (PCTs) slightly above the forecasted peak traffic level in the production environment. This is nothing new; we have been doing that for several years. What was new in the last year was that we decided to forecast the traffic more scientifically than relying on ad hoc back-of-the-napkin calculations.</p><p>We built a custom machine learning model for this purpose. At the same time, we tried the <a href="https://facebook.github.io/prophet/">Facebook Prophet</a> tool, which forecasts time-series data with seasonality. Here, I’ll explain how we approached building our own machine learning model and will explain step-by-step how we achieved that goal. The reason to build this model was to have a tool we could use to fine-tune and add more features for better predictions in the future. I have to admit that it was fun too!</p><p>Since I was more familiar with the <a href="https://www.r-project.org/">R</a> statistical computing language, I decided to use it in this project. We generated a dataset from our application metrics using our Application Performance Management (APM) tool and used it in both of the models (our model and the Prophet model) for validating the results. We then monitored the predicted values vs. the actual peak traffic values every week and tuned our model. This approach was very useful, as our predictions for future peak seasons were adjusted every week based on new data points. Let’s see how we built our model.</p><h3>How to set up an R environment</h3><p>First, I installed the <a href="https://www.r-project.org/">R</a> SDK using Homebrew on my MacBook. Alternatively, the R SDK can be downloaded directly from their <a href="https://cran.r-project.org/mirrors.html">mirrors</a> for other operating systems.</p><figure><img alt="Screenshot of terminal commands to install R using Homebrew on a Mac" src="https://cdn-images-1.medium.com/max/1024/1*zVryusJgyj1HekpPUF055A.png" /></figure><p>Then, I installed RStudio as my IDE for R development. The community edition of <a href="https://rstudio.com/products/rstudio/download/">RStudio Desktop</a> was good enough. The following diagram should help you to get familiar with RStudio Desktop quickly.</p><figure><img alt="Screenshot of the RStudio Desktop IDE" src="https://cdn-images-1.medium.com/max/1024/1*p-EYKyzAJ4JACNcO2n8Daw.png" /><figcaption>RStudio Desktop</figcaption></figure><ul><li>Panel 1— Coding area where you write your R programs and save them to files.</li><li>Panel 2 — Shows the variables and data loaded in the environment.</li><li>Panel 3 — R console shows the running commands and output. You can also run ad hoc R commands in it.</li><li>Panel 4 — Shows various information including the visual output of the program (e.g., generated graphs and plots).</li></ul><p>Getting familiar with a language and an IDE is very useful for any kind of programming. You don’t have to use R. These days, Python-based tools are widely used for data science work. You should be able to achieve what I am describing here with your preferred language and tools.</p><h3>How we captured data</h3><p>In machine learning, data is the the key ingredient of the recipe. You have to be aware of your systems and know what types of data are available before using them in your models. I talked to a data scientist at Vrbo™ (part of Expedia Group™) about my model and his immediate reaction was:</p><blockquote>How did you capture that dataset?</blockquote><p>I realized that we engineers have access to various forms of rich datasets, but we may underestimate the importance of that data.</p><p>For this project, we manually generated the dataset from the APM metrics. We use a third-party <a href="https://en.wikipedia.org/wiki/Content_delivery_network">CDN</a> service provider at the edge of our network. We send access log metrics data from the CDN to our APM tool for both cached requests (static contents served from CDN) and non-cached requests (dynamic content coming from web services). This shows the daily access-log metrics data for a week in a time-series graph:</p><figure><img alt="Graph showing traffic undulating with daily rhythm, with one slightly higher than its peers" src="https://cdn-images-1.medium.com/max/1024/1*iiDz8YJ5DrNk5vtWHsLW4Q.png" /><figcaption>Daily traffic data for a single week</figcaption></figure><p>In this exercise, we were interested only in predicting peak traffic, so we generated a <a href="https://github.com/lasanthak/traffic-forcast/blob/main/src/sample-data.csv">time-series dataset</a> from the metrics data for non-cached requests by taking the highest weekly traffic values as individual data points. Traffic was calculated as requests-per-second (rps) and time was represented by the week number (w). Due to the retention policy of our metrics data, we had a limited dataset to work with (slightly over 120 data points for training).</p><figure><img alt="Table showing requests per second by year, month, and week" src="https://cdn-images-1.medium.com/max/1024/1*PzSYIzAX34cefet9bCyt6w.png" /><figcaption>Sample dataset for the time series data</figcaption></figure><h3>Studying the dataset</h3><p>Understanding the dataset by analyzing it using various methods is essential for better prediction. Simply by drawing a quick plot of data, I was able to identify the following features about the dataset.</p><figure><img alt="R code: plot(past_data$value~past_data$w, type=’b’, cex=0.3, col=’Red’)" src="https://cdn-images-1.medium.com/max/1024/1*btGjKJBVMwlR0FEjzRAVTw.png" /></figure><figure><img alt="Graph of dataset showing peaks and an overall upward trend" src="https://cdn-images-1.medium.com/max/1024/1*XKhfpaEObnepNHfAYVjYqw.png" /><figcaption>Basic plot of the dataset</figcaption></figure><ul><li>Data shows seasonality (with yearly cycles).</li><li>Every year the peak traffic occurs around the same time.</li><li>The traffic is trending upward over several years.</li><li>There is an anomaly due to the COVID-19 downtime.</li></ul><p>It was also useful to understand how the weekly peak traffic values were spread across each month. The following box plots show the mean and variance of the monthly peak traffic values. Insights into the dataset would help in the process of designing machine learning models.</p><figure><img alt="R code: ggplot(data, group=month, aes(x=month, y=value)) + geom_boxplot(aes(color(=factor(month)))" src="https://cdn-images-1.medium.com/max/1024/1*4BYCKijf5RMC5cd48r2Iaw.png" /></figure><figure><img alt="A box plot graph" src="https://cdn-images-1.medium.com/max/1024/1*NtGskvhv4B9zsdqQFE_6og.png" /><figcaption>A box plot for 2019 data</figcaption></figure><figure><img alt="A box plot graph" src="https://cdn-images-1.medium.com/max/1024/1*X-Ur3uQHAigjNr3DLlb6GA.png" /><figcaption>A box plot for 2020 data</figcaption></figure><p>From what we learned to this point, it was clear that there was an anomaly during the downtime due to the COVID-19 shutdown period in 2020. I decided to exclude the data points for those months from the training set and to use an 80–20 split for training and testing. Make sure you are spending enough time in this phase to get to know the dataset. That will lead to a better forecasting model for your data.</p><h3>Modeling seasonality</h3><p>Now that we have an idea about the dataset, we know that the data shows seasonality. How can we model seasonal data? That’s where we can use a well-known mathematical formula called the <a href="https://en.wikipedia.org/wiki/Fourier_series">Fourier series</a>. The Fourier series has been used to compose periodic functions using sine and cosine values of harmonically related frequencies. The mathematical formula goes like this:</p><figure><img alt="Mathematical formula for the Fourier series" src="https://cdn-images-1.medium.com/max/1024/1*M7kZZv72qEmbcSxPaOp2nA.png" /></figure><p>We learned that the Facebook Prophet tool also uses the Fourier series to model the seasonality in the data. This <a href="https://www.analyticsvidhya.com/blog/2018/05/generate-accurate-forecasts-facebook-prophet-python-r/">article</a> about Facebook Prophet explains their approach in details. I also used the same technique to model the seasonality in our data into a linear regression model. I had to do some simple <a href="https://en.wikipedia.org/wiki/Feature_engineering">feature engineering</a> work to define a new time variable t using a 52-week period P to go with a yearly seasonality.</p><figure><img alt="R code: data$t = (2 * pi * data$w) / 52" src="https://cdn-images-1.medium.com/max/1024/1*4jTnDJ0jVZ8k36ArHrkcKg.png" /></figure><figure><img alt="Graph compare training data with fitted curve" src="https://cdn-images-1.medium.com/max/1024/1*1qD7WRGzWVPvfeSb2inrXw.png" /><figcaption>Fitted curve modeling seasonality with Fourier series</figcaption></figure><p>I like to highlight the importance of interpreting the data correctly for the purpose of creating a model that best fits your dataset.</p><h3>Trend analysis</h3><p>Even if the Fourier series did a good job in modeling seasonality, the model was not sensitive enough for the trending shifts (see Picture 10). We decided to do further feature engineering work to introduce a new variable trend that predicts the trending of the traffic over time. For that purpose we have used a <a href="https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/smooth.spline">smooth spline</a> function with smoothing parameter (spar) set to 1.</p><figure><img alt="R code: train.trend_spl = smooth.spline(train$value~train$t, spar=1.0); data$trend = predict(train.trend_sp1, data$t)$y" src="https://cdn-images-1.medium.com/max/1024/1*HPRHFx0W4l-wt2Dzt8l8ww.png" /></figure><p>That gave us a smooth trend line that was sensitive enough to major shifts of the traffic patterns over time (see graph below). The data points for the COVID downtime (the shaded area) were excluded from trend analysis. We can further experiment with the spar parameter to adjust the sensitivity of trending to our liking.</p><figure><img alt="Seasonal graph with a smooth upward trend" src="https://cdn-images-1.medium.com/max/1024/1*Q140aWLdIVJlTxcSJA9W_w.png" /><figcaption>Trend line modeled with a smooth spline</figcaption></figure><p>You will have to model different aspects of the data using different strategies to get a better fit.</p><h3>Predicting</h3><p>After analyzing and preparing the dataset, it was time for the fun part: Predicting the future traffic! I used a <a href="https://github.com/lasanthak/traffic-forcast/blob/main/src/traffic-forcast.R">linear regression</a> model with the aforementioned techniques to model the seasonality and trending because linear regression models are usually good at extrapolating future values for time-series data. I used the actual traffic data (value), time representing the week number (t) and the trending data (trend) from training. With some trial and error, I came up with this model:</p><figure><img alt="R code: lm.fit = lm(value ~ trend + t + I(cos(t)) + I(sin(t)) + I(cos(2*t)) + I(sin(2*t)) +… (and so on up to 7), data = train)" src="https://cdn-images-1.medium.com/max/1024/1*GY9r8WnCn4iTXSJ2zVnwHA.png" /></figure><p>The following graph shows the predictions and the fitted curve plotted against the real data points. The data points for the COVID downtime (the shaded area) were excluded from this training set as well.</p><figure><img alt="Graph showing fitted curve" src="https://cdn-images-1.medium.com/max/1024/1*BnnHE0dSSBkNeoNzVmP56Q.png" /><figcaption>Predictions and fitted curve against the real data points</figcaption></figure><p>The main benefits of having our own model are that we can fine-tune its parameters and improve its predictions with the addition of new features to the dataset. Why not use the Facebook Prophet for the peak traffic forecasting? The short answer is that we can and do. But one problem with the Prophet tool is that it can only support a single variable (it is univariate). Our model can support multiple variables (it is multivariate). For example, we experimented with adding the “<em>lead time</em>” (time between the book date and the actual trip start date) into our model and found a correlation between the lead time and the peak traffic data.</p><p>We are very satisfied with the results of this project. This linear regression model was compared against the predictions from the Prophet model every week and it proved to be very reliable. The learnings we got out of this excercise were very valuable. At Vrbo, we plan to apply these learnings elsewhere to improve the reliability of our systems.</p><h3>Final thoughts</h3><p>My main goal in this story was not to share numbers or specifics. I wanted to share the story of my journey. I hope this will help or inspire someone who is pondering on applying data science techniques to a problem they are trying to solve. Reach out to us if you have questions or comments about the methods described in here. Thank you for letting me to tell this story. Bon voyage!</p><figure><img alt="Display quote: Life is not about the destination. It’s about the journey." src="https://cdn-images-1.medium.com/max/1024/1*Dv-wCSrjgm7WAX5vUpaylg.png" /></figure><blockquote><a href="https://lifeatexpediagroup.com/technology?utm_source=techblog&amp;utm_medium=career&amp;utm_campaign=postfooter">Learn more about technology at Expedia Group</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62f660f51877" width="1" height="1" alt=""><hr><p><a href="https://medium.com/expedia-group-tech/predict-it-an-engineers-journey-into-data-science-62f660f51877">Predict It! An Engineer’s Journey into Data Science</a> was originally published in <a href="https://medium.com/expedia-group-tech">Expedia Group Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Jarviz, Can You Show My Application Couplings?]]></title>
            <link>https://medium.com/expedia-group-tech/jarviz-can-you-show-my-application-couplings-a9e57f7f76a7?source=rss-4aa892ba9ae9------2</link>
            <guid isPermaLink="false">https://medium.com/p/a9e57f7f76a7</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[dependencies]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Lasantha Kularatne]]></dc:creator>
            <pubDate>Thu, 09 Apr 2020 19:35:24 GMT</pubDate>
            <atom:updated>2021-05-25T17:28:38.847Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/expedia-group-tech/software/home">EXPEDIA GROUP TECHNOLOGY — SOFTWARE</a></h4><h4>Java bytecode analysis reveals field-level dependencies</h4><p>Interdependencies between software modules are hard-to-avoid conundrums. When your organization grows, these interdependencies multiply, potentially leading to low cohesion and high coupling among your software modules.</p><p>When I reflect back on the software solutions we’ve built at Vrbo™, part of <a href="https://www.expediagroup.com/">Expedia Group</a>™, I see countless dependencies within our software. We have created so many couplings! Are you in that situation too? What can we do about it? How can we at least identify them?</p><figure><img alt="Jarvis logo © 2020 Expedia Group" src="https://cdn-images-1.medium.com/max/720/1*BRGhxaupGa6dexrSdFFZEQ.png" /><figcaption>Jarvis logo © 2020 Expedia Group</figcaption></figure><h3>What is Jarviz?</h3><p><a href="https://github.com/ExpediaGroup/jarviz">Jarviz</a> is a dependency analysis tool designed for Java applications. Jarviz can provide a complete graph of couplings between Java methods across artifacts. Since non-private methods in a class can be accessed by other classes, method calls can be intertwined creating manifold couplings among each other. Jarviz deeply analyzes Java bytecode to reveal these couplings in a user-friendly format.</p><p>Jarviz consists of three components.</p><p><strong>Jarviz Java Library</strong><br>This Java library scans the Java <a href="https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-6.html">bytecode</a> of binary artifacts using a custom classloader and generates the dependency coupling data as a <a href="http://jsonlines.org/">JSON Lines</a> (.jsonl) file. Currently JAR and WAR artifact formats are supported. To find the dependency couplings, Jarviz inspects the opcodes using static bytecode analysis.</p><p>Here is a sample JSON representation of a dependency coupling generated from the Jarviz Java library:</p><pre>{<br>  &quot;applicationName&quot;: &quot;MyApp&quot;,<br>  &quot;artifactFileName&quot;: &quot;foo-product-1.2.1.jar&quot;,<br>  &quot;artifactId&quot;: &quot;foo-product&quot;,<br>  &quot;artifactGroup&quot;: &quot;com.foo.bar&quot;,<br>  &quot;artifactVersion&quot;: &quot;1.2.1&quot;,<br>  &quot;sourceClass&quot;: &quot;foo.bar.MyClass&quot;,<br>  &quot;sourceMethod&quot;: &quot;doFirstTask&quot;,<br>  &quot;targetClass&quot;: &quot;foo.bar.YourClass&quot;,<br>  &quot;targetMethod&quot;: &quot;getProductId&quot;<br>}</pre><p><strong>Jarviz Graph Tool</strong><br>Jarviz graph tool is a Node.js application designed for graphing the dependency coupling data for Java applications. The dependency coupling data, provided as a <a href="http://jsonlines.org/">JSON Lines</a> (.jsonl) input file, can be generated from the Jarviz Java library. Here is the graphical representation of sample dependency couplings generated using the Jarviz graph tool:</p><figure><img alt="Large graph showing many nodes arranged radially around larger nodes and connected by straight edges" src="https://cdn-images-1.medium.com/max/1024/1*qQMYsP1aYLnCI6Aha1QN-w.png" /></figure><p>The image below zooms in on the highlighted cluster of dependencies in the previous image. The tooltips for each node and edge will provide useful information about particular dependencies.</p><figure><img alt="Radial graph showing many software components depended on by a central one" src="https://cdn-images-1.medium.com/max/1024/1*szhoFEY6jCAPZFcrrrQMoA.png" /></figure><p><strong>Jarviz CLI Tool</strong><br>Jarviz CLI is a command-line tool designed for *nix systems to perform dependency analysis for Java applications. Internally it uses both the Jarviz Java library and Jarviz Graph Tool</p><p>Most users will just need to use the Jarviz CLI tool to find application couplings, so this article will describe the technical details of the Jarviz CLI tool.</p><figure><img alt="decorative separator" src="https://cdn-images-1.medium.com/max/1024/1*vsv7RejfqSHcIc2Q2awEOg.png" /></figure><h3>Dependency couplings</h3><h4>What is a coupling?</h4><p>A coupling is a dependency between two methods. Let’s say method hello of class com.abc.Alpha calls the method world of class com.xyz.Beta. Then there is a dependency coupling between those two methods and Jarviz represents that coupling this way:</p><figure><img alt="Table illustrating the source and target of a coupling." src="https://cdn-images-1.medium.com/max/673/1*4eM3ZS1Zh15T4Ug5WfQa8w.png" /></figure><h4>Why field-level dependencies?</h4><p>There are many tools capable of generating dependency graphs either at to class level or at the library level. This is useful information to have when you are supporting an application. It is also very important to know the dependencies down to the field-level access in certain situations, such as when a class with many fields is shared in many applications that only use a handful of fields each. Jarviz provides dependency coupling data at the field level provided that the internal state of the fields are accessed via method calls. Jarviz analyzes deeply inside the methods, even lambda functions, to extract the dependency data.</p><h4>JVM Internals</h4><p>Java method-level couplings happen when a method of a particular class (call site) invokes a target method of another class (receiver). In order to view these couplings, analyzing bytecode is a better way to view these couplings compared to analyzing the source code. This static code code analysis examines the compiled, optimized Java code that the JVM interprets. The bytecode also contains metadata with more information about the dependency couplings. Therefore, the couplings identified by static code analysis of bytecode are almost identical to the dependency couplings seen by the JVM at runtime.</p><p>The Java virtual machine supports the following invoke commands (opcodes) which can generate couplings:</p><ul><li>invokestatic — Invokes a class (static) method</li><li>invokespecial — Invokes instance initialization methods, private methods, and methods of a superclass of the current class</li><li>invokevirtual — Invokes public and protected non-static methods via dynamic dispatch</li><li>invokeinterface — Similar to invokevirtual, but for dispatching methods based on an interface type</li><li>invokedynamic — Invokes dynamic methods (lambdas) to enabe dynamic languages</li></ul><p>Analysing invokedynamic can be tricky as the method handle can change dynamically at runtime. Java provides a performance optimization for these invokedynamic invocations by providing a bootstrap method handle initially, and executes that method handle if the call site does not change. We are using that strategy to evaluate usages from invokedynamic, but note that static code analysis may not be accurate if the code extensively uses invokedynamic calls to change behavior at runtime.</p><h4>Jarviz Class Loader</h4><p>Jarviz creates a custom class loader based on the <a href="https://docs.oracle.com/javase/8/docs/technotes/tools/findingclasses.html">bootstrap class loader</a> to safely load and unload the classes from the artifact files. The bootstrap classloader is mainly responsible for loading JDK internal classes, typically rt.jar and other core libraries located in the $JAVA_HOME/jre/lib directory. Given that the classes from the artifact have not been already loaded in the bootstrap classloader, we can guarantee that there are no collisions. Jarviz creates a new classloader to analyze each artifact and destroys it and the loaded classes after each.</p><h4>Using the Jarviz CLI</h4><p>Clone <a href="https://github.com/ExpediaGroup/jarviz">the repository</a> and run the jarviz script in the top-level directory. Jarviz will download artifacts from the remote Maven repository if they haven’t been already.</p><p><strong>Format</strong><br>$ jarviz &lt;command&gt; [parameters]</p><p><strong>Commands</strong><br>analyze — Generate dependency data as a JSON file by analyzing Java artifacts.<br>graph — First generate dependency data as a JSON file by analyzing Java artifacts, and then generate the dependency graph as a standalone HTML file.</p><p><strong>Parameters</strong><br> -a &lt;arg&gt;—Path to an input file specifies an application set containing artifacts to be analyzed (JSON)<br> -f &lt;arg&gt;— Path to an input file with RegEx patterns to be used for filtering out the couplings found from the analysis (JSON)<br> -h— To display help<br> -v — To display version</p><p><strong>Example</strong><br>$ jarviz graph -a artifacts.json -f filter.json</p><figure><img alt="decorative separator" src="https://cdn-images-1.medium.com/max/1024/1*YFelTg0GTTuYvgFyeJS18A.png" /></figure><h4>Conclusion</h4><p>This first version of Jarviz provides features to address needs we identified within Vrbo in our cloud migration effort. We found Jarviz useful for identifying application couplings and felt Java engineers elsewhere in the industry could enjoy its benefits. With the help of broader Expedia Group engineering talent and the open source community, we hope that Jarviz will continue to grow and deliver value.</p><blockquote><a href="https://lifeatexpediagroup.com/brands">https://lifeatexpediagroup.com/brands</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a9e57f7f76a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/expedia-group-tech/jarviz-can-you-show-my-application-couplings-a9e57f7f76a7">Jarviz, Can You Show My Application Couplings?</a> was originally published in <a href="https://medium.com/expedia-group-tech">Expedia Group Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>