<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Lime Engineering - Medium]]></title>
        <description><![CDATA[Lime‘s Engineering Blog for Software Engineering and Data Science - Medium]]></description>
        <link>https://medium.com/lime-eng?source=rss----266a88f54bca---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Lime Engineering - Medium</title>
            <link>https://medium.com/lime-eng?source=rss----266a88f54bca---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 18:14:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/lime-eng" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Lime Analytics Engineering: Modeling GovTech]]></title>
            <link>https://medium.com/lime-eng/lime-analytics-engineering-modeling-govtech-67134e58a89d?source=rss----266a88f54bca---4</link>
            <guid isPermaLink="false">https://medium.com/p/67134e58a89d</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[analytics-engineering]]></category>
            <dc:creator><![CDATA[Cody Heaton]]></dc:creator>
            <pubDate>Wed, 21 Aug 2024 18:08:03 GMT</pubDate>
            <atom:updated>2024-08-21T18:08:03.092Z</atom:updated>
            <content:encoded><![CDATA[<p>GovTech systems, designed to help businesses manage locality-specific compliance, require data modeling that involves navigating the complexities of spatial, temporal, and density attributes — each of which can be multivariate and intricate. At Lime, where managing a global fleet of vehicles is critical for operations, the stakes are especially high. Accessibility and availability are paramount to supporting our broad spectrum of stakeholders, from analysts to government relations and operations leaders.</p><p>As Analytics Engineers, our challenge is to transform these complex attributes into digestible and useful data products. But how do we achieve this in a way that is both performant and maintainable? This blog post delves into how our team employs dbt, a powerful tool in our data engineering toolkit, to implement advanced dimensional modeling techniques. We’ll explore the use of dimension-to-dimension joins, bridge tables, and SCD Type-2 tables, which are typically maintenance-heavy. Our goal is to showcase how we model operational data sources with complex relationships in a flexible, user-friendly manner, enhancing understanding and usability of the dimensional model.</p><h3>GovTech Specifically at Lime and why it’s not Unique</h3><p>Predicting the exact needs of city governments is a formidable challenge due to the diverse and unique requirements of each cityscape. This complexity isn’t unique to Lime — many businesses face similar challenges when operating across numerous jurisdictions, each with its own set of rules and expectations.</p><p>At Lime, we address this variability through a system we call “Operational Targets.” This system allows our operations team to collect a wide range of inputs that reflect the specific requirements of each market. For example, these inputs might include data on local traffic patterns, regulatory restrictions, or public transit integration, ensuring that our services align closely with local needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*emukRqUfj3MGb3V0" /><figcaption><a href="https://www.sfmta.com/maps/map-powered-scooter-neighborhood-san-francisco">An Example of Areas in San Francisco with Fleet Requirements</a></figcaption></figure><p>But collecting this data is only part of the solution. It’s equally crucial to integrate these inputs effectively into our data warehouse, making them accessible and actionable for our stakeholders. This integration enables us to transform raw data into valuable insights, which in turn supports decision-making across functions — from government relations to strategic planning.</p><p>By surfacing this tailored data in our data warehouse, we empower our analysts and product leaders with the nuanced understanding needed to optimize operations and meet diverse regulatory environments effectively.</p><h3>GovTech Complexity</h3><p>The needs of modeling compliance are ever-evolving, often without much notice. In the context of understanding where our fleet was and when, it’s essential to maintain a view of history while planning for the future.</p><p><strong>Methods of Counting:</strong> We need to account for varied methods of compliance. Some cities prefer to measure vehicles in the public right of way (PROW) using hourly average while others prefer to look at the max deployed at any given time. In addition, opposed to absolute counts of vehicles, we have percentage based counts where x% of vehicles in the entire city need to be in a given subsection of the city. Point being, we will want to calculate the data in different ways.</p><p><strong>Temporal: </strong>Our vehicles are deployed 24/7, however, from a compliance standpoint we must be able to measure where they are at certain times of the day. For example, we may need to maintain a minimum number of vehicles in a given area between 6–9 AM and then 5–8 PM. From a modeling perspective, we’re inclined to create columns for each set; time_span_one &amp; time_span_two. However, it’s difficult to predict the number of time spans, what if tomorrow a city decided they now want 3 time spans? We’d need to rebuild our table!</p><p><strong>Spatial:</strong> Where our vehicles are deployed in a given city matters <em>a lot</em>, both from a revenue and compliance perspective. There are two main problems to address: measure changes to shapes of city sections and managing denominator shapes for percentage based counts.</p><p>For example, if we need to have a minimum of 500 vehicles in a section of a city, we’ll want to track over the changes to the boundary of this section over time. This example could just as easily be deploying 10% of all vehicles to a certain subsection. Not only will we have to manage the ever-changing spatial shapes, but also the shapes of the denominator spatial shapes (which could be more than 1 shape)!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/0*PMIc5tBK_CdXI-tG" /><figcaption><a href="https://www.minneapolismn.gov/government/government-data/datasource/bike-scooter-dashboard/"><em>Cities Track the Maximum Deployment of Fleet in Administrative Areas (Downtown)</em></a></figcaption></figure><h3>Our Technical Solution</h3><p>In our pursuit to manage complex compliance and operational data effectively, <a href="https://docs.getdbt.com/reference/commands/snapshot">dbt snapshots</a> emerge as a pivotal solution. These snapshots are especially adept at creating and managing Slowly Changing Dimension (SCD) Type-2 tables, which allow us to look back in history and understand performance within the context of past configurations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8_5bfOnkuZL26LHWSEM2DQ.png" /><figcaption>Example of Operational Target Table Column Attributes</figcaption></figure><p><strong>Taking inventory of the modeling concepts:</strong></p><ul><li><strong>count_method:<em> </em></strong><em>standard SCD type-2 attribute</em></li><li><strong>timeframe: </strong>managed with <em>a bridge table to manage the multivariate nature of time</em></li><li><strong>Spatial_id &amp; denominator_spatial_id:</strong> <em>utilize a dimension-to-dimension join to capture spatial attributes over time and manage the relationships between multiple spatial dimensions.</em></li></ul><p>Let’s dive into how we’ll leverage <strong>dbt snapshot</strong> to make managing these techniques a breeze.</p><h3><strong>dbt snapshot</strong></h3><p>dbt’s snapshot feature simplifies the heavy lifting involved in SCD type-2 transformations and the integration of complex bridge and dimension-to-dimension joins. Here’s how we implement it:</p><ol><li><strong>Load Dimension Attributes</strong>: All dimension attributes are loaded into an intermediate layer model using the current state of the attributes. In addition, we keep the arrays for time and spatial data unflattened. This setup preserves the flexibility to adapt to changes in compliance requirements.</li><li><strong>Generate Surrogate Keys</strong>: Using dbt’s <strong>generate_surrogate_key</strong> function, we create unique keys for both the timeframe and denominator spatial attributes. This facilitates the creation of bridge tables and ensures consistent reference points across different snapshots.</li><li><strong>SCD Type-2 Transformations</strong>: Lastly, a dbt snapshot model performs the SCD Type-2 transformation.</li></ol><p>That is the bulk of the dimensional modeling; dbt snapshot handles the SCD type-2 transformations and our bridge tables are effectively already built within the snapshot table.</p><h3>Surfacing Models to the Data Mart</h3><p>Once the dbt snapshot has completed the foundational data transformations, our next step is to make this enriched data accessible and practical for end-user queries. This involves surfacing the snapshot table to our data mart, which represents the most refined and business-ready data in our warehouse.</p><p>To maximize the usability of our snapshot table, we implement a series of lightweight views:</p><ul><li><strong>Dimension View</strong>: Focuses on single-variate attributes, providing a clean and straightforward representation of each operational target.</li><li><strong>Timeframe View</strong>: Isolates the timeframe attribute, facilitating temporal analysis by flattening the relevant arrays and aligning this data with our standard time dimensions.</li><li><strong>Spatial View</strong>: Handles the denominator spatial attributes, ensuring that spatial calculations and visualizations are consistent and accurate.</li></ul><p>These views serve not only to align the column naming conventions across our datasets but also to simplify the orchestration of data queries. By flattening arrays in dedicated views, we ensure that each aspect of our data can be easily accessed and manipulated as needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*41SW2rCS4uK5JFSc" /><figcaption>General Lineage</figcaption></figure><p>The resulting data lineage is streamlined and efficient, designed to support a wide range of queries and analyses. This structure allows stakeholders to trace changes over time through a dimension table that contains one row per operational target, meeting the general use-case of querying operational target changes.</p><p>Furthermore, for stakeholders requiring more detailed analysis, the setup democratizes access to array transformations. Easy joins to the two bridge tables allow for scalable handling of complex queries. For instance, the pipeline computing fleet distribution can now effortlessly filter and group vehicles using the criteria established in these views.</p><h3>Final ERD Diagram</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K24DE_nP17ka-Jqx" /></figure><h3>Elevate Your Data Strategy with dbt Snapshots</h3><p>dbt is the next-generation tool in data stack technology, the robust capabilities of dbt snapshots are a microcosm of the full feature set. Uplevel your data modeling processes with dbt’s cutting-edge approach that simplifies management of complex, evolving datasets.</p><p>Join the Community: Connect with us and other data professionals on the <a href="http://getdbt.slack.com/">dbt Slack community</a>. Exchange ideas, ask questions, and stay at the forefront of data engineering innovations. Tell us how you’re leveraging dbt to transform your data!</p><p>Career Opportunities: Inspired by what you’ve read? We’re looking for talented individuals to join our team. Explore our <a href="https://www.li.me/about/careers">careers page</a> to find opportunities to work on exciting projects like those discussed here. Help us shape the future of transportation and solve tough problems along the way.</p><p>Take your data architecture to the next level with dbt. Join us in redefining the possibilities of analytics engineering.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67134e58a89d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lime-eng/lime-analytics-engineering-modeling-govtech-67134e58a89d">Lime Analytics Engineering: Modeling GovTech</a> was originally published in <a href="https://medium.com/lime-eng">Lime Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Refine, Retrieve, Rebuttal: Lime’s RAG Flow]]></title>
            <link>https://medium.com/lime-eng/refine-retrieve-rebuttal-limes-rag-flow-90fd4d13c1a4?source=rss----266a88f54bca---4</link>
            <guid isPermaLink="false">https://medium.com/p/90fd4d13c1a4</guid>
            <dc:creator><![CDATA[Chao Ma]]></dc:creator>
            <pubDate>Mon, 22 Apr 2024 23:54:25 GMT</pubDate>
            <atom:updated>2024-04-22T23:54:25.798Z</atom:updated>
            <content:encoded><![CDATA[<p>Starting the new year with a tech project is exciting, especially when it involves cutting-edge AI tools. Imagine being tasked with creating an RAG (Retrieval-Augmented Generation) chatbot for your organization — what an adventure! This task initially seems like a breeze thanks to the latest OpenAI models, like the remarkably accurate GPT-4 and its turbo version.</p><p>Partner this with the LangChain framework, and you have a solid foundation for rapidly developing an efficient RAG application.</p><p>Envision seamlessly integrates your Google Drive repository or JIRA with the chatbot. Combine that with powerful GPT models, and voilà! It’s ready to be launched as a handy Slack bot or a convenient web application.</p><p>However, as the project unfolds, unexpected challenges bubble up. As we are developing our own RAG chatbot, we have to be prepared to deal with challenges such as these: Here’s a few of these imaginative hurdles:</p><ul><li><strong>Curbing Hallucinations: </strong>Despite LLM’s prowess, it sometimes fabricates answers, known as ‘hallucinations’, particularly when relevant data is absent.</li><li><strong>Dealing with Jokes: </strong>Humor is part of any workplace, and people love to test AI’s wit. How can you detect and block jokes?</li><li><strong>Contextual Integration: </strong>Merging conversations can be challenging because it’s only sometimes appropriate to do so. We need to consider the context carefully because if a question relies on information from a previous one, merging without context could lead to the retrieval of incorrect knowledge. Conversely, if we indiscriminately merge all questions, we might retrieve irrelevant or inaccurate information when questions are unrelated.</li><li><strong>Knowledge staleness: </strong>Knowledge can age like fine wine or sour milk. Old documents with outdated information may have high similarity to the question.</li><li><strong>Unknown Answer Accuracy:</strong> LLM models have no confidence scores, so we are not able to know how confident the answer is. How doyou get to know the quality of answers?</li><li><strong>Language Barriers: </strong>Language diversity adds complexity. If the question is in Italian, the knowledge is in French, and your prompt is in English, it is challenging for your LLM.</li><li><strong>Sophisticating Retrieval: </strong>Distinguishing between a sea of similar knowledge is vital, especially when crucial details differ. If the embedding model cannot cover all the cases, how can you handle the failures?</li></ul><p>While there are countless challenges we could dissect, today our spotlight shines brightly on the solutions. It is these creative fixes and strategic improvements that will transform our RAG retrieval chatbot from a concept to a cornerstone of efficiency in our organization. Let’s dive into the world of possibilities and explore how we can optimize our AI assistant to its full potential.</p><p><strong>Diagram of Lime RAG Flow</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/0*--QpuZ5u8ihFGJNk" /></figure><p><strong>Pre-Processing: Refine</strong></p><p>Instead of using the original question to retrieve knowledge — which often leads to numerous inaccuracies directly — it’s better first to refine the question using an LLM to mitigate potential issues. Here’s how to improve the questioning process:</p><ul><li><strong>Detect Jokes:</strong> Ask your LLM whether the question is humorous. If it confirms the question is a joke, do not proceed with it.</li><li><strong>Context Merge</strong>: Present the LLM with the current question as well as several preceding ones from the same user. Inquire whether the question is entirely new or if it needs to be merged with previous ones. If merging is required, ask the LLM to formulate a new question that encapsulates the user’s overarching intent.</li><li><strong>Entity Extraction: </strong>This step is crucial. Have the LLM identify key entities within the question. This prevents scenarios where the user is asking about your generation-4 products, but the knowledge base returns information on generation-3 products due to similarity in questions. By extracting entities such as “generation-4,” the LLM can help target the response more accurately.</li><li><strong>Language Detection</strong>: Request the LLM to identify the language the user is currently employing.</li></ul><p><strong>Similarity Search: Retrieve</strong></p><p>To enhance the quality of our similarity search, we’ve incorporated the following factors:</p><ul><li><strong>Fundamental Similarity: </strong>This is the degree of similarity between the user’s question and the knowledge available in our database.</li><li><strong>Entity Similarity:</strong> We assess the similarity of entities detected in the question to ensure the documents retrieved are relevant and contain the queried entities.</li><li><strong>Time Decay:</strong> We consider the age of each piece of knowledge, which may be indicated by the creation date of JIRA tickets or the last modification timestamp of documents.</li><li><strong>User Feedback:</strong> User responses to our search results are used to refine the system, with positive feedback increasing and negative feedback decreasing the knowledge’s relevance score.</li></ul><p>Be wary of a common pitfall: most Retriever-Augmented Generation (RAG) sample codes instruct you to configure ‘K,’ which determines the count of knowledge pieces retrieved. However, based on our experience, it’s also crucial to set another parameter, ‘min-token,’ which requires the retrieval process to collect at least ‘X’ tokens before stopping. This step is important because if ‘K’ is set to 1 and the retrieval only fetches a piece of knowledge that consists solely of a title from your repository, it would be insufficient for answering questions.</p><p>At Lime, we’ve developed our retrieval framework to maintain complete control over all aspects of the retrieval process, instead of using third-party frameworks like Langchain.</p><p><strong>Post Processing: Rebuttal</strong></p><p>After sending the refined question along with the retrieved knowledge to your LLM for an answer, it’s crucial to be prepared for the subsequent step: Rebuttal.</p><p>There are instances where the knowledge sourced may not provide an adequate answer due to it being unavailable in the knowledge base, perhaps because of a technical issue. In such situations, our bot might encounter three specific problems:</p><ul><li>Failure: If the LLM cannot provide an answer and responds with an “I don’t know” or a similar statement, it’s essential to gracefully handle the inability to generate an appropriate answer. Craft a polite and understanding response to the user instead of displaying the unhelpful answer.</li><li>Hallucination: Your bot might generate an answer that is unrelated to the knowledge within your domain. It’s a significant concern in current LLMs. To mitigate this, verify the accuracy by checking if the LLM’s answer aligns with the retrieved knowledge. If there’s no correlation, it’s safer to withhold the answer.</li><li>Tone discrepancies. The LLM might respond with overly technical details to a non-technical user or provide an excessively brief reply. Reassess and adjust the tone of the answer with the LLM, prompting it to rephrase accordingly to meet the expected conversational tone, if necessary.</li></ul><p>Lastly, asking the user if the bot’s response met their expectations is important as it serves as the final safeguard. If they give a thumbs up, make sure to enhance the knowledge base for similar future inquiries. Conversely, if the response is unsatisfactory, consider reducing reliance on that knowledge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EUT31zVwjx34K0CF" /></figure><p>Building an AI-driven chatbot is quite challenging. Some organizations opt to fine-tune open-source models, such as Gemma or Llama, we gave up this solution due to the rapid growth of our knowledge repository and the impracticality of daily retraining. Others have suggested using LLMs that can now handle inputs up to 1 million tokens, potentially reducing retrieval efforts. However, this raises issues of increased cost and latency.</p><p>We have maintained our commitment to the RAG solution and, after months of persistent effort, we have created a unique process called the Refine/Retrieve/Rebuttal flow. This method may diverge significantly from the workflows found in open-source projects, but it is the result of extensive analysis of hundreds of problematic cases, meticulously considered for weeks.</p><p>Finally, imagine that your chatbot is a diligent, tireless entity that thinks quickly but sometimes lacks knowledge and is prone to hallucinations. Constructing a workflow that helps it comprehend questions, locate knowledge, and verify its responses is, without a doubt, your most crucial task, right?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90fd4d13c1a4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lime-eng/refine-retrieve-rebuttal-limes-rag-flow-90fd4d13c1a4">Refine, Retrieve, Rebuttal: Lime’s RAG Flow</a> was originally published in <a href="https://medium.com/lime-eng">Lime Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leverage Daily Emails with Key Metrics at Lime]]></title>
            <link>https://medium.com/lime-eng/leverage-daily-emails-with-key-metrics-at-lime-5c8dccb44f89?source=rss----266a88f54bca---4</link>
            <guid isPermaLink="false">https://medium.com/p/5c8dccb44f89</guid>
            <category><![CDATA[reporting]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[tooling]]></category>
            <category><![CDATA[micromobility]]></category>
            <dc:creator><![CDATA[Yin Zhang]]></dc:creator>
            <pubDate>Wed, 04 Aug 2021 23:56:10 GMT</pubDate>
            <atom:updated>2021-08-10T17:50:14.932Z</atom:updated>
            <content:encoded><![CDATA[<p>At Lime, we rely on internal data reports to guide many of our key business decisions, and we’re constantly looking for ways to make these reports as accessible as possible to our employees. To that end, historically we’ve leaned heavily on our custom-built web portal, called the “data portal,” which consolidates data from many different avenues such as Google Sheets, Tableau Dashboards, and our own custom-built, web-based dashboards.</p><p>However, our employees experience significant friction as users of the data portal. To get the data they need, employees must log into the data portal, find the dashboard they need, set their desired filters, and view and analyze the data returned. Sometimes, users of our data portal need to repeat this process on a daily basis, especially when monitoring our high-level business metrics.</p><p>Instead of requiring our employees to visit the data portal every single time they need to access the same set of metrics, we wanted to figure out a way to push that data directly to them. We ultimately decided to do this via automated email reports that are sent directly into our employees’ inboxes.</p><h3><strong>High Level Design</strong></h3><p>To build these email reports, we need to address these problems:</p><ul><li>Where do we get accurate data to compose the report?</li><li>How do we send the reports to users periodically?</li><li>How do we customize the content of reports for each user?</li></ul><p>Given that the data engineering team at Lime developed a data ecosystem with Lime Data Warehouse (LDW), which serves as our single source of truth for data analysis, and the data portal web application that serves as a single entry point for data consuming, we were able to leverage the LDW and the data portal to build out the email reporting system.</p><p>So to answer the questions in the previous section accordingly:</p><ul><li>We can query the LDW reporting layer in SnowFlake to get the aggregated data for reporting purposes.</li><li>The reports can be generated with prebuilt HTML/CSS/Javascript templates and sent to users with SendGrid by running a scheduled job on the data portal server.</li><li>Users can customize and subscribe/unsubscribe to reports on the data portal client with preferences data stored in the MySQL database.</li></ul><p>The following diagram shows what the overall design looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d6sy4i0DsOlIwmw0" /><figcaption>Diagram for high level design</figcaption></figure><p>In the end, we designed two solutions for the email reporting: the Daily Overview Report and the Metrics Summary Report. These two reports serve different purposes:</p><ol><li>The Daily Overview Report is the “One Report For All,” meaning that it is the same report for all subscribers. Users subscribe to these reports by joining a specific mailing list. This report provides a generalized overview of company key business metrics everyday.</li><li>The Metrics Summary Report is a customizable report for any user. Users can specify what data they want through the data portal client, and a report with specialized content will be sent to each subscriber based on their preferences. This report especially enables the operators to watch multiple regional data together side by side.</li></ol><h3><strong>Implementation details</strong></h3><p>The design and implementation behind the scene for the two reports are mostly the same. The data portal server first connects to Snowflake ( “<a href="https://www.npmjs.com/package/snowflake-sdk">snowflake-sdk”</a>) and queries the data from LDW to create the reports. It then uses SendGrid to send the emails. This process is run on a daily cron job (“<a href="https://www.npmjs.com/package/node-schedule">node-schedule</a>” ).</p><h4><strong>Daily Overview Report</strong></h4><p>The Daily Overview Report serves as a performance summary for all of our markets and products.</p><p>The report is broken down into sections/subtables that display key metrics for each region, broken down by product. A product represents a mode of transportation, and today we provide 3 modes: scooters, e-bikes, and e-mopeds. Having dedicated sections for each mode allows our internal stakeholders to see how our metrics are doing on both a market level and a product level.</p><p>In addition to product-specific sections, the report contains an “All Products” section to show how each market is doing across all modes, as well as a “summary” section that shows how the company overall is doing for each mode. With all of this in mind, we fetch the relevant metrics (and calculate their week-over-week change) for the appropriate product and markets and use that data to populate our report. Below is an example of what the daily email report looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LruElxqehFyJjiVfoOsPXg.png" /><figcaption>Example for daily overview report</figcaption></figure><p>We also maintain a preset list of metrics to display. These metrics are decided upon by business and data analysts where they determine which metrics can provide the best insights on how the company is performing. These metrics are then calculated and stored within a Snowflake Table that we had mentioned before. When we build out our report, we maintain a hard-coded list of metric config objects that help us determine things like which Snowflake table column do we reference and what format do we use to display the value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*muuJR7sz2M4UyII6iQdB3A.png" /><figcaption>Example for daily overview report configs</figcaption></figure><h4><strong>Metrics Summary Report</strong></h4><p>While the Daily Overview Report sends the same email to all subscribers, the Metrics Summary Report is meant to be more personalized. The report provides configuration options that allows users to select regions and products for which they want to receive daily reports.</p><p>As with the Daily Overview Report, we query Snowflake to get data for the summary report for all regions and products first. With the query results, we construct a hash-map like object in the following shape:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pRIFozyfA8KelO_0hXAjsQ.png" /><figcaption>Type definitions for data used for reporting</figcaption></figure><p>Here, the key is the region that represents each market for reporting, the value is its corresponding reporting data. In the value object, the current and prev object are used to calculate the week over week difference for metrics.</p><p>Unlike the Daily Overview Report where we broke it down into smaller subtables, the Metrics Summary Report is sent as one consolidated table. This is because one single Metrics Summary Report will contain fewer regions due to users typically selecting only a few regions that are relevant to them. Furthermore, consolidating region-based and product-based data into one table makes it easier to do a side-by-side comparison as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jsdhkmWc_ixv4Ve_HxkLFQ.png" /><figcaption>Example for metrics summary report</figcaption></figure><p>Another key difference between the two email reports is that the product and region values of the Metrics Summary are dynamic as they are based on the user’s preference. This is where the hashmap object comes into play. Instead of making a query for each user preference, we run a single query to fetch metrics for all regions and products, and preload them into the hashmap to reduce database I/O. When we go on to create an email report for a user, we use the hashmap to quickly fetch the metrics for that specific region and product, and then populate the table content.</p><p>To store a user’s email report preferences, we created a table `email_report_subscription` in data portal’s MySQL database:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2RiJQSxlfLrdHnt0ecmuNA.png" /><figcaption>schema for email_report_subscription table</figcaption></figure><p>The report_id field is used to identify which report the user is subscribing to. This column allows us to easily add more email reports with customizations in the future. The regions and products columns are used to keep track of a user’s preferences (for which regions and products does the user want to receive daily metrics for?). The active column is a boolean value which indicates whether or not the user is subscribed to the report.</p><p>Based on these configurations, if a user selected N regions and M products, the table in the email report will have N * M * X cells where X is the number of metrics.</p><p>Although we consolidate all of the data into one table, we still break the table down into subsections for better readability. To implement this, we maintain a list of Section Objects where we store the name of the section as well as the columns/metrics that the section contains, similar to the ones we used in the Daily Overview Report but with an extra section layer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MJ1vMxpvUabG2uafQhj2-A.png" /><figcaption>Example for metric summary report configs (sections)</figcaption></figure><h3><strong>Challenges and Solutions</strong></h3><p><strong>Google Groups were difficult to maintain</strong></p><p>As mentioned before, the Daily Overview Report is sent to certain google groups. However, over time, these email groups were not being maintained regularly which made it difficult to see who was or was not in the list. People would later find out about the report and request to be added to the email group, causing more manual work. To solve this, we came up with a self-serve solution and created a designated group for data reporting. With this new email group, we included a link within the email report that users can use to add/remove themselves from the group.</p><p><strong>Data was not ready when report is sent out</strong></p><p>As mentioned earlier, the content of our email reports came from a Snowflake table in LDW that was updated periodically by Airflow DAGs. We encountered an issue where there was a race condition in which we sent out an email report before the DAG was able to update the table. This resulted in incomplete data within our reports. To fix this, we configured the cron job to send the email at a later hour to give more buffer time for the table to be updated as a temporarily solution. And moving forward we’ll need to optimize the DAG to make the run faster.</p><h3><strong>Future improvement</strong></h3><p><strong>Set up monitoring and alerting</strong></p><p>The daily overview report is something that many people expect to receive. It would be problematic if something unexpected happened and the report was not sent. It would be beneficial to add additional monitoring/alerting to be notified of these issues and enable the team to take more proactive actions.</p><p><strong>Add resend button under admin console</strong></p><p>There are cases when the report is not accurate for external reasons and needs to be resent that day. The current process is very manual where we need to resend the email report through our own dev environment. A future improvement is to surface a resend button in the admin console of our data portal client so that any admin user can go there and resend the report.</p><h3>Lastly</h3><p>Thanks for reading and stay tuned for future posts about the investments at Lime on the toolings that leverages our daily work.</p><p>Also big thanks to Garvit Patel, Can Wang, Jue Zou, the Data-Eng Team and Tooling Team at Lime for initializing and helping out the project.</p><p>Thanks to Steve Jin, Xiuming Chen, Charlie Fang, Ian hook and many others for reviewing this article.</p><p>Co-Author: Yin Zhang, Jiaying Liu</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c8dccb44f89" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lime-eng/leverage-daily-emails-with-key-metrics-at-lime-5c8dccb44f89">Leverage Daily Emails with Key Metrics at Lime</a> was originally published in <a href="https://medium.com/lime-eng">Lime Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Identifying Commute Trips at Lime]]></title>
            <link>https://medium.com/lime-eng/identifying-commute-trips-at-lime-f10c79d7744f?source=rss----266a88f54bca---4</link>
            <guid isPermaLink="false">https://medium.com/p/f10c79d7744f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[user-segmentation]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[micromobility]]></category>
            <category><![CDATA[commuting]]></category>
            <dc:creator><![CDATA[Nick Dulchin]]></dc:creator>
            <pubDate>Tue, 06 Apr 2021 21:31:01 GMT</pubDate>
            <atom:updated>2021-04-06T21:31:01.614Z</atom:updated>
            <content:encoded><![CDATA[<p>Understanding why riders choose to ride with Lime is key to designing new product features, customizing how we talk to different riders, deciding which promotions to offer, and understanding what’s driving some of the long-term trends we are observing on the ground.</p><p>In this post, we will walk through how and why we developed a machine learning model to identify commute trips specifically. We follow the commute use case very closely because it helps us understand if we are meeting our reliability goals; reliability must be high for riders to trust us with their commutes.</p><h3>Why Build a Classification Model for Commute Trips?</h3><p>Lime regularly polls riders and non-riders about why they are using (or not) our service. Some of those surveys are administered by our User Research team, some by third parties, and some are directly embedded into our product. In fact, we ask most riders at the end of their trip, to tell us why they used Lime (see below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*CUHnoNug-Go38wmcKVzMmg.png" /><figcaption>End Trip Survey</figcaption></figure><p>While this data can give us general information about Lime usage over a large region, it is too sparse to give us a precise understanding of local or individual behavior. That is, we cannot use this data to micro-adjust our strategy at the local level or send customized marketing or promotions to individual riders.</p><p>For that we need an approach that can generalize our knowledge to all trips, given that we have information on a few trips.</p><p>Enter, machine learning.</p><h3>Building the Model — Feature and Model Selection</h3><p>We choose to model the problem as a binary classification. Every trip gets classified as either “commute” or “not commute”.</p><h3>Step 1 — Feature generation</h3><p>The first step is to collect and choose features to build the model. We initially considered a very large set of potential features based on our understanding of what might correlate with a commute trip. They can be categorized into trip-level features, geographic context features, and rider context features.</p><h4>Trip level features (examples)</h4><ul><li><strong>Total trip duration, trip distance, trip displacement (straight line distance between start and finish): </strong>Commute trips are usually shorter in duration and more direct in route than recreational trips.</li><li><strong>Time of day, day of week: </strong>Commute trips tend to concentrate on weekdays and rush hours.</li><li><strong>Group ride: </strong><a href="https://help.li.me/hc/en-us/articles/360035073193-Group-Ride">Group rides</a> are unlikely to be commutes.</li><li><strong>Weather: </strong>Bad weather usually reduces the number of recreational trips more so than commute trips.</li></ul><h4>Geographic context features (examples)</h4><ul><li><strong>Trip region: </strong>Different cities and neighborhoods have different base rates of commuting.</li><li><strong>OpenStreetMap points of interest: </strong>Simple counts of different point of interest categories (e.g. tourist landmarks, public transit, etc.) can encode information about the origin or destination of a trip, which can correlate with commute or non-commute trips.</li></ul><h4>Rider context features¹ (examples)</h4><ul><li><strong>Count of past trips: </strong>Having many past trips over a long time period makes it more likely that a trip is a commute; having many trips on the same day or past few hours makes it less likely that a trip is a commute.</li><li><strong>Repeat trips: </strong>Commutes are more likely to be repeats of past trips. We can measure this by looking at past trip start and end locations.</li></ul><h3>Step 2 — Feature selection</h3><p>We use <a href="https://christophm.github.io/interpretable-ml-book/shapley.html#shapley">Shapley values</a> as our feature importance metric.² We started with ~100 features and iteratively trained, tuned, assessed performance, and dropped features until model performance degraded significantly.</p><p>The excellent <a href="https://github.com/slundberg/shap">SHAP package</a> <a href="http://google.com/">implements</a> computationally efficient ways to calculate Shapley values and allows us to visually communicate feature importance. Each dot below is a trip. The color of the dot indicates high (red) or low (blue) values for that feature. Being to the left of 0 decreases the likelihood of being a commute, being to the right increases it. Using the day of week feature as an example, we see that the lightest red values (which represent the encodings for Saturday and Sunday) are highly predictive of non-commute trips.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/0*8iiVutR7kLHL8_hh" /><figcaption>Feature Importance</figcaption></figure><h3>Step 3 — Model selection</h3><p>We evaluate different models using ROC curves. An ROC curve illustrates the trade-off between the true positive rate (labeling a true commute as a commute) and false positive rate (labeling a true <em>non</em>-commute as a commute). Better models are located in the top-left hand side of the chart because they accurately label many true commutes (high true-positive rate) while avoiding labeling non-commutes (low false-positive rate). The plot below shows how different models and simple rules of thumb compare. We looked at other metrics as well, but chose to plot ROC over, for example, precision-recall, <a href="https://minds.wisconsin.edu/handle/1793/60482">because</a> there is no extreme imbalance between the proportion of commute and non-commute trips.</p><p>As illustrated in the chart below, machine learning models improve significantly upon simple heuristics including:</p><ul><li>classifying all weekday trips between 7AM-10AM and 4PM-7PM as commutes</li><li>labeling all repeat trips as commutes</li><li>flipping a coin (random classification)</li></ul><p>Many models and feature sets were evaluated in the course of this exercise — and we should stress the iterative part of this process — but only a handful of them are shown below. In the graph we plot the performance of an L1-regularized logistic regression (<a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html">scikit-learn</a> implementation, light-blue), a gradient boosting model (<a href="https://lightgbm.readthedocs.io/en/latest/">LightGBM</a> implementation, dark blue), and an ensemble model that combines those two models into one meta-model (<a href="http://rasbt.github.io/mlxtend/user_guide/classifier/StackingClassifier/">mlxtend</a> implementation, pink).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/0*CSNiYz5aEP1dhBCw" /><figcaption>Model Performance</figcaption></figure><h3>Generalizing the model to all Lime trips</h3><p>The model described in the previous section is trained and tested on a subset of all trips — those that have labels generated by the end-trip survey. This subset is not generated at random and we could run into some issues if the model’s performance degrades when it is exposed to different data.</p><p>In other words — how can we trust that the new model performs well over <em>all </em>Lime trips when it was developed on a small, non-random subset of trips?</p><p>As an example, it could be that healthcare workers — who are likely over-represented in our dataset because it was collected during the Covid lockdown— tend to commute very early in the morning. Our model would then learn that commute trips generally take place early in the morning. When the lockdown ends and the rest of the commuters come back, our model might fail to detect commutes that occur later in the day.</p><p>To be sure, not all biases will cause a degradation in the model’s performance and some of them can be readily fixed with reweighting techniques. But some of them cannot and we still want to estimate how much of an issue they represent.</p><h3>Identifying potential sources of bias and measuring their impact on model performance</h3><p>In this section, we discuss three potential sources of bias and how we addressed them:</p><ul><li>Covid might be disrupting commute patterns</li><li>End-trip survey data (i.e. our training set) is only collected on 4 and 5-star trips</li><li>End-trip survey response rates vary across regions</li></ul><p>We used non-random splits of training and test data to measure the first two sources of bias and used a modified loss function to address the third source of bias.</p><h4>Covid or seasonality induced bias</h4><p>Covid is disrupting commute patterns over time and a model trained during the height of Covid might not generalize to a post-Covid world. To understand the extent of this bias, we compare the model’s performance when the training and testing datasets are generated at random to when the model is trained on (chronologically) earlier data and tested on later data. All of the survey data was collected during the pandemic so this is not perfect, but mobility behavior has still changed over time during this period, so we consider it a useful proxy.</p><p>We do not see significant loss of performance with the chronological split (vs. the randomized split). We therefore estimate that the model generalizes well enough for now. We are continuously monitoring this as more of our riders get vaccinated and we collect new data.</p><h4>Trip rating bias</h4><p>We only collect survey data for 4 and 5 star rated trips yet want to generalize the model to all rated and unrated trips. Again we do not see a significant loss in model performance when training on 5 star trips and testing on 4 star trips and so feel that the model generalizes well enough in this respect.</p><h4>Regional distribution bias</h4><p>Survey response rates vary across regions and we do not have enough labeled data to train region-specific models for all regions. In order to make sure that the model “cares about” or focuses on optimizing for the right data, we need a way to tell the model which regions occur more often in the survey dataset than in the all trips dataset and vice versa.</p><p>To do that, we calculate the percentage of all trips and the percentage of surveyed trips in each region and pass the ratio as a weight in the model’s loss function. This further penalizes bad predictions in regions underrepresented in the survey and lesser penalizes bad predictions in regions overrepresented.</p><h3>Pseudo-labeling and active learning as data augmentation methods</h3><p>In the remainder of this section, we discuss pseudo-labeling and active learning, which are techniques that increase the amount of labeled data available and in doing so attempt to increase the generalizability and performance of the model. Note that these techniques are currently under consideration but have yet to be implemented in the training or serving of our model.</p><p>Pseudo-labeling uses semi-supervised learning to create labels for previously unlabeled data. Unlabeled data in our case are trips without a survey response.</p><p>Different algorithms exist and they generally work by calculating some measure of distance between data points and then performing clustering. Unlabeled data points in close distance to highly uniform labeled data are assumed to have that label. This is done iteratively until clusters converge. Finally, the model is retrained using the new pseudo-labels in addition to the originally labeled data.</p><p>This approach is often used when only a small amount of data is labeled. This is the case for us in smaller and newer regions where we operate. Initial tests show small performance increases and we see the method as useful for expanding the distributional coverage of our data and giving us increased model performance as a result.</p><p>The plot below illustrates the intuition behind pseudo-labeling. Note that only labeled data are shown: commutes in blue, non-commutes in orange. Pseudo-labeling takes advantage of clusters that exist in the data. If an unlabeled data point (again no unlabeled data is plotted below) is close to many commute data points, we might pseudo-label it a commute trip and use it as training data. As a side note, to provide this type of visual intuition we first reduce the many-dimensional feature space used by the model into 3-dimensions using t-distributed Stochastic Neighbor Embedding (t-SNE). t-SNE is a dimensionality reduction technique that attempts to maintain the relative distances between data points that exist in the original space in a lower dimensional space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*YXwZyoUL8Cg9ev60" /><figcaption>Dimensionality Reduced Visualization of Data</figcaption></figure><p>Finally, we are also considering active learning to improve the generalization of the model to all Lime trips. Active learning focuses on the efficient use of manual labeling resources. The goal is to determine which unlabeled data points would be most informative to have labels for. We could then manually derive labels for those data points and in doing so improve the performance and generalizability of the model at a relatively low cost.</p><p>Here is a look at the entire process, from raw data, through transformation, train-test splitting, feature selection, hyperparameter tuning, and finally model selection and deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/0*e2wS32VsrhOi9VOr" /><figcaption>ML Pipeline</figcaption></figure><h3>Summary</h3><p>Identifying why a rider took a trip enables a better understanding of the business, a tailored experience for different rider types, and better tracking of product and operational goals. At Lime, we use a rider survey to generate labeled commute data and ML modeling to generalize detection capabilities to all Lime trips.</p><p>In the post above, we go through feature and model selection, assessment of model performance, and key considerations for generalizing model output to all trips. We also discuss pseudo-labeling and active learning techniques that we are exploring to further improve performance. The current model significantly outperforms previously existing heuristics, is driving new thinking to better serve commuters, and is used to assess service reliability.</p><p>There are always new approaches to consider and improvements to make and we would love to hear your thoughts — please comment below to continue the conversation.</p><p>Thanks for reading and stay tuned for future posts about how we are using modeling and machine learning techniques to extend our rider behavior insights!</p><p>Thanks to Dounan Tang, Justin Bozonier, Tristan Taru, Ruben Kogel, Jeh Lokhande and the rest of the Data Science and Analytics team for their valuable comments and suggestions.</p><p>Thanks to Michael Kronthal and Zach Kahn for their user research insights.</p><p>Lastly, thanks to Jianfeng Hu and Arne Huang for helping troubleshoot the unforeseen complexities of putting this model in production.</p><p>[1]: The decision to use rider context features requires consideration. Downstream, we plan to use commute trips to classify riders as commuters so ideally each trip is considered independently from rider context. Unfortunately (and unsurprisingly), it is hard to get good performance without considering the rider context so ultimately we decided to include these features.</p><p>[2]: Shapley values are a way to assign credit for a joint task among a group of contributors that satisfies a particular set of important properties. Applied to predictive models, the features are the contributors, prediction is the task, and Shapley scores are the credit that each feature deserves. The method reduces the inconsistency and arbitrariness that plagues some other feature importance measures by considering all possible feature permutations when calculating values.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f10c79d7744f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lime-eng/identifying-commute-trips-at-lime-f10c79d7744f">Identifying Commute Trips at Lime</a> was originally published in <a href="https://medium.com/lime-eng">Lime Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why and How We Built a Data Catalog at Lime]]></title>
            <link>https://medium.com/lime-eng/why-and-how-we-built-a-data-catalog-at-lime-6cb79419b7e2?source=rss----266a88f54bca---4</link>
            <guid isPermaLink="false">https://medium.com/p/6cb79419b7e2</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Jiaying Liu]]></dc:creator>
            <pubDate>Wed, 28 Oct 2020 19:31:44 GMT</pubDate>
            <atom:updated>2020-10-28T19:31:44.201Z</atom:updated>
            <content:encoded><![CDATA[<p>At Lime, we serve millions of rides every month. With all of these trips, we collect massive amounts of data that are eventually funneled to our Snowflake data warehouse where the data engineering team creates sanitized and optimized tables for data analysis. These new tables are added to the warehouse along with the raw tables obtained from our production app and other third party services. The end result is that we end up with thousands of tables.</p><p>While it is good that we have all these tables to get data from, the sheer number of them can make it difficult to determine which tables we can use for particular data analysis. Often, we ask ourselves the following questions:</p><ul><li>Which tables can I use to get X?</li><li>What information can I get from table X?</li><li>What does column X in table Y mean?</li><li>How are some tables related via foreign keys?</li></ul><p>With more and more tables, these questions become increasingly difficult to answer. Our solution to this problem was to use a data catalog.</p><h3>What is a Data Catalog?</h3><p>The following is a definition made by <a href="https://www.oracle.com/big-data/what-is-a-data-catalog/#:~:text=Simply%20put%2C%20a%20data%20catalog,support%20data%20discovery%20and%20governance">Oracle</a>:</p><blockquote>“Simply put, a data catalog is an organized inventory of data assets in the organization. It uses metadata to help organizations manage their data. It also helps data professionals collect, organize, access, and enrich metadata to support data discovery and governance”</blockquote><p>Based on that definition, we can view a data catalog as a tool that uses metadata (data about data) to help organize our data assets (in our case, the tables). Our original problem is that we need a way to help our data users discover tables that could help answer their data questions. We need our data catalog to be able to read the metadata that is currently available in our data warehouse and display that metadata to our users in a way that they can easily discover the tables they need.</p><h3>Buy or Build?</h3><p>There are many companies that provide data catalogs as a consumer product. Many of them include interesting features such as usage recommendations and automated data classification. The question that remains is whether or not we should buy the data catalog service from one of those companies or simply build our own. After some thought, we decided to implement our own data catalog due to two main reasons:</p><ol><li>We did not need all of the features that these companies provided in their data catalog products. While a recommendation system based on usage would be nice, we were simply looking for a way for our data users to find data assets that they explicitly searched for. Automated data classification is also not needed in our case because our tables tend to follow the same naming conventions which makes it easier to determine what kind of data it holds. Simply put, we felt that the return on investment for these extra features was not worth it. It was much more cost-effective to simply build out our own data catalog that suits our current needs.</li><li>By making our own data catalog, it was much easier to integrate it into our data tooling platform. This platform is meant to be the one-stop-shop for resources that our data users can use to achieve their data-related tasks. Naturally, it made sense to incorporate our data catalog into our data tooling platform rather than having it exist on a separate, third-party platform.</li></ol><h3>Getting Table Metadata</h3><p>The first challenge is to retrieve the table metadata in our data warehouse. In Snowflake, we can easily figure out the metadata of all tables and views in a particular database with the following query:</p><pre>SHOW OBJECTS IN DATABASE &lt;database_name&gt;;</pre><p>This query will display information such as the name, the schema it belongs to, and the number of rows.</p><p>Now that we have a list of tables/views for a particular database, we can run the following query to get metadata of all columns for a specific table/view:</p><pre>DESCRIBE &lt;object_type&gt; &lt;object_name&gt;;</pre><p><strong>&lt;object_type&gt;</strong> can be either “table” or “view” and <strong>&lt;object_name&gt;</strong> is the name of the table/view we want to get column metadata for.</p><p>Now that we have the queries we need, we need to run them in the data tooling platform’s Node.js backend server to obtain the metadata, process it, and save it in some persistent data storage. We run the queries above using <a href="https://docs.snowflake.com/en/user-guide/nodejs-driver.html">Snowflake’s Node.js Driver</a>. With the data received from these queries, we build out a JSON object that contains an array of table metadata. Once the JSON object is complete, we store it in AWS S3 as a JSON file for future reference. Since tables can be created, modified, and deleted every day, we run this process every day within a scheduled job using <a href="https://www.npmjs.com/package/node-schedule">node-schedule</a>.</p><h3>Displaying the Table Metadata</h3><p>With our table metadata stored as a JSON file in S3, we can create an endpoint on our server that will simply retrieve the file from S3 and return a JSON response. The front-end of our application will then need to display the metadata in a way that makes it easy for our users to access and discover it. An easy to use UI is essential to allow our users to explore our data catalog. Let’s revisit each question that our users could potentially ask.</p><h3>Which Tables Can I Use to Get X?</h3><p>This question focuses more on the problem of discovery. A data user knows what kind of data and information they want, but they do not know where to find it. Because of this, we need our data catalog to have a simple-to-use search tool that filters for specific tables based on the criteria inputted by the user. After planning out the use cases, we decided on this design for the search tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*dhgLAQocf5GNYuSQ2VpIZg.png" /></figure><p>To make searching simple, users can simply type in keywords and the data catalog will return tables that contain the keywords in their metadata. The data catalog will search for occurrences of the keyword(s) within the table name, its columns, and its annotations (more on that later). Users can also restrict their search even further by specifying which fields of the metadata they want to do keyword matching on and which database/schema they want the table to be in.</p><h3>What Information Can I Get From Table X?</h3><p>The user has now found a table based on their search, but now they need to see the actual metadata. For each table, we display key metadata such as its name, which database/schema it belongs to, and the name of each of its columns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hjf0wp_giL9uvDMV3x2AtA.png" /></figure><p>Furthermore, each table result in the data catalog allows users to view sample data to help them get a sense of what the data looks like within the table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yn1Oix5nGAEFSgSYTM8T9A.png" /></figure><h3>What Does Column X in Table Y Mean?</h3><p>When a user finds a table, the part that they are most interested in is the content. Columns are important because they tell the reader of the table what a particular value is supposed to represent. This is why column naming is very important when designing a table. Column names should be descriptive and tell the user what kind of data it contains. When a user sees column X in table Y, they should be able to deduce what column X represents in table Y’s perspective. However, making column names perfectly descriptive is a difficult task on its own. Some column names require some sort of business context to understand. Some columns cannot be described in just a few words, and we cannot make the names too long because it will make the table more difficult to query.</p><p>The solution that the data engineering team decided on was to write column annotations for our tables. Column annotations are essentially comments we leave on our table columns that give more context and insight as to what kind of data a particular column contains. On Snowflake, you can annotate a specific table column with the following query:</p><pre>COMMENT ON COLUMN &lt;table_name&gt;.&lt;column_name&gt; IS &lt;comment&gt;;</pre><p>We save these comment queries within the table’s <strong>Data Definition Language (DDL)</strong> file for better management and maintenance. These annotations serve as notes that help readers understand what the column represents should the name not be enough to understand its meaning. We collect these annotations as part of our metadata fetching mechanism and display them in our data catalog.</p><p>But how do we make all of our table columns annotated? A single table can easily have more than 10 columns, and we have thousands of tables in our Snowflake warehouse. The task of annotating tens of thousands of columns was simply too time consuming and was not worth the investment.</p><p>Instead, we decided to only annotate a specific group of tables. These tables are reviewed and sanitized by our data teams to ensure that they have the highest quality of data and are performant. These are generally the first tables recommended for high-quality data analysis. Because these tables are meant to be used for data analysis, it made sense that we wanted to prioritize them for annotation. Furthermore, because these tables go through a rigorous approval process, these types of tables are created less frequently; as a result, it is easier to maintain annotations for these tables. In order to ensure that our tables are properly annotated before being released, we have an automated checker that ensures the table’s DDL file includes the column annotations.</p><p>It’s great that these high-quality tables are annotated, but what about another table? How will I get the annotations for those? On our data tooling platform, we have a forum where users can ask data-related questions. The questions and answers are saved to facilitate knowledge sharing. If someone has a question about a particular column that is not annotated, they can post a question on the forum and someone on the data team can answer. The answers are saved for other users to refer to should they have similar questions.</p><h3>How are Some Tables Related via Foreign Keys?</h3><p>Some users may also be interested in the relationship between various tables. How does table X relate to table Y? Some tables contain many foreign keys that point to many different tables, and some tables are referred to by many other tables’ foreign keys. To help explain how these complicated relationships work, we created an Entity-Relationship (ER) diagram feature within our Data Catalog to visualize these relationships.</p><p>Some columns exist as foreign keys and their annotations will describe which table and column they are pointing to. From this, we can parse the values we need (the table and column name), and create an Entity-Relationship (ER) diagram to visualize the relationships between our tables. If you are only interested in seeing the relationships for table X, you can search for table X, and the ER diagram will only display tables that have a relationship with table X with a degree of at most 1. This means that it will only show table X, tables that are pointed at by a foreign key in table X, and tables that have a foreign key pointing to table X. Below is a sample of what the ER diagram feature would look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Lg6YibmUt_4Yxf03Z23iA.png" /></figure><p>The ER diagram is built with <a href="https://js.cytoscape.org/">cytoscape.js</a>. In the above example, we see that our TRIP table has a foreign key <strong>REGION_ID</strong> that points to the <strong>ID</strong> column of the REGION table.</p><h3>Conclusion</h3><p>First, let’s summarize the process of how we built out our own data catalog:</p><ol><li>Have our server run queries to our Snowflake data warehouse to get our table metadata.</li><li>Process the metadata obtained from the query results into a JSON object.</li><li>Store the JSON object on S3.</li><li>Repeat steps 1–3 within a daily, scheduled job.</li><li>Have the client retrieve the JSON object from an endpoint on the server that retrieves the JSON from S3.</li><li>Render the table metadata in an easy-to-use UI that provides the following capabilities:</li></ol><ul><li>Search for specific tables based on multiple criteria</li><li>Show sample data of each table</li><li>Display annotations of columns</li><li>Show relationships between tables with an ER diagram</li></ul><p>With complex data schemas, it is difficult for our internal users to find the data that they need to complete their tasks. With the data catalog, we empower users to discover and learn about the tables in our data warehouse by providing an easy-to-use search tool and formatting our table metadata in an easy-to-read format. As our business continues to expand, so will our data and its complexity. As a result, we will eventually need to make enhancements to our data catalog to handle this. These enhancements can include features that exist in third-party data catalogs such as usage recommendations, or features more specific to our company such as matching synonyms based on business context.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6cb79419b7e2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lime-eng/why-and-how-we-built-a-data-catalog-at-lime-6cb79419b7e2">Why and How We Built a Data Catalog at Lime</a> was originally published in <a href="https://medium.com/lime-eng">Lime Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Experimentation Analysis at Lime]]></title>
            <link>https://medium.com/lime-eng/experimentation-analysis-at-lime-bee846d62dd?source=rss----266a88f54bca---4</link>
            <guid isPermaLink="false">https://medium.com/p/bee846d62dd</guid>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Kamya Jagadish]]></dc:creator>
            <pubDate>Sat, 18 Jul 2020 01:29:09 GMT</pubDate>
            <atom:updated>2020-07-18T01:29:09.776Z</atom:updated>
            <content:encoded><![CDATA[<p>Inaccurate experimentation analysis leads to suboptimal business decisions. As Lime grows, the importance of standardized experiment analysis increases commensurately— with hundred of thousands more riders on our platform today than two years ago, the wrong business decision now can impact thousands more riders than it could before. In order to be agile with our testing while maintaining rigor in our experiment analysis, we built an experimentation handbook for teams to reference. The Handbook not only consists of guidance on how to best set up and analyze tests, but has also created a standard set of methods for the Data Science and Analytics team to employ. This blog post includes some sections from our Handbook, which is organized into Pre-Test (test setup), During Test (checking in), and After Test (results) sections.</p><h3>Pre-Test (Set Up)</h3><h4>Picking an Experimentation Setup</h4><p>When determining which experiment setup to use at Lime, there are a few key factors to consider.</p><p><strong>1) What kind of test are we running (i.e. what are we randomizing on)?</strong></p><ul><li><strong>Rider Tests: </strong>Tests affecting the Rider experience such as UI changes or new promotions are Rider Tests with KPI metrics like Rider Retention. These tests are normally randomized on Riders.</li><li><strong>Deployment Tests: </strong>Tests that affect deployment strategies, such as changes to Scooter Hotspots (if permitted to do so), are deployment tests with KPI metrics like trip start conversion. These tests are generally randomized on Scooters.</li><li><strong>Hardware Tests</strong>: Tests on the Scooter hardware or firmware itself, such as an improvement to scooter parts, are generally randomized on scooters.</li></ul><p><strong>2) Do we anticipate the treatment to have network effects?</strong></p><p>After determining whether our treatment should be tested via a Rider, Deployment, or Hardware test, we then need to consider whether our test treatment will have network effects, also known as “interference bias”. (As a quick summary, “network effects” occur when the change in behavior for users in the treatment group also ends up affecting the users in the control group, thereby biasing the results. As an example, if we made Lime scooters free for treatment group users, then likely scooters rides would increase enough such that the users in the control group would have a harder than normal time finding an available scooter to ride, which would make the control group no longer acting at the “normal” baseline.)</p><p>If we don’t anticipate having network effects, then we can often use an A/B test setup. However, if we do anticipate network effects, then we usually end up using a Pre-Post experiment method or a Switchback test (both of which are defined in section 3b).</p><p><strong>3) Which specific test setup to use?</strong></p><p><strong>3a)</strong> When there are no anticipated network effects, generally an A/B test is applicable. A/B tests are controlled tests where some portion of users get the test treatment and some portion of users get the control treatment. The main next step is to determine the size of the test and control groups. Here is a rough guide for how we decide the sizes at Lime:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aLNtHLcGsW3HqiIQ_PJyvg.png" /></figure><p><strong>3b)</strong> When you do have anticipated network effects, then there are two main test options we use at Lime:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OxvaBHFW2VquflcAXSSVyQ.png" /></figure><p>The decision tree below summarizes the methodology explained above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nPCNDcnUob8X3VTZ_GfITQ.png" /></figure><h4>Dealing with Overlapping Tests</h4><p>As Lime scales, we have more tests running simultaneously that could be affecting each other thereby introducing bias. If separate tests’ treatments are impacting the same users and/or same metrics at the same time, then we can’t be sure what the individual impact of a test may truly be (e.g. running two tests that both aim to increase the adoption of the Lime Wallet feature). This table outlines Lime’s 4 solutions to run more than one test at a time with similar impacts :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iwuIHryNTc2NRClq8QlxQw.png" /></figure><h3>During Test (Checking In)</h3><h4>Checking for Sampling Bias</h4><p>Whenever running a test, at Lime or elsewhere, we need to ensure that there aren’t inherent biases in our samples in order to trust any insights we draw by comparing a treatment and control group. The most basic way of ensuring there’s no bias is by running the test for a long enough time to have a large enough sample size such that it decreases random bias. In addition to this:</p><ol><li>If the test is pre-assigned (users are all exposed at one static point in time), then we do a quick A/A test, which entails splitting our test and control populations to compare basic metrics between the two groups <em>without </em>yet having the test users exposed to any treatment conditions. Once we’ve confirmed that there are no differences in metrics such as trips per user or app sessions per day, we can rule out the potential for sampling bias.</li><li>If the test has live assignment (more users are exposed each day), then an A/A test is not a viable method to rule-out sampling bias. Instead, once the test has reached completion, we check the similarity between metrics pre-exposure between the test and control groups (similar to a post-factual A/A test).</li></ol><p>When comparing groups to ensure similarity, we also do a sample ratio mismatch check (i.e. run a significant test on the volume of exposed users) to ensure that a difference in the exposure volumes themselves don’t introduce bias.</p><h4>Dealing with the Peeking Problem</h4><p>Peeking (looking at test results before reaching the needed sample size (n)) causes the probability of a Type 1 error (rejecting a true null hypothesis) to increase. The most common way to deal with this is to calculate the required n before starting the test and then only making a decision when you have reached that sample size. However, the drawbacks to that solution are that:</p><ol><li>We want to peek to stop bad things from happening or flawed experimental design &gt;&gt; Fix: we can still do this as long as we don’t report the interim results officially; we can just use it as a safeguard in case of negative things occurring.</li><li>We are too reliant on our power analysis calculations before the test starts (i.e. a priori effect), e.g. we overestimate the effect we expect to see or that we need to see from a business perspective &gt;&gt; Fix: we can always err on the side of a low effect size like 2% over 5%. (Means higher n necessary though so not very sustainable)</li><li>The p = .08 problem: It’s clear that we are almost at significance and directionally moving that way. But we need to let the experiment run a bit more, collect more n to get to the 5% threshold &gt;&gt; Fix: we can always err on the side of requiring a larger n than we think we need. But again this doesn’t seem like the optimal solution.</li></ol><p>At Lime, our solution is to run a normal power analysis before the experiment starts to calculate the day on which we should have sufficient sample size to check results. We can only check the final results <em>one time</em> on that planned day and need a 95% confidence level to claim the results are statistically significant. If we want to check results before that time (or are requested to do so), then we can only claim results are stat sig if they meet a 99.9% confidence level as opposed to the initial 95%. This reduces the likelihood that we are falsely rejecting the null hypothesis by recalculating the impact multiple times. We decided that this solution is better suited than using Bayes Factors (another common solution in the industry) since it is a simpler calculation and we are still building out our internal experimentation analysis tools.</p><h3>After Test (Results and Roll Out)</h3><h4>Picking a Statistical Test to use</h4><p>It’s important for our teams to be aligned on which statistical test to use when, so that we are reading impact in a consistent way. Here are the guidelines that we use at Lime:</p><ul><li>For continuous metrics (e.g. revenue per trip), we are comparing means and can generally use a two-tailed t-test. If there is a small sample size, then we use a bootstrap t-test.</li><li>For ratios (e.g. WoW retention), we also recommend a t-test. We use t-test instead of a z-test because z-tests require more statistical power and are generally only more efficient and powerful when the metric being analyzed is binary.</li><li>The above two bullet points assume the metrics have a normal distribution, but at Lime, we often deal with skewed data (e.g. trips per user is left-skewed), in which case we use the Wilcoxon Rank-Sum test for which we don’t need to have an assumption about the distribution.</li><li>For categorical variables, we would propose using a chi-square test. However, categorical metrics are currently rarely used at Lime, so we haven’t built this into our experiment analysis pipelines.</li></ul><h4>How to Accurately Run Multiple Comparisons</h4><p>This issue with running multiple comparisons (e.g. if you compare a metric for each Country exposed to the treatment) is that the statistical probability of incorrectly rejecting a true null hypothesis will significantly increase as the number of simultaneously tested hypotheses increases. There are many different ways to help correct this issue — each correction method introduces a trade-off between correcting for false positives and reporting too many false negatives. Given each experiment is nuanced, this trade off of values must be made to best suit the needs of the business. At Lime, we use the Benjamini–Hochberg (BH) procedure to adjust the p-value, which is not only straightforward to implement but also less strict than the Bonferroni correction method (another method widely used in industry), which assumes that each test is independent of each other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/1*BDitGWgSGCQWjZU_yIh4nQ.png" /></figure><p>This is a simplified version of how we choose a hypothesis test but shows the main framework. The most common additional test we use is ANOVA for comparing metric impact across multiple variants.</p><h4>Extrapolating to Topline</h4><p>Very often people look at the impact seen in a test (e.g. +5% retention) and then assume that’s the lift on topline; however, this is inaccurate if the test is only exposed to a certain population of riders (e.g. only new riders or only Paris riders). As an example, in San Francisco, Lime Riders need to lock their scooters to a bike rack after ending a ride. If we make a change to the locking process, test the improvement only in San Francisco, and see a +10% increase in trips taken, can we claim that we’ll see +10% trips globally? No, because only San Francisco was exposed in this test treatment affecting a feature (lock-to) that is not present in all global markets.</p><p>Accurately calculating the topline impact is pretty straightforward — it’s generally solved by multiplying the impact we see by the percentage of users who are eligible for the treatment. Therefore our proposed solution is to determine this percentage when sizing the test (this is often the same as the trigger point, e.g. the proportion of people who open wallet out of all exposed users who open their app) and then to create a table like this for stakeholders to easily understand what the global topline impact is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DOKIu5pomB1uaDahFoCNqQ.png" /></figure><h3>Summary</h3><p>By taking the time to standardize experiment analysis methodologies within the engineering team at Lime, we are not only able to maintain a consistent rigor in our analysis across the board, but we are also now able to build our first experimentation analysis platform and incorporate some of these principles. Our analysis platform already incorporates principles such as using BH corrections when measuring significance and will soon include other automated principles such as A/A test checks. As a team, we have many more topics to discuss and standardize (e.g. the best way to check for network effects via counter-metrics), so stay tuned for future blog posts!</p><h3>Acknowledgments</h3><p>Many members of Lime’s Data Science and Analytics team contributed to our handbook including Tristan Taru, Dounan Tang, Jeh Lokhande, Siyi Luo, and Ben Laufer.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bee846d62dd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lime-eng/experimentation-analysis-at-lime-bee846d62dd">Experimentation Analysis at Lime</a> was originally published in <a href="https://medium.com/lime-eng">Lime Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing JS for Native-Like Webviews]]></title>
            <link>https://medium.com/lime-eng/optimizing-js-for-native-like-webviews-f109683e081a?source=rss----266a88f54bca---4</link>
            <guid isPermaLink="false">https://medium.com/p/f109683e081a</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[mobile-app-development]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Leo Jiang]]></dc:creator>
            <pubDate>Thu, 26 Mar 2020 18:02:05 GMT</pubDate>
            <atom:updated>2024-04-16T18:43:04.910Z</atom:updated>
            <content:encoded><![CDATA[<p>We at Lime recognized early on that one of the main bottlenecks in our product development process is the mobile release cycle. For both Android and iOS, it can take over two weeks for native code changes to propagate to users. We publish batched changes weekly, then it takes a few days for Google or Apple to approve our changes and an additional few days for users to update their apps. To speed up our development process, we explored building mobile experiments using HTML/JavaScript and running them in a webview framework. This way, our app would load web pages from our servers, which we can update in just minutes.</p><p>When designing this platform, our main challenge was loading web pages quickly enough to feel native. Our platform loads files from the Internet, which is slower than using local native code. Other apps usually display an empty screen while loading, but we wanted to avoid this. We also had other challenges such as making the UI look native and authenticating users seamlessly, but these were relatively easy to address. This article will explain how we optimized our web stack to serve content fast enough to feel native.</p><h3>User Flow</h3><p>Here’s an overview of how our webview platform works:</p><ol><li>A user navigates to a webview page</li><li>The app creates a webview</li><li>The app sets cookies in the webview containing the user’s auth token, language, location, etc.</li><li>The webview loads an HTML file from a CDN</li><li>The CDN runs <a href="https://varnish-cache.org/docs/trunk/users-guide/vcl.html">VCL</a> scripts to fetch an HTML file from S3, based on the user’s language</li><li>The webview renders the initial screen using the HTML and embedded <a href="https://www.sitepoint.com/how-and-why-you-should-inline-your-critical-css/">critical CSS</a></li><li>The webview loads external dependencies: CSS, JS, images, and fonts</li><li><a href="https://github.com/preactjs/preact">Preact</a> (lightweight React) runs and updates the DOM (there should be no changes)</li><li>JS uses the auth token in a cookie to fetch data from our <a href="https://github.com/rails/rails">Rails</a> server</li><li>Preact re-renders based on the fetched data</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/0*V7uCZmP8wSZnPRws" /></figure><p>After Preact runs, the webview should become nearly indistinguishable from a native page. However, users won’t see any content until the initial HTML render and they can’t interact with the page until JS runs. In contrast, for native pages, both of these steps are nearly instantaneous. To deliver a native-like experience, we managed to significantly reduce the loading time for both the initial screen and the JS files.</p><h3>Rendering Initial Screen Quickly</h3><p>It’s much faster to render content using static HTML and inline CSS than waiting for JS to load. We could write raw HTML and use a DOM manipulation library, but Lime’s other frontends are all React or Vue, so we decided to do the same here. To generate static HTML with a JS UI library like React, we’d need either server-side rendering (SSR) or a static site generator (SSG). SSR can be slow with high traffic unless we invest significantly in our infrastructure, so we went with SSG.</p><p>We weren’t happy with the existing SSGs because they weren’t optimized enough for speed. Therefore, we built our own SSG using <a href="https://github.com/preactjs/preact-render-to-string">preact-render-to-string</a>. During the build process, we use <a href="https://github.com/mde/ejs">EJS</a> templates to generate a separate static HTML file for each experiment and each language. For example, we’d have limepass-en.html, limepass-es.html, loyalty-en.html, loyalty-fr.html, etc. We also use <a href="https://github.com/FullHuman/purgecss">PurgeCSS</a> to embed critical CSS. Then, we upload the static HTML files to S3.</p><p>Since our mobile clients don’t know which languages are available for each experiment, we use Fastly CDN’s VCL scripts to dynamically serve the correct HTML file based on the user’s language. During the build, we upload the supported languages to Fastly’s key-value store.</p><p>SSG lets users see the initial screen much sooner than using JS to render it. However, an issue with SSG is that users still need to wait for JS to load before seeing dynamic content. We render placeholders in our static HTML files and progressively replace them with dynamic data. For example, the price for our LimePass experiment can be different for each user, so we can’t include them in our static HTML file.</p><p>Fully rendered after JS loads:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/0*ArftnGLNNK34r1Ni" /></figure><p>Static HTML:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/0*igqGm3tgiK9WZk1L" /></figure><p>On our office network, the browser downloads the HTML file and renders it in 150ms. If we rely on the CSS and JS files to render content, it would take 500ms.</p><h3>Loading JS Quickly</h3><p>Before the page can load dynamic data or be interactive, JS must be loaded. To load JS quickly, there are several things we can do:</p><ul><li>Have fewer JS files</li><li>Make JS files smaller</li><li>Transfer JS files faster</li></ul><h4>A Single JS File Per Page</h4><p>The normal way to build JS files is to have one entry point which conditionally loads other JS files as “chunks”. However, this is slower because of the extra network requests. Instead, we built a single JS file for each experiment, with all the critical dependencies included, so only one JS network request is needed.</p><p>A typical build would produce:</p><pre>js/</pre><pre>  chunks/</pre><pre>    en.js</pre><pre>    es.js</pre><pre>    fr.js</pre><pre>    limepass.js</pre><pre>    loyalty.js</pre><pre>    main.js</pre><p>First, the browser loads main.js, which then loads a language chunk and an experiment chunk. Our output files look like:</p><pre>js/</pre><pre>  limepass-en.js</pre><pre>  limepass-es.js</pre><pre>  loyalty-en.js</pre><pre>  loyalty-fr.js</pre><p>All the JS a page needs is included in a single file. We still use chunks for non-critical dependencies such as <a href="https://amplitude.com/">Amplitude</a> logging, which we load after Preact renders.</p><h4>Optimizing JS File Size</h4><p>There are nearly endless amounts of things you could do to reduce JS file size, so we’ll provide a brief summary here and share more details in a separate blog post. Our build uses Webpack, Babel, and SASS.</p><ul><li>Used <a href="https://github.com/google/closure-compiler">Google Closure Compiler</a> for compression</li><li>Used <a href="https://github.com/mixtur/webpack-spritesmith">Spritesmith</a> and inline SVGs to have at most one image file per page</li><li>Wrote a <a href="https://github.com/webpack-contrib/css-loader">css-loader</a> extension to generate shortened CSS classes</li><li>Used <a href="https://github.com/webpack-contrib/webpack-bundle-analyzer">Webpack Bundle Analyzer</a></li></ul><p>⠀⠀⠀⠀⠀E.g. <a href="https://github.com/zloirock/core-js">co</a>‎‎‎‎<a href="https://github.com/zloirock/core-js">re-js</a>’s Symbol polyfill was 3.5kB, but all we needed was its constructor, so we aliased it with a 30B polyfill</p><p>⠀⠀⠀⠀⠀E.g. re-implemented some <a href="https://github.com/lodash/lodash">Lodash</a> functions that were adding 25% to the bundle size</p><ul><li>Disabled async/await</li></ul><p>⠀⠀⠀⠀⠀Async/await polyfills are huge, the trade-off between code cleanliness and file size isn’t worth it because our experiments are simple</p><ul><li>Used Webpack’s <a href="https://webpack.js.org/plugins/module-concatenation-plugin/#debugging-optimization-bailouts">optimizationBailout</a></li></ul><p>⠀⠀⠀⠀⠀E.g. many modules weren’t treated as ES modules, which we fixed</p><p>As a comparison, an empty page generated with <a href="https://github.com/gatsbyjs/gatsby">Gatsby</a> loads 5 JS files totaling 300kB, while our framework would load a single 40kB JS file.</p><h4>Transferring JS Files Quickly</h4><p>We use a CDN which handles most delivery optimizations for us. It uses low latency infrastructure, serves files from the geographically closest servers, and uses HTTP/2. In addition, we added <a href="https://github.com/google/brotli">Brotli</a> support by uploading Brotli-compressed files to S3. A VCL script would serve the Brotli version if the user’s browser supports it.</p><h3>Results</h3><p>We’ve successfully launched several experiments on our webview platform. We were able to build, launch, and iterate on experiments much more quickly than building them natively. On average, it takes about 3 minutes for webview changes to reach users, compared to 2 weeks for native apps.</p><p>Making the content load quickly was crucial for making the experiment feel native. We can still make a lot of performance gains on the mobile or infra sides, such as pre-fetching or better caching. If you’re interested in working on challenging problems at Lime, check out our <a href="https://jobs.lever.co/limebike?department=Engineering">career page</a>.</p><p><a href="https://www.linkedin.com/in/leojiang1/">LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f109683e081a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lime-eng/optimizing-js-for-native-like-webviews-f109683e081a">Optimizing JS for Native-Like Webviews</a> was originally published in <a href="https://medium.com/lime-eng">Lime Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[foo(bar) vs bar.foo()]]></title>
            <link>https://medium.com/lime-eng/foo-bar-vs-bar-foo-16593a7b3f65?source=rss----266a88f54bca---4</link>
            <guid isPermaLink="false">https://medium.com/p/16593a7b3f65</guid>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Ian Hook]]></dc:creator>
            <pubDate>Mon, 02 Mar 2020 18:55:33 GMT</pubDate>
            <atom:updated>2020-03-03T04:10:47.351Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to code functions balancing scalability and usability</h4><p>As our engineering team at Lime has grown, and our code base with it, it has become more difficult for any given engineer to know everything there is to know about the modules that they work on. We set out to try to solve this problem and dove into the code to see what we could do to fix it. In doing so we came across an interesting pattern that we’re using in our main codebase.</p><p>Let’s say we have some function foo which takes as one of its parameters an instance of class Bar. We have two ways of writing this code:</p><pre>foo(bar)</pre><p>or</p><pre>bar.foo()</pre><p>We firmly believe in taking a data-driven approach to adding new features, and when the data doesn’t exist we run A/B tests. When we’re running a test on scooters, we assign all of the scooters to a single test group. We need to make sure that whenever the scooter needs to exhibit the behavior under test, it must always exhibit the behavior it was originally assigned. If we thought about this as a drug test, our data would be bad if we randomly gave all participant placebos for one dose and the test drug for the next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p_DuwWuHGqmpTjavDTbMgw.jpeg" /><figcaption>happy lime users</figcaption></figure><p>As we were looking for ways that we could improve our code, we saw that the way that we were implementing our A/B tests always preferred going the bar.foo() route.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bdec8d6b502536dcab8ab1b6554eee9f/href">https://medium.com/media/bdec8d6b502536dcab8ab1b6554eee9f/href</a></iframe><p>Using bar.foo(), is essentially asking the object for information about itself, and foo(bar) uses an outside expert to retrieve information about an object. When thinking about A/B testing as a drug test, should we ask the patient whether they’re getting the placebo or not? No, an external export should be keeping records based on the information they have about the patient.</p><p>As a coder though, just because doctors can’t do a thing doesn’t mean that it won’t work for us. Indeed this code does work and has powered some of our best feature decisions.</p><p>When given multiple ways to achieve the same goal, we should lean toward the solution that provides the most maintainability going forward. <a href="https://en.wikipedia.org/wiki/SOLID">SOLID principles</a> often serve as a good starting point to look for guidance on maintainability.</p><p>The first principle that stands out as providing insight is the <a href="https://en.wikipedia.org/wiki/Open%E2%80%93closed_principle">Open/Closed Principle</a>.</p><p>We want our modules to be closed to changes arising from new features being added to our code base and open to be used in a wide variety of ways. Essentially, all our core modules should be black boxes that give others powerful (but not unlimited) control over the objects. As developers want to use our modules, they should not be tinkering inside the black box.</p><p>Again, if we think about medical studies, we don’t go implanting subdermal tags to assign patients to groups, we use information that the patient has readily available, like their name. Similarly, we shouldn’t open up our <em>Scooter</em> class to add a new function to help the scooter identify which group it belongs to. Instead, we should make sure that a <em>Scooter</em> instance exposes enough information that an outside expert could determine whether it is in a test group.</p><p>Fortunately, ab_test_subject points to the fact that we probably do, by passing the full object to AbTestSubject the code above doesn’t really point to how we determine what’s important to identify a scooter as a part of a group.</p><p>Unfortunately, the Open/Close principle only points to a problem without providing any solutions.</p><p><a href="https://en.wikipedia.org/wiki/Interface_segregation_principle">Interface segregation principle</a> also seems to apply to this problem and also points to a possible solution. It states, no client should be forced to depend on methods it does not use. It recommends that our interface be split into multiple pieces.</p><p>In this case, we can think of the interface for the Scooter class as the list of all its public methods and variables. The split here is to take Scooter#sidewalk_detection? and move it to a different interface. Where?</p><p>Turns out that sidewalk_detection?<em> </em>is only used in two places in our code: when we start the logging and when we stop it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0e76b36b33fd04c2b7a25a5611130e85/href">https://medium.com/media/0e76b36b33fd04c2b7a25a5611130e85/href</a></iframe><p>We could move the definition of sidewalk_detection? to <em>Trip</em>, but this is the same road with the same problems. Instead of opening Scooter to add the experiment, we instead have to open Trip. And if Trip functionality isn’t part of the experiment, are we forced to put this back on Scooter? Even then, many things that rely on Trip don’t really rely on whether or not we’re trying to detect sidewalks, and the Integration Segregation principle says that we should split it out.</p><p>Enter the brand new module SidewalkDetection:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a2b5def1a704bbfd39de4980a51e071a/href">https://medium.com/media/a2b5def1a704bbfd39de4980a51e071a/href</a></iframe><p>We can also rewrite Trip to do the bare minimum to support sidewalk detection:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/768b64bab3ccef0c88c1710469bdc2e2/href">https://medium.com/media/768b64bab3ccef0c88c1710469bdc2e2/href</a></iframe><p>Finally, all mention of sidewalk detection is removed from our Scooter class allowing it to follow the Open/Closed Principle!</p><p>Making the new SidewalkDetection module clearly groups code that cares about sidewalk detection in a single location, limiting the locations that need to change as the feature changes.</p><p>As we worked through this approach on all of the experiment code in the <em>Scooter</em> class, we found some experiments had already run their course, the associated code had already been deleted, and what remained in the Scooter class was extremely dead. New experiments that follow the example of the SidewalkDetection module, this should stop happening this problem, which is another nice bonus to this approach.</p><p>The last big plus to this approach is, once other vehicle technologies (like hoverboards) become consumer-ready, our Hoverboard instance can also easily be passed to the module to enable sidewalk detection there too.</p><p>There’s a difference between writing functional code and writing good code. I like to think that functional code solves the problem you know about, while good code solves problems you didn’t even know you had.</p><ul><li>When coding a new feature, there is often a choice of adding it to an existing class or creating a new location for that feature and passing in the class as a parameter.</li><li>If the feature is not core to the existing class, passing the class as a parameter helps to maintain the boundary between different pieces of code, and by extension different parts of the codebase.</li><li>Make life easier for future developers by not expanding code interfaces that they’ll have to read through to understand how something works.</li><li>Sometimes attaching a piece of code directly to a class can severely limit its reusability.</li><li>SOLID principles are a great place to start to understand how to write more maintainable code.</li></ul><p>If you are interested in working on challenging problems at Lime, check out <a href="https://www.li.me/careers">our career page</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16593a7b3f65" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lime-eng/foo-bar-vs-bar-foo-16593a7b3f65">foo(bar) vs bar.foo()</a> was originally published in <a href="https://medium.com/lime-eng">Lime Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>