<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Upstart Tech - Medium]]></title>
        <description><![CDATA[Enabling effortless credit based on true risk - Medium]]></description>
        <link>https://medium.com/upstart-tech?source=rss----f8ac0e656cc0---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Upstart Tech - Medium</title>
            <link>https://medium.com/upstart-tech?source=rss----f8ac0e656cc0---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 01:33:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/upstart-tech" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Upstart’s Model 19 Leverages Richer Data for Even Greater Accuracy]]></title>
            <link>https://medium.com/upstart-tech/upstarts-model-19-leverages-richer-data-for-even-greater-accuracy-6d7873efaaac?source=rss----f8ac0e656cc0---4</link>
            <guid isPermaLink="false">https://medium.com/p/6d7873efaaac</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[loans]]></category>
            <category><![CDATA[credit]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Brett Manning]]></dc:creator>
            <pubDate>Fri, 21 Feb 2025 18:51:01 GMT</pubDate>
            <atom:updated>2025-02-21T18:51:01.654Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dvsXlxSMue1vwAaZY5-DNg.png" /></figure><p>In December, Upstart launched its latest personal loan underwriting model: Model Version 19.0.0.</p><p>Model 19 added two standout features that allow the model to make even more accurate predictions when assessing credit risk: the Payment Transition Model (PTM) and Trade Account Embeddings. PTM allows us to consider the different stages of a loan payment (current, delinquent, paid off, and charged off) rather than only whether or not a loan is current right now. Embeddings make it possible for the model to learn subtle patterns in historic credit behavior.</p><p>Together, these features increase model complexity by 88%, representing both a significant advance in data analysis and a massive feat of engineering. Let’s have a deeper look at them.</p><p><strong>Payment Transition Model</strong></p><p>While our previous <a href="https://www.upstart.com/ptcc#:~:text=Understanding%20&#39;loan%20month&#39;&amp;text=With%20our%20patented%20loan%2Dmonth,default%20in%20month%2030!).">loan-month model</a> architecture predicted whether or not a borrower would make each specific monthly payment, it left some information on the table, such as the exact payment pattern of a borrower.</p><p>To capture this payment behavior, the PTM framework models the probabilities for a loan to transition among different payment statuses. The ability to tell the difference between on-time and late payments delivers a greater level of sophistication when predicting risk. For example, Model 19 can now estimate the probability that a borrower who is one month past due will either recover to “current” status or move further past due for each month of payment.</p><p>Adding PTM to the model results in a richer dataset, with up to four additional observations per loan in delinquency scenarios. This improves predictive accuracy while aligning loan behavior more directly with the charge-off timelines of Upstart’s lending partners.</p><p><strong>Trade Account Embeddings</strong></p><p>Embeddings are a method for transforming categorical, or text-like, data into numerical representations. In the case of Model 19, embeddings transform trade account attributes — such as the portfolio type or current balance amount you’d see listed on a typical credit report — into compact, high-dimensional numerical features (which is to say, they pack a lot of information into more easily digestible numbers for the model).</p><p>Imagine each borrower’s credit history as a series of “words” built from trade account attributes. For example, a single account could yield a word like “XYJN50011,” which would represent the connection among the different types of credit held, account balances, and payment details that feeds into a risk assessment of that borrower. The embedding model converts these words into a compressed numerical vector space. These vectors allow the model to capture complex relationships among credit accounts and identify subtle patterns in borrower behavior that enhance the model’s ability to make more accurate predictions. Moreover, embeddings make it possible to scale the amount and complexity of information the model can process and do so efficiently.</p><p>One challenge with embeddings is their perceived complexity. Regulators, consumers, and lending partners need clear explanations for loan decisions and compliance with fair lending laws. Upstart solved this by leveraging its existing approach to <a href="https://www.upstart.com/lenders/regulatory-compliance/adverse-action-notices/">Adverse Action Notices</a> and by applying comprehensive fair lending controls, including rigorous <a href="https://www.upstart.com/lenders/regulatory-compliance/detecting-eliminating-proxies/">quantitative proxy analysis</a> and disparate impact testing.</p><p>These safeguards ensure that embeddings improve model performance without sacrificing explainability or fairness — a critical balance for expanding access to credit.</p><p><strong>Enabling Advanced Models</strong></p><p>The advances outlined above, in addition to the “<a href="https://medium.com/upstart-tech/how-we-built-our-most-advanced-underwriting-model-yet-a6b46ca3225f">APR as a feature</a>” that we added to <a href="https://medium.com/upstart-tech/how-we-built-our-most-advanced-underwriting-model-yet-a6b46ca3225f">Model 18</a>, have resulted in a 55x increase in model complexity. Between Model 17 and now, we’ve seen a 16x increase in calculations downstream of the model as a result of an increase in model predictions and more lending programs (i.e., the credit parameters and target returns laid out by our lending partners on the Upstart platform). To keep pace with complexity and deliver these accuracy improvements at scale, we’ve optimized multiple areas of our stack for better performance, including service-to-service communication, high parallelization, and balanced distribution of workloads across our compute infrastructure.</p><p>We’re excited about the results. As our CEO, Dave Girouard, <a href="https://www.upstart.com/news/q4-and-full-year-2024-earnings-call-transcript/">said</a> on our most recent earnings call: “We’re often surprised by the increase in accuracy these types of innovations deliver. But concepts like <a href="https://medium.com/upstart-tech/how-we-built-our-most-advanced-underwriting-model-yet-a6b46ca3225f">“APR as a Feature”</a> and “PTM” aren’t one-time boosts — they’re new model forms entirely. You can think of them as innovation vectors that offer our team many ways to refine and build on their advantages for a long time to come.”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d7873efaaac" width="1" height="1" alt=""><hr><p><a href="https://medium.com/upstart-tech/upstarts-model-19-leverages-richer-data-for-even-greater-accuracy-6d7873efaaac">Upstart’s Model 19 Leverages Richer Data for Even Greater Accuracy</a> was originally published in <a href="https://medium.com/upstart-tech">Upstart Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Effortless Okta Policies with Terraform]]></title>
            <link>https://medium.com/upstart-tech/effortless-okta-policies-with-terraform-b6574af23d70?source=rss----f8ac0e656cc0---4</link>
            <guid isPermaLink="false">https://medium.com/p/b6574af23d70</guid>
            <category><![CDATA[identity-management]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[okta]]></category>
            <dc:creator><![CDATA[Lukas Indre]]></dc:creator>
            <pubDate>Wed, 19 Feb 2025 20:47:00 GMT</pubDate>
            <atom:updated>2025-02-19T20:47:00.666Z</atom:updated>
            <content:encoded><![CDATA[<p>Cybersecurity risks are on the rise — Are you doing everything you can to help your company secure access to its resources? Thankfully, Okta provides a robust suite of tools that empower companies like Upstart to secure their business applications from multiple angles. With Okta, you can enforce access requirements, such as being on a specific network, using a company-managed device, belonging to a particular group, or any combination of these conditions. And that’s just the beginning — Okta offers a wide range of policy options to meet diverse security needs.</p><p>However, with great flexibility comes the risk of misconfiguration. The Okta Identity Engine’s policy system is highly customizable but doesn’t come with guardrails. This means it’s possible to configure policy rules in ways that lead to unexpected or non-intuitive behaviors. To address these challenges, I developed a <a href="https://registry.terraform.io/modules/lukasindre/authentication-policy-stack/okta/latest">Terraform module</a> that abstracts and simplifies Okta’s authentication policies, leveraging the <a href="https://registry.terraform.io/providers/okta/okta/latest/docs">Okta Terraform provider</a> to streamline and standardize configurations. Detailed below are examples of this module in action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*vMBYJuORQJrWVQ4g.jpeg" /></figure><h3>Network Zone Abstraction</h3><p>A common practice in most companies is to ensure that users access company resources through a private network, typically via a VPN. Okta supports this configuration by leveraging groups of gateways or IP addresses, referred to as network zones. In authentication policies, you can require users to route their traffic through these defined network zones. A sample configuration might look like this:</p><pre>network_connection = &quot;ZONE&quot;<br>network_includes = [&quot;zone_id_1&quot;, &quot;zone_id_2&quot;]</pre><p>This requires defining two arguments: one for the type of network connection and another for a list of network zone IDs.</p><p>To simplify this process, the module allows you to use a single key, `network_zones`, to handle the otherwise opaque zone IDs for you. All you need to do is ensure the network zone is configured within your Terraform project, and provide its ID as items in the list. For example:</p><pre>network_zones = [<br>  okta_network_zone.my_zone.id,<br>  okta_network_zone.their_zone.id<br>]</pre><p>This approach makes the configuration more intuitive and eliminates the need to reference complex zone IDs manually.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LlsOXa1rmS5sHMpx" /></figure><h3>Managed Device Abstraction</h3><p>Managed device access is critical to ensure secure, seamless connectivity for employees while protecting sensitive data. By enforcing device compliance policies, organizations can reduce risk, maintain productivity, and adapt to evolving hybrid work environments.</p><p>Let’s say you want to allow only managed iOS devices to access a specific Okta application in your stack. You could attempt to configure this using the Okta Terraform provider like so:</p><pre>device_is_managed = true<br>platform_include {<br>  os_type = &quot;IOS&quot;<br>  type = &quot;MOBILE&quot;<br>}</pre><p>However, there are a few issues with this configuration. One issue, as highlighted in the documentation, is that if you set `device_is_managed` to `true`, you must also set `device_is_registered` to `true`. While it’s easy to understand that a managed device must first be registered through Okta Verify, it’s less intuitive why you need to explicitly declare this dependency in the configuration.</p><p>To address this, you’d revise the configuration like so:</p><pre>device_is_registered = true<br>device_is_managed = true<br>platform_include {<br>  os_type = &quot;IOS&quot;<br>  type = &quot;MOBILE&quot;<br>}</pre><p>While this resolves the issue and creates a valid configuration, it introduces another problem: The resulting code is neither succinct nor easy to read. This is where the abstraction provided by my Terraform module shines. With the module, you can achieve the same functionality by simply defining the managed platforms in list format:</p><pre>managed_devices = [&quot;ios&quot;]</pre><p>This approach is straightforward, concise, and far easier to understand. It abstracts the burden of configuring dynamic blocks for each platform you specify. It minimizes the cognitive load for developers while ensuring that your configuration is both correct and maintainable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EQZUtwRr4_4MrioG" /></figure><h3>Priority Abstraction</h3><p>Okta allows you to set a priority for policies to determine the order in which rules are evaluated against the authentication context. However, managing priorities can become cumbersome as your authentication policies grow to include 10 or more rules. From a readability standpoint, having 10+ policy rules is challenging, as they won’t all fit on one screen. Now imagine needing to change the order in which these rules are evaluated — You’d have to manually adjust the priority of the rule you want to move and then increment or decrement the priorities of the preceding or succeeding rules accordingly.</p><p>This is where the module simplifies the process. It enhances readability and manageability by handling rule priorities for you. Simply declare your policy rules in your desired order, highest priority first, within the `rules` list argument. The module will automatically assign priorities based on that order. This approach eliminates the need for manual priority adjustments, making your configuration more intuitive and easier to maintain.</p><h3>Default Rule Abstraction</h3><p>Okta policy framework sets rule priority with integers. The lower the number, the higher the priority. Zero and 99 are the highest and lowest priorities, respectively. When you create an authentication policy rule in Okta, a default rule is automatically attached to the policy at priority 99. This default rule is configured as an “access allow” rule. However, following best practices, it’s recommended to deny all traffic by default and only allow explicitly defined traffic.</p><p>To manage this default rule in Terraform, you would typically need to run an import operation (as <a href="https://registry.terraform.io/providers/okta/okta/latest/docs/resources/app_signon_policy_rule">outlined in the documentation</a>). Instead of handling this through manual imports, writing automation for the import, or editing the default rule directly in the Okta console, this module simplifies the process by creating its own default rule. The module assigns this rule a priority of 98 — the second lowest — ensuring it sits below all explicitly defined rules. This rule is configured as a “deny any request” rule (an implicit deny), reinforcing secure access by default while allowing your explicitly defined rules to take precedence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oRyt-PGBBy2QM6h3" /></figure><h3>MFA Abstraction</h3><p>Session timeouts in Okta are configured at the authentication policy rule level, which governs access to an application. The <a href="https://registry.terraform.io/providers/okta/okta/latest/docs/resources/app_signon_policy_rule#re_authentication_frequency-1">values</a> for session timeouts must be set using the <a href="https://www.digi.com/resources/documentation/digidocs/90001488-13/reference/r_iso_8601_duration_format.htm">ISO 8601 Period</a> format. Although this format is great for computers, it can be less intuitive for humans. For instance, if you want to require multi-factor authentication (MFA) for every sign-in attempt (“step-up authentication”), the re-authentication period must be set to the smallest allowable interval. In ISO 8601, the smallest unit is seconds (`S`), and the format requires a time separator (`T`) between the date and time components. You prefix these rules with Period (`P`) to signify its compliance with the format. Therefore, to configure step-up authentication, you would set the period to `PT0S`.</p><p>Although this is easy for machines to parse, it’s not particularly human-friendly. To address this, the module introduces a simple step-up authentication Boolean setting, along with human-readable options for session timeouts. Instead of dealing with cryptic expressions, you can configure authentication frequencies in familiar terms, such as:</p><ul><li>daily (`PT24H`)</li><li>hourly (`PT1H`)</li></ul><p>This abstraction makes configurations more accessible and easier to maintain while still aligning with Okta’s requirements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k9JH06NEKwAhw2ot" /></figure><h3>Phishing-Resistant Authenticator Abstraction</h3><p>Okta allows you to define which categories of authenticators a user is permitted to use in a policy. This functionality is implemented via a `<a href="https://registry.terraform.io/providers/okta/okta/latest/docs/resources/app_signon_policy_rule#constraints-1">constraints</a>` argument. At first glance, this might seem straightforward, as the provider expects `constraints` to be a list of strings. However, upon reviewing the <a href="https://registry.terraform.io/providers/okta/okta/latest/docs/resources/app_signon_policy_rule#constraints-1">documentation</a>, you’ll find that these strings are actually string representations of nested JSON objects. For example:</p><pre>constraints = [<br>  jsonencode(<br>    {<br>      possession = {<br>        required = true<br>        phishingResistant = &quot;REQUIRED&quot;<br>        userPresence = &quot;REQUIRED&quot;<br>      }<br>    }<br>  )<br>]</pre><p>This makes understanding and defining constraints more complex. Also, passing invalid JSON strings can sometimes fail silently, leading to unexpected behavior.</p><p>To simplify this process, the module abstracts these JSON strings by allowing you to use Boolean values instead. For example, let’s say you want your users to authenticate only with phishing-resistant authenticators (and who wouldn’t?). Simply set `phishing_resistant = true`, and the module will take care of generating and encoding the correct object manifest for you. This not only reduces the risk of errors but also makes your configuration far easier to read and maintain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BE1AlLKaVfHSuzcf" /></figure><h3>Conclusion</h3><p>Managing Okta authentication policies can be complex, especially as your organization scales and your security needs grow. While Okta provides powerful tools to secure access, its flexibility can lead to intricate configurations that are difficult to manage and prone to misconfiguration. This is where the <a href="https://registry.terraform.io/modules/lukasindre/authentication-policy-stack/okta/latest">Terraform module</a> steps in, offering abstractions that simplify policy management while maintaining robust security practices.</p><p>From managing network zones and device policies to automating rule priorities, creating secure default rules, and enabling human-readable configurations for MFA and phishing-resistant authenticators, this module eliminates much of the manual work and complexity involved in using Okta. By translating complicated JSON objects and cryptic formats into intuitive, declarative configurations, it enables teams to focus on building and maintaining secure systems without getting bogged down by the intricacies of policy definitions.</p><p>With cybersecurity always a top priority, adopting tools that balance flexibility, security, and usability is more important than ever. This Terraform module helps you leverage Okta’s full potential and also ensures that your authentication policies are easy to understand, maintain, and adapt to your organization’s evolving needs. Effortless Okta policies are no longer just a goal — They’re a reality.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6574af23d70" width="1" height="1" alt=""><hr><p><a href="https://medium.com/upstart-tech/effortless-okta-policies-with-terraform-b6574af23d70">Effortless Okta Policies with Terraform</a> was originally published in <a href="https://medium.com/upstart-tech">Upstart Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Upstart’s New Low-Code/No-Code Data Platform: Aqueduct]]></title>
            <link>https://medium.com/upstart-tech/upstarts-new-low-code-no-code-data-platform-aqueduct-282799dbdfed?source=rss----f8ac0e656cc0---4</link>
            <guid isPermaLink="false">https://medium.com/p/282799dbdfed</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Bhargavi Damodaran]]></dc:creator>
            <pubDate>Mon, 03 Feb 2025 18:34:00 GMT</pubDate>
            <atom:updated>2025-02-04T17:42:34.942Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By Srinivasan Hariharan, Puneet Bysani, Bhargavi Damodaran &amp; Satheesh Boopathi</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CyFx50B2tA4PhYEwBdNgqg.png" /></figure><p>Data is the fuel that powers machine-learning algorithms and enables AI systems to learn, adapt, and improve over time. The more reliable data points an AI system can access, the more accurate and effective it becomes.</p><p>That’s where Upstart’s Data Platform team comes in. <a href="https://www.upstart.com/">Upstart</a> is dedicated to <a href="https://www.upstart.com/our-story">expanding access to credit through the use of AI</a>, and our role is to provide fast access to high-quality data to enable quicker model iteration, regular improvements in model accuracy, the uncovering of new business insights, and increased confidence in financial reporting through low-code and easy-to-use tools.</p><p>In an effort to better serve other Upstart teams — engineers, product managers, data analysts — and help them overcome challenges in the development process, we built a new solution that allows them to create data engineering workflows at scale. It’s called Aqueduct, a low-code/no-code data platform that, we like to think, has changed the game when it comes to data management. Here’s how it came together:</p><p><strong>Speed Bumps to Productivity</strong></p><p>By conducting interviews across the organization, we were able to identify the main issues slowing productivity and efficiency for our colleagues:</p><p><strong>1. High Development Time:</strong> The in-house requests for data from sources as varied as S3, Kafka, and third-party APIs (for use in the Lakehouse for model training and analytics) required us to build bespoke data pipelines to extract and transform that data. This led to a lot of extra dev time, redundant code across projects and teams, a lack of standardization and best practices, and inconsistent testing practices among teams, which ultimately resulted in an inefficient development lifecycle.</p><p><strong>2. Reliance on Specialized Data-Engineering Skills: </strong>A dependency on data engineers for common ingestion patterns and transformations — along with the specialized skills needed to handle high volumes of complex data (e.g., flattening large JSON blobs or querying nested JSON structures) — became a bottleneck. This slowed down the development of new product insights and hampered rapid model iterations.</p><p><strong>3. Infrastructure and Setup Time:</strong> Analysts and machine-learning scientists faced significant friction in setting up development environments, spending excessive time on non-core tasks like securing permissions for sensitive data (e.g., Personally Identifiable Information) and managing infrastructure, rather than focusing on developing and testing models.</p><p><strong>4. Support for Various Data Pipelines:</strong> Data consumers inside Upstart increasingly requested access to near real-time data to support critical business processes such as online models, real-time analytics dashboards, fraud detection, and velocity controls. In response, we needed to build scalable and reliable infrastructure that could support a variety of pipeline types — batch, streaming, event-driven — Change Data Capture, and Single Message Transforms.</p><p><strong>Our North Stars</strong></p><p>Yes, there was more than just one. By developing scalable infrastructure and self-service tooling for these various pipeline types, we aimed to meet Upstart’s diverse business needs and unlock more value from our data ecosystem. To get there, we established a set of guiding principles that would shape how Aqueduct was built.</p><p><strong>1. Domain-Specific Language (DSL) for Data Engineering: </strong>To reduce development time, we drew inspiration from well-established tools such as DBT and domain-specific languages such as SQL. We implemented DSL support in our platform, enabling users to interact with various SaaS and cloud provider tools — Databricks Spark Streaming, Delta Lake, S3, and Kafka — through simple commands defined in configuration files. This abstraction enables users to build complex data pipelines without needing to write extensive custom code, enabling them to focus on business logic rather than technical and infrastructure details.</p><p><strong>2. Low-Code/No-Code: </strong>To eliminate the need for specialized data engineering skills and enforce best practices, we built our platform with a low-code/no-code approach. This automates critical tasks such as deployment, compute, and infrastructure provisioning, while enabling developers to inject custom data processing code using popular languages such as Python or SQL. This flexibility enables users to format data for model training or analytics while still adhering to enforced best practices, such as cost attribution tags to track resource consumption, table indexes to optimize read/write, and auto-scaling to manage resource efficiency. This approach ensures that performance and cost-related issues are minimized without burdening users with complex infrastructure management.</p><p><strong>3. Support for Various Types of Pipelines:</strong> Our platform is designed to support a diverse range of data pipeline types in order to cater to different business needs. By supporting multiple types of pipelines, we enable users to build and manage data workflows that suit their specific use cases, from real-time analytics to large-scale batch processing.</p><p><strong>4. Extensible and Modular Design:</strong> We aimed to modularize the platform’s architecture to enable seamless addition of source connectors (for pulling events) and sink connectors (for persisting events). This modular approach ensures that the platform remains highly extensible — making it easy to integrate new sources or sinks without major refactoring. By managing SaaS integrations as separate modules (e.g., Databricks Spark, Delta Lake, Confluent Kafka, AWS SQS), we avoid vendor lock-in, giving users the flexibility to switch or expand their toolset as needed. This design not only enhances flexibility but also supports scalability as new technologies emerge.</p><p><strong>5. Security and Deployment: </strong>To simplify access management and enhance security, we abstracted the handling of credentials such as IAM roles, service accounts, and access tokens, along with role-based access controls (RBAC) to various SaaS and cloud provider services. This ensures that sensitive credentials are managed securely and automatically, reducing the operational burden on users. Also, we sought to streamline production deployments by offering pre-built continuous integration/continuous delivery (CI/CD) templates. These templates simplify the process of validating, submitting, and testing configuration changes, enabling teams to deploy their pipelines quickly and with greater confidence in production environments. This approach reduces manual overhead, enhances consistency across deployments, and promotes best practices for production readiness.</p><p><strong>Rolling It Out</strong></p><p><em>Main Components of Aqueduct</em></p><p>Aqueduct is designed to empower users to create data engineering workflows at scale. It uses a Domain Specific Language in JSON format to define workflows that ingest, transform, and write data while abstracting away infrastructure complexities and scheduling details. It supports a wide variety of use cases aimed at the needs of data engineers, software engineers, machine learning engineers, and analytics engineers.</p><p>Aqueduct is based on the Workflow Design Pattern. The main modules in Aqueduct are Workflow Engine and Connectors. Each workflow job has an input stream, processing logic, and output stream. An Aqueduct job can contain multiple workflows packaged with their own dependencies that can be executed in parallel. For example, a datalake_ingestion DSL can contain multiple workflows, and when translated into Databricks, it creates multiple jobs to scale out and optimize resources, typically managing up to 100 workflows per job. It also abstracts the job compute configuration, infrastructure management, scheduling, and workflow orchestration into the DSL configurations. This enables Aqueduct users to focus on defining their data workflows without worrying about the implementation details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rk-lgXP0e6FTObE5" /></figure><p><em>Self-Serve Workflows</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3jHnYexUIm1POx78" /></figure><p>Aqueduct users clone the template repository so that each team has their own DSL repo. Users configure the DSL to instruct the platform to perform tasks such as reading data from different sources, applying transformations, and writing data to various destinations. This declarative way of defining workflows makes Aqueduct easily accessible for both technical and non-technical users in Upstart. Below is a sample DSL to ingest Salesforce marketing cloud data from S3 to Databricks Lakehouse:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*oBrqFXPzXGflzfrjvKFtEA.png" /></figure><p>As seen above, users can specify different workflow types such as data_lake_ingestion, data_lake_transformation, streaming transformations, kafka_connect_sink, etc. Based on the workflow type, Aqueduct provides execution abstraction. For example, DSLs for data_lake_ingestion and data_lake_transformation are translated into Databricks jobs, while DSLs for kafka_connect_sink are translated into Kafka sink connectors. For the data_lake_ingestion workflow type, all users need to do is specify input properties, output properties, compute configuration, and schedule. Aqueduct handles everything behind the scenes to translate the DSL into a job and execute it in Databricks.</p><p>In some cases, users need to write custom transformations beyond just ingesting data from one source to a destination. For those types of transformations, Aqueduct supports custom processing built using Python code and PySpark DataFrames. Users can also specify these process scripts in the DSL.</p><p>This workflow as a config enables users to scale their data workflows for volume and latency requirements. For example, a data pipeline that ingests data from thousands of Kafka topics into thousands of corresponding Delta Lake tables translates into multiple Databricks jobs, complete with auto-scaling, optimal instance types, monitoring, and observability. Additionally, Aqueduct offers out-of-the-box performance optimizations, including auto-optimization, indexing, partitioning, and Z-ordering. These features enable users to focus on the business logic without the need to manage the complexities of the pipeline.</p><p><em>Production Deployment</em></p><p>Now that we understand how the pipeline is configured, let’s take a look at how CI/CD works. All services running in Aqueduct are stateless. Users commit the DSL through GitHub, and once the PR is approved and merged, the build process validates the DSL. Upon successful validation, the DSL is copied to an S3 bucket. The arrival of the DSL in the S3 bucket triggers an event in the Aqueduct Kafka topic. The Aqueduct translator service continuously listens to this topic. Once the new DSL arrives, the translator parses and submits it to Databricks or Confluent based on the specified workflow type. Once the job is created, it runs as a streaming or batch job based on the schedule.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z6wMS_cJT_LipBj7" /></figure><p><strong>Conclusion</strong></p><p>Aqueduct has proven to be a transformative tool for streamlining data engineering workflows at scale in Upstart. While challenges in roll-out and early-stage use presented themselves, we are implementing new features to support better orchestration and dependency management, observability, and data governance. We continue to improve the platform to deliver a better experience for our teams. The following shows a summary of the overall impact of Aqueduct at Upstart:</p><ul><li>Empowers teams to create data pipelines without relying on data engineers.</li><li>Accelerates development of data pipelines by reducing the effort from weeks to days. We observed a 50% reduction in average time to ingest raw data or transform data within the Lakehouse.</li><li>Enables easy plugging of custom transformation code without any infrastructure overhead.</li><li>Out-of-the-box handling for ingestion scaling and optimizing the written datasets for optimal read performances.</li><li>Speeds up time-to-insight using reusable pipeline templates.</li><li>Seamlessly integrates with popular tools like Databricks, Kafka, and AWS.</li><li>Auto-scaling of infrastructure based on data volume.</li><li>Automated alerting and notifications of workflow failures to the responsible teams.</li></ul><p>There were other challenges encountered when adopting Aqueduct for niche use cases in Upstart, such as scaling of real-time data processing and dynamically generating DSLs to reduce manual intervention. We will dive into those, and discuss specific use cases where Aqueduct has provided significant value to the Upstart, in future blog posts.</p><p><em>Read more about Upstart’s innovations in data, engineering, machine learning, and AI </em><a href="https://medium.com/upstart-tech"><em>here</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=282799dbdfed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/upstart-tech/upstarts-new-low-code-no-code-data-platform-aqueduct-282799dbdfed">Upstart’s New Low-Code/No-Code Data Platform: Aqueduct</a> was originally published in <a href="https://medium.com/upstart-tech">Upstart Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Built Our Most Advanced Underwriting Model Yet]]></title>
            <link>https://medium.com/upstart-tech/how-we-built-our-most-advanced-underwriting-model-yet-a6b46ca3225f?source=rss----f8ac0e656cc0---4</link>
            <guid isPermaLink="false">https://medium.com/p/a6b46ca3225f</guid>
            <category><![CDATA[credit]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[lending]]></category>
            <category><![CDATA[persona-loan]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Brett Manning]]></dc:creator>
            <pubDate>Wed, 14 Aug 2024 19:01:09 GMT</pubDate>
            <atom:updated>2024-08-14T19:01:09.732Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By Brett Manning &amp; Omar Venado</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Azxv4XL8J1t3PnfjZW1vOw.jpeg" /></figure><p>Last week we released Upstart’s personal loan underwriting model 18, or M18 as we call it internally. <a href="https://www.upstart.com/news/upstart-q2-2024-earnings-materials-ceo-transcript/">As our CEO Dave Girouard mentioned during our most recent earnings call</a>, this model represents one of the largest and most impactful improvements to our core credit pricing model in our history. In fact, with this launch, 18% of all accuracy gains in this model since our inception have been delivered by our machine-learning team in the last 12 months.</p><p>Model 18 includes our latest implementation of APR as a feature. This means APR is an input to the model, as a measure of the total cost of the loan. The model delivers significant gains as a result of modeling improvements for two key determinants of repayment behavior:</p><ol><li>Loan Affordability — all else equal a higher interest rate means higher repayments which are harder to afford</li><li>Adverse Selection — accepting a very high interest rate might be a signal of particular financial stress</li></ol><p>To understand how Model 18 achieves this breakthrough, let’s first explain how an underwriting model works. The model takes what we know about the borrower, and predicts their probability to default or repay the loan. For traditional lending this might be a mapping from FICO score and debt-to-income ratio to a single default probability.</p><p>But at Upstart we look deeper. Our advanced ML model is at the core of our difference. First, we take into account a much broader set of borrower characteristics than a traditional credit model. That’s well more than 1,600 features about borrowers, including their traditional credit file but also new data like their employment and how well they manage their finances. Second, we use cutting-edge AI to predict the probability of repayment on a month-by-month basis, which gives us a granular view of every borrower.</p><p>The probability of default and prepayment is combined with lender risk tolerance to calculate the interest rate the borrower pays. The interest rate plus any fees are used to create an APR and an associated repayment schedule. The APR is itself a key driver of the default probability, which raises an interesting question: How can the APR be both an input and an output of the model?</p><p>The mathematician reading this will recognize this as a “fixed point problem.” In a simple linear model this fixed point is quite easy to solve for (similarly it’s easy to add in a crude adjustment to an ML model). However, to truly solve for a large scale multilayered ML model you have to think creatively.</p><p>The brute force solution would be to run an iterative scheme for each calculation, but given the sophistication of our models that would leave borrowers waiting hours for offers. To solve it, we came up with a clever approach that uses discretization to capture much of the effect, while reducing the number of calculations at runtime. This enables us to run our risk models many times in parallel to arrive at the appropriate answer. In fact, M18 generates approximately 1 million predictions for each applicant in order to converge to the correct APR, which is six times the number of predictions of our prior model.</p><p>Beyond simply accounting for APR in our model, we went a step further and modeled for adverse selection effects. That is, when a borrower accepts an offer that is significantly higher than similar borrowers, it can be seen as higher risk. To address this, we used cutting-edge ML-in-ML to identify offers likely to be subject to adverse selection and price that risk appropriately, avoiding any spurious correlations.</p><p>Even with clever use of math and statistics, our calculations increased tenfold which raised our compute costs by 3x and latency by 70%. We had a challenge to deliver these model benefits at scale without negatively affecting the customer experience and keeping our costs in check. To speed up our predictions by 85%, we made extensive low-level changes to our XGBoost library to optimize tree traversal while maintaining full decimal parity — creating a brand new innovative inference algorithm as a result. This allowed us to reduce those higher compute costs by 37%, as well as model inference latency by 26%.</p><p>Given the increase in complexity and scale of Model 18 compared to its predecessor, we needed to thoroughly validate the model’s end-to-end performance and how well it was integrated throughout our systems. To accomplish this, we added the ability to “shadow price” loans, which entails asynchronously replaying all pricing transactions in production using Model 18 and comparing the outcomes of both models in near-real time. This piece of infrastructure allowed us to test the new model at scale with zero customer impact, ensure the correctness of our loan offers across thousands of transactions, and validate that production outcomes aligned with those obtained during research.</p><p>For the borrower, this model means (1) higher likelihood of approval as we can lend to a large population and (2) getting a more accurate rate. For some borrowers that will mean a lower rate. For others it will help us better ensure that our loans are affordable and do not contribute to financial stress. For our lenders, this new model means greater confidence that they will realize their expected returns thanks to lower losses, even as borrowers continue to feel the effect of high inflation in an uncertain economy.</p><p>This kind of innovation is what Upstart is all about: Bringing together the best in ML/AI research and engineering to offer loans that drive the best possible outcomes for consumers and lenders. But we’re just getting started. We have a pipeline of research and new data sources ready to roll out over the next six months with the potential for even bigger accuracy wins along with more new ideas on the horizon. By continuing to drive improvements in our model, we can approve more borrowers and offer even more reliable returns to our lenders.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6b46ca3225f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/upstart-tech/how-we-built-our-most-advanced-underwriting-model-yet-a6b46ca3225f">How We Built Our Most Advanced Underwriting Model Yet</a> was originally published in <a href="https://medium.com/upstart-tech">Upstart Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Mobile App: Zero to One]]></title>
            <link>https://medium.com/upstart-tech/building-a-mobile-app-zero-to-one-79b862f6d847?source=rss----f8ac0e656cc0---4</link>
            <guid isPermaLink="false">https://medium.com/p/79b862f6d847</guid>
            <category><![CDATA[mobile-app-development]]></category>
            <category><![CDATA[react-native]]></category>
            <category><![CDATA[mobile]]></category>
            <category><![CDATA[ios-app-development]]></category>
            <dc:creator><![CDATA[stephskardal]]></dc:creator>
            <pubDate>Thu, 16 Nov 2023 16:03:13 GMT</pubDate>
            <atom:updated>2023-11-16T16:03:13.053Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1TbmTiHCMYlM5Yq_-wIMNw.png" /></figure><p>Hi! In this blog post, I’ll be sharing my experience of working on <a href="https://www.upstart.com/">Upstart</a>’s borrower-facing mobile iOS app, now released and available in the <a href="https://apps.apple.com/app/apple-store/id6450968733?pt=126471806&amp;ct=TechBlogMobileZeroToOne&amp;mt=8">App Store</a>!</p><p>Upstart has been primarily a monolithic web stack during our time as a company, with a recent increase in engineering effort to move to a microservice architecture. We have been focused on expanding product verticals and serving borrowers with a significant investment in responsive web. There had been growing interest internally and externally in offering a mobile app, so we put a small team together to build a proof of concept that could evolve into a MVP (minimum viable product). I joined the mobile team to begin working on that proof of concept, with additional goals of shaping early technical direction and quickly adding a second mobile engineer. Here are some highlights from this journey!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UOX5QOIzh__fYthg" /><figcaption>Learn more about what Upstart is and how it uses AI <a href="https://www.upstart.com/our-story">here</a><strong>.</strong></figcaption></figure><h3>Determining the Architecture</h3><p>Early on, we were tasked with determining the tech stack of choice for our mobile app. We expected we would remain a small team, therefore building out two separate teams (iOS and Android) was going to be untenable for this launch. We also believed at the time that the MVP wasn’t going to be extremely technically complicated; most likely it would target functional parity with our web experience. In our initial launch, we’d have no need for the power and capabilities of a native implementation.</p><p>The general mantra in those early months was “walk before we run,” i.e., get an app out the door, provide value to our users, iterate over time to expand on functionality, and then leverage mobile features such as notifications or new experiences that could drive engagement. We believed at that time the most complicated element would be integration with an existing, constantly evolving monolith with no mobile infrastructure and many years of tech debt.</p><p>We felt we had a couple of clear cross-platform options (React Native, Flutter, MAUI, Cordova) to get the most value given our small team. The most viable option that jumped out was React Native with TypeScript, for the following reasons:</p><ul><li><strong>Upstart Investment in React and TypeScript: </strong>Upstart has invested a large amount of frontend development in React and TypeScript, and its use across Upstart’s ecosystem is mature. New frontend projects follow the adoption of React, TypeScript, and our internal component library (for web).</li><li><strong>Expert Engineers: </strong>Because of this investment, we have an abundance of React and TypeScript engineers who could contribute in meaningful ways during this proof of concept development and beyond. The interoperability between React and React Native is high.</li><li><strong>Community support:</strong> Generally speaking, the React Native community and ecosystem are quite active. Other cross-platform communities have a similar ecosystem, but they didn’t intersect with Upstart’s experience and ecosystem.</li></ul><p>Because one of the goals would be to propel the proof of concept into a MVP app, we felt that React Native would require the least ramping up and potentially the easiest throwaway work if down the line we wanted to abandon it due to any technical limitations. We never want to throw away work, but it made sense to ramp up and demonstrate potential quickly with a framework that a small team could execute on.</p><p>A number of people weighed in on this early tech stack decision and the team moved forward on the proof-of-concept app in those early months.</p><h3>Webviews vs. Native vs. Hybrid</h3><p>Much of our experimental time was spent evaluating between approaches and patterns in webviews, native, and hybrid. For context, a “webview” is a full browser embedded in a mobile app (HTML, JS, CSS, without all the browser packaging), similar to an iFrame in the web world. A native implementation means that content and all interactions live in native code. “Native” here means React Native, which under the hood translates to native app code for Android and iOS. A hybrid approach blends webviews and native as seamlessly as possible — ideally an end user has no idea what is native and what is a webview, while the engineers benefit from the control of deciding which content-serving mechanism is best.</p><p>For a tenured mobile engineer at a mobile-first company or even a company that was built with a strong external API architecture, the technical decision to go with webviews versus native might not be interesting. But for a zero-to-one app where there is no existing webview or API infrastructure in a complex, mostly monolithic architecture, the decisioning around where to build native and webviews is perhaps more interesting and challenging than the implementation itself. Framed another way: It is challenging to take a high-velocity monolithic architecture that is already serving customers and build a new client consumer requiring a service-oriented architecture without migrating the entire monolith to the ideal architecture.</p><p>In our early experiments, a hybrid approach came out as the winning choice. We learned a number of important lessons in supporting that approach:</p><p><strong>UX Interactivity</strong></p><p>Having a significant amount of UX interactivity (dynamic height, modals, animations) lends itself to native implementation. Generally speaking, the native animation will be snappier and smoother than those inside a webview, i.e., native UX typically “feels” better than webview interactivity on a mobile device.</p><p><strong>API Support</strong></p><p>If there are no APIs to provide backend data, they need to be built. In a post-MVP world, backend APIs should support versioning and multiple consumers like web and mobile. In a monolith web world, API versioning may never have been a consideration because everything ships together!</p><p><strong>Content Complexity</strong></p><p>The complexity of the content may lend itself to webviews if an organization can’t support maintaining logic in two places. For example, a team or organization may not be structured to best support two different codebases if the complexity is high. If shared logic** doesn’t work, webviews are appealing.</p><p>**Examples of shared logic:</p><ul><li>Backend / service only, which returns data structures that the different consumers can maneuver into the right form factor at the presentation layer.</li><li>Presentation layer, which returns fully formed API JSON data consumed by the web and mobile app.</li><li>Frontend component based, which is shared in both frontend web and frontend mobile components.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xSMOWtB6u2RIXyNO" /><figcaption>Examples of shared logic options among web and mobile.</figcaption></figure><p><strong>High-Velocity Changes</strong></p><p>A culture of updating content at a high frequency lends itself to webviews as well. Here at Upstart, we now strive for at least three daily releases of our monolith, and many of our microservices release more frequently. Meanwhile, the app release cycle is slower as it is dependent on App Store approval and user update rate. Any content that has a high churn of code can be served in a webview that lives in its own source code release cycle, decoupled from the app release cycle.</p><p><strong>Serving Partial Content</strong></p><p>If there is no ability to serve a section of a webpage for a webview, the code must be modified to support that. This can be done either via additive or subtractive methods, i.e., creating small HTML chunks that contain only the smallest pieces of content, or by rendering a larger piece of content and making only the relevant content visible via style rules. The former is optimal from a content performance perspective. Navigating legacy code in a performant and scalable way may require attention to detail to achieve this.</p><p>After a few months and a lot of experimentation in webview-landia, the proof of concept gained support. Our team saw alignment with the product and design teams on the scope of a MVP, shaped by a strong product vision combined with early technical experimentation. The engineering team also grew by one — bringing in a much appreciated tenured mobile engineer!</p><p>Our team felt confident about the scope of work except for a pretty big piece: A major part of the mobile app functionality used webviews from a user experience that hadn’t evolved in several years. There was an opportunity to improve the user experience for both web and mobile with the improvement of the webview building blocks. Additionally, the code driving that experience was behind the curve on Upstart’s latest frontend tech architecture. Any time spent updating or improving “legacy” code could become throwaway work if the code was to be migrated to the new frontend solution.</p><p>So, the mobile team committed to a large effort on the update of the web experience for users, which would be built in a way to support shared API-level integration and/or webview integration and offer parity in the mobile app. Another intentional decision was made at this time, as well, to align with the direction of our identity platform team, which was set to release and support a centralized identity service in early Q3.</p><h3>Execution &amp; Delivery</h3><p>By this point, we had a growing team and a generous number of contributors from all parts of Upstart (legal, compliance, growth, quality, operational support, and more!) with a few major goals:</p><ul><li>Finish the implementation in the app for those parts that would be native.</li><li>Contribute as major stakeholders in the migration and update of the web experience that would be a large part of the mobile app, shaping the technical direction to allow consumption by web and mobile. Update the mobile app to consume these pieces as they were delivered incrementally.</li><li>Update our mobile authorization to adopt the centralized auth service built by our identity team, working closely with them on specific mobile needs.</li></ul><p>We overcame a few bumps and were able to execute on all of these deliverables within a few months. While there were early proof-of-concept decisions made around native vs. webview, a handful of those decisions evolved as the shared UX for web and mobile was finalized.</p><p>This project phase also saw the completion of a handful of quality and maintenance concerns:</p><ul><li>We put significant effort into optimizing the user experience in-app by preloading data in the background.</li><li>With the help of a quality engineer, we supplemented unit tests by adding end-to-end tests that focused on integration signals like login and webview form submission processes.</li><li>With support from our React engineers, we implemented stricter linting rules and cleaned up technical debt (yes, we still created tech debt in a POC!).</li></ul><p>Earlier last month, we submitted our app to the App Store, and it was approved! Find it <a href="https://apps.apple.com/app/apple-store/id6450968733?pt=126471806&amp;ct=TechBlogMobileZeroToOne&amp;mt=8">here</a>.</p><p>For the remainder of this quarter, we plan on ramping up and monitoring our app through observability and metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*758rIgoqrOheCyvx" /><figcaption>Find our app <a href="https://apps.apple.com/app/apple-store/id6450968733?pt=126471806&amp;ct=TechBlogMobileZeroToOne&amp;mt=8">here</a>.</figcaption></figure><h3>Lessons Learned and What’s Next?</h3><p>While we learned a lot about working with a strong product vision to build a mobile app for zero-to-one, one important lesson is this: Technical decisions aren’t necessarily objective. The people and organizational structure of the team will influence the technical direction of a project. No doubt the journey from idea to launch for this app could have looked different if approached by a different team or organization. But the end result, if done right, should be the same: a strong, ready-to-launch product.</p><p>What’s next for us? First and foremost, we want to continue to have stable releases and analytics that help us shape our direction (walk before we run, with data!).</p><p>After that, we’re considering a few possible next steps: capitalizing on the cross-platform tech decision of React Native to offer the app in Android, building out notifications, bringing more web functionality over to mobile, or offering new mobile-featured functionality to engage both new Upstart users and existing users. We haven’t decided on the exact direction yet, but exciting opportunities await in our <a href="https://apps.apple.com/app/apple-store/id6450968733?pt=126471806&amp;ct=TechBlogMobileZeroToOne&amp;mt=8">now-released mobile app</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79b862f6d847" width="1" height="1" alt=""><hr><p><a href="https://medium.com/upstart-tech/building-a-mobile-app-zero-to-one-79b862f6d847">Building a Mobile App: Zero to One</a> was originally published in <a href="https://medium.com/upstart-tech">Upstart Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Numerical Optimization Unlocked Value for Upstart]]></title>
            <link>https://medium.com/upstart-tech/how-numerical-optimization-unlocked-value-for-upstart-14ab3e7abff8?source=rss----f8ac0e656cc0---4</link>
            <guid isPermaLink="false">https://medium.com/p/14ab3e7abff8</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[numerical-optimization]]></category>
            <category><![CDATA[loans]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[John Vandivier]]></dc:creator>
            <pubDate>Thu, 10 Aug 2023 21:43:51 GMT</pubDate>
            <atom:updated>2023-08-10T21:46:28.972Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By John Vandivier and Xingwen Zhang</em></p><p>Upstart is the leading AI lending marketplace, working at the intersection of finance and machine learning (ML). The application of ML to finance involves many interesting problems related to <a href="https://en.wikipedia.org/wiki/Mathematical_optimization">numerical optimization</a>, which is the mathematical process of selecting particular numbers that best satisfy criteria that are important to the outcome you’re trying to produce. In this article, we’ll focus on a specific kind of problem in this area — convergence errors — and show how we found a solution that allowed Upstart to recover substantial value for the company, our customers, and our lending partners.</p><p>First, let’s show how both numerical optimization and convergence errors apply to Upstart. For numerical optimization, the outcome we’re trying to produce could be loan terms that account for a borrower’s default risk on that loan, while a criteria we’d optimize for would be the right level for the interest rate, one that’s neither too low nor too high. A convergence error in this case would mean we were unable to mathematically determine a single ideal interest rate for a given loan. That’s a problem for Upstart because it impacts our accuracy in determining risk and our revenue, two key business metrics.</p><p>This played out in a feature we launched called Dynamic Pricing. As a lending marketplace, Upstart connects prospective borrowers with loan offers from our lending partners. One aspect of Dynamic Pricing allows our lenders to state their desired return on a loan using a table of applicant FICO scores and risk tiers, where the applicant risk is assessed and graded by Upstart.</p><p>Dynamic Pricing is a valuable feature for our partners because each partner has a unique appetite for risk, a unique risk strategy, and differing requirements on return for different levels of risk. But this feature caused errors to occur in our prior mathematical approach to loan offer creation, which involved using a <a href="https://en.wikipedia.org/w/index.php?title=Root-finding_algorithms&amp;oldid=1145256963">root-finding algorithm</a> to calculate the accompanying interest rate on a loan. This calculation integrates a borrower’s risk assessment with the desired return expressed by our lending partner.</p><p>The Dynamic Pricing rollout included an ensemble of testing and observability measures. One concrete observability tactic in particular allowed us to rapidly catch and resolve the convergence issues: a tool called <a href="https://docs.celeryq.dev/en/stable/getting-started/introduction.html">Celery</a>. Because real-time numerical optimization is resource intensive, Upstart engineers strategically encapsulate this work into asynchronous jobs using Celery. By ensuring that Celery is integrated into our production logging system, we were able to quickly observe and react to any kind of error that is internal to this complex logic.</p><p>Dynamic Pricings were observed to produce convergence errors in a low single-digit percentage of cases. Upstart engineers rolled out a quick fix that cut this value in half. The quick fix was a simple mathematical change: We reduced the convergence precision requirement, which enabled more root-finding routines to claim that they had converged, thereby decreasing the errors. But the problem is that lower precision means Upstart becomes less effective in our main service — accurately assessing risk. Therefore, this fix wasn’t a fix at all by our standards.</p><p>Upstart engineers quickly coordinated with our experts in machine learning to form a longer-term solution. A simple solution that proved effective was a linear interpolation using a smoothing factor. This solution is diagrammed below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-SEgUisIYU1KPhah" /></figure><p>Mathematically, the smoothing factor is a distance on the interest rate axis, over which a new and flatter line is calculated to replace the prior vertical line. In the graph above, B is the interest rate location where the vertical discontinuity, and its associated convergence issue, were originally encountered. At the point of convergence failure, our root-finding algorithm is modified to look around and interpolate linearly within a distance of the smoothing factor in either direction from the discontinuity.</p><p>In this context, as shown in the prior diagram, the output variable is the <a href="https://www.investopedia.com/terms/n/npv.asp">net present value (NPV)</a> of the loan. The NPV represents the sum of the loan’s future cash flow, discounted to account for the time value of money, which reflects the concept that money available today is worth more than the same amount in the future.</p><p>Upstart calculates an interest rate that satisfies a target <a href="https://www.investopedia.com/terms/i/irr.asp">internal rate of return (IRR)</a> — the annual rate at which an investment grows. This target IRR is the return our lending partners expect on the capital invested in the loan. It’s the rate that yields an NPV of zero with respect to Upstart because every dollar is accounted for: The net cash flows from the borrower exactly match the net payments to the lending partner.</p><p>To find a loan that matches the target IRR, Upstart calculates an expected internal rate of return based on the default risk and prepayment probability of the loan. The higher this risk, the lower the expected IRR at a particular interest rate. We iterate through the set of possible interest rates until we find an interest rate that yields a net present value of approximately zero, indicating that the target IRR is approximately equal to the expected IRR.</p><p>Alongside the smoothing factor, engineers added log messages to enable Upstart analyst correlation between the smoothing factor and convergence errors. This enabled Upstart to tune the smoothing factor as a hyperparameter, finding the smoothing factor number that minimized the frequency of convergence errors. Larger values eliminate more errors at the cost of precision. Smaller values increase both precision and the risk of a convergence failure. The smoothing factor was implemented as a <a href="https://docs.launchdarkly.com/home/flags/variations">LaunchDarkly multivariate flag</a>. This implementation enabled Upstart to update the value many times without many code changes or deployments. This implementation is also strategically defensive. If needed, the feature can be disabled without a code release by assigning a value of zero.</p><p>An initial smoothing factor was rolled out in Q4 of 2022. Within weeks, an ideal value was found that reduced the daily error frequency to zero while retaining maximum precision in the convergence calculation. An error frequency of zero means more applicants are awarded loans, lending partners are able to profit by funding more loans, and Upstart is able to improve our bottom line through increased revenue, reduced cost, and higher accuracy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14ab3e7abff8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/upstart-tech/how-numerical-optimization-unlocked-value-for-upstart-14ab3e7abff8">How Numerical Optimization Unlocked Value for Upstart</a> was originally published in <a href="https://medium.com/upstart-tech">Upstart Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Upstart Cut Near $10M in AWS Cloud Costs Using the 80/20 Rule]]></title>
            <link>https://medium.com/upstart-tech/how-upstart-cut-near-10m-in-aws-cloud-costs-using-the-80-20-rule-2cc634c59324?source=rss----f8ac0e656cc0---4</link>
            <guid isPermaLink="false">https://medium.com/p/2cc634c59324</guid>
            <category><![CDATA[finops]]></category>
            <category><![CDATA[80-20-rule]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cost-savings]]></category>
            <dc:creator><![CDATA[Christopher Gray]]></dc:creator>
            <pubDate>Thu, 22 Jun 2023 21:53:28 GMT</pubDate>
            <atom:updated>2023-06-22T21:53:28.142Z</atom:updated>
            <content:encoded><![CDATA[<p>At Upstart, we’ve successfully saved approximately $10 million on our <a href="https://aws.amazon.com/">AWS cloud</a> costs by using <a href="https://www.investopedia.com/terms/1/80-20-rule.asp">the 80/20 rule</a> to optimize our compute, database, storage, and network expenses.</p><p>The 80/20 rule, also known as the Pareto principle, is a widely recognized principle in business that suggests 80% of a given output comes from 20% of the input. In the context of cloud cost optimization, this means that a majority of the cost savings can be achieved by focusing on a few key areas that account for a large portion of the overall expenses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FeFsjtYdhpkGPjO7Sxbmlw.jpeg" /></figure><p>Here’s how we were able to achieve significant cost savings in those areas while maintaining the same level of performance and reliability.</p><p><strong>Compute Optimization</strong></p><p>Upstart optimized our compute costs by analyzing our fleet of Kubernetes clusters to identify under utilization of both CPU and Memory, and to perform rightsizing efforts. To accomplish this, we took a look at the workloads and appropriately resized the CPU and Memory of applications based on historical performance. Additionally, we identified that in some cases, the fleet of workloads would be better optimized on different classes of AWS Instances, moving from the C class to R class instances, which have an 8:1 Memory ratio instead of the 2:1 Memory ratio of the C class. This helped to fit more of the applications onto the EC2 instances and reduce the size of the fleet by up to 80% per cluster.</p><p>Additionally, a review was performed on EC2 instances to identify abandoned workloads that could be safely cleaned up. In the effort of identifying the machines, monitors were put in place to alert on instances with low to no utilization to keep the cloud environment free of abandoned workloads.</p><p><strong>Database Optimization</strong></p><p>Upstart optimized our database costs by identifying and moving data out of large databases and moving them to S3, and updating the application functions to pull data from S3 buckets rather than directly out of the database. Shrinking the database storage aided in reducing the IOPS, storage costs, as well as backup costs. In addition to considering our current retention and deletion process and database cleanup, we conducted a review of our backup policies and made updates to those allowed for reducing the unit costs of our database backups with automated lifecycle policies and safely purging of nonessential redundant backups without the risk of losing any essential data.</p><p><strong>Storage Optimization</strong></p><p>Upstart optimized our storage costs by using Amazon S3 for object storage and Amazon Elastic Block Storage (EBS) for block storage. With Amazon S3, Intelligent Tiering and Lifecycle Policies were put into place to archive data to greatly reduce the unit cost at scale. For the EBS Volumes, in addition to considering our current retention and deletion process, analysis was performed to find non-attached volumes that could be purged, as well as finding rightsizing opportunities for volumes that were too large or had more IOPS than were required. A migration from GP2 to GP3 volumes was utilized in multiple cases to further reduce cost while increasing performance capabilities.</p><p><strong>Network Optimization</strong></p><p>We optimized our network costs by analyzing traffic patterns and identifying places where VPC Endpoints or PrivateLink could be implemented to reduce network fees, as well as adding to our security for the private network traffic. The implementation of VPC Endpoints and PrivateLink greatly reduced the network fees going over the NAT Gateways.</p><p><strong>Conclusion</strong></p><p>Upstart’s successful implementation of the 80/20 rule demonstrates the effectiveness of this principle in achieving cost savings in cloud infrastructure. By focusing on the key areas of compute, databases, storage and network, Upstart was able to achieve significant cost savings while maintaining the same level of performance and reliability. This approach can be applied to any cloud infrastructure to achieve similar cost savings and optimize cloud expenses.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cc634c59324" width="1" height="1" alt=""><hr><p><a href="https://medium.com/upstart-tech/how-upstart-cut-near-10m-in-aws-cloud-costs-using-the-80-20-rule-2cc634c59324">How Upstart Cut Near $10M in AWS Cloud Costs Using the 80/20 Rule</a> was originally published in <a href="https://medium.com/upstart-tech">Upstart Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we estimate the impact of the macroeconomy on loans]]></title>
            <link>https://medium.com/upstart-tech/how-we-estimate-the-impact-of-the-macroeconomy-on-loans-b02ec701d7c0?source=rss----f8ac0e656cc0---4</link>
            <guid isPermaLink="false">https://medium.com/p/b02ec701d7c0</guid>
            <category><![CDATA[lending]]></category>
            <category><![CDATA[macroeconomics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Grant Schneider]]></dc:creator>
            <pubDate>Thu, 23 Mar 2023 15:01:34 GMT</pubDate>
            <atom:updated>2023-03-30T14:38:19.397Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Grant Schneider, Vice President of Machine Learning</em></p><p>Over my eight and a half years at Upstart, I’ve had the pleasure of working on a lot of interesting and challenging problems that were essential to our business. (See <a href="https://www.upstart.com/updates/meet-grant-schneider-head-of-upstart-columbus">here</a> for just how many!) Recently, the Machine Learning team has been working on yet another: quantifying one of the most dynamic macroeconomic environments seen in decades.</p><p>The past year reinforced the big impact that changing macroeconomic conditions like inflation and unemployment can have on Americans’ finances, and in turn, the credit performance of loans.</p><p>So, we wondered: Is there a way to measure the impact of the changing macro on Upstart-powered loans so that our bank and credit union partners could better manage their lending?</p><p>Turns out there is. We call it the <a href="http://www.upstart.com/umi">Upstart Macro Index (UMI)</a>. We launched it yesterday, and I’m going to explain how it was built.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fj8hw2KpMaBOWCYvVfGriA.png" /></figure><p>First, we needed a way to quantify the macro impact on our borrowers across different months in a way that was both rigorous and actionable. As a classically trained statistician, my first instinct would be to run an experiment holding everything constant except the month. But that doesn’t work because no such constants exist. We can’t give the same person the same loan in multiple months in order to A/B test the effect of a given pair of months. Even if the same person came back for a second loan, they would be in different financial circumstances and, at the very least, have an additional loan.</p><p>In theory, we could instead randomly sample a group of loan applicants, slow down the loan process for them to get a set of loans that spanned multiple months, and track their performance in relation to the macroeconomy during that period. However, we know our <a href="https://www.trustpilot.com/review/www.upstart.com">borrowers really love our industry-leading speed</a>, and intentionally slowing things down is <a href="https://review.firstround.com/speed-as-a-habit">near heresy around these parts</a>.</p><p>So, we can’t run an experiment. Could we try a simple comparison of the same cohort’s loan performance in two different months? This won’t work, either, because the risk profile of a loan will change while it’s in payment. This “month on book” factor — the number of months since the loan was originated — is an incredibly important feature for predicting risk, and we’ve spent a significant amount of time modeling and thinking about the relationship between them at both the individual loan and portfolio level.</p><p>But what about comparing the performance of different cohorts in the same month on book? For example, August 2022 loan originations in November 2022 and October 2022 loan originations in January 2023 would both be scheduled for their third payments. The problem with this approach is that upgrades to our underwriting model, changes in our marketing efforts, macroeconomic shifts, seasonality, and any number of other factors can make monthly cohorts very different, especially when there is a long period of time between them.</p><p>You may also ask if we could look at a subset of borrowers within these cohorts based on their credit score or income. But that’s too simplistic. Our entire business is premised on the idea that a person is more than just a three-digit credit score and that artificial intelligence can be used to better determine if someone is creditworthy.</p><p>If none of these methods work, then how did we do it?</p><p>At a high level, we evaluate our underwriting model’s predictions in comparison to observed defaults to assess any uncaptured effects in a given month. Fortunately, our innovative (and <a href="https://patentimages.storage.googleapis.com/c0/f9/14/e1b80540e2545b/US11568312.pdf">patented</a>!) “loan-month model” already provides us predictions on a per-loan, per-month basis. This is in contrast to more traditional models, which predict losses at the individual loan level or even just the portfolio level.</p><p>We model default hazard ratios (for those of you familiar with survival analysis) as the dependent variable here. These ratios are the predicted probabilities of default at a given time in the life of a loan conditional on having not already defaulted or fully paid off the loan, and there are multiple ways to adjust them to try to quantify the macro effect. The simplest would be to apply a post-hoc adjustment to match the actual losses in a given month we’re tracking — called an “observation month” — to those predicted by the model. But in circumstances where monthly macro effects are correlated with other inputs to the model, such as credit or employment variables, this sequential approach of first fitting the model and then estimating the effects could cause the macro effects to be under or overweighted and therefore incorrect.</p><p>Instead, we add an indicator variable for each observation month to the model inputs in our model-training pipeline and retrain it. The indicator variable assigns a specific value to the observation month in such a way that the model won’t make that mistake in correlation. As a result, we can jointly estimate the effect of a given month alongside the more than a thousand other variables we track, allowing the model to detect nuanced relationships between them.</p><pre>model_pl = Pipeline([<br>    (&#39;features&#39;, feature_eng_step),<br>    (<br>        &#39;add_obs_month&#39;,<br>        ColumnTransformer([<br>            (<br>                &#39;observation_month&#39;,<br>                convert_to_obs_month_cat_feature,<br>                [&#39;month_on_book&#39;, &#39;first_payment_month&#39;]<br>            ),<br>            (&#39;other_features&#39;, &#39;passthrough&#39;, other_features)<br>        ])<br>    ),<br>    (&#39;estimator&#39;, estimator)<br>])<br>model_pl.fit(X, y)</pre><p>We then take a sample of 2 million historical loan payment events (meaning whether a payment was made or not and what month on book) and regenerate predictions for the entire set of possible observation months since January 2017. Why this particular point in time? Because it was one of the earliest time points where we had enough loan volume for the predictions to be stable. As an example, say that in our sample of 2 million payment events we isolate payments two and five for a specific loan originating in May 2022, corresponding to observation months of July 2022 and October 2022. We would then recompute the predicted hazard ratios for each of these two payment events for each of the possible observation months to date (January 2017 to February 2023), yielding 74 months × 2 million sampled payments for a total of 148 million predictions.</p><pre>for first_pmt_month_i in payment_months_of_interest:<br>    X[:, first_pmt_month_idx] = first_pmt_month_i<br>    preds = model_pl.predict(X)</pre><p>Next we take the average of the 2 million predicted hazard ratios for each of the calendar months, resulting in a vector of length 74, or the average of that 2 million for each of the 74 months. Finally, we normalize these average predicted hazards by a long-term average hazard ratio. This makes the number more interpretable, so that a UMI value of 1 means that defaults should be in line with long-term average expectations, a value of 2 means they should be doubled, a value of 0.5 means they should be halved, and so on.</p><p>Now that we have our quantification of the relative macro effects, we (and you, as of today!) can update our perspectives monthly on how the creditworthiness of American consumers is evolving. Furthermore, we can apply an adjustment to the hazard ratios for loans we’re originating today, which is informed by UMI. This forward adjustment is a powerful tool that allows our lending partners to combine their unique outlook on macro with our state-of-the-art risk models to obtain a bespoke model at the cutting edge of credit risk.</p><p>A final question you might ask is whether we’re able to use this observation-month variable in prediction given we won’t have any training data for future months. This is an exciting area of active research, and we have multiple promising ideas to pursue here — so, stay tuned!</p><p><em>Upstart intends to release UMI monthly, including revisions to prior months when applicable. Subscribe for first access to those updates here: </em><a href="http://upstart.com/umi"><em>upstart.com/umi</em></a></p><p><strong>LEGAL DISCLAIMER</strong></p><p>The statements and information on this site are current as of March 21, 2023, unless another date with respect to any information is indicated, and are provided for informational purposes only. Past UMI performance can provide no assurance and is not indicative of future UMI results. UMI is based on historical data and Upstart’s analysis of the losses within Upstart-powered loan portfolios and is specific to Upstart’s borrower base. UMI is not intended to measure the macroeconomic risks in terms of losses of loan portfolios or asset classes that are not Upstart-powered loans, including loans held by other segments of the U.S. population. It is not designed to measure the current state of the overall economy or to measure or predict future macroeconomic conditions, trends or risks. It is also not designed to measure or predict the future performance of Upstart-powered loans or of Upstart’s other products, overall financial results of operations or stock price. We expect that our research and development efforts to improve UMI could result in changes or revisions to current or past UMI values.</p><p>All forward-looking statements or information on this site are subject to risks and uncertainties that may cause actual results to differ materially from those that Upstart expected. Any forward-looking statements or information on this site are only as of the date hereof. Upstart undertakes no obligation to update or revise any forward-looking statements or information on this site as a result of new information, future events or otherwise. More information about these risks and uncertainties is provided in Upstart’s public filings with the Securities and Exchange Commission, copies of which may be obtained by visiting Upstart’s investor relations website at <a href="https://www.upstart.com/">www.upstart.com</a> or the SEC’s website at <a href="https://www.sec.gov/">www.sec.gov</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b02ec701d7c0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/upstart-tech/how-we-estimate-the-impact-of-the-macroeconomy-on-loans-b02ec701d7c0">How we estimate the impact of the macroeconomy on loans</a> was originally published in <a href="https://medium.com/upstart-tech">Upstart Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Self-Service OAuth with Okta and Terraform]]></title>
            <link>https://medium.com/upstart-tech/self-service-oauth-with-okta-and-terraform-ede1555465ab?source=rss----f8ac0e656cc0---4</link>
            <guid isPermaLink="false">https://medium.com/p/ede1555465ab</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Lukas Indre]]></dc:creator>
            <pubDate>Wed, 18 Jan 2023 22:37:21 GMT</pubDate>
            <atom:updated>2023-01-18T23:02:21.949Z</atom:updated>
            <content:encoded><![CDATA[<p>Here at Upstart, our engineering architecture is growing to be ever more complex, both on the product side and internally. This includes both internal application communication as well as integrating external applications. With software, everyone is chomping at the bit to write that amazing tool to solve problems X, Y, and Z as the opportunity for solving these problems grows.</p><p>For example, it feels great to finally finish that code snippet to automate a mindless, time-consuming manual task that’s been eating away at you for a while. But before you share that tool, there’s a feeling in your stomach that seems impossible to shake. You have tested locally, operating under implicit trust from one system to another, but how is trust handled when your pipeline is performed by a computer hosted in the cloud? Authorization, that was the pit in your stomach! Fortunately, authorization (“authz”) can be both easy and self-service, while keeping security in mind, using a few simple tools: Okta, Terraform, and a secrets vault (AWS Secrets Manager in this case).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uh6oJpOsqyF4sa1q3KnxVg.jpeg" /></figure><h3>The Current State and Its Problems</h3><p>We have all seen it; that super-privileged (hopefully encrypted) API key that sits waiting to be accessed and used in Authorization headers for your API calls. In most cases, when interacting with external services, you have the option to paste a generated API key into a form-field to be used for HTTP requests. This is super convenient and easy, but here are a couple issues that may come to light from using this option:</p><ol><li>You just pasted a secret key into a field that you have no control over, exposing that key to external databases or services owned by others.</li><li>There is nothing programmatically telling you what constraints govern what this key is able to access and do. Once you create this key, you have no control on what requests can be made under its privileges. If this key is compromised, your attacker only has to try a handful of authorization tactics that are common amongst most static credentials.</li><li>Which brings me to my next point: In most cases, this key is static. The longer a static key exists, the more likely it is to be compromised. This may motivate key rotation, which is an entirely separate, fruitfully fault-prone process.</li></ol><h3>Welcome, OAuth</h3><p>OAuth, or Open Authorization, is an authz protocol that can address most limitations of a static API key. Rather than a service inherently trusting an HTTP header, there is a series of handshake transactions to gain access to resources, prior to accessing any secure servers. The transactional flow sequence looks like this:</p><ol><li>The service requests an access token restricted to certain scopes from the authorization server (in this case, with a client credentials flow).</li><li>The authorization server checks the token request against its defined access policies for the OAuth client. If the client passes the checks, an access token is returned, scoped to the necessary resources.</li><li>The service uses the token (typically as a Bearer) to make subsequent requests to the secured resources.</li><li>Each request received by the server is tested. In this case, a token introspection endpoint is used by the server to determine if the token is valid (not expired, correct scopes for resources and operations requested). If the token is valid, the HTTP request is successful, otherwise the request should throw an error.</li></ol><p>This flow may seem very confusing and a lot of extra work compared to a static key, and the benefits may not seem explicitly clear.</p><h3>Benefits of OAuth</h3><p>Now that we have an understanding of the general OAuth flow, let us outline its benefits as compared to static API credentials. From the above explanation, it may seem like we are just creating another static credential, and calling it something other than “API key.” Very literally, yes, we are! Even though there is a higher up-front cost to set OAuth up, we still have a much better security posture for authorization based on the following non-exhaustive list of reasons:</p><ul><li>While client credentials are static, the access token you are provided via successful OAuth transaction is not. Typically, the OAuth provider will configure access tokens minted for their servers to be dynamic and expirable. Therefore, if a Bearer token is compromised, your attacker will only have access to your resources for the amount of time your token is configured to be active. After that, your service should return HTTP errors (since the service has ongoing token introspection per request).</li><li>You may now be wondering about the scenario of a compromised client credential pair. Let us go through that exercise and imagine that an attacker gained access to your application’s client credentials. Oh no! Had they gotten your API key, they could try a few different ways to pass in Authorization headers to your server, and likely figure out pretty quickly how to make things go poorly for your team. This is where OAuth excels! If a hacker gets your client credentials, they have to figure out what kind of access policies you have configured in your OAuth provider to gain an access token. Typically, when making a token request to an OAuth server, you must provide a scope(s) to the authorization server, so the authorization server can check to see whether your client credentials have the ability to get an access token with that scope. The good thing about these scopes is that they are totally arbitrary! You define them as a developer in your application, and configure them in your OAuth provider. A hacker will have a harder time figuring out what client can access custom scopes that you defined, versus figuring out whether to pass a static API key in an Authorization header by way of Single Sign-On for Web Systems (SSWS) or Bearer.</li></ul><p>Here are a couple very common, well-known ways to use an API key, that are sure to be known by your attacker. You’ll notice that the credential never has to change and that the methods for authorization below are probably some that you have used before.</p><pre># Bearer authorization<br>curl https://my-very-secure-service.com/databases/123/delete \<br>  -H “Authorization: Bearer your-hacked-key-xyz”<br><br># Basic authorization<br>curl https://my-very-secure-service.com/databases/123/delete \<br>  -H “Authorization: Basic your-hacked-key-xyz”<br><br># SSWS Authorization (Okta’s API token authorization method<br>curl https://my-very-secure-service.com/databases/123/delete \<br>  -H “Authorization: SSWS your-hacked-key-xyz”<br><br># custom api key header, can be found in AWS API gateways, for example<br>curl https://my-very-secure-service.com/databases/123/delete \<br>  -H “x-api-key: your-hacked-key-xyz”</pre><p>Comparatively, here is what would happen if a client credential pair was compromised</p><pre># Token request without scopes parameter<br>curl \<br>https://my-very-secure-auth-server/token?grant_type=client_credentials&amp;client_id=your-hacked-id&amp;client_secret=your-hacked-secret</pre><p>In the above scenario, it is quite possible that you are able to retrieve a token without requesting scopes, but if you configure your application to only perform operations under a scoped token, the token generated here is useless.</p><pre># Token request with invalid scopes<br>curl \<br>https://my-very-secure-auth-server/token?grant_type=client_credentials&amp;client_id=your-hacked-id&amp;client_secret=your-hacked-secret&amp;scopes=fake.scope.guess</pre><p>In this scenario, you’d get an error return value, as fake.scope.guess is an invalid scope that is not defined in your authorization server. This is how OAuth adds another layer of security that is totally custom to each individual implementation of it.</p><ul><li>Access tokens are dynamic, both in expiration and scope. Another benefit of OAuth is flexibility for least-privileged access on a per-request basis! For example, given an application with read and write operations, it is likely that you will separate read and write privileges to minimize risk, assigning each privilege to a different set of accessors. This is where OAuth flexibility shines. All you have to do is request read scope with your client credentials, and then you don’t have to worry about accidental (or threat actor) write operations. If someone were to compromise this token, it would not have write access, which reduces the surface area of the threat. Conversely, if someone were to compromise your static API key that has all the permissions baked into it, they could do whatever they want.</li></ul><h3>Okta OAuth Provider with Terraform</h3><p>If I haven’t convinced you that OAuth is better than an API key, or you think the overhead of setting this up is too great, let’s move on to self-service OAuth.</p><p>Now understanding the benefits of OAuth, we must weigh the overhead cost against the security gain. There definitely is a high overhead, thinking in terms of standing up your own authorization server to mint access tokens, scopes, access policies, etc. Fortunately, at Upstart we have a great set of resources to handle this with. In the simplest case, as few as five lines of code is all it takes to provision OAuth resources, along with bringing your own credentials for your applications.</p><p>If you are lucky enough to be working with Okta, AWS, and Terraform, this is a breeze! We have recently gone through the exercise of configuring OAuth applications, scopes, access policies, and access policy rules in Okta. While this is a relatively easy task for a seasoned Okta developer, why should the Okta developer be a bottleneck for product development? I believe there is no reason to be the bottleneck, so I decided to write a self-service module for OAuth, using Okta as the OAuth Provider. Terraform acts as the self-service piece and AWS Secrets Manager stores client credentials.</p><p>All a developer needs to do is create a secret containing credentials for both client and server side applications, and then utilize <a href="https://github.com/lukasindre/terraform-okta-oauth-generator/tree/v2.0.0">this Terraform module</a> to provision the Okta resources needed. With a couple lines of code, and a little bit of knowledge surrounding Okta’s OAuth API endpoints, developers can be up and running with OAuth in just a couple of minutes! Wrapped in this module are plenty of configurable things, namely your service’s name, scopes, and token lifetime. In comparison, the end-to-end workflow for this without self-service would be:</p><ul><li>Developer submits request to Okta team to provision OAuth resources</li><li>Back and forth questionnaire between Okta team and developer</li><li>Okta team clicks what seems to be a million buttons to provision resources</li><li>Okta team shares client credentials with developer</li></ul><p>With change management, and time zone considerations for workers, the above workflow could take days, if not weeks, to complete. Below we can see a couple declarations of this module from the examples directory:</p><p>This is the most simple declaration of the module, if you do not require that much customization.</p><pre>module &quot;example_app&quot; {<br>  source = &quot;github.com/lukasindre/terraform-okta-oauth-generator.git?ref=v2.0.0&quot;<br>  label  = &quot;Test App&quot;<br>  client_credentials = {<br>    client_client_id     = local.creds[&quot;client_client_id&quot;]<br>    client_client_secret = local.creds[&quot;client_client_secret&quot;]<br>    server_client_id     = local.creds[&quot;server_client_id&quot;]<br>    server_client_secret = local.creds[&quot;server_client_secret&quot;]<br>  }<br>}<br><br>data &quot;aws_secretsmanager_secret_version&quot; &quot;creds&quot; {<br>  secret_id = local.secret_id<br>}<br><br>locals {<br>  secret_id          = &quot;my-secret-id&quot;<br>  credentials_object = jsondecode(data.aws_secretsmanager_secret_version.creds.secret_string)<br>}</pre><p>On the other hand, if you want to configure your own scopes, add token lifetime, and policy descriptions, you can see that in the following declaration:</p><pre>module &quot;easy_bake_service&quot; {<br>  source = &quot;github.com/lukasindre/terraform-okta-oauth-generator.git?ref=v2.0.0&quot;<br>  label  = &quot;Easy Bake Oven&quot;<br>  scopes = [<br>    {<br>      &quot;name&quot; : &quot;dough.cookie.mix&quot;,<br>      &quot;description&quot; : &quot;grant permission to mix cookie dough&quot;<br>    },<br>    {<br>      &quot;name&quot; : &quot;dough.cookie.bake&quot;,<br>      &quot;description&quot; : &quot;grant permission to bake the cookie dough&quot;<br>    },<br>    {<br>      &quot;name&quot; : &quot;cupcake.batter.mix&quot;,<br>      &quot;description&quot; : &quot;grant permission to mix cupcake batter&quot;<br>    }<br>  ]<br>  access_token_lifetime_minutes = 60<br>  client_credentials = {<br>    client_client_id     = local.creds[&quot;client_client_id&quot;]<br>    client_client_secret = local.creds[&quot;client_client_secret&quot;]<br>    server_client_id     = local.creds[&quot;server_client_id&quot;]<br>    server_client_secret = local.creds[&quot;server_client_secret&quot;]<br>  }<br>  policy_description = &quot;This policy restricts what the OAuth app is allowed to do with the easy bake oven service.&quot;<br>}<br><br>data &quot;aws_secretsmanager_secret_version&quot; &quot;creds&quot; {<br>  secret_id = local.secret_id<br>}<br><br>locals {<br>  secret_id          = &quot;my-secret-id&quot;<br>  credentials_object = jsondecode(data.aws_secretsmanager_secret_version.creds.secret_string)<br>}</pre><p>With this proposed self-service model, all the developer has to do is provide client credentials, know a little bit of Terraform and they can be on their way to OAuth as quickly as the Terraform build happens in their CICD pipeline.</p><h3>The Module</h3><p>This Terraform module takes care of all the nitty-gritty, Okta-nuanced details imposed by the Okta API, so developers don’t have to learn an entirely new framework to interact with to provision their resources. For the developer’s interest and needs, it creates the OAuth clients and developer-defined scopes and lets them self-service changes, speeding up development. For Security’s interest, it defines access policy and rules so that only the clients provisioned per module are able to access the defined scopes, versus an “All Clients” trust scenario, minimizing risk. Additionally, using this module frees resources from the Okta development team to improve our corporate identity posture in other great ways. Going forward, the use and evolution of this module will ensure that both Okta developers and software developers are spending their resources and time on developing, while keeping our Information Security team happy by both using OAuth and a templated authorization framework.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ede1555465ab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/upstart-tech/self-service-oauth-with-okta-and-terraform-ede1555465ab">Self-Service OAuth with Okta and Terraform</a> was originally published in <a href="https://medium.com/upstart-tech">Upstart Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Flaky Tests and CI Build Stability: A 2022 Summary]]></title>
            <link>https://medium.com/upstart-tech/flaky-tests-and-ci-build-stability-a-2022-summary-2ca92f9299ff?source=rss----f8ac0e656cc0---4</link>
            <guid isPermaLink="false">https://medium.com/p/2ca92f9299ff</guid>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[flaky-tests]]></category>
            <dc:creator><![CDATA[stephskardal]]></dc:creator>
            <pubDate>Mon, 09 Jan 2023 20:07:14 GMT</pubDate>
            <atom:updated>2023-01-09T20:07:14.871Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dOe5zmHiqWMA9PTsJlxHHw.png" /><figcaption><a href="https://www.upstart.com/">https://www.upstart.com/</a></figcaption></figure><p>Here at Upstart, we’ve done quite a bit of work over 2022 to improve continuous integration (CI) build stability for our monolith as part of a greater effort to improve our tech platform. We’ve done other great things, but CI build stability and dev experience is the focal lens I often choose to write about.</p><p>We are actively building out microservices outside of our Ruby on Rails monolith. In order to extract microservices, we must have stability in our CI and continuous deployment (CD) to increment out of the monolith. We see more than 100 engineers contributing to our monolith weekly. Instability in the form of tests or architecture impacts those engineers on our get-out-of-the-monolith train!</p><h3>Q2: The Merge Queue</h3><p>Early in the year, an automated merge robot (‘borg’) was introduced to minimize implicit merge conflict failures for those &gt;100 engineers. Prior to that, we had gone through many experiments around optimal merge patterns on our high velocity monolith:</p><ul><li>Merge when your build is green — take to production when you want!</li><li>Merge when your build is green. Enforce a schedule to release at least once per day.</li><li>Weeks long merge freeze to get everything under control when things have gotten out of control.</li><li>Merge all the mergeable things in the evening. Work through conflicts the next morning to get a release out daily.</li><li>Create a team to support daily releases to own conflict resolution. Build up a team of individuals from each product team to own daily releases.</li><li>Merge via a queue, with high priority labels jumping to the front of the queue.</li></ul><p>Each of these strategies had various trade offs. A major trade off introduced in merging everything at once was CI build instability due to merge conflicts and flaky tests. Once we merged everything at the end of the day, the next morning was spent actively triage errors to identify if failures were new issues, implicit conflict failures, or test flakes.</p><p>The ‘borg’ was introduced at the end of March to eliminate merge queue issues and automate many of our challenges around conflicts. Every half hour, the borg runs CI with batches of mergeable PRs. The largest passing-build batch (by PR count) gets merged at the end of the build and the next half hour the process starts all over again. This happens all day, every day.</p><p>The introduction of ‘borg’ yielded a build success rate increase from ~30% to ~70%, effectively eliminating merge conflicts and the time spent by many triaging those conflicts. The borg has been happily chugging away since March with minimal tweaks!</p><h3>Q3: Dissatisfaction in Flakes and Isolated Retries</h3><p>Around the middle of the year, a productivity survey was sent out to engineers. Despite the major success with the introduction of the automated merge queue and the beloved ‘borg’, release stability and flaky tests remained a blocker per the survey results. The consumer experience required that you click a “Retry” button on your <a href="https://www.jenkins.io/">Jenkins</a> build whenever you had a test failure, not knowing whether it was a result of your change or a test flake. There was an active Slack channel where people shared experiences over flakes to help triage whether they had a flake or a real failure.</p><p>Based on the survey results, we immediately enacted a plan to retry failed tests in isolation after all 54k tests were run. Credit for that idea goes to an engineer who had brought it up a few times before — this time we finally listened to him. “Isolated retries” were incrementally added for two test frameworks:</p><ul><li><a href="http://rspec.info/">Rspec</a> tests, via the <a href="https://relishapp.com/rspec/rspec-core/docs/command-line/only-failures">–only-failures tag</a></li><li><a href="https://cucumber.io/">Cucumber</a> (feature) tests, via a homegrown retry feature — Cucumber does not have an equivalent to –only-failures</li></ul><p>We surfaced isolated retries in <a href="https://medium.com/upstart-tech/test-reporting-in-a-monolith-from-concept-to-delivery-7b9bf1b73c39">our test reporting tool</a> to monitor root causes. We saw an improvement in build success rate from ~70% to ~80% with isolated retries. Tests that passed in isolation experienced less data leaking (rspec) and less simultaneous resource demand (feature), leading to higher pass rates.</p><p>This 10% improvement translated to an improved user experience:</p><ul><li>less confusion and frustration</li><li>less context switching</li><li>less triaging</li><li>less manually clicking the Jenkins retry button expecting a different outcome</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*SJQplM68BvPc8Wsc" /><figcaption>Monolith build pass rate progress throughout the year.</figcaption></figure><h3>Q4: More Flaky Test Work</h3><p>Even with the iterative improvements throughout 2022, we wanted to surpass the 80% pass rate. We aimed to increase our deployment frequency while improving stability. We set a milestone to reach and maintain 90% build pass rates. By increasing the pass rate, we build trust for end users to ensure in CI that failures are due to their change and not flaky tests.</p><p>We aimed to minimize the manual intervention required to notify and support our build stability manifested in flaky tests. Our overall strategy included a few pieces:</p><ul><li>Reporting/Monitoring: Examination across many builds to understand impact of flakes to prioritize or triage.</li><li>Prevention: Linting, when we know what causes flakes.</li><li>Notification: Notify responsible and relevant teams when flakes are failing builds. Also leveraging DataDog for automated alerts around “build health”.</li><li>Education: Building out and evangelizing causes of flakes and strategies for flakes to support prevention when linting can not.</li></ul><p>Throughout Q4, we made progress on many of these components and maintained a 90% build pass rate.</p><h3>2023: What’s Next?</h3><p>Our build pass rate hovers around 90% heading into 2023. The reality is that flaky tests are regularly exposed and added to our test suite of ~54k tests. I believe that we should aim to maintain a 90%+ pass rate, add tools that minimize the introduction of flakes, and adjust processes to reduce flake blast radius.</p><p>As a retrospective, I examined the last three months of flakes to guide next steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/0*J3ITLjnIF9Y7uJ3b" /></figure><p>Our flakes from the last few months can be categorized (with nuance) into a few groups:</p><ul><li><strong>Time specific</strong>: This set of flakes comes from assertions that begin to fail at a specific date or time. The test may have passed at the time it was written, and usually fails at some point in the future or under a specific time condition (e.g. time of day / holiday).</li><li><strong>Mock leak</strong>: This set of flakes comes from mocks leaking between tests, e.g. allow_any_instance_of(*ClassName*) has shown to leak in our tests. We advise on using allow(object) instead of allow_any_instance_of mocking.</li><li><strong>Global def leak</strong>: This set of flakes comes from global definitions living inside the root level of feature/ or spec/ files. These methods can leak into other code with methods of the same name. We now have a blocking linter to prevent this flake cause.</li><li><strong>ENV leak</strong>: This set of flakes comes from cases where setting ENV values in one test leaks to another. We lint this now (blocking) — with the recommended approach of using ClimateControl mocking instead.</li><li><strong>Constant leak</strong>: Similar to the ENV leakages, this set of flakes comes from cases where we set a constant in one test and fail to reset it before other tests. Test ordering can then fail downstream tests that rely on this constant set upstream.</li><li><strong>Data Type comparison</strong>: This set of flakes comes from scenarios where we are asserting values from data types that are not equivalent, e.g. we have seen it most in comparing BigDecimals to Floats. We remediate these flakes by updating the data type to match.</li><li><strong>Capybara / browser issue</strong>: This category of flakes includes a variety of causes related to browser issues — including but not limited to timing issues, delay issues, or other third party reliability issues. Of our 54k tests, ~2% of them are feature tests, but we see a disproportionate amount of flakes here. We can mitigate these, but I advise shifting tests left to minimize unexpected browser behavior and third party impact.</li><li><strong>Bad data assertion</strong>: This set of flakes occurs when we are asserting around data that we expect to be in a specific state, and at test run time it’s not in that state. For example, we may assert a specific attribute on an object, but a previous test or randomized behavior has modified the data resulting in failure. This type of error can be mitigated by ensuring data is cleaned between tests.</li><li><strong>Data Creation</strong>: This is a broad category of flakes that fail due to data creation, either that the test relies on data to be created in a specific order, the data created is not random enough, the random data created breaks validation, or the assertion is not robust enough.</li></ul><h4>Where do we go from here?</h4><p>The above data reveals opportunities for further flaky test prevention, targeting high frequency issues. I also believe that many of the other flake types are difficult to lint, so my general guidelines for minimizing flaky tests are:</p><ul><li>Don’t write tests asserting around a specific time unless you can freeze in the past (Note: browsers don’t have awareness around server side time freezing. This means integration tests are unable to take advantage of this testing technique).</li><li>Try to shift tests left, separating the logic and data layers. Reduce dependencies on browsers, third parties and data, generally speaking.</li><li>Don’t leak your constants.</li><li>Enact stricter typing in your code and tests to minimize type flakes. We are accomplishing this through a combination of 1) implementing <a href="https://sorbet.org/">Sorbet</a> in our application code, and 2) raising awareness of this type of test failure.</li></ul><p>We’ve taken great strides in improving the development experience to iterate and release code in our monolith, and I hope it enables us to support our move to a microservice ecosystem throughout 2023!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ca92f9299ff" width="1" height="1" alt=""><hr><p><a href="https://medium.com/upstart-tech/flaky-tests-and-ci-build-stability-a-2022-summary-2ca92f9299ff">Flaky Tests and CI Build Stability: A 2022 Summary</a> was originally published in <a href="https://medium.com/upstart-tech">Upstart Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>