<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Martin Chaov on Medium]]></title>
        <description><![CDATA[Stories by Martin Chaov on Medium]]></description>
        <link>https://medium.com/@martin.chaov?source=rss-c923e3c536c1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3BoVD3-VhMRPGL8hAuLj9g.jpeg</url>
            <title>Stories by Martin Chaov on Medium</title>
            <link>https://medium.com/@martin.chaov?source=rss-c923e3c536c1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 02:33:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@martin.chaov/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Your WebSocket limit is not where you think [Part 2]]]></title>
            <link>https://medium.com/draftkings-engineering/your-websocket-limit-is-not-where-you-think-part-2-43774f630b67?source=rss-c923e3c536c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/43774f630b67</guid>
            <category><![CDATA[bff]]></category>
            <category><![CDATA[scale]]></category>
            <category><![CDATA[websocket]]></category>
            <category><![CDATA[load-balancing-websockets]]></category>
            <category><![CDATA[backend-engineering]]></category>
            <dc:creator><![CDATA[Martin Chaov]]></dc:creator>
            <pubDate>Fri, 23 Oct 2026 08:40:18 GMT</pubDate>
            <atom:updated>2026-10-23T08:41:45.357Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>The hidden fan-out behind BFFs and why adding pods may not be enough</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3VB7wT41Kmem8PkxGgK3LQ.png" /><figcaption>The hidden fan-out behind BFFs and why adding pods may not be enough</figcaption></figure><p>Well… <a href="https://medium.com/draftkings-engineering/your-websocket-limit-is-not-where-you-think-part-1-3b1cc1b17259">I promised more</a>… If you somehow stumbled on this one without going through the first part, consider checking it out <a href="https://medium.com/draftkings-engineering/your-websocket-limit-is-not-where-you-think-part-1-3b1cc1b17259">here</a>.</p><p>A Backend-for-Frontend, or BFF, is a backend service tailored to the needs of a specific frontend interface, such as a mobile app, desktop web, native app, partner API, internal dashboard, or another client-specific surface.</p><p>Because clients differ by use case and capabilities, they often need different payloads, contracts, SLAs, auth flows, caching rules, latency tolerances, etc. A BFF lets the frontend consume a clean API without knowing which backend services contribute to the final experience.</p><p>In normal request/response systems, this is often a reasonable trade-off. In WebSocket-heavy systems, the trade-off can become expensive very quickly.</p><p>The client sees this:</p><p>Mobile Client -&gt; WebSocket -&gt; Mobile BFF</p><p>The client application sees a single connection and a clean API. From the infrastructure side, there is a variety of outbound connections, streams, subscriptions, RPC calls, etc.</p><p>The platform may see this:</p><p>Mobile Client -&gt; WebSocket -&gt; Mobile BFF</p><p>But the BFF itself handles the fanout to other services, such as:</p><ul><li>Odds API</li><li>Wallet API</li><li>Promotions API</li><li>Notifications API</li><li>User State API</li><li>Risk API</li></ul><p>The BFFs may hide this from the client, but cannot hide it from the infrastructure provider.</p><h3>BFF as a connection multiplier</h3><p>For example:</p><p>1 client WebSocket + 1 odds subscription + 1 wallet subscription + 1 notification subscription + 1 user-state subscription + 1 risk subscription</p><p>= 1 inbound connection and 5 upstream dependencies</p><p>At a small scale, this may be acceptable.</p><p>On a large scale, it becomes a multiplication problem:</p><ul><li><strong>100,000 users × 5 backend dependencies = 500,000 upstream contexts</strong></li><li><strong>1,000,000 users × 5 backend dependencies = 5,000,000 upstream contexts</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KDQN1dXNgMVQbTf0rHNGFQ.png" /><figcaption>A context may be a TCP socket, a multiplexed stream, or a subscription. Count transport connections and logical state separately.</figcaption></figure><p>Here, an upstream context means a logical subscription maintained for a user. Subscriptions consume application state; their underlying TCP connections consume sockets and source ports, plus conntrack or NAT entries where those mechanisms apply. Pooling and multiplexing reduce transport connections. Sharing subscriptions can also reduce duplicated subscription state and backend work.</p><h3>Cardinality matters more than the diagram</h3><p>A normal architecture diagram:</p><p><strong>BFF -&gt; Backend Service</strong></p><p>Such diagrams rarely show which subscriptions are shared and which are duplicated across users or replicas. Multiply dimensions only when the implementation actually creates every combination:</p><p><strong>users × jurisdictions × market segments × backend services × replicas × retry paths</strong></p><p>Example, a system that is split into market segments and jurisdictions across a few back-end services where each user can hold a subscription to each market in each jurisdiction for monitoring purposes:</p><p><strong>100,000 concurrent users × 50 jurisdictions × 20 market segments × 5 backend services</strong></p><p>If every connected user could subscribe across every jurisdiction and every market segment, the logical subscription space becomes:</p><p><strong>100,000 × 50 × 20 = 100,000,000 logical user-market-jurisdiction combinations</strong></p><p>For that same demand, where subscriptions are shared, changes the number maintained against the backends:</p><ol><li>Shared across the cluster:</li><li>50 jurisdictions × 20 market segments = 1,000 jurisdiction–segment pairs;</li><li>1,000 pairs × 5 backends = <strong>5,000 upstream subscriptions</strong>.</li><li>Duplicated per pod: 1,000 pairs × 20 pods × 5 backends = <strong>100,000 upstream subscriptions.</strong></li><li>Duplicated per user: 100,000 users × 5 backends = <strong>500,000 upstream subscriptions</strong>.</li></ol><h3>How this maps to pods, IPs, and ports?</h3><p>More details on the mechanics of this in Part 1.</p><p>At the network level, every outbound TCP connection consumes a tuple:</p><p><strong>source IP:source port -&gt; destination IP: destination port / protocol</strong></p><p>For TCP, the tuple ceiling depends on the source IP and destination IP:port, subject to the allocation rules at that hop.</p><p>The common ephemeral-port range configuration on many Linux systems is:</p><p><strong>32768–60999 = 28,232 ports</strong></p><p>The usable range depends on the host configuration, network namespace, NAT layer, cloud provider, and egress topology. Cloud NAT gateways and managed load balancers often have provider-specific allocation rules.</p><p>The tuple model is useful for a rough estimate. Example:</p><p><strong>users = 100,000</strong></p><p><strong>pods = 20</strong></p><p><strong>fanout_ratio = 5</strong></p><p>Per pod:</p><p><strong>users_per_pod = 100,000 / 20 = 5,000</strong></p><p><strong>outbound_connections = 5,000 × 5 = 25,000</strong></p><p>The above means that each pod may create 25,000 backend-facing connection contexts. But this does not automatically mean 25,000 connections to the same destination tuple. If the five dependencies are five different backend destination IP pairs, the pressure may be closer to:</p><p><strong>5,000 connections per backend destination</strong></p><p>If those dependencies collapse through one backend gateway, one service VIP, one NAT path, one egress gateway, or one destination IP, the pressure can approach:</p><p><strong>25,000 connections competing for the same tuple space</strong></p><p>This is where many production surprises happen.</p><p>The diagram shows:</p><p><strong>BFF -&gt; Backend Services</strong></p><p>The network sees:</p><p><strong>one pod IP -&gt; one backend IP:port</strong></p><p><strong>one node IP -&gt; one backend IP:port</strong></p><p><strong>one NAT gateway IP -&gt; one backend IP:port</strong></p><p><strong>one egress gateway IP -&gt; one backend IP:port</strong></p><p>The source IP may be the pod IP, node IP, NAT gateway IP, or egress gateway IP, depending on the CNI, masquerade/SNAT rules, Service implementation, and egress path.</p><p>Now add a sprinkle of production realities:</p><ul><li>jurisdictional isolation</li><li>data locality rules</li><li>backend failover</li><li>retries</li><li>uneven load distribution</li><li>connection churn</li><li>deployment overlap</li><li>blue/green environments</li><li>canary releases</li><li>load-balancer stickiness</li><li>node-level conntrack pressure</li><li>noisy neighbours</li></ul><p>…the estimated safe capacity can disappear very quickly.</p><p>Connection pooling, multiplexing, shared subscriptions and backend endpoint distribution can reduce pressure. Adding pods helps when they add capacity to the constrained resource. It may not relieve a shared NAT or node limit, and it can increase backend load when subscriptions are duplicated per pod. Spreading connection-heavy pods across nodes can reduce shared-node contention; affinity rules alone do not provide resource isolation.</p><p>The BFF capacity question includes:</p><ul><li>How many source IPs do we have?</li><li>How many NAT IPs do we have?</li><li>How many backend destination IP:port pairs do we use?</li><li>Are connections direct, proxied, or translated?</li><li>Are upstream connections pooled, reused, multiplexed, or per-user?</li><li>Do backend subscriptions happen once per cluster, once per pod, or once per user?</li></ul><p>Without those answers, autoscaling is operating on an incomplete model.</p><h3>Why HPA does not automatically fix WebSocket BFFs</h3><p>For WebSockets, adding pods mostly helps future connections. Existing connections usually stay attached to the pod that accepted them. If one BFF pod already owns 100,000 WebSocket connections, adding ten more pods does not automatically move those existing sockets. Existing connections remain on the old pod until they close or the application arranges reconnection. Draining can stop new admissions while established connections continue; it does not necessarily close them immediately. There is a way to make the load-balancing process predictable and manageable. To load-balance long-lived connections, they should be cut off periodically and force clients to reconnect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lfto2zDwclcslnhhv4hJyw.png" /><figcaption>New connections may reach either pod. Use jitter and backoff for controlled reconnects; scale on the resource under pressure.</figcaption></figure><blockquote><strong>In fact, WebSockets should not be used without planned periodic interruptions at the gateway level!</strong></blockquote><p>Monitoring only CPU can miss connection or memory pressure in mostly idle workloads. Keep CPU as a scaling signal when message processing is the constrained resource.</p><p>A BFF can own many idle connections that consume memory, file descriptors, conntrack entries, NAT tuples, and backend subscriptions while using little CPU. HPA may see low CPU and decide not to scale, even though the system is close to a connection limit.</p><p>HPA can use CPU, memory, custom metrics, and external metrics. But it is still a replica-count mechanism. It cannot redistribute existing WebSockets on its own. Scaling on CPU while the system is running out of file descriptors, NAT ports, or backend subscription capacity does not fix the actual limit.</p><p>Choose scaling signals that correspond to capacity another replica can add. Per-pod connections, process FD utilization, memory usage and CPU can be useful, depending on the workload. Track message rate, event-loop lag, reconnect rate and per-user upstream demand to explain pressure and plan headroom. Treat node conntrack usage, NAT allocation and backend errors as evidence of shared constraints: increasing BFF replicas may not relieve them. KEDA or custom metrics can expose signals to an autoscaler; they do not establish that adding replicas fixes the bottleneck.</p><h3>Fine-tuning</h3><p>There are two classes of fixes: fine-tuning and architecture.</p><p>Tuning increases the pressure that the current architecture can withstand. It does not change the shape of the pressure.</p><p>Common tuning fixes include:</p><ul><li>increase process file descriptor limits</li><li>increase ip_local_port_range</li><li>increase conntrack limits</li><li>increase ingress worker connection limits</li><li>increase pod replicas</li><li>increase ingress replicas</li><li>spread pods across more nodes</li><li>use dedicated node pools</li><li>avoid unnecessary SNAT</li><li>add NAT IPs or source IP capacity</li><li>increase backend endpoint count</li><li>increase load-balancer idle timeouts where appropriate</li><li>tune WebSocket heartbeat behavior</li><li>reduce reconnect storms</li></ul><p>For example, increasing file descriptor limits may be appropriate if the process is artificially capped below the expected number of sockets.</p><p>Increasing the ephemeral port range may be useful if outbound connections are legitimate and the host range is too narrow.</p><p>Increasing conntrack limits may be necessary if nodes are expected to track a large number of flows.</p><p>Adding NAT IPs may help when many connections are translated through too few source IPs.</p><p>But tuning has a failure mode: it is only symptomatic while not addressing the architectural root cause.</p><ul><li>If the BFF opens 5 upstream connections per user, raising file descriptor limits only lets the bad model run longer.</li><li>If all traffic is SNATed through too few NAT IPs, adding pods may not help.</li><li>If each BFF pod duplicates the same upstream subscriptions, increasing pod count may multiply backend load.</li><li>If WebSockets are unevenly distributed, adding new pods may not relieve old hot pods.</li></ul><p>The more durable fix is usually architectural: reduce the amount of backend work created per connected user.</p><p>Instead of this:</p><p><strong>10,000 users × 5 backend connections = 50,000 backend connections</strong></p><p>Prefer this:</p><ul><li>10,000 users</li><li>shared backend connection pools</li><li>shared subscriptions</li><li>internal fan-out</li></ul><p>The goal is to change the multiplication factor.</p><p>Better patterns include:</p><ul><li>connection pooling</li><li>shared upstream subscriptions</li><li>subscription multiplexing</li><li>HTTP/2 or gRPC streams where appropriate</li><li>pub/sub broker</li><li>stream-processing layer</li><li>application-level fan-out</li><li>user partitioning or sharding</li><li>backend endpoint distribution</li><li>dedicated WebSocket gateway layer</li><li>separate BFF edge from fan-out layer</li></ul><p>A weak model looks like this:</p><p>User WebSocket -&gt; BFF</p><ul><li>per-user backend subscription</li><li>backend service</li></ul><p>A better model looks like this:</p><p>User WebSocket -&gt; BFF</p><ul><li>shared subscription pool</li><li>internal fan-out</li></ul><p>A more scalable model separates edge connection ownership from backend fan-out:</p><p>User WebSocket -&gt; WebSocket gateway</p><ul><li>session/routing layer</li><li>shared stream processor or pub/sub layer</li><li>backend services</li></ul><p>Connection pooling does not solve every problem. It helps when backend work can reuse connections. It does not remove the need for per-user authorization, per-user risk state, wallet state, or session state.</p><p>Sharing upstream work does not remove per-client delivery costs. Bound each client’s outgoing queue and define what happens when a consumer falls behind: apply backpressure, coalesce updates, drop messages only where the delivery contract permits it, or disconnect and recover. A separate broker or fan-out layer also adds operational cost and failure modes.</p><p>The goal is not to pretend users have no individual state.</p><p>The goal is to avoid creating multiple long-lived TCP connections for each connected user, duplicating backend subscriptions, or performing repeated upstream work when that work can be shared.</p><p>Large subscriber systems generally rely on architecture, routing, replication, and pub/sub design. They are not just one process accepting sockets.</p><h3>Capacity model for a BFF</h3><p>For a BFF that owns WebSockets, capacity planning should include the BFF explicitly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vnk5K3A6m4wT-6c_VWtAjQ.png" /><figcaption>Evaluate tuple and conntrack budgets at the actual hop or shared scope, using measured per-user consumption.</figcaption></figure><p>The BFF-specific values matter:</p><p>fds_per_user = inbound WebSocket + upstream sockets held for that user + broker/cache/database/client handles + runtime overhead</p><p>upstream_connections_per_user = number of backend destinations × connections or subscriptions per destination</p><p>fanout_ratio = backend-facing streams, calls, or contexts / connected users</p><p>The most important value is the fan-out ratio.</p><ul><li>If the fan-out ratio is close to zero because upstream subscriptions are shared, the BFF can scale far beyond the per-user upstream model.</li><li>If the fan-out ratio is five or ten, the BFF may hit file descriptors, ephemeral ports, conntrack, memory, backend limits, or NAT limits long before the inbound WebSocket count looks impressive.</li></ul><p>Use the TCP connection ratio to describe transport demand, the subscription ratio to describe backend state, and call or message rates to describe processing demand. These aggregate ratios are not automatically incremental per-user costs: measure shared-pool growth before extrapolating. A ratio of five subscriptions per user does not imply five TCP connections per user.</p><p>Before scaling, answer:</p><ul><li>How many active WebSockets exist per pod?</li><li>How much memory is used per active connection?</li><li>How many file descriptors are used per user?</li><li>How many upstream connections or contexts exist per user?</li><li>Are subscriptions shared or per-user?</li><li>Are subscriptions duplicated per pod?</li><li>Which source IPs are used for each outbound hop?</li><li>Which destination IP:port pairs receive the traffic?</li><li>Is traffic direct, proxied, SNATed, or routed through an egress gateway?</li><li>What are the NAT and conntrack limits?</li><li>What happens during scale-out?</li><li>What happens during reconnect storms?</li><li>What happens during deployment overlap?</li></ul><p>The capacity question is:</p><p><strong>How much backend work does each connected user create, and where is that work multiplied?</strong></p><h3>Summary</h3><p>BFFs are useful because they hide backend complexity from clients.</p><p>In WebSocket-heavy systems, that same abstraction can hide fan-out, long-lived backend state, duplicated subscriptions, and tuple pressure.</p><p>The BFF may look simple from the frontend:</p><p><strong>Client -&gt; WebSocket -&gt; BFF</strong></p><p>But the production system may behave like this:</p><p><strong>Client -&gt; WebSocket -&gt; BFF -&gt; many backend contexts</strong></p><p>Autoscaling helps only when it scales the constrained resource. Adding pods does not automatically redistribute existing WebSockets, reduce per-user fan-out, fix NAT exhaustion, or remove backend subscription pressure.</p><p>Tuning can increase a system’s capacity to a point. Architecture changes the multiplication factor.</p><p>As a rule of thumb:</p><ul><li>Count users.</li><li>Count sockets.</li><li>Count upstream contexts.</li><li>Count tuple pressure.</li><li>Count duplicated subscriptions.</li><li>Measure the narrowest hop.</li><li>Then scale the actual bottleneck.</li></ul><h3>References</h3><p>[1] Microsoft Azure Architecture Center. Backends for Frontends pattern. Microsoft Learn.</p><p><a href="https://learn.microsoft.com/en-us/azure/architecture/patterns/backends-for-frontends">https://learn.microsoft.com/en-us/azure/architecture/patterns/backends-for-frontends</a></p><p>[2] Fette, I., Melnikov, A. (2011). RFC 6455: The WebSocket Protocol. RFC Editor.</p><p><a href="https://www.rfc-editor.org/rfc/rfc6455">https://www.rfc-editor.org/rfc/rfc6455</a></p><p>[3] McManus, P. (2018). RFC 8441: Bootstrapping WebSockets with HTTP/2. RFC Editor.</p><p><a href="https://www.rfc-editor.org/rfc/rfc8441">https://www.rfc-editor.org/rfc/rfc8441</a></p><p>[4] Larsen, M., Gont, F. (2011). RFC 6056: Recommendations for Transport-Protocol Port Randomization. RFC Editor.</p><p><a href="https://www.rfc-editor.org/rfc/rfc6056">https://www.rfc-editor.org/rfc/rfc6056</a></p><p>[5] Linux Kernel Documentation. IP Sysctl.</p><p><a href="https://docs.kernel.org/networking/ip-sysctl.html">https://docs.kernel.org/networking/ip-sysctl.html</a></p><p>[6] Linux Kernel Documentation. Netfilter conntrack sysctl variables.</p><p><a href="https://docs.kernel.org/networking/nf_conntrack-sysctl.html">https://docs.kernel.org/networking/nf_conntrack-sysctl.html</a></p><p>[7] Kubernetes Documentation. Horizontal Pod Autoscaling.</p><p><a href="https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/">https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/</a></p><p>[8] Kubernetes Documentation. Horizontal Pod Autoscaling walkthrough and custom metrics.</p><p><a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/">https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/</a></p><p>[9] Kubernetes Documentation. Pods.</p><p><a href="https://kubernetes.io/docs/concepts/workloads/pods/">https://kubernetes.io/docs/concepts/workloads/pods/</a></p><p>[10] Kubernetes Documentation. Virtual IPs and Service Proxies.</p><p><a href="https://kubernetes.io/docs/reference/networking/virtual-ips/">https://kubernetes.io/docs/reference/networking/virtual-ips/</a></p><p>[11] Google Cloud Documentation. Cloud NAT: IP addresses and ports.</p><p><a href="https://docs.cloud.google.com/nat/docs/ports-and-addresses">https://docs.cloud.google.com/nat/docs/ports-and-addresses</a></p><p>[12] Amazon Web Services Documentation. Work with NAT gateways.</p><p><a href="https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-working-with.html">https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-working-with.html</a></p><p>[13] Microsoft Learn. Troubleshoot SNAT port exhaustion in Azure App Service.</p><p><a href="https://learn.microsoft.com/en-us/azure/app-service/troubleshoot-intermittent-outbound-connection-errors">https://learn.microsoft.com/en-us/azure/app-service/troubleshoot-intermittent-outbound-connection-errors</a></p><p>[14] Microsoft Learn. Source Network Address Translation for outbound connections.</p><p><a href="https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-outbound-connections">https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-outbound-connections</a></p><p>[15] KEDA Documentation. Scaling Deployments, StatefulSets &amp; Custom Resources.</p><p><a href="https://keda.sh/docs/latest/concepts/scaling-deployments/">https://keda.sh/docs/latest/concepts/scaling-deployments/</a></p><p>[16] NGINX Documentation. WebSocket proxying.</p><p><a href="https://nginx.org/en/docs/http/websocket.html">https://nginx.org/en/docs/http/websocket.html</a></p><p>[17] Envoy Documentation. Timeouts FAQ.</p><p><a href="https://www.envoyproxy.io/docs/envoy/latest/faq/configuration/timeouts">https://www.envoyproxy.io/docs/envoy/latest/faq/configuration/timeouts</a></p><p>[18] gRPC Documentation. Core concepts, architecture and lifecycle.</p><p><a href="https://grpc.io/docs/what-is-grpc/core-concepts/">https://grpc.io/docs/what-is-grpc/core-concepts/</a></p><p>[19] Paracha, M., Sadiq, M., Liang, J., Liang, J., Durad, M., &amp; Sheeraz, M. (2023). Multi-Layered Filtration Framework for Efficient Detection of Network Attacks Using Machine Learning. Sensors, 23(13), 5829.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43774f630b67" width="1" height="1" alt=""><hr><p><a href="https://medium.com/draftkings-engineering/your-websocket-limit-is-not-where-you-think-part-2-43774f630b67">Your WebSocket limit is not where you think [Part 2]</a> was originally published in <a href="https://medium.com/draftkings-engineering">DraftKings Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your WebSocket limit is not where you think [Part 1]]]></title>
            <link>https://medium.com/draftkings-engineering/your-websocket-limit-is-not-where-you-think-part-1-3b1cc1b17259?source=rss-c923e3c536c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b1cc1b17259</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[websocket]]></category>
            <category><![CDATA[backend-engineering]]></category>
            <category><![CDATA[load-balancing-websockets]]></category>
            <category><![CDATA[scale]]></category>
            <dc:creator><![CDATA[Martin Chaov]]></dc:creator>
            <pubDate>Fri, 23 Oct 2026 08:40:10 GMT</pubDate>
            <atom:updated>2026-10-23T08:42:36.558Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>The hidden infrastructure limits behind WebSockets at scale</blockquote><p>After writing about <a href="https://medium.com/draftkings-engineering/lessons-learned-websocketapi-at-scale-604617a54cdb">lessons learned from WebSockets at scale</a>, I wanted to focus on the often underestimated part: load balancing and scaling WebSocket systems in the cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WKd3h87h0WjdK9OkxiHT8w.png" /><figcaption>The hidden infrastructure limits behind WebSockets at scale</figcaption></figure><p>A typical WebSocket setup is often shown as a direct client-to-server connection, but real systems are more complex. To reduce the number of connections opened by each client, multiple APIs and services can be exposed through an aggregator service or a back-end for frontend (BFF). This service manages client connections while upstream services handle business logic. As concurrency grows, file descriptors, source ports or shared connection-tracking capacity can become bottlenecks before CPU or memory. Production deployments introduce additional complexity, including multiple data centers, service discovery, clustering, CDNs, and WAFs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CwbHrAFL2ETfq7biFSVHFg.png" /></figure><p>If the above is not designed and set up correctly, there are myriad pitfalls to fall into. In short, this article shows how to estimate connection capacity across a WebSocket server and the network paths it depends on*. Bear with me; the answer involves many intertwined variables.</p><p><em>* Illustrative assumptions, not universal defaults or measured operating limits. Unless stated otherwise, each user has one WebSocket over a dedicated TCP connection.</em></p><h3>The theory</h3><p>A single Linux server can hold a very large number of WebSocket connections, in theory. This is possible when the setup is a tuned process, one listening socket, controlled clients, and little or no upstream fan-out. However, operating a production Kubernetes system behind load balancers, ingress proxies, Kubernetes Services, NAT, connection tracking, shared worker nodes, and other upstream/downstream services is a completely different picture.</p><p>Instead of asking: <em>“Can this process hold a million sockets?”</em></p><p>The better question is: <em>“How many connection tuples and shared kernel resources can every </em><strong><em>network hop</em></strong><em> sustain?”</em></p><p>The maximum capacity is the minimum of several limits: file descriptors, socket memory, ephemeral port range, NAT port allocation, conntrack table capacity, load-balancers behavior, backend endpoints, per-user upstream fan-out, and application memory model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Ud3SvQycBY3XTWx2wqpyQ.png" /><figcaption>Service routing may select pod endpoints without terminating TCP. Measure the constrained resource at each actual network hop.</figcaption></figure><h3>Starting simple: one client -&gt; one server</h3><p>In the classic HTTP/1.1 case, WebSocket starts as an HTTP request and uses the HTTP Upgrade mechanism to switch to the WebSocket protocol. After the upgrade, the client and server exchange WebSocket frames over a long-lived connection.</p><p><em>WebSocket can also be bootstrapped over HTTP/2 using the Extended CONNECT method, so the one-WebSocket-to-one-TCP-connection model is common but not universal.</em></p><p>A TCP connection is identified by local and remote socket information, commonly modeled as source IP, source port, destination IP, destination port, and protocol.</p><p>For example:</p><ul><li><strong>client_a:50123 -&gt; server:443</strong></li><li><strong>client_b:50123 -&gt; server:443</strong></li><li><strong>client_c:50123 -&gt; server:443</strong></li></ul><p>All three clients use the same destination port 443. The connections are still distinct because the source IP addresses differ. Therefore, a server listening on port 443 is not limited to about 65,000 inbound clients merely because TCP ports are 16-bit values.</p><p>A limit can arise when TCP connections concentrate from one source IP toward one destination IP:port, or when sockets, descriptors, buffers or tracked flows exhaust a resource budget.</p><p><em>Idle timeout configuration also falls under capacity planning. Set the heartbeat interval below the relevant idle timeout and verify which traffic resets that timer. TCP keepalive packets may not reset an application-layer idle timeout; WebSocket ping/pong or application messages may be required. When many clients reconnect at once, the system transitions from a steady-state connection problem to a connection storm.</em></p><h3>A bit more Linux stuff</h3><p>A single Linux process can hold many TCP connections, but each connection consumes operating-system and application resources. The first resource is a file descriptor. In Linux, <strong>socket()</strong> returns a file descriptor, and the enforced per-process file-descriptor ceiling is the soft <strong>RLIMIT_NOFILE</strong> value, which host or container configuration can set.</p><p>A way to look at this for a network server:</p><p><strong>1 TCP socket ≈ 1 file descriptor</strong></p><p><strong>1 WebSocket over one TCP connection ≈ 1 file descriptor</strong></p><p>If the server holds one inbound WebSocket per user, then 100,000 connected users require roughly 100,000 file descriptors. If each connected user also causes the server to open two outbound backend connections, the process needs about three sockets per user:</p><p><strong>1 inbound WebSocket + 2 outbound backend connections = 3 sockets per user</strong></p><p>Ignoring other descriptors and operating headroom, a process limit of 65,535 gives a theoretical ceiling of <strong>65,535 / 3 = 21,845 users</strong>.</p><p>For planning, use: <strong>user capacity = floor((FD limit − fixed FD usage − reserve) / FDs per user)</strong>.</p><p>That number is the result of a specific file-descriptor limit and a specific per-user socket model. If the process limit is raised to <strong>1,048,576</strong> and the application still uses three sockets per user, the file-descriptor side rises to about <strong>349,525</strong> users.</p><p>Sockets also consume memory. TCP has receive and send buffers, and Linux exposes TCP buffer defaults and autotuning parameters through tcp_rmem, tcp_wmem, rmem_max, and wmem_max. A WebSocket stack can also hold TLS state, protocol state, compression state, authentication metadata, application session objects, runtime allocations, and observability state.</p><p>A rough estimate such as this is useful only as a placeholder:</p><ul><li><strong>100,000 connections × 50 KB ≈ 5 GB</strong></li><li><strong>1,000,000 connections × 50 KB ≈ 50 GB</strong></li></ul><p>The actual per-connection footprint must be measured from the target runtime and workload; the numbers above are for illustrative purposes. Additionally, one should consider the CPUs distribution. 1 CPU with 50 GB RAM is hardly an efficient configuration for this use case.</p><p><em>Note: backlog settings affect connection establishment, not the number of established WebSockets the server can hold. They matter during reconnect storms, when many clients attempt to reconnect in a short interval. This is a different failure mode and is not discussed in this article.</em></p><h3>Ephemeral ports and outbound fan-out</h3><p>Ephemeral ports are temporary local ports used for outbound connections. When a process opens a TCP connection to another service and does not bind a specific local port, the operating system chooses a local source port from the configured ephemeral-port range.</p><p>Example:</p><ul><li><strong>server_ip:49152 -&gt; backend_ip:443</strong></li><li><strong>server_ip:49153 -&gt; backend_ip:443</strong></li><li><strong>server_ip:49154 -&gt; backend_ip:443</strong></li></ul><p>For simultaneous TCP connections from one source IP to one destination IP:port, each connection needs a distinct source port. Read <strong>net.ipv4.ip_local_port_range</strong> in the namespace opening the connections, and account for reserved and otherwise unavailable ports before estimating usable capacity.</p><p>If the range is: <strong>32768–60999</strong></p><p>Then it contains: <strong>60999–32768 + 1 = 28,232 ports</strong></p><p>The Linux default range is 32768–60999, but deployed settings vary. TCP source ports can be reused across different destination IP:port pairs when the complete connection tuple remains unique. Explicit bind-before-connect behavior and NAT allocation rules can impose additional constraints.</p><p>This does not normally cap inbound WebSockets from many unrelated client IP addresses. It can constrain outbound connections concentrated on a small number of source-IP/destination-IP:port combinations.</p><p>For example, many pods may connect to an external service through one node IP, NAT gateway IP, or egress gateway IP:</p><ul><li><strong>pod_A:40001 -&gt; external_service:443</strong></li><li><strong>pod_B:40002 -&gt; external_service:443</strong></li><li><strong>pod_C:40003 -&gt; external_service:443</strong></li></ul><p>After SNAT, the external service may see:</p><ul><li><strong>nat_gateway_ip:50001 -&gt; external_service:443</strong></li><li><strong>nat_gateway_ip:50002 -&gt; external_service:443</strong></li><li><strong>nat_gateway_ip:50003 -&gt; external_service:443</strong></li></ul><p>From the external service’s point of view, the pod IPs disappeared. Many internal sources collapsed behind fewer external source IPs. This is operationally useful, but it creates source-port pressure because the NAT layer must keep translated connection tuples distinct for each destination IP:port and protocol; its allocation policy determines how source ports can be reused across destinations.</p><p>Many pods can share the same outbound bottleneck:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eKy85YfGLZNQDej0g8L8EQ.png" /><figcaption>NAT allocation rules are provider-specific. Adding pods does not necessarily add public source IPs or NAT port capacity.</figcaption></figure><p>This is the part that many application engineers miss while counting users, pods, or replicas, but the network device may be counting translated connection tuples. Once thousands of internal connections are translated through a single source IP to the same destination IP:port and protocol, the NAT layer can run out of usable source ports, even though the application still has CPU and memory available.</p><p>Example cloud behaviors for popular providers:</p><ul><li><strong>Google Cloud NAT </strong>documents that each NAT IP address offers 64,512 TCP source ports and that, for private GKE clusters, NAT source IP address and source-port tuples are assigned to the node VM and shared by all pods on that node. It also documents that closed connections can make the NAT tuple unusable for 120 seconds by default.</li><li><strong>AWS documents NAT</strong> gateway scaling in terms of the number of simultaneous connections per destination and recommends adding secondary IPv4 addresses when more connections are required.</li><li><strong>Azure documents SNAT </strong>port exhaustion as a cause of intermittent outbound connection failures and recommends connection pooling and reuse.</li></ul><p>To verify if an issue is related to port exhaustion, there are a few useful questions to ask:</p><ul><li>How many source IPs?</li><li>How many destination IP:port pairs?</li><li>Is there NAT or SNAT/DNAT?</li><li>Is traffic direct, proxied, or translated?</li><li>Are connections reused, pooled, or multiplexed?</li></ul><h3>But wait, there’s more… Conntrack!</h3><p>Conntrack means connection tracking. Linux connection tracking records network flow state so NAT, firewall rules, and Service routing can treat packets from the same flow consistently.</p><p>Kubernetes commonly uses conntrack in kube-proxy and iptables/nftables-based Service implementations, although this depends on the datapath.</p><p>Conntrack pressure matters because it is usually node-scoped in common Linux datapaths. Multiple pods on the same worker node can contribute to the same conntrack table. If one workload creates many connections, it can affect other workloads on the node.</p><p>The failure mode can look unrelated to the original service:</p><ul><li>new connections fail</li><li>DNS becomes unreliable</li><li>some services time out</li><li>packet drops increase</li><li>only some nodes are affected</li></ul><p>If <strong>nf_conntrack_count</strong> approaches <strong>nf_conntrack_max</strong>, the node is near its tracking limit. That is a shared node-level networking constraint.</p><h3>The production path never looks like the examples</h3><p>The simple view is:</p><p><strong>Client -&gt; WebSocket -&gt; Server</strong></p><p>A real production path may look more like this:</p><p><strong>Clients</strong></p><p><strong>-&gt; Cloud Load Balancer</strong></p><p><strong>-&gt; Ingress or Gateway</strong></p><p><strong>-&gt; Kubernetes Service</strong></p><p><strong>-&gt; Server Pod</strong></p><p><strong>— -&gt; Backend APIs</strong></p><p><strong>— -&gt; Backend streams or subscription services</strong></p><p><strong>— -&gt; Storages, brokers, or caches</strong></p><p>This matters because every NAT, proxy, or translation layer can alter the distribution of tuples. At the application level, the system may look distributed across many services. At the network level, it may be concentrated through a small number of source IPs, pod endpoints, backend endpoints, NAT IPs, or load-balancer targets.</p><p>The first hop is usually:</p><p><strong>Client -&gt; Gateway / Load Balancer</strong></p><p>Since clients normally come from many source IP addresses and networks, this is very rarely the first ephemeral-port bottleneck. There are limits, though: maximum concurrent connections, TLS termination capacity, idle timeout, WebSocket support, per-target behavior, and cloud-provider quotas.</p><p>The next hop is:</p><p><strong>Gateway / Ingress -&gt; Server Pod</strong></p><p>A hidden capacity exhaustion can occur when many long-lived connections originate from a small number of ingress, gateway, node, or proxy source IPs to a small number of pod IP endpoints. In that case, the relevant capacity is not the total number of external client IPs, but the tuple space between the proxy and pod layers.</p><p>For an ingress opening TCP connections directly to distinct pod IP:port endpoints:</p><p><strong>tuple ceiling ≈ ingress_source_ips × server_pod_endpoints × usable_source_ports</strong>.</p><p>If the ingress connects to a single Service VIP:port instead, inspect the socket-facing destination and Service datapath before multiplying by backend pod count.</p><p>If one ingress source IP connects to one server pod IP, and the usable source-port range is <strong>28,232</strong> ports, the theoretical tuple count for that specific pair is about <strong>28,232</strong>. If the same traffic is evenly spread across two server pods, the example becomes about <strong>56,464</strong> connections. With two ingress source IPs and four pod endpoints, it becomes about <strong>225,856</strong> connections.</p><p><em>Those examples assume even load distribution, no protocol multiplexing, no additional proxy constraints, and no other lower limit. Autoscaling does not automatically rebalance old WebSockets. New pods receive new connections, but established connections stay where they are unless the application, proxy, or load-balancer layer deliberately drains or reconnects them.</em></p><p>The third hop is often the dangerous one:</p><p><strong>Server Pod -&gt; Upstream Services</strong></p><p><em>The source IP may be the pod IP, node IP, NAT gateway IP, or egress gateway IP, depending on the CNI, masquerade/SNAT rules, and egress path.</em></p><p>If each user has two upstream subscriptions and each uses a dedicated TCP connection, the model becomes:</p><p><strong>server_pod_ip:ephemeral_port -&gt; backend_A_ip:443</strong></p><p><strong>server_pod_ip:ephemeral_port -&gt; backend_B_ip:443</strong></p><p>For TCP connections from the same source IP, different destination IP:port pairs have distinct tuple spaces, subject to port-allocation rules. If both connections use the same destination IP:port, they compete for that pair’s source-port budget.</p><p>Example:</p><ul><li><strong>usable source ports = 28,232</strong></li><li><strong>connections per user to the same backend IP:port = 2</strong></li><li><strong>28,232 / 2 = 14,116 users</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WP3I-03njCgWndc1RgND2Q.png" /><figcaption>Illustrative ceilings for one source IP per hop. Real capacity also includes overhead, memory, conntrack, NAT policies, and headroom.</figcaption></figure><p>Under these assumptions, outbound tuple capacity limits this source IP to about 14,116 users, even if the server has capacity for more inbound WebSockets.</p><h3>A bit of Kubernetes stuff…</h3><p>A Kubernetes pod is a shared execution context using Linux namespaces and cgroups. Containers in a pod share a network namespace, including IP address and port space, while the TCP/IP implementation and many kernel resources still come from the node kernel.</p><p>Important consequences:</p><ul><li><strong>RLIMIT_NOFILE</strong> applies per process; container configuration can set the limits inherited by its processes</li><li>TCP implementation comes from the node kernel</li><li>conntrack is shared in common node-level datapaths</li><li>network bandwidth is shared at the node</li><li>CPU is scheduled and may be throttled by cgroups</li><li>memory is constrained by container and pod limits</li></ul><p>A noisy neighbor can therefore affect a connection-heavy workload. Another pod can contribute to conntrack pressure, network bandwidth saturation, CPU softirq pressure, packet drops, or NAT pressure on the same node, depending on the CNI and Service implementation, as well as the cloud network path.</p><p>Kubernetes scheduling does not automatically know that a workload is connection-heavy.</p><ul><li>Resource requests and placement constraints control where pods run and how they share nodes.</li><li>Autoscaling metrics adjust replica counts;</li></ul><p>custom metrics do not automatically make the scheduler account for connection pressure.</p><p>Kubernetes Services add another abstraction between clients and pods. The implementation can use kube-proxy in different modes or an alternative datapath, and that choice changes packet handling, NAT behavior, conntrack dependency, and source-IP visibility.</p><p>A request may pass through some of these layers:</p><ul><li>cloud load balancer</li><li>ingress controller or gateway</li><li>Service virtual IP</li><li>kube-proxy iptables or nftables rules</li><li>alternative eBPF datapath</li><li>pod IP</li><li>node SNAT or masquerade</li><li>cloud NAT gateway</li><li>firewall or egress gateway</li></ul><p>This is not a universal path. The actual path depends on cloud provider, Service type, externalTrafficPolicy, ingress controller, CNI, kube-proxy mode, egress gateway, and whether source IP is preserved or translated.</p><h3>Conservative capacity model</h3><p>Estimate the tuple ceiling of each hop, then compare it with the other resource ceilings for the same user population.</p><p>If every source can use every counted destination and each pair has the same port budget: <strong>hop_tuple_ceiling ≈ source_ips × destination_IP_port_pairs × usable_source_ports_per_pair.</strong></p><p>With traffic distributed in proportion to those budgets:</p><p><strong>hop_user_ceiling ≈ hop_tuple_ceiling / TCP_connections_per_user_on_that_hop.</strong></p><p>Otherwise, calculate each constrained pair’s user ceiling separately for the workload’s actual connection demands and take the limiting result.</p><p>Example (these numbers vary wildly based on configuration and design):</p><ul><li>ephemeral range: 32768–60999 = 28,232 ports</li><li>1 user = 1 LB-&gt;Server connection + 2 Server-&gt;Backend connections</li><li>3 sockets per user in the server process</li><li>Backend A and Backend B are different destinations</li><li>process file descriptor limit = 65,535</li></ul><p>Under those assumptions:</p><ul><li>LB -&gt; Server: about 28k users per ingress source IP to one pod IP:port</li><li>Server -&gt; Backend A: about 28k users per server source IP to one backend IP:port</li><li>Server -&gt; Backend B: about 28k users per server source IP to one backend IP:port</li><li>FD limit: 65,535 / 3 = about 21.8k users</li></ul><p>Among the limits modeled here, the theoretical ceiling is about 21.8k users before fixed descriptor usage and operating headroom. If both upstream connections use the same backend IP:port, the tuple ceiling becomes <strong>28,232 / 2 = about 14.1k users</strong>.</p><p><em>A safe operating target must also account for the other resources and failure conditions.</em></p><p>What one should take from that is that the real limit can be much lower than anticipated when the production topology concentrates connections through narrow source-IP, destination-IP, file-descriptor, memory, NAT, or conntrack resources.</p><p>A checklist that can help in quantifying the state of a system:</p><ul><li>Count inbound sockets.</li><li>Count outbound sockets per user.</li><li>Identify source IPs per hop.</li><li>Identify destination IP:port pairs per hop.</li><li>Check FD limit, memory, conntrack, NAT/SNAT, LB idle timeout.</li><li>Load test reconnect storms, not only steady state.</li></ul><h3>I want more!</h3><p><a href="https://medium.com/draftkings-engineering/your-websocket-limit-is-not-where-you-think-part-2-43774f630b67">Following up next with an article about BFFs, tuning, and fan-out!</a></p><h3>References</h3><p>[1] Eddy, W. (2022). RFC 9293: Transmission Control Protocol (TCP). RFC Editor. https://www.rfc-editor.org/rfc/rfc9293</p><p>[2] Larsen, M., Gont, F. (2011). RFC 6056: Recommendations for Transport-Protocol Port Randomization. RFC Editor. https://www.rfc-editor.org/rfc/rfc6056</p><p>[3] Fette, I., Melnikov, A. (2011). RFC 6455: The WebSocket Protocol. RFC Editor. https://www.rfc-editor.org/rfc/rfc6455</p><p>[4] McManus, P. (2018). RFC 8441: Bootstrapping WebSockets with HTTP/2. RFC Editor. https://www.rfc-editor.org/rfc/rfc8441</p><p>[5] Linux man-pages project. socket(2) — Linux manual page. https://man7.org/linux/man-pages/man2/socket.2.html</p><p>[6] Linux man-pages project. getrlimit(2) — Linux manual page. https://man7.org/linux/man-pages/man2/getrlimit.2.html</p><p>[7] Linux man-pages project. tcp(7) — Linux manual page. https://man7.org/linux/man-pages/man7/tcp.7.html</p><p>[8] Linux man-pages project. listen(2) — Linux manual page. https://man7.org/linux/man-pages/man2/listen.2.html</p><p>[9] Linux Kernel Documentation. IP Sysctl. https://docs.kernel.org/networking/ip-sysctl.html</p><p>[10] Kubernetes Documentation. Pods. https://kubernetes.io/docs/concepts/workloads/pods/</p><p>[11] Kubernetes Documentation. Using Source IP. https://kubernetes.io/docs/tutorials/services/source-ip/</p><p>[12] Kubernetes Documentation. Virtual IPs and Service Proxies. https://kubernetes.io/docs/reference/networking/virtual-ips/</p><p>[13] Kubernetes Blog. kube-proxy Subtleties: Debugging an Intermittent Connection Reset. https://kubernetes.io/blog/2019/03/29/kube-proxy-subtleties-debugging-an-intermittent-connection-reset/</p><p>[14] Kubernetes Documentation. Horizontal Pod Autoscaling. https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/</p><p>[15] Kubernetes Documentation. Assigning Pods to Nodes. https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/</p><p>[16] Kubernetes Documentation. Pod Topology Spread Constraints. https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/</p><p>[17] Cilium Documentation. Kubernetes Without kube-proxy. https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/</p><p>[18] Google Cloud Documentation. Cloud NAT: IP addresses and ports. https://docs.cloud.google.com/nat/docs/ports-and-addresses</p><p>[19] Amazon Web Services Documentation. Work with NAT gateways. https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-working-with.html</p><p>[20] Microsoft Learn. Troubleshoot intermittent outbound connection errors in Azure App Service. https://learn.microsoft.com/en-us/azure/app-service/troubleshoot-intermittent-outbound-connection-errors</p><p>[21] Amazon Web Services Documentation. Edit attributes for your Application Load Balancer. https://docs.aws.amazon.com/elasticloadbalancing/latest/application/edit-load-balancer-attributes.html</p><p>[22] NGINX Documentation. WebSocket proxying. https://nginx.org/en/docs/http/websocket.html</p><p>[23] Envoy Documentation. How do I configure timeouts? https://www.envoyproxy.io/docs/envoy/latest/faq/configuration/timeouts</p><p>[24] HAProxy Documentation. Configuration Manual. https://www.haproxy.com/documentation/haproxy-configuration-manual/latest/</p><p>[25] Rotaru, M., Olariu, F., Onica, E., Rivière, E. (2017). Reliable Messaging to Millions of Users with MigratoryData. Middleware 2017 / arXiv. https://arxiv.org/abs/1712.09876</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b1cc1b17259" width="1" height="1" alt=""><hr><p><a href="https://medium.com/draftkings-engineering/your-websocket-limit-is-not-where-you-think-part-1-3b1cc1b17259">Your WebSocket limit is not where you think [Part 1]</a> was originally published in <a href="https://medium.com/draftkings-engineering">DraftKings Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Feature Flags: Performance and Resilience]]></title>
            <link>https://medium.com/draftkings-engineering/mastering-feature-flags-performance-and-resilience-69b7351abe56?source=rss-c923e3c536c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/69b7351abe56</guid>
            <category><![CDATA[managing-complexity]]></category>
            <category><![CDATA[feature-toggles]]></category>
            <category><![CDATA[feature-flags]]></category>
            <category><![CDATA[feature-flagging]]></category>
            <category><![CDATA[operational-switches]]></category>
            <dc:creator><![CDATA[Martin Chaov]]></dc:creator>
            <pubDate>Tue, 30 Apr 2024 13:59:25 GMT</pubDate>
            <atom:updated>2024-09-23T14:06:31.703Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*17gdl2BwVqJBg_cK" /><figcaption>Performance and Resilience when using feature flags according to DALLe3</figcaption></figure><p>This article is part of a series; if you got here first, it might be worth it to check some of the previous ones:</p><ul><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-the-basics-f292e64a4b4b">The basics</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-types-of-feature-flags-b57fa613ce38">Types of feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-life-cycle-3303c746d227">Life cycle</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-testing-feature-flags-2bfdff31905f">Testing feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-risk-mitigation-3238c1a247d9">Risk mitigation</a></li><li><strong>Performance and Resilience (you are here)</strong></li></ul><p>Performance and Resilience are essential aspects of software development, and feature flags bring the intricacies of their requirements. This article explores the nuances of performance and resilience challenges introduced by feature flagging on client applications and outlines strategies for their mitigation.</p><h3>Latency, Resource utilization, throughput</h3><p>Latency is not solely a network concern; it encompasses the time from cause to effect within an application. A common source of latency in applications using feature flags is the decision-making overhead. Every time a feature flag check occurs, it introduces a slight delay as the system determines whether to enable a feature. This is compounded when new logic must be downloaded or when flags are evaluated sequentially. Depending on product requirements, SLAs, and implementation details, a feature flag could trigger the download of new application logic, which brings another layer of complexity (ability to load and execute code on demand) and latency before the feature can be run. It is not intuitive to notice how one flag could degrade the application’s performance. A feature flag could also bring latency if its value has to be re-validated with a back-end service periodically during an active session or on application bootstrap.</p><p>While resource utilization should not be significantly affected in most cases, budgeting how much CPU time and memory allocations for feature flags is a good practice. Does storing a 100MB flag configuration on the device make sense vs. downloading it every time? Is there a way to download only what is required at the cost of additional latency or the inability to download it on demand? What is the worst case for the user if a flag is unavailable or cannot be switched on/off?</p><p>Throughput is a business-defined metric expressed with technical values. The basic definition of throughput is the rate at which a system can process requests. Periodic polling for feature flags can increase the load on different components in the system (gateways, authentication, authorization, etc.). One source of degradation can be the system serving the flag values itself if it becomes overwhelmed with traffic or requests. Moreover, it is a simple transaction of returning what a flag value is for a user in a segment. However, user segments can change dynamically; ideally, a user can be in multiple segments, which may overlap. Determining which value is appropriate for a user can quickly become time-consuming for the back end.</p><p><strong>Mitigation strategies:</strong></p><h4><strong>Caching</strong></h4><p>Caching strategies should be tailored to <a href="https://medium.com/draftkings-engineering/mastering-feature-flags-types-of-feature-flags-b57fa613ce38">flag types</a>, with micro-caching implemented to reduce load for frequently accessed flags.:</p><ul><li><strong>Release </strong>— short-term caching with quick invalidation. Release flags are often used to control the rollout of new features. These flags may change rarely, but when they do have to change, especially in case of a rollback, it should be near instantaneous. Cache the values for a few minutes before revalidation and default to the “off” state rather than using a stale “on” state. This approach balances the need for up-to-date flag states with the benefits of the reduced number of calls reaching the back-end in cases where the back-end system is not available and can’t respond with up-to-date values.</li><li><strong>Experiment </strong>— session-based caching. Experiment flags control A/B tests or other user experiments. User experience must remain consistent throughout a session once an experiment variant is assigned. Cache the flag state at the beginning of a user session and maintain that state for the session’s duration (a clear definition of the user session is required in the context of the experiment). This ensures users don’t switch between experiment groups mid-session, preserving the experiment’s integrity. Any flag should have the ability to be switched off if it hurts the business.</li><li><strong>Operational </strong>— long-term caching with manual invalidation. These flags toggle operational aspects of the application, such as feature maintenance modes, back-end service endpoints, logging levels, etc. These flags change infrequently. Since operational flags don’t change often, caching their states for extended periods (hours, days, months) until an update is manually triggered can significantly reduce system load.</li><li><strong>Permission </strong>— user-specific caching with dynamic invalidation. Permission flags control access to features based on user roles, which can change as user segments are updated (user transitioning from casual to professional or VIP tier). Such caching should be done per user, with mechanisms to invalidate the cache when the user segment changes. This requires a more sophisticated mechanism, potentially integrating with the system’s authentication and authorization layer to detect permission changes.</li></ul><p>The caching strategy should not lead to stale or inconsistent flag states across different parts of the application or multi-device applications. The caching mechanisms must scale with the application, considering the number of flags and the volume of requests.</p><h4><strong>Optimized flag evaluation with strict SLAs</strong></h4><ul><li><strong>Lazy evaluation </strong>— evaluate flags only when necessary, rather than at the start of the application, to reduce unnecessary loading times.</li><li><strong>Consider Flag SDKs</strong> — most flag management platforms provide an SDK, which often comes with all the required optimizations (local caching, background polling, push updates, etc.)</li><li><strong>User segmentation</strong> — pre-compute segment/flag/environment combinations and store them for efficient access to reduce the need for complex evaluation logic upon each request.</li><li><strong>Monitoring and alerting</strong> — track the performance impact of feature flags and set up alerts based on the SLA breaches. A close to real-time feedback loop allows for quick adjustments to flags, especially for release and experimentation types.</li></ul><h4><strong>Data model efficiency</strong></h4><p>Different <a href="https://medium.com/draftkings-engineering/mastering-feature-flags-types-of-feature-flags-b57fa613ce38">flag types</a> can require different data modeling from an efficiency point of view. Here are a few examples to consider when modeling them based on the type of flag:</p><ul><li><strong>Release </strong>— to support gradual rollout strategies, these flags should be able to track targeted segments, rollout percentages and timing (10% ramp up per day), state tracking (not started, in progress, completed)</li><li><strong>Experiment </strong>— they require a model that can handle variant assignment and track user engagement metrics as defined by the experiment or A/B test. They should be able to track values per user or user segment assignments, user engagement metrics, and others.</li><li><strong>Operational </strong>— requires a straightforward and accessible data model. Such flags usually need to be easy to access and evaluate. Besides their current value, a timestamp of when the last change occurred can help synchronize their state in a distributed system.</li><li><strong>Permission </strong>is standardized according to RBAC and supports dynamic updates to avoid edge cases with users moving between segments.</li></ul><p>An example of a standard data model for feature flags:</p><pre>type Flag = {<br>    id: String                  // GUID<br>    name: String<br>    type: String                // Release | Experimentation | Operational | Permission<br>    description: String<br>    scope: []                   // short-lived | long-lived | user-based | system-based<br>    environments: {<br>        development: Boolean<br>        staging: Boolean<br>        production: Boolean<br>    }<br>    targeting: {<br>        segments: []            // segment identifiers<br>        conditions: {<br>            attribute: String   // user attribute<br>            operation: Enum     // equals, greater than, lesser than, etc.<br>            value: String       // value if attribute meets condition<br>        }[]<br>    }<br>    rollout: {<br>        strategy: Enum          // gradual, targeted, all, etc.<br>        percentage: Number      // required<br>    }<br>    operationalParameters: {    // holds all the values required for the application to use this flag<br>        param1: value1<br>    }<br>    createdAt: Number           // timestamp<br>    updatedAt: Number           // timestamp<br>}</pre><h4><strong>Monitor performance</strong></h4><p>The different flag types have very distinct purposes and reasons for existing. Thus, monitoring them can vary greatly. What could be considered “performant” for the different types of flags can be affected by the business niche and system constraints.</p><ul><li><strong>Release </strong>— rolling new system updates mainly affects the system stability and availability. Important metrics to track (before and after the update) include response times, error rates, user metrics, conversion rates, RUM, and APM.</li><li><strong>Experiment </strong>— A/B testing requires close monitoring of user behavior and experiment outcomes. Important metrics to track include conversion rates, click-through rates, funnels, and other business-specific KPIs to assess the variants. In addition, system monitoring for performance degradation should be performed to understand if that can skew the results. Google Analytics, APM, RUM.</li><li><strong>Operational </strong>— as these control operational aspects of the system (i.e., log level), they should be tracked in terms of CPU, memory, network, and other relevant resource utilization. In addition, error rates will be helpful to ensure operational changes do not introduce instability. ELK, Grafana, PRTG, etc.</li><li><strong>Permission </strong>— monitoring focus should be on access patterns, access violations, malicious behavior (brute force attempts, unauthorized access attempts), and tracking how often and by whom such features are accessed.</li></ul><h4><strong>Selective use</strong></h4><p>Feature flags should be applied with great care and attention; in many cases, using a feature flag can increase the TCO of doing something disproportionate to the benefits provided. You can read more about this topic in our article about <a href="https://www.imot.bg/pcgi/imot.cgi?act=5&amp;adv=1r162547854159487&amp;slink=amwjkr&amp;f1=1&amp;fe7=1">risk mitigation</a>.</p><h3>Error rates, Availability, Failover time</h3><p>Feature flags can significantly increase the complexity of tracking error rates due to the permutations of flag states. An application with multiple flags might see a manifold increase in potential error states. Ensuring high availability and swift failover in the event of errors requires the following:</p><ul><li><strong>Rate limiting</strong> — to manage excessive polling for flag values, the goal is to avoid service degradation or failures due to sudden spikes in traffic.</li><li><strong>Circuit breakers</strong> — when the application detects a failure threshold has been reached, it should temporarily turn off the failure. In such cases, retries should implement an incremental back-off with a random seed to avoid multiple clients stressing the back end simultaneously.</li><li>The previous section already covers real-time monitoring (RUM) and alerting.</li></ul><p>Feature flags that control critical system aspects can complicate the failover mechanisms. System downtime can be prolonged if there is no easy way to switch to the feature flags’ confirmed working state/values.</p><p>Third-party services integrated via feature flags must not compromise the system’s performance and Resilience. Implementing rate limiting and circuit breakers can mitigate the risk of third-party failures. Additionally, asynchronous loading of feature flags can enhance startup times without sacrificing user experience.</p><p>Often, a third-party system would handle the heavy lifting related to the operational aspects of using feature flags.</p><p>Measuring the performance of feature flags involves assessing both the impact of the flags on the system and the effectiveness of the flags in achieving their intended goals. A few starting points:</p><ol><li>Collect system performance metrics before/after a flag is enabled: response times, error rates, resource usage, etc.</li><li>Measure feature usage and how often the end users use the code behind the flag.</li><li>Business metrics such as session length and click-through rate.</li><li>Compare behavior and revenue data during A/B testing.</li><li>Flag management metrics include how often it gets toggled, related incidents, how long it remains in the codebase, etc.</li></ol><h3>Glossary</h3><ul><li><strong>Lazy Evaluation</strong>: A strategy to delay the checking and applying feature flags until necessary, minimizing the performance impact on application startup and operation.</li><li><strong>Flag SDKs</strong>: Software Development Kits provided by feature flag management platforms, offering built-in optimizations like local caching, background polling, and push updates to facilitate efficient flag handling.</li><li><strong>Pre-compute Segment/Flag/Environment Combinations</strong>: The practice of calculating and storing the results of complex feature flag evaluations in advance to speed up flag checks during runtime.</li><li><strong>RBAC (Role-Based Access Control)</strong>: A method of managing users’ permissions based on their organizational role relevant to permission flags for controlling access to certain features.</li><li><strong>Failover Time</strong>: The duration required to switch to a stable system state in case of failure, which can be extended by the complexity introduced by feature flags.</li><li><strong>Throughput</strong>: The rate at which a system can process requests, which can be impacted by feature flag checks and the management of flag values, particularly in high-load scenarios.</li><li><strong>Caching</strong>: Storing copies of feature flag states to reduce system load and improve performance. Different strategies (micro-caching, session-based, long-term, user-specific) are used based on flag type and use case.</li><li><strong>Error Rates:</strong> The frequency of errors occurring within a system or application, often used to measure reliability or quality.</li><li><strong>Availability:</strong> The proportion of time a system is operational and accessible to users, typically expressed as a percentage of total time.</li><li><strong>Failover:</strong> Switching to a standby database, server, or network if the primary system fails or is temporarily unavailable.</li><li><strong>Rate Limiting:</strong> The practice of controlling the amount of incoming or outgoing traffic to or from a network or application to prevent overload.</li><li><strong>Circuit Breakers:</strong> A system design pattern that stops the flow of operations in an application to prevent failures from cascading to other system parts.</li><li><strong>A/B Testing:</strong> Comparing two versions of a webpage or app against each other to determine which one performs better.</li><li><strong>RUM (Real User Monitoring):</strong> A type of performance monitoring that captures and analyzes every transaction by users of a website or application.</li><li><strong>APM (Application Performance Monitoring):</strong> Tools or platforms that monitor and manage the performance and availability of software applications.</li><li><strong>TCO (Total Cost of Ownership):</strong> The purchase price of an asset plus the costs of operation, reflecting the total cost of ownership over the life of an asset.</li><li><strong>User Segmentation:</strong> The practice of dividing users into groups based on specific criteria, such as behavior or demographics, to provide tailored experiences.</li></ul><p>This article is part of a series; if you got here first, it might be worth it to check some of the previous ones:</p><ul><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-the-basics-f292e64a4b4b">The basics</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-types-of-feature-flags-b57fa613ce38">Types of feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-life-cycle-3303c746d227">Life cycle</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-testing-feature-flags-2bfdff31905f">Testing feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-risk-mitigation-3238c1a247d9">Risk mitigation</a></li><li><strong>Performance and Resilience (you are here)</strong></li></ul><p>Want to learn more about DraftKings’ global Engineering team and culture? Check out our <a href="https://careers.draftkings.com/life-at-draftkings/engineering/?utm_source=medium&amp;utm_medium=medium&amp;utm_campaign=medium_blog">Engineer Spotlights</a> and <a href="https://careers.draftkings.com/jobs/?utm_source=medium&amp;utm_medium=medium&amp;utm_campaign=medium_blog">current openings</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69b7351abe56" width="1" height="1" alt=""><hr><p><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-performance-and-resilience-69b7351abe56">Mastering Feature Flags: Performance and Resilience</a> was originally published in <a href="https://medium.com/draftkings-engineering">DraftKings Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Feature Flags: Risk Mitigation]]></title>
            <link>https://medium.com/draftkings-engineering/mastering-feature-flags-risk-mitigation-3238c1a247d9?source=rss-c923e3c536c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/3238c1a247d9</guid>
            <category><![CDATA[managing-complexity]]></category>
            <category><![CDATA[feature-toggles]]></category>
            <category><![CDATA[operational-switches]]></category>
            <category><![CDATA[feature-flagging]]></category>
            <category><![CDATA[feature-flags]]></category>
            <dc:creator><![CDATA[Martin Chaov]]></dc:creator>
            <pubDate>Wed, 28 Feb 2024 07:15:30 GMT</pubDate>
            <atom:updated>2024-09-23T14:08:24.793Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OKmFil_543sPTUE-" /><figcaption>Risk mitigation of Feature Flags according to DALLe3</figcaption></figure><p>This article is part of a series; if you got here first, it might be worth it to check some of the previous ones:</p><ul><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-the-basics-f292e64a4b4b">The basics</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-types-of-feature-flags-b57fa613ce38">Types of feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-life-cycle-3303c746d227">Life cycle</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-testing-feature-flags-2bfdff31905f">Testing feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-risk-mitigation-3238c1a247d9"><strong>Risk mitigation </strong></a><strong>(you are here)</strong></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-performance-and-resilience-69b7351abe56">Performance and Resilience</a></li></ul><p>Feature flags offer a powerful way to manage software features but come with risks.</p><p>The first step in risk mitigation is identifying the potential risks. They may include code complexity, security vulnerabilities, and increased testing overhead. Risk assessment should be conducted before implementing feature flags. Moreover, the risk assessment should be part of implementing a feature behind a flag. Once they are identified, appropriate mitigation strategies should be applied. The goal is to minimize the negative impact while maximizing the benefits of using feature flags.</p><h3>Code complexity</h3><p>Introducing feature flags into a software system can inadvertently escalate code complexity — probably the most prominent risk. Multiple flags can interact unpredictably, leading to an exponential increase in system states. This complexity can compromise code readability and maintainability and elevate the risk of defects. To mitigate these challenges, organizations are advised to adopt best practices such as limiting the lifespan of temporary flags, employing clear naming conventions, and conducting focused code reviews. Utilizing feature flag management systems with centralized dashboards can also provide valuable oversight, enabling teams to monitor active flags, their states, and interdependencies effectively.</p><p>Let’s take as an example 4 flags, one of each <a href="https://medium.com/draftkings-engineering/mastering-feature-flags-types-of-feature-flags-b57fa613ce38">type</a>:</p><ul><li>System_Release_DepositV2 — a release flag for incremental release of a new deposit page</li><li>User_Experimentation_ColorSchemeTest — A/B testing for the effect color has on the conversion of customers</li><li>System_Operational_LoginSwitch — system flag used to migrate users from to a new login provider gradually</li><li>User_Permission_VIPDeposit — flag providing a customized deposit experience to the VIP user segment</li></ul><p>At first glance, these flags are not related. However, depending on the segmentation of users, one user could hit two or more of these flags at the same time. In a vast system (even a well-designed one), dynamic deviations from functional requirements could lead to unpredictable system behavior. One way to mitigate this, albeit a bit coarse, is to limit the number of flags and their dependencies for the system in question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*DGRQKL06qRp66pABUurA5A.png" /><figcaption>Dependencies between flags.</figcaption></figure><p>To take a VIP user enrolled into the A/B test and affected by the login migration and the gradual rollout of the new deposit flow. Which of these flags should take precedence? If a user can be in more than one user segment, such cases should be handled in code or configuration of the flags.</p><h3>Adoption of new application versions</h3><p>Sometimes, new software versions cannot be easily propagated to the end users. Things can be as easy on the web as introducing a new character in the URL. However, in native apps, the user must take action; they might not even have the Internet bandwidth required to download and install the new application to their devices. Since engineers cannot enforce instant updates, a gradual new version adoption transition must be considered when introducing or decommissioning feature flags. This fragmentation introduces additional complexity to the codebase as the engineering team must ensure compatibility across multiple application versions.</p><h3>Testing overhead</h3><p>Using feature flags can substantially increase the scope and complexity of <a href="https://medium.com/draftkings-engineering/mastering-feature-flags-testing-feature-flags-2bfdff31905f">testing requirements</a>. Each flag introduces new possible permutations of the system state that must be rigorously tested to ensure stability and performance. This increased testing overhead can strain resources and extend timelines. To manage this, organizations often adopt automated testing frameworks designed to handle multiple flag states. Additionally, it’s advisable to clearly define testing scopes and objectives based on the type and purpose of each feature flag. This targeted approach can help maintain testing efficiency while thoroughly evaluating critical flag interactions. The math behind this is relatively trivial: 3 flags with two possible values each equals eight distinct system configurations. Even a few concurrent flags, like 20 with three different values (ex., one per user segment), give <strong>3 486 784 401</strong> permutations.</p><p>At DraftKings Inc., we have hundreds of flags with an average value of 4.2 at any given time. Testing all the possible configurations is <strong>not </strong>feasible. Thus proper mapping out of dependencies and restrictions between flags should be applied. Analysis of the user segments for intersections and flag collision is one way to find hidden dependencies between the feature flags.</p><p>Testing is expanded upon in the previous article from the series: <a href="https://medium.com/draftkings-engineering/mastering-feature-flags-testing-feature-flags-2bfdff31905f">Testing Feature Flags</a>.</p><h3>Cross-application A/B testing</h3><p>In cases where the users access a system through multiple entry points (web, native, remote terminals, etc.) A/B testing becomes particularly challenging. In the majority of cases, different types of client applications have different life cycles and feature sets, making it difficult to have a consistent and controlled testing environment. Experiments must be designed with these limitations in mind or be closed down to a single type of application.</p><p>While users can expect inconsistent user experience between different mediums, accurately segmenting users across various applications for A/B testing can be complex. They can have different usage patterns or preferences in multiple mediums — checking statuses on the phone vs. doing work on the desktop vs. doing work on the phone vs. checking reports on the desktop. Cross-application A/B tests generate data from different sources with varying formats, metrics, reliability, volume, and frequency. Integrating this data for a cohesive analysis can become a significant challenge and could easily lead to misleading insights or wrong conclusions supported by numbers.</p><h3>Zombie features</h3><p>Residual feature flags, often termed “zombie features,” pose a risk to system integrity. These flags have outlived their utility but remain embedded in the codebase. They contribute to system bloat, complicate maintenance, and potentially introduce security vulnerabilities. Regular audits for identifying and removing these redundant flags should be performed. Updated documentation that accurately reflects the status of each flag is also crucial. Some organizations employ automated tools to identify flags not accessed or modified within a specific timeframe, signaling them for review or removal.</p><p>A typical example can be found in e-commerce platforms that frequently run seasonal promotions or limited-time offers. Once the promotion period ends, the feature flags controlling these offers may remain in the codebase, becoming a “zombie feature.” Moreover, the same flag could be re-used periodically, accumulating code dependencies over time. Some companies conduct regular audits using internal tools to identify such flags, marking them for deprecation or removal to maintain a clean codebase.</p><h3>Security risks</h3><p>Unauthorized access to premium features and accidental exposure to sensitive functionalities can pose a security risk. For example, if a feature flag controlling access to a new administrative tool is not properly authorized, it could be exploited to gain unauthorized access.</p><p>Mitigation strategies:</p><ul><li>Secure coding practices: This involves validating input, sanitizing data, and ensuring default settings are available as a backup in the application code.</li><li>Encryption and secure channels: Encrypt feature flag data and use specific channels for transmitting flag changes. This prevents unauthorized interception or tampering with flag settings.</li><li>Access controls: Implement strict access controls for feature flag management. Only authorized personnel should be able to create, modify, or delete feature flags, of which there should be a company procedure to document, approve, and then record the change in an audit log.</li><li>Regular audits: Conduct security audits focusing on feature flags to identify potential vulnerabilities and ensure compliance with security policies.</li><li>Automated monitoring: Utilize automated monitoring tools to detect unusual activities related to feature flags, such as unexpected changes or access patterns.</li></ul><p>Consider the scenario of integrating a third-party payment processing tool using feature flags. The flag controls the rollout of this new feature. If this flag is misconfigured or lacks adequate security measures, it can expose the application to financial fraud risks.</p><p>When flags are used for granular access control to new features, additional verification should be in place to confirm that the user who makes these requests is, in fact, authorized and part of the targeted audience.</p><h3>Rollback plans</h3><p>In software development, a rollback plan serves as a contingency measure to revert changes in the event of a failure or unintended consequences.</p><p>For instance, a financial technology company may introduce a feature flag to enable a new payment gateway. If issues arise during the rollout — such as increased transaction failures — the rollback plan is activated to turn off the feature flag, reverting the system to its previous state.</p><p>A well-structured rollback plan includes:</p><ul><li>predefined triggers for activation,</li><li>a list of steps to execute the rollback,</li><li>a communication strategy to inform stakeholders and sometimes even the customers.</li></ul><p>Tools like Spinnaker or Kubernetes can automate the source-code rollback process, reducing the time and manual effort required to restore the system to a previous version. Rollbacks may also be triggered based on business KPIs such as negative impact on revenue or customer satisfaction. However, rollbacks are hard with feature flags, even with well-established planning and execution processes. In the best-case scenario, only the flag value is reverted to the default “off” state, and the engineering team can troubleshoot the system safely. In the worst case, the source code should be rolled back entirely, maybe even including some back-end services. There should be a known good state of the feature flags for the previous version of the system’s source code that can be reverted to.</p><p>The line between which component is part of the version of the system becomes blurry and challenging to draw in a system with hundreds of flags and thousands of running services. The whole application code + underlying services + monitoring infrastructure + flags and their values are considered a working snapshot of the system and should be versioned accordingly. Feature flag rollouts should be treated as code changes — it is better to have minor changes at a higher frequency than a big bulk of changes with many moving parts.</p><p>Sometimes, code rollback might not even be a feasible option — native apps are much more complicated to revert to previous versions due to the requirement of the customers’ interaction.</p><h3>Monitoring and alerts</h3><p>Monitoring and alerts are integral to the effective management of feature flags. Consider a cloud-based CRM platform that uses feature flags to control access to a new analytics dashboard. Monitoring tools can track key performance indicators like load times, error rates, and user engagement. If these metrics fall outside predefined thresholds, alert systems can notify the development team in real time. This enables quick identification and resolution of issues, minimizing impact on the user experience. Solutions like Grafana for monitoring and PagerDuty for alerts can be integrated into the feature flag management system to provide a comprehensive overview of feature performance and system health.</p><h3>Customer support impact</h3><p>The implementation of feature flags can significantly affect customer support operations. The customer experience (CX) team should be adequately trained to handle queries and issues from users who are part of a segment under an active experiment. This includes being aware of the specific features or changes being tested, understanding how to troubleshoot related problems, and knowing when to escalate issues to the development team. Effective communication between the CX and engineering teams ensures customer feedback is promptly addressed and used to improve the feature flag implementation. Creating a knowledge base with FAQs and common issues related to active feature flags can also empower the CX team to provide faster and more accurate responses to customer inquiries.</p><h3>Glossary</h3><ul><li><strong>Rollback Plans:</strong> Strategies or procedures for reverting software to a previous state in case of failure or negative consequences after deploying a new feature or update.</li><li><strong>CRM (Customer Relationship Management) Platform:</strong> Software used by businesses to manage and analyze customer interactions and data throughout the customer lifecycle.</li><li><strong>KPIs (Key Performance Indicators):</strong> Quantifiable measures used to evaluate the success of an organization, employee, or other entity in meeting objectives for performance.</li></ul><p>This article is part of a series; if you got here first, it might be worth it to check some of the previous ones:</p><ul><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-the-basics-f292e64a4b4b">The basics</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-types-of-feature-flags-b57fa613ce38">Types of feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-life-cycle-3303c746d227">Life cycle</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-testing-feature-flags-2bfdff31905f">Testing feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-risk-mitigation-3238c1a247d9"><strong>Risk mitigation </strong></a><strong>(you are here)</strong></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-performance-and-resilience-69b7351abe56">Performance and Resilience</a></li></ul><p>Want to learn more about DraftKings’ global Engineering team and culture? Check out our <a href="https://careers.draftkings.com/life-at-draftkings/engineering/?utm_source=medium&amp;utm_medium=medium&amp;utm_campaign=medium_blog">Engineer Spotlights</a> and <a href="https://careers.draftkings.com/jobs/?utm_source=medium&amp;utm_medium=medium&amp;utm_campaign=medium_blog">current openings</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3238c1a247d9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-risk-mitigation-3238c1a247d9">Mastering Feature Flags: Risk Mitigation</a> was originally published in <a href="https://medium.com/draftkings-engineering">DraftKings Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Feature Flags: Testing Feature Flags]]></title>
            <link>https://medium.com/draftkings-engineering/mastering-feature-flags-testing-feature-flags-2bfdff31905f?source=rss-c923e3c536c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bfdff31905f</guid>
            <category><![CDATA[feature-flagging]]></category>
            <category><![CDATA[feature-toggles]]></category>
            <category><![CDATA[managing-complexity]]></category>
            <category><![CDATA[feature-flags]]></category>
            <category><![CDATA[operational-switches]]></category>
            <dc:creator><![CDATA[Martin Chaov]]></dc:creator>
            <pubDate>Wed, 28 Feb 2024 07:13:12 GMT</pubDate>
            <atom:updated>2024-09-23T14:08:59.125Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TdbhOCF8S1uqrY05" /><figcaption>Testing feature flags according to DALLe3</figcaption></figure><p>This article is part of a series; if you got here first, it might be worth it to check some of the previous ones:</p><ul><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-the-basics-f292e64a4b4b">The basics</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-types-of-feature-flags-b57fa613ce38">Types of feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-life-cycle-3303c746d227">Life cycle</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-testing-feature-flags-2bfdff31905f"><strong>Testing feature flags</strong></a><strong> (you are here)</strong></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-risk-mitigation-3238c1a247d9">Risk mitigation</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-performance-and-resilience-69b7351abe56">Performance and Resilience</a></li></ul><p>Feature flags serve as a mechanism in software development for managing features in a controlled environment. This article delineates a structured approach for testing feature flags, encompassing planning, execution, and monitoring.</p><p>Using feature flags can substantially increase the scope and complexity of testing requirements. Each flag introduces new permutations of system states that must be rigorously tested to ensure stability and performance. This testing overhead can strain resources and extend timelines. To manage this, organizations often adopt automated testing frameworks designed to handle multiple flag states. Additionally, it’s advisable to clearly define testing scopes and objectives based on the type and purpose of each feature flag.</p><p><strong>The primary way to keep the feature flags easy to test and maintain is to make sure their number in the system is kept to the bare minimum required to meet said system’s goals!</strong></p><p>To expand on the flag types from the <a href="https://medium.com/draftkings-engineering/mastering-feature-flags-types-of-feature-flags-b57fa613ce38">previous article</a>…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LQZeLJ2iUtzsFEMb8bDtpw.png" /><figcaption>Testing of feature flags based on the flag type.</figcaption></figure><h3>Planning &amp; Designing Test Scenarios</h3><p>Initial planning establishes the framework for the testing process.</p><p><strong>Objectives and Goals</strong></p><p>Objectives should be clearly defined. For example, Experimentation flags focus on user metrics, while Operational flags could be geared toward system performance.</p><ul><li>Define the testing objectives based on the feature flag type.</li><li>Specify the expected behavior and features impacted.</li></ul><p><strong>Dependencies and Scope</strong></p><p>Dependencies and scope require careful consideration.</p><ul><li>Identify dependencies with other feature flags.</li><li>Determine the testing scope, including environments (test, staging, production) and platforms (web, native).</li></ul><p>A few flags with a few possible values (just the — on/off) can explode in complexity.</p><p>The test design aims to cover all potential scenarios. Assume we have the following four flags of different types:</p><ul><li>System_Release_DepositV2 — a release flag for incremental release of a new deposit page</li><li>User_Experimentation_ColorSchemeTest — A/B testing for the effect color has on the conversion of customers</li><li>System_Operational_LoginSwitch — system flag used to migrate users from to a new login provider gradually</li><li>User_Permission_VIPDeposit — flag providing a customized deposit experience to the VIP user segment</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*DGRQKL06qRp66pABUurA5A.png" /><figcaption>Possibly hidden relationships between different flags.</figcaption></figure><p>The test scenarios should cover all the cases from the combinations of the different possible flag values based on the assigned user segments. Consider if a customer could simultaneously have to be a part of all four flags due to participating in a few of the targeted user segments. Most of the time, tests should cover the system before the flag is enabled; thus, only the new scenarios should be developed. However, whenever flag dependencies are discovered, there should be tests related to the system behavior if the said system is misconfigured.</p><p>Test scenarios form the core of the testing strategy.</p><ul><li>Clearly define the states: off, on, default, and others… including the combination of flags (which becomes the value itself) in case of dependencies.</li><li>Develop test scenarios for both the enabled and unwanted states. The existing test coverage should be enough to ensure the disabled state is handled.</li><li>Formulate scenarios for different user segments.</li><li>Formulate scenarios for the dependent flags.</li><li>Formulate cases that assess the collected data’s accuracy.</li><li>Scenarios for the logging and monitoring related to the new feature, especially for experiment flags — it is not uncommon for decisions to be taken based on faulty statistics.</li></ul><p>Appropriate test data and environment configurations are essential for effective test execution, as is their proper documentation. The test data and environment configurations should be documented in the test scenarios and plan. Performance, Resilience, and security-related topics are going to be expanded in upcoming articles.</p><h3>Functional and End-to-end Testing</h3><p>This section examines how the feature flag interacts with the overall system. Feature flags should not be chained in more ways than “if flag X is on, flag Y is off” — most interesting cases arise from user segmentation and users in more than one segment. This is where collision detection and prioritization should be implemented.</p><p>Test execution and validation of feature flag behavior:</p><ul><li>Configure the feature flag settings for each test scenario.</li><li>Execute the test scenarios and validate outcomes.</li><li>Document any anomalies or unexpected behavior.</li></ul><p>Automation and parallelized testing can be enabled via environments configured based on various available user segments and their combinability.</p><p><strong>Enabled and Disabled States</strong></p><p>Testing should include both enabled and disabled states of the feature flag.</p><ul><li>Conduct thorough testing when the feature is enabled.</li><li>Confirm that the feature is deactivated when disabled.</li></ul><p><strong>Boundary Cases and Dependencies</strong></p><p>Testing should also consider edge cases and dependencies, particularly for complex flags.</p><ul><li>Examine edge cases and scenarios influenced by the feature flag.</li><li>Involve engineers in monitoring logs and metrics.</li></ul><p><strong>Cross-Browser and Cross-Platform</strong></p><p>Compatibility testing is necessary to ensure consistent behavior across different platforms.</p><ul><li>Validate compatibility across various browsers and platforms.</li></ul><h3>Test Rollbacks</h3><p>Rollbacks serve as a contingency measure.</p><ul><li>Validate the interaction between the feature flag and other system components.</li><li>Ensure no conflicts arise when integrated with other functionalities.</li><li>Prepare a disaster recovery plan. This involves outlining steps to recover from any adverse effects caused by the feature flag.</li></ul><h3>Deployment and Release Testing</h3><p>Before going live, validating the feature flag in various deployment environments is essential. This ensures a smooth transition from staging to production.</p><ul><li>Validate behavior across different environments and during the release process.</li></ul><p><strong>Minimizing User Impact</strong></p><p>Testing in various deployment environments is necessary for a seamless transition to production.</p><ul><li>Validate behavior in different environments.</li><li>Implement a phased approach for enabling the feature flag in production.</li><li>Define specific metrics or KPIs for assessing the success of the feature flag.</li></ul><h3>Monitoring and Maintenance</h3><p>Ongoing monitoring is required post-deployment.</p><ul><li>Implement logging and monitoring mechanisms.</li><li>Monitor performance and stability continuously.</li></ul><h3>Cost-Benefit Analysis</h3><p>Evaluating the return on investment (ROI) for feature flags involves assessing costs and benefits. Prices may include development, testing, and ongoing maintenance. The complexity the flag introduces, such as the need for additional test scenarios, should be accounted for. Benefits may include reduced time-to-market, variable release strategies, and system stability. The analysis informs decision-making regarding the feature flag’s long-term viability.</p><h3>Version Control</h3><p>Managing feature flags across different software versions requires a robust version control system. Compatibility with current and legacy versions is essential, especially for long-lived flags or those spanning multiple versions. The snapshot of the system’s current state, combining running services, configurations, feature flags and their values, and others, could be considered for a system version.</p><h3>Glossary</h3><ul><li><strong>Feature Flag</strong>: A toggle for turning features on or off.</li><li><strong>Test Cases</strong>: Specific conditions under which a feature flag is tested.</li><li><strong>Functional Testing</strong>: Verifying that the feature works as expected.</li><li><strong>Integration Testing</strong>: Ensuring the feature works within the overall system.</li><li><strong>User Acceptance Testing (UAT)</strong>: Validation by the end-users or stakeholders.</li><li><strong>Deployment Testing</strong>: Verifying feature flag behavior in different environments.</li><li><strong>Monitoring</strong>: Ongoing tracking of feature flag performance and issues.</li><li><strong>Release Flag</strong>: A flag used for controlling the release of new features.</li><li><strong>Experimentation Flag</strong>: A flag used for A/B testing.</li><li><strong>Operational Flag</strong>: A flag used for controlling operational settings.</li><li><strong>Permission Flag</strong>: A flag used for controlling access to features.</li></ul><p>This article is part of a series; if you got here first, it might be worth it to check some of the previous ones first:</p><ul><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-the-basics-f292e64a4b4b">The basics</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-types-of-feature-flags-b57fa613ce38">Types of feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-life-cycle-3303c746d227">Life cycle</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-testing-feature-flags-2bfdff31905f"><strong>Testing feature flags</strong></a><strong> (you are here)</strong></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-risk-mitigation-3238c1a247d9">Risk mitigation</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-performance-and-resilience-69b7351abe56">Performance and Resilience</a></li></ul><p>Want to learn more about DraftKings’ global Engineering team and culture? Check out our <a href="https://careers.draftkings.com/life-at-draftkings/engineering/?utm_source=medium&amp;utm_medium=medium&amp;utm_campaign=medium_blog">Engineer Spotlights</a> and <a href="https://careers.draftkings.com/jobs/?utm_source=medium&amp;utm_medium=medium&amp;utm_campaign=medium_blog">current openings</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bfdff31905f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-testing-feature-flags-2bfdff31905f">Mastering Feature Flags: Testing Feature Flags</a> was originally published in <a href="https://medium.com/draftkings-engineering">DraftKings Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Feature Flags: Life Cycle]]></title>
            <link>https://medium.com/draftkings-engineering/mastering-feature-flags-life-cycle-3303c746d227?source=rss-c923e3c536c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/3303c746d227</guid>
            <category><![CDATA[managing-complexity]]></category>
            <category><![CDATA[feature-flags]]></category>
            <category><![CDATA[operational-switches]]></category>
            <category><![CDATA[feature-toggles]]></category>
            <category><![CDATA[feature-flagging]]></category>
            <dc:creator><![CDATA[Martin Chaov]]></dc:creator>
            <pubDate>Sat, 18 Nov 2023 08:48:48 GMT</pubDate>
            <atom:updated>2024-09-23T14:27:35.463Z</atom:updated>
            <content:encoded><![CDATA[<p>This article is part of a series; if you got here first, it might be worth it to check some of the previous ones first:</p><ul><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-the-basics-f292e64a4b4b">The basics</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-types-of-feature-flags-b57fa613ce38">Types of feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-life-cycle-3303c746d227"><strong>Life cycle</strong></a> (you are here)</li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-testing-feature-flags-2bfdff31905f">Testing feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-risk-mitigation-3238c1a247d9">Risk mitigation</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-performance-and-resilience-69b7351abe56">Performance and Resilience</a></li></ul><p>This article provides an overview of the typical life cycle of a feature flag, from its inception to eventual decommissioning. Similarly to the software development life cycle, the feature flag life cycle may vary depending on the business domain and system criticality. This article presents a generalized life cycle without making assumptions about specific business requirements.</p><h3>The life cycle at a glance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*4aNscKtPr9IGuLHE6-cmSQ.png" /></figure><h3>Creation</h3><p>The first step entails documenting the flag’s objectives, stakeholders, and expected results. A detailed testing plan should be developed as well. The flag is then “created” in the team’s chosen flag management system and added to the codebase, usually in its default “off” state.</p><p>Risk assessment is required to evaluate the impact of the feature flag on the system and business and any potential dependencies to other features and flags. KPIs should be set to measure the flag objectives. Especially in regulated industries, it is vital to consider how feature flags interact with legal requirements.</p><h4>Naming convention</h4><p>A standardized naming convention for feature flags is required to maintain clarity and consistency across development teams. It should eliminate ambiguity and simplify identifying each flag’s purpose, scope, and ownership, reducing the risk of misconfiguration or misuse. Moreover, a well-defined naming structure streamlines documentation and monitoring processes, enhancing overall project manageability.</p><p>Depending on the company’s scale and size, there are many ways to approach this. This section outlines a somewhat generic approach to naming that should be distilled based on one’s reality and needs.</p><ul><li>Names should be split into categories like “global” and “team” flags.</li><li>Include prefix for the type of flag:</li><li>Release: RLS</li><li>Experimentation: EXP</li><li>Operational: OPS</li><li>Permissions: PRM</li></ul><ol><li>Flags owned by a specific team → prefix-productCode-teamCode-flagName:</li><li><strong>RLS-ShoppingCart-CartTeam-ShoppingCartV2.1</strong></li><li><strong>EXP-ShoppingCart-DataScienceTeam-FloatingShoppingCart</strong></li><li><strong>OPS-ShoppingCart-DataScienceTeam-IncreasedLogLevelForFloatingShoppingCart</strong></li><li><strong>PRM-ShoppingCart-DataScienceTeam-EnabledFloatingShoppingCart</strong></li><li>Global flags → prefix-globalPrefix-domain-flagName:<strong> OPS-Global-FeatureFlags-InvalidateAllCachedFlags</strong></li></ol><h3>Testing</h3><p>A flag undergoes validation in controlled environments. Its various states are verified against the testing plan, and interactions/dependencies to other flags are also tested, where applicable. There is an <a href="https://medium.com/draftkings-engineering/mastering-feature-flags-testing-feature-flags-2bfdff31905f">article</a> later in the series that does a deep dive in flag testing.</p><h3>Deployment</h3><p>Before deployment, the feature flag is configured to its default state in the flag management system. The application code, which includes the feature flag, is then deployed. Verification should be performed to ensure the application functions correctly with the flag in its default “off” state — turning the flag off would be the first action during an incident response.</p><h3>Activation</h3><p>There should be no change to the flag value without a “green light.” Necessary approvals for activation should be obtained following a process similar to the one required to deploy code to the production environment. The rollout strategy should be determined (gradual, percentile, canary, etc.). Subsequently, the flag is turned on for the desired user group.</p><p>After the necessary approvals for activation are secured, and an audit log is maintained — the rollout strategy, whether gradual or percentile-based, is established. The flag is activated for its intended audience.</p><p>Any change to the value of a flag should be considered a rollout if it affects the system’s end users. In the case of feature flags, there are a few different rollback considerations:</p><ul><li>Return to a previously known good state of the flag in the flag management system.</li><li>Redeployment of an older known good state of the codebase.</li></ul><p>Changes in feature flags on production should be treated as a production deployment.</p><h3>Monitoring</h3><p>Key performance indicators (KPIs) are closely monitored to assess the flag’s impact on the system and the user experience. Measuring the performance of feature flags involves considering both the effect of the flag on the design and the effectiveness of the flag in achieving its intended goals.</p><p>There will be a more detailed chapter on monitoring in a follow-up article.</p><h3>Deactivation (optional)</h3><p>The flag may be deactivated for a variety of reasons:</p><ul><li>Adverse impact on revenue</li><li>System instability</li><li>Flag reaching its end-of-life</li></ul><h3>Clean up</h3><p>Once the flag is stable, it is removed from the codebase. The associated feature either becomes a permanent part of the system or is removed. The flag remains in the management system until full user adoption.</p><h3>Decommission</h3><p>The flag is officially retired! It should be removed from the flags management system, and all related documentation should be updated. Both the planned and actual decommissioning dates are noted.</p><p>The flag’s name should remain unusable for some time to ensure that rollback to the previous application version would not require the re-creation of the flag.</p><p>Post-decommission, the flag’s data and audit log should be archived for compliance or historical analysis.</p><h3>The process around the life cycle</h3><p>Managing the life cycle of a feature flag is not just a technical endeavor but also involves strategic planning, cost considerations, and stakeholder engagement.</p><h3>Documenting feature flags</h3><p>Good documentation helps ensure that everyone on the team understands what the flag does, when it should be used, who can approve activation/deactivation, and aids in troubleshooting.</p><p>At the very least, a feature flag should have the following:</p><ul><li><strong>Name</strong> — According to the approved naming convention.</li><li><strong>Status</strong> — Active/Inactive, Environments availability: Production, Staging, Testing, etc.</li><li><strong>Creation date </strong>— Date of creation in the feature flag management system.</li><li><strong>Expected to decommission date</strong> — Date when all code and feature flag management system references would be removed.</li><li><strong>Description </strong>— Detailed information about what the flag does, what features or settings it controls — links to relevant documentation, disaster recovery plan, etc.</li><li><strong>Type of the flag </strong>— Experimentation, Operational, Permissions, Release.</li><li><strong>Scope of the flag </strong>— Long/Short lived, User/System based.</li><li><strong>Change log </strong>— Record of changes to the flag (audit log exists in feature flags management system).</li><li><strong>Owner of the flag</strong> — Stakeholders responsible for the creation of the flag.</li><li><strong>Values descriptions </strong>— What are the possible values of the flag and their expected behavior?</li><li><strong>Dependencies</strong> — What features, teams, and domains are affected by changes to the value? Who should be notified before and after modifications to the flag?</li><li><strong>Fallback behavior </strong>— What is the expected default behavior in a feature flag management system outage?</li></ul><h3>Maintenance</h3><p>Ongoing maintenance is a cornerstone of effective feature flag management. Regular audits help identify obsolete flags, minimize technical debt, and optimize system performance. As part of the maintenance, an awareness of the total cost of ownership of the system should be kept up to date.</p><h3>Cost Analysis</h3><p>A comprehensive understanding of the cost structure is essential for sustainable feature flag management. This encompasses development, operational overhead, and the complexities introduced by an increasing number of active flags. Keeping the number of active flags in the system low should be one of the priorities. Three flags with two values each would give eight possible configuration permutations. Taking a more extensive system as an example, where there could easily be 100 flags with an average of 4 values per flag, the number of possible permutations is exponentially growing to 1.6069380442589902755419620923412e+60 permutations.</p><h3>Versioning</h3><p>Maintaining consistency across different software versions is crucial when a feature flag is involved. Proper versioning ensures that the flag behaves as expected, irrespective of the software release it interacts with. Versioning is also an aspect to consider when defining a rollback strategy.</p><h3>Stakeholder Involvement</h3><p>The life cycle of a feature flag is a collaborative process that requires input from various stakeholders. Their involvement is critical in defining the flag’s objectives, approving its activation, monitoring its impact, and ensuring alignment with organizational goals. For example, there shouldn’t be a case where the data analyst team is unaware that an experiment is stopped.</p><h3>Additional categorization</h3><p>Splitting the flags based on other properties could be beneficial for tracking and documenting purposes. While these categories could vary wildly based on business niche and development process, here are a few flags to consider: time-based, geolocation, device-specific, dependency, visibility level, environment, etc.</p><h3>Variations based on flag type</h3><h3>Release Flags</h3><ol><li><strong>Creation</strong>: Often short-lived, these flags may require less documentation than other types.</li><li><strong>Testing</strong>: Primarily tested in staging to ensure the new feature doesn’t break existing functionalities.</li><li><strong>Monitoring</strong>: Less emphasis on user behavior and more on system stability.</li><li><strong>Decommission</strong>: Usually decommissioned quickly once the feature is stable.</li></ol><h3>Experimentation Flags</h3><ol><li><strong>Creation</strong>: Requires detailed documentation outlining the A/B test hypotheses and metrics.</li><li><strong>Testing</strong>: Rigorous testing in both staging and production to ensure that data collection is accurate.</li><li><strong>Monitoring</strong>: Heavy focus on analytics and user behavior metrics.</li><li><strong>Decommission</strong>: This may remain longer in the code for future experiments.</li><li><strong>Customer support:</strong> the CX team should be aware of variations and be able to support the customers if any issues arise.</li></ol><h3>Operational Flags</h3><ol><li><strong>Creation</strong>: Documentation focuses on system parameters and flag activation/deactivation conditions.</li><li><strong>Testing</strong>: Often tested under simulated operational conditions.</li><li><strong>Monitoring</strong>: Emphasis on system performance metrics.</li><li><strong>Decommission</strong>: May remain in the system long-term for operational flexibility.</li></ol><h3>Permission Flags</h3><ol><li><strong>Creation</strong>: Documentation includes user roles/segments and permissions — analysis on dependencies and if user segments overlap.</li><li><strong>Testing</strong>: Requires testing for different user roles and segments.</li><li><strong>Monitoring</strong>: Focus on user access logs and potential security issues.</li><li><strong>Decommission</strong>: It is often long-lived and may only be decommissioned if the feature it controls becomes universally accessible. The customer support team should also be aware of variations and be able to support the customers if any issues have arisen.</li></ol><h3>Long-lived vs. Short-lived Flags</h3><ul><li><strong>Long-lived Flags</strong>: These often go through a more rigorous documentation and approval process and may require periodic review.</li><li><strong>Short-lived Flags</strong>: These may skip some life cycle stages like long-term monitoring and may be decommissioned more quickly.</li></ul><h3>User-based vs. System-based Flags</h3><ul><li><strong>User-based Flags</strong>: More focus on monitoring user behavior and user-specific metrics. For a flag to be user-based, it has to use at least one piece of user context to operate.</li><li><strong>System-based Flags</strong>: Emphasis on system performance and stability metrics. System-based flags have a hard requirement not to use any user context.</li></ul><h3>Summary</h3><p>Effective management of feature flags requires a disciplined approach through each stage of their life cycle, from creation to decommissioning. Adherence to testing plans, securing necessary approvals, and vigilant monitoring are crucial. Understanding this life cycle helps teams mitigate risks while maximizing the benefits of feature flags, such as agile development and dynamic feature control.</p><h3>Glossary</h3><ul><li><strong>Feature Flag Management System</strong>: A tool for managing feature flags’ creation, deployment, and monitoring.</li><li><strong>Default State</strong>: The initial state of a feature flag, usually set to ‘off.’</li><li><strong>Rollout Strategy</strong>: The method used for gradually releasing a new feature to users.</li><li><strong>Audit Log</strong>: A record of changes, often used for compliance and monitoring.</li><li><strong>Key Performance Indicators (KPIs)</strong>: Metrics used to evaluate the success of a feature or experiment.</li></ul><p>This article is part of a series, if you enjoyed it you should check out some of the other parts as well:</p><ul><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-the-basics-f292e64a4b4b">The basics</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-types-of-feature-flags-b57fa613ce38">Types of feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-life-cycle-3303c746d227"><strong>Life cycle</strong></a> (you are here)</li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-testing-feature-flags-2bfdff31905f">Testing feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-risk-mitigation-3238c1a247d9">Risk mitigation</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-performance-and-resilience-69b7351abe56">Performance and Resilience</a></li></ul><p>Want to learn more about DraftKings’ global Engineering team and culture? Check out our <a href="https://careers.draftkings.com/life-at-draftkings/engineering/?utm_source=medium&amp;utm_medium=medium&amp;utm_campaign=medium_blog">Engineer Spotlights</a> and <a href="https://careers.draftkings.com/jobs/?utm_source=medium&amp;utm_medium=medium&amp;utm_campaign=medium_blog">current openings</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3303c746d227" width="1" height="1" alt=""><hr><p><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-life-cycle-3303c746d227">Mastering Feature Flags: Life Cycle</a> was originally published in <a href="https://medium.com/draftkings-engineering">DraftKings Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Feature Flags: Types of Feature flags]]></title>
            <link>https://medium.com/draftkings-engineering/mastering-feature-flags-types-of-feature-flags-b57fa613ce38?source=rss-c923e3c536c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/b57fa613ce38</guid>
            <category><![CDATA[feature-flagging]]></category>
            <category><![CDATA[feature-flags]]></category>
            <category><![CDATA[feature-toggles]]></category>
            <category><![CDATA[operational-switches]]></category>
            <category><![CDATA[managing-complexity]]></category>
            <dc:creator><![CDATA[Martin Chaov]]></dc:creator>
            <pubDate>Sat, 18 Nov 2023 08:48:18 GMT</pubDate>
            <atom:updated>2024-09-23T14:28:25.368Z</atom:updated>
            <content:encoded><![CDATA[<p>This article is part of a series, if you got here first, it might be worth it to check the previous one first:</p><ul><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-the-basics-f292e64a4b4b">The basics</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-types-of-feature-flags-b57fa613ce38"><strong>Types of feature flags</strong></a><strong> </strong>(you are here)</li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-life-cycle-3303c746d227">Life cycle</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-testing-feature-flags-2bfdff31905f">Testing feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-risk-mitigation-3238c1a247d9">Risk mitigation</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-performance-and-resilience-69b7351abe56">Performance and Resilience</a></li></ul><p>This article delves into the various types of feature flags, a powerful technique that allows teams to modify system behavior without changing code. Understanding the different types of flags — Release, Experimentation, Operational, and Permission — is crucial for effective feature management. Whether you’re a developer, a product manager, or a system administrator, this guide will help you comprehend the utility and application of each flag type.</p><h3>Types of Feature Flags</h3><p>Categorizing feature flags helps in better management and understanding of their purpose.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*lv2leN-CfKf4LmNqo_7quw.png" /></figure><p>The main classification is based on the flag’s purpose:</p><ol><li>Release Flags</li><li>Experimentation Flags</li><li>Operational Flags</li><li>Permission Flags</li></ol><p>Two additional dimensions are time: short/long-lived and scope: user/system-based.</p><ul><li>short-lived — flag exists for a single release/experiment cycle</li><li>long-lived — flag exists for multiple (but still limited) release cycles</li><li>user-based — flag directly affects UX and what users see</li><li>system based — flag is controlling the operational aspect of the application</li></ul><h3>Release flags</h3><p>They are also known as release toggles and separate feature deployment from feature release. Using them correctly allows engineers to merge code to the main code base and deploy it to production without changing the application behavior. Once the feature is thoroughly tested and signed off for release, the flag can be toggled to make it available for all users.</p><ul><li>Purpose: to decouple the process of deploying code from releasing it to end-users.</li><li>Application: phased rollouts and canary releases, allowing engineers to expose new features to a small user base before a full-scale launch.</li><li>Stakeholders: software and QA engineers are the primary users of this type of flag. Engineers can merge code without affecting production, while QA teams use them to test new features in a live environment.</li></ul><h3>Experimentation flags</h3><p>They are often used in A/B testing and allow engineers to expose different versions of a feature to other groups of users. They are a valuable instrument to verify how users perceive feature changes, their impact on their behavior, and our bottom line. The key requirement with experimentation flags is to capture data about the exposure and which user got what.</p><p>Various stakeholders can then use data gathered during such experiments to make informed decisions about direction. Different stakeholders could perform such experiments for other purposes, such as engineering, comparing different content searching, or sorting algorithms.</p><ul><li>Purpose: used for A/B testing and feature experimentation by allowing user groups to experience different feature sets.</li><li>Application: optimizing user experience or comparing different algorithms (i.e., engagement with new chat vs. old chat).</li><li>Stakeholders: product managers set the criteria for A/B tests, while data analysts evaluate the results to inform future development.</li></ul><h3>Operational flags</h3><p>They are used to control the operational aspects of a system. They are handy for managing system behavior during different conditions or incidents. Operational flags are strictly system-based flags and should affect the application as a whole. They are instrumental during migrations from one back-end to another, or gradual adoption of a new type of service, etc. Other examples could include changing log level, throttle calls to the back-end to manage system load, enabling/disabling debug and maintenance modes, etc. However, these should be limited in scope and targeted towards a specific function of the application, such as while migrating to a new SSO provider; it could be helpful to be able to change the log level or throttle calls. This shouldn’t be the practice for features or system behavior that is considered standard.</p><ul><li>Purpose: control system behavior.</li><li>Application: dynamically adjusting system logging levels without requiring restarts, toggling maintenance, or limited functionality mode during incidents, etc.</li><li>Stakeholders: DevOps teams use the flags to manage system behavior during incidents, while system administrators may use them for routine maintenance tasks.</li></ul><h3>Permissions flags</h3><p>They control access to features based on user segmentation, which allows for fine-grained control over who and what they can access. Example: Early adopter users’ segment gets a new product experience; VIP customers get a different deposit experience, etc. These flags also enable engineers to test on production without affecting the general player population.</p><ul><li>Purpose: control feature access based on user roles or segments.</li><li>Application: early access, beta testing, customized user experience based on user segment.</li><li>Stakeholders: product managers may set these flags to segment user access, while customer success teams can toggle them to provide premium features to specific accounts.</li></ul><h3>Summary</h3><p>In summary, feature flags offer a versatile approach to dynamically managing features and system behavior. We’ve explored four primary types of flags: Release Flags for decoupling deployment from release, Experimentation Flags for A/B testing, Operational Flags for system management, and Permission Flags for user-based access control. Knowing when and how to use these flags can significantly enhance your software development lifecycle, providing flexibility, reducing risk, and enabling data-driven decisions.</p><p>This article is part of a series, if you enjoyed it you should check out some of the other parts as well:</p><ul><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-the-basics-f292e64a4b4b">The basics</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-types-of-feature-flags-b57fa613ce38"><strong>Types of feature flags</strong></a><strong> </strong>(you are here)</li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-life-cycle-3303c746d227">Life cycle</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-testing-feature-flags-2bfdff31905f">Testing feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-risk-mitigation-3238c1a247d9">Risk mitigation</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-performance-and-resilience-69b7351abe56">Performance and Resilience</a></li></ul><p>Want to learn more about DraftKings’ global Engineering team and culture? Check out our <a href="https://careers.draftkings.com/life-at-draftkings/engineering/?utm_source=medium&amp;utm_medium=medium&amp;utm_campaign=medium_blog">Engineer Spotlights</a> and <a href="https://careers.draftkings.com/jobs/?utm_source=medium&amp;utm_medium=medium&amp;utm_campaign=medium_blog">current openings</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b57fa613ce38" width="1" height="1" alt=""><hr><p><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-types-of-feature-flags-b57fa613ce38">Mastering Feature Flags: Types of Feature flags</a> was originally published in <a href="https://medium.com/draftkings-engineering">DraftKings Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Feature Flags: The Basics]]></title>
            <link>https://medium.com/draftkings-engineering/mastering-feature-flags-the-basics-f292e64a4b4b?source=rss-c923e3c536c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/f292e64a4b4b</guid>
            <category><![CDATA[feature-flagging]]></category>
            <category><![CDATA[managing-complexity]]></category>
            <category><![CDATA[feature-toggles]]></category>
            <category><![CDATA[operational-switches]]></category>
            <category><![CDATA[feature-flags]]></category>
            <dc:creator><![CDATA[Martin Chaov]]></dc:creator>
            <pubDate>Sat, 18 Nov 2023 08:47:23 GMT</pubDate>
            <atom:updated>2024-09-23T14:29:21.752Z</atom:updated>
            <content:encoded><![CDATA[<p>This article is the first one in a series, more advanced topics are covered in the subsequent issues:</p><ul><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-the-basics-f292e64a4b4b"><strong>The basics</strong></a> (you are here)</li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-types-of-feature-flags-b57fa613ce38">Types of feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-life-cycle-3303c746d227">Life cycle</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-testing-feature-flags-2bfdff31905f">Testing feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-risk-mitigation-3238c1a247d9">Risk mitigation</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-performance-and-resilience-69b7351abe56">Performance and Resilience</a></li></ul><h3>What Are Feature Flags?</h3><p>Feature flags, also known as feature toggles, have become a cornerstone in modern software development, enabling teams to deliver features incrementally, test in production, and respond quickly to market changes. This guide covers everything one needs to know about feature flags, from best practices to resilience and performance considerations. This article is not a tutorial on using a specific third-party service to handle feature flags but a guide on what is required to run a successful operation depending on feature flags.</p><p>Feature flags are conditional code statements that allow developers to toggle features on or off without redeploying the application. They facilitate continuous delivery, enabling code to be merged into the main codebase and deployed while keeping new or incomplete features hidden from the end-users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*mPzwRMOx7rrQ_bKEgMltZQ.png" /><figcaption>A feature flag can be configured to serve different values based on a user segment.</figcaption></figure><h3>Feature Flags != Remote Configuration</h3><p>While feature flags and remote configurations control system behavior, they serve different purposes. Feature flags control access to specific features, whereas remote configurations set operational parameters. Feature flags are naturally binary, on or off, while remote configurations can manage more complex settings.</p><p>Feature flags (a.k.a. feature toggles) are a way for engineering teams to instrument an application to turn features on or off under specific conditions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*KTzsiaKFu8J1p8WkKeodFA.png" /><figcaption>A feature flag is a fine-grained business behavior control mechanism. While achieving the same with RC is somewhat possible, there are <strong>brutal</strong> fundamental differences.</figcaption></figure><p>With remote configuration, an application typically retrieves a block of settings from its back-end or configuration service. Then, the application configures its settings and features based on it.</p><ul><li>Configurations are settings that control the behavior of a system. They are typically used to set the system&#39;s parameters or operating options.</li><li>Configurations are often set at startup and do not change while the system is running unless the system is restarted or specifically designed to allow dynamic configuration changes.</li><li>Testing in multiple environments</li></ul><h3>Why Use Feature Flags?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*XQBdlBIrPe5I4wUotDdFYQ.png" /></figure><ul><li><strong>Operational Risk Management:</strong> Safely test new features in a live environment.</li><li><strong>Product Experience Optimization:</strong> Conduct A/B tests to improve user experience.</li><li><strong>Revenue/Engagement Tests:</strong> Measure the impact of new features on key performance indicators.</li><li><strong>Decouple deployment and release</strong>: Engineers can deploy new features but not release them to the end users.</li></ul><h3>When to use feature flags</h3><ol><li>Gradual feature rollout — roll out new features to a small percentage of users while gradually ramping up the % while monitoring the overall system.</li><li>A/B Testing and Experimentation — show feature variations to determine the impact on various KPIs.</li><li>Canary releases — releasing features to a small segment of users: beta testers, early adopters, etc.</li><li>Trunk-based development and CD — used to turn off incomplete features but still merge into the codebase and deploy to production.</li><li>Operational switches — application level controls such as log level, rate limiting calls to BE, etc.</li><li>Permissions — use flags to control access to features based on user segmentation.</li></ol><h3>When not to use feature flags</h3><ol><li>Control of sensitive data — flags should not be used to control access to sensitive data or critical system functionality.</li><li>Managing uncertainty — feature flags are not a way to avoid deciding &quot;if&quot; to release a feature.</li><li>Permanent features — some features will always be enabled and won&#39;t need to be hidden behind a flag: login, deposit, withdrawal, etc.</li><li>Instead of configuration — feature flags should not substitute remote configuration.</li></ol><h3>Pitfalls of Feature Flagging</h3><h3>Increased code complexity</h3><p>Implementing a new Boolean flag adds two new code paths in an application, but it could also mean two different versions of a service (or entirely different back-end) running side-by-side. Multiple flags related to the same feature create a permutation of possible code paths. Example 3 flags with two possible states is 2³, which gives eight different code paths. Only a few of these eight permutations could be related to viable business flows, and the rest should not be possible — for which the application must ensure they are not.</p><h3>Increased testing complexity</h3><p>Increased cyclomatic complexity in the code comes with overhead for quality assurance. 3 Boolean flags can be combined in 8 ways; 10 gives 1024 possible combinations. This is added to the required features so a feature flagging system can manage them. This becomes exponentially more complex when flags get more than two possible values each.</p><h3>Overuse</h3><p>Some of the use cases for feature flags are prone to overuse. It is easy to push code to the stream behind a flag. However, is that code going to be ever released? Ideally — yes. However, with the dynamic nature of business niches and priority shifts, this code could become a &quot;zombie&quot; of a feature that will never be released. Still, it got merged into the codebase and deployed to production. It is an unfinished feature, never to be completed, hidden behind a flag that will never be turned on.</p><p>A problem with putting every feature behind a flag is that eventually, it is not only the different combinations of flags that should be tested but also compatibility between what features are enabled.</p><p>In addition, this could also lead to increased package size for native client applications.</p><h3>Glossary</h3><ol><li><strong>Feature Flags</strong>: Conditional code statements that allow developers to toggle features on or off without redeploying the application.</li><li><strong>Remote Configuration</strong>: A block of settings retrieved from a back-end or configuration service that controls system behavior.</li><li><strong>Phased Rollouts</strong>: The practice of releasing new features to a subset of users, gradually increasing the percentage of users who see the quality.</li><li><strong>Kill Switch</strong>: A feature flag that can immediately disable a feature in case of an issue.</li><li><strong>Chain of Dependencies</strong>: The interrelationship between multiple feature flags, where one flag&#39;s state may depend on another&#39;s state.</li><li><strong>Operational Risk Management</strong>: Using feature flags to test new features in a live environment safely.</li><li><strong>Product Experience Optimization</strong>: Using feature flags for A/B testing to improve user experience.</li><li><strong>Revenue/Engagement Tests</strong>: Experiments using feature flags to measure the impact of new features on key performance indicators (KPIs).</li><li><strong>Canary Releases</strong>: The practice of releasing features to a small segment of users, such as beta testers or early adopters.</li><li><strong>Trunk-Based Development</strong>: A practice where all developers work in a single branch, using feature flags to toggle incomplete features.</li><li><strong>Operational Switches</strong>: Feature flags for application-level controls like log levels or rate-limiting back-end calls.</li><li><strong>Cyclomatic Complexity</strong>: A measure of the number of linearly independent paths through a program&#39;s source code, increased using feature flags.</li><li><strong>Zombie Feature</strong>: A feature hidden behind a flag and never released, leading to codebase clutter.</li><li><strong>Package Size</strong>: The total size of the application, which can increase due to the overuse of feature flags in native client applications.</li></ol><h3>Summary</h3><p>Feature flags are potent tools for agile development, balancing speed and safety. Effective use requires a disciplined approach to testing, documentation, and ongoing lifecycle management. Understanding and implementing these best practices can maximize the benefits while mitigating risks.</p><p>This guide aims to be the go-to document for mastering feature flags, covering everything from basic concepts to advanced best practices. A standardized approach can ensure consistent, efficient, and safe use of feature flags across all projects.</p><p>This article is part of a series; if you enjoyed it, you should check out some of the other parts as well:</p><ul><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-the-basics-f292e64a4b4b"><strong>The basics</strong></a> (you are here)</li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-types-of-feature-flags-b57fa613ce38">Types of feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-life-cycle-3303c746d227">Life cycle</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-testing-feature-flags-2bfdff31905f">Testing feature flags</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-risk-mitigation-3238c1a247d9">Risk mitigation</a></li><li><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-performance-and-resilience-69b7351abe56">Performance and Resilience</a></li></ul><p>Want to learn more about DraftKings’ global Engineering team and culture? Check out our <a href="https://careers.draftkings.com/life-at-draftkings/engineering/?utm_source=medium&amp;utm_medium=medium&amp;utm_campaign=medium_blog">Engineer Spotlights</a> and <a href="https://careers.draftkings.com/jobs/?utm_source=medium&amp;utm_medium=medium&amp;utm_campaign=medium_blog">current openings</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f292e64a4b4b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/draftkings-engineering/mastering-feature-flags-the-basics-f292e64a4b4b">Mastering Feature Flags: The Basics</a> was originally published in <a href="https://medium.com/draftkings-engineering">DraftKings Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lazy rendering Web UIs with IntersectionObserver API]]></title>
            <link>https://medium.com/draftkings-engineering/lazy-rendering-web-uis-with-intersectionobserver-api-bc69a4b61325?source=rss-c923e3c536c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc69a4b61325</guid>
            <category><![CDATA[lazy-rendering]]></category>
            <category><![CDATA[virtualized-rendering]]></category>
            <category><![CDATA[intersection-observer]]></category>
            <category><![CDATA[lazy-loading]]></category>
            <category><![CDATA[windowing]]></category>
            <dc:creator><![CDATA[Martin Chaov]]></dc:creator>
            <pubDate>Wed, 16 Aug 2023 15:56:44 GMT</pubDate>
            <atom:updated>2024-09-23T15:42:49.938Z</atom:updated>
            <content:encoded><![CDATA[<h3>Optimizing Web UI Performance: A Deep Dive into Lazy Rendering and IntersectionObserver API</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*_llDXDY7rmvm5JgFD8KE6w.png" /></figure><p>Lazy rendering, is a design pattern in web development that delays the initialization or rendering of elements until they are needed. This technique is particularly useful for improving the performance and efficiency of web applications, especially those which need to present large amounts of content to their users.</p><p>Traditionally, web pages load all elements at once, which can lead to slower load times and unnecessary data usage. However, with lazy rendering, only what is necessary is loaded while the rest loads on-demand based on an user-controlled trigger such as scrolling. If utilized correctly, this technique could reduce initial load time, and save bandwidth and system resources.</p><p>Lazy rendering can be implemented using various techniques, including JavaScript events, IntersectionObserver API, and third-party libraries.</p><p>At DraftKings, we utilize lazy rendering in a few ways:</p><ul><li>triggering download of data when user scrolls to a certain place and rendering data as soon as it is ready</li><li>quickly downloading data in the background but not rendering it until needed</li></ul><p>Both techniques have their use cases and could mean the difference between sluggish application and a satisfying user experience.</p><p>The goals of lazy rendering are to:</p><ul><li>improve perceived performance</li><li>reduce various resource utilization</li></ul><p>This article is going to present a distilled solution using IntersectionObserver API and TypeScript.</p><h3>What is IntersectionObserver?</h3><p>“The Intersection Observer API provides a way to asynchronously observe changes in the intersection of a target element with an ancestor element or with a top-level document’s viewport.” — MDN</p><p>To simplify — the IntersectionObserver executes a callback function when a specified HTML element enters the viewport (“root” element in the example below).</p><pre>const observer = new IntersectionObserver(<br>    handleIntersection,             // reference to callback function<br>    {<br>        root: htmlElementReference, // which element acts as viewport, null for document&#39;s viewport<br>        rootMargin: &quot;0px&quot;,          // margin around the target to add to the size of the box before calculating visibility, defaults to zero<br>        threshold: 1.0              // how much of the element should be visible before triggering the callback<br>    }<br>);<br> <br>observer.observe(htmlElement)       // reference to HTML element to check for visibility against the viewport</pre><p>The options object has additional intricacies which are not discussed in this article. For more details, check the IntersectionObserver parameters interface <a href="https://developer.mozilla.org/en-US/docs/Web/API/IntersectionObserver/IntersectionObserver#parameters"><strong>here</strong></a>.</p><h3>Example Implementation</h3><p>The following example takes into account these guidelines:</p><ul><li>do it more than once — abstract away the observer logic</li><li>make it reasonably optimal — no overoptimization, include basic things like synchronization with the animation frame</li><li>have a manageable scrolling experience — use a placeholder element to keep the scrollbar position and size consistent</li><li>use one observer per viewport</li><li>register different handlers for different elements</li></ul><p>If you want to jump to the code repo in GitHub and start tinkering, <a href="https://github.com/mchaov/lazy-renderer-ts-react"><strong>here is the example code</strong></a><strong>. </strong>There is also a code pen linked further down in the article.</p><h3>Renderer class</h3><p>Abstracting away some of the logic in a reusable class.</p><pre>/**<br> * IntersectionObserverInit<br> * Setup some reasonable defaults to make initialization straight forward<br> * @see https://developer.mozilla.org/en-US/docs/Web/API/IntersectionObserver#constructor<br> */<br>const defaultsOptions: IntersectionObserverInit = {<br>    // root: null,      // null === document&#39;s viewport<br>    rootMargin: &quot;0px&quot;,  // assume the box doesn&#39;t need margins<br>    threshold: .01      // assume the box should show as soon as the first pixels should become visible<br>};<br> <br>/**<br> * VisibilityChangeCb<br> * Export a type that will make it easier for implementers to work with callbacks<br> * @see https://developer.mozilla.org/en-US/docs/Web/API/IntersectionObserverEntry<br> */<br>export type VisibilityChangeCb = (e: IntersectionObserverEntry) =&gt; void<br> <br>/**<br> * Renderer<br> * This class should be instantiated for every viewport that we would like to control.<br> * There could be multiple instances for one viewport. As with everything, performance impact<br> * should be measured.<br> * @example<br>    const observer = new Renderer()<br>    observer.registerDom(<br>        htmlElementReference,<br>        intersection =&gt; entry.isIntersecting ? &quot;Yes&quot; : &quot;No&quot;<br>    )<br> */<br>export class Renderer {<br> <br>    private observer: IntersectionObserver                              // observer instance<br>    private animationFrameID: number = -1                               // animation frame id for cleaning<br>    private callBacks: Map&lt;Element, VisibilityChangeCb&gt; = new Map()     // store all callbacks<br> <br>    /**<br>     * Constructor<br>     * @param {IntersectionObserverInit} props default options for the IntersectionObserver constructor<br>     * @returns {Renderer} renderer instance<br>     */<br>    constructor(props: IntersectionObserverInit = {}) {<br> <br>        this.observer = new IntersectionObserver(<br>            this.scheduleUpdate,<br>            {<br>                ...defaultsOptions, ...props                            // merge defaults with user provided options<br>            }<br>        );<br>    }<br> <br>    /**<br>     * registerDom<br>     * Start observing the specified element and store its callback for later use<br>     * @param {HTMLElement} element reference to HTML element<br>     * @param {VisibilityChangeCb} callback function that takes IntersectionObserverEntry as parameter<br>     * @void<br>     */<br>    registerDom(element: Element, callback: VisibilityChangeCb) {<br>        this.callBacks.set(element, callback)<br>        this.observer.observe(element)<br>    }<br> <br>    /**<br>     * unRegisterDom<br>     * Remove element from observe list and delete theh reference to its callback<br>     * @param {HTMLElement} element reference to HTML element<br>     * @void<br>     */<br>    unRegisterDom(element: Element) {<br>        this.callBacks.delete(element)<br>        this.observer.unobserve(element)<br>    }<br> <br>    /**<br>     * Schedule render at the next convenient moment<br>     * @param {IntersectionObserverEntry[]} entries automatically passed by the IntersectionObserver upon invocation<br>     * @void<br>     */<br>    private scheduleUpdate = (entries: IntersectionObserverEntry[]) =&gt; {<br> <br>        window.cancelAnimationFrame(this.animationFrameID)              // drop the previous frame if it was not rendered<br> <br>        this.animationFrameID = window.requestAnimationFrame(() =&gt; {    // setup next animation frame<br>            this.updateVisibility(entries)                              // execute the callbacks of each registered element<br>        })<br> <br>    }<br> <br>    /**<br>     * updateVisibility<br>     * The callback function invoked when the browser schedules an animation frame<br>     * @param {IntersectionObserverEntry[]} entries all observable entries that were registered<br>     * @void<br>     */<br>    private updateVisibility = (entries: IntersectionObserverEntry[]): void =&gt; {<br>        entries.forEach(e =&gt; {<br>            const callback = this.callBacks.get(e.target)<br>            if (typeof callback === &quot;function&quot;) {<br>                callback(e)<br>            }<br>        })<br>    }<br>}</pre><p>Using this class would be quite simple.</p><pre>import { Renderer } from &quot;./renderer&quot;<br> <br>/**<br> * intersectionHandler<br> * Function is called each time the viewport detetcts a change<br> * @param {IntersectionObserverEntry} entry<br> * @void<br> */<br>function intersectionHandler(entry: IntersectionObserverEntry) {    // takes an entry and processes it<br>    if (entry.isIntersecting) {                                     // do something if content is visible<br>        entry.target.classList.add(&quot;isVisible&quot;)<br>    } else {                                                        // do something if content is not visible<br>        entry.target.classList.remove(&quot;isVisible&quot;)<br>    }<br>}<br> <br>const observer = new Renderer()                                     // bootstrap an observer<br> <br>const boxes = document.querySelectorAll(&quot;.box&quot;)                     // get all of the boxes we wish to observe<br> <br>boxes.forEach(x =&gt; observer.registerDom(x, intersectionHandler))    // register all of the boxes with the observer</pre><p>Below is a working demo that is also adding and removing content based on visibility.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcodepen.io%2FMchaov%2Fembed%2Fpreview%2FPoxKbPQ%3Fdefault-tabs%3Djs%252Cresult%26height%3D600%26host%3Dhttps%253A%252F%252Fcodepen.io%26slug-hash%3DPoxKbPQ&amp;display_name=CodePen&amp;url=https%3A%2F%2Fcodepen.io%2FMchaov%2Fpen%2FPoxKbPQ&amp;image=https%3A%2F%2Fshots.codepen.io%2Fusername%2Fpen%2FPoxKbPQ-512.jpg%3Fversion%3D1688711407&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=codepen" width="800" height="600" frameborder="0" scrolling="no"><a href="https://medium.com/media/79b897d9442c7d24e8c9623800229726/href">https://medium.com/media/79b897d9442c7d24e8c9623800229726/href</a></iframe><h3>Best practices</h3><p>In our applications at DraftKings, we often have enormous amounts of content to present to our customers. Especially in our Sportsbook, where we also utilize push updates, the content should be up to date to enable users to make informed decisions. A phone screen or even desktop screen cannot fit this amount of information in a comprehensible way, so the users end up seeing one screen worth of data at a time. While there is content above and below the point where the user has scrolled, all updates for that content are not relevant until they are presented on the screen, so they could wait for a more appropriate time to be propagated to the Document Object Model (DOM).</p><p>Below are some thoughts to consider when designing and implementing lazy rendering.</p><h4>Batching content</h4><p>Batching refers to the practice of loading multiple elements at once, rather than individually. There are different types of batching in relation to lazy rendering:</p><ul><li>batching when downloading content — for example, pulling the next page when user scrolls near the bottom of the current one</li><li>batching when rendering content — proactively adding or removing content from the DOM to optimize the performance of a page by reducing the amount of nodes the browser has to keep in memory</li></ul><p>Batching when downloading tends to be more efficient than downloading single entities for both the client and the backend, but also in terms of network utilization. Especially when using HTTP, a lot of the payload when downloading small amounts of data tends to be HTTP headers. In that case, it becomes more efficient to download content in batches (or pages, if you will).</p><p>Batching when rendering is a different type of a beast. Rendering a lot of individual elements all the time could end up increasing the number of reflows and repaints which leads to layout thrashing. Batching is a balancing act, as loading too many elements at once can also slow down the rendering of the application.</p><p>Finding the sweet spot between how much and when it should be rendered can be tricky. One way to manage this is by using the <a href="https://developer.mozilla.org/en-US/docs/Web/API/window/requestAnimationFrame"><strong>requestAnimationFrame API</strong></a> and making sure frames are executed within a certain time threshold.</p><p>Lazy rendering has design implications on the backend services as well. Instead of getting everything with one call, content is being pulled based on some type of user interaction. The backend should be tailored to enable such implementations by providing a reasonable low latency API with variable pagination support and content filtering capabilities. Moreover, depending on the use case, the backend should also provide metadata such as: amount of available content, rate limit for number of calls, counters for different types of content, and others.</p><h4>Testing</h4><p>Testing lazy rendering can be quite challenging. Apart from simply testing to see if content generates, UX should be tested to make sure content is presented at the right time to ensure a smooth experience. This may include testing how the application behaves in different network conditions, device types, etc. Tests should inform the engineers if there should be an adjustment to the user experience based on the conditions. Moreover, they should be designed to provide meaningful feedback beyond the basic pass/fail. Some points to consider:</p><ul><li>Testing edge cases such as rapid scrolling up/down or sudden changes in network conditions to stress the lazy rendering implementation.</li><li>Tracking metrics such as CPU and memory usage, number of calls to the backend, download times, number of DOM elements, backend load.</li><li>How lazy rendering affects perceived performance of the application? Is there any scroll jumping or other disruptive behaviors?</li></ul><p>IntersectionObserver works directly with the DOM and since it is hard to mock for unit tests, E2E tests would be required to validate its use. A work-around for rendering DOM in memory would be to substitute the real IntersectionObserver with a mock implementation that is always returning visible elements, thus you could verify what is being generated. However, this approach leaves showing and hiding untested and waiting for a manual test or additional automation created specifically for it.</p><h4>Avoid scroll jumping</h4><p>Depending on the use-case, scroll jumping might not be 100% avoidable, but there are definitely ways to lessen its impact. Scroll jumping can occur when elements are loaded in as the user scrolls, thereby changing the page length, thus forcing the browser to re-calculate the scroll size and position. This can be disorienting for the users. To improve this, placeholders could be used to reserve space for elements before they’re loaded. From our experience, placeholders work best when they can expand with the “min-height” CSS property set to a reasonable number, rather than fixed height or auto. Having zero height placeholders would just trigger all content to be loaded as soon as the page becomes operational.</p><h4>Handling re-renders</h4><p>Re-renders could be costly in terms of performance. Techniques like memorization could help reduce them by sacrificing some memory to unload the CPU. While one of the goals of lazy rendering is to reduce resource utilization, this technique still has some cost associated with it. One way to keep things in check is to remove elements from the DOM but keep them referenced so that the garbage collector won’t delete them, and insert them back when appropriate. This added complexity could be too much for some use-cases though.</p><h4>Performance</h4><p>IntersectionObserver is just an enabler and depending on how it is being implemented and used it could improve things quite significantly. However, this is also a tradeoff → adding complexity to the code to improve the users’ perception and resource allocation. IntersectionObserver could be used together with list virtualization (a.k.a. windowing) and DOM reuse to drop the number of elements on the page to the bare minimum, 1.5–2 screens worth of content. Batching could also significantly improve but might also hurt performance if not appropriate for the use case.</p><h3>Downsides of lazy rendering</h3><ol><li>Complexity — implementing lazy rendering in any shape or form (even someone else’s NPM package) is adding complexity to the codebase and testing process. It requires careful management of state and timing, and might not be fully compatible with all libraries and frameworks.</li><li>SEO impact — search engines may not be able to fully render the content that is loaded lazily, especially if it is triggered only by user interaction.</li><li>Searching issues — similarly to how search engines cannot search content that is not on the page, users who try to do it with “ctrl+f” will face the same challenges. Engineers would have to add additional searching capability to their web sites so that users could find what they need.</li><li>Initial load times — lazy rendering could improve overall performance. However, it could also increase initial loading times, as the browser has to download and execute the lazy rendering logic before it could start showing content.</li><li>Increased complexity in managing backend calls — since content is loaded piece by piece and triggered by user interaction it could be challenging to discover potential use-cases and issues coming from them before releasing it to a wider audience.</li><li>Libraries and frameworks compatibility — modern JS frameworks hide a lot of the complexity of managing the DOM by abstracting away the updates and rendering from the engineer. IntersectionObserver is used to do DOM level optimizations and requires direct access to the references of the DOM elements. Moreover, to pull it off requires intimate knowledge of both the DOM and the library in question.</li></ol><p>Want to learn more about DraftKings’ global Engineering team and culture? Check out our <a href="https://careers.draftkings.com/life-at-draftkings/engineering/?utm_source=medium&amp;utm_medium=medium&amp;utm_campaign=medium_blog">Engineer Spotlights</a> and <a href="https://careers.draftkings.com/jobs/?utm_source=medium&amp;utm_medium=medium&amp;utm_campaign=medium_blog">current openings</a>!</p><h3>References</h3><ul><li><a href="https://github.com/mchaov/lazy-renderer-ts-react">GitHub repo with the example code.</a></li><li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API">IntersectionObserver documentation.</a></li><li><a href="https://web.dev/avoid-large-complex-layouts-and-layout-thrashing/">Layout thrashing</a></li><li><a href="https://web.dev/how-to-optimize-inp/">Optimizing interaction to next paint</a></li><li><a href="https://stackoverflow.com/questions/66450070/are-layout-thrashing-reflow-the-same-meaning-in-html">Reflow vs layout thrashing</a></li><li><a href="https://developer.mozilla.org/en-US/docs/Web/API/window/requestAnimationFrame">Request animation frame API</a></li><li><a href="https://www.npmjs.com/package/react-intersection-observer">Popular React Hook</a></li><li><a href="https://react-window.vercel.app/#/examples/list/fixed-size">React-window</a></li><li><a href="https://bvaughn.github.io/react-virtualized/#/components/List">React-virtulized</a></li><li><a href="https://blog.bitsrc.io/angular-maximizing-performance-with-the-intersection-observer-api-23d81312f178">Angular and IntersectionObserver</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc69a4b61325" width="1" height="1" alt=""><hr><p><a href="https://medium.com/draftkings-engineering/lazy-rendering-web-uis-with-intersectionobserver-api-bc69a4b61325">Lazy rendering Web UIs with IntersectionObserver API</a> was originally published in <a href="https://medium.com/draftkings-engineering">DraftKings Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lessons Learned: WebSocketAPI at scale]]></title>
            <link>https://medium.com/draftkings-engineering/lessons-learned-websocketapi-at-scale-604617a54cdb?source=rss-c923e3c536c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/604617a54cdb</guid>
            <category><![CDATA[websocket-api]]></category>
            <category><![CDATA[backpressure]]></category>
            <category><![CDATA[websocket-api-at-scale]]></category>
            <category><![CDATA[protocol-efficiency]]></category>
            <category><![CDATA[load-balancing]]></category>
            <dc:creator><![CDATA[Martin Chaov]]></dc:creator>
            <pubDate>Fri, 05 May 2023 13:43:10 GMT</pubDate>
            <atom:updated>2024-02-16T06:54:19.620Z</atom:updated>
            <content:encoded><![CDATA[<p>Here at Draftkings Inc. we utilize the WebSocket API for a variety of use-cases where low latency and push updates are required. Websockets are a powerful tool for real-time communication, but managing them at a large scale can present a number of interesting challenges. This article will explore some of these challenges, including:</p><ul><li>State management</li><li>Load balancing long-lived connections</li><li>Network and application security</li><li>Backpressure</li><li>Protocol efficiency</li><li>Application performance monitoring (APM)</li></ul><p>These challenges can vary depending on the characteristics of the business and the system in question; whether it is an internal system, if there a need for an SDK to abstract communication, and the scale and expected system load of the operation.</p><p>This article is not intended to provide a definitive answer on how to handle these challenges, but to provide an overview of the key issues to consider when managing WebSockets at scale. The solutions discussed will also depend on the specific requirements of the business and the system. Furthermore, it’s good to mention that there are plenty of tutorials available online showing how to set up WebSockets in a variety of languages such as TypeScript, Swift, Kotlin, C#, and Java.</p><p><strong>Important to note: </strong>Some of the difficulties with the WebSocket API are inherited from the underlying abstractions (TCP/IP) while others are broadly applicable to any request/response type of communication. In that sense, none of the challenges discussed in this article are inherently a “WebSocket API problem.” They are applicable to any form of streaming communication.</p><h3>When to use a WebSocket?</h3><p>WebSocketAPI is used for real-time, bi-directional communication between clients and servers over the web. It is particularly useful in applications that require low-latency, high-frequency communication, such as:</p><ul><li>Chat/video conferencing</li><li>Collaboration tools</li><li>Multiplayer games</li><li>Financial applications</li><li>IoT applications</li><li>Data visualization</li><li>Notifications or alerts</li><li>Real-time analytics</li><li>Location tracking</li><li>Online education</li><li>Social media applications</li></ul><h3>Challenges</h3><p>The majority of challenges with WebSocket API come from the lack of documentation on the <strong>what </strong>and <strong>how </strong>beyond their basic “how-to” implementation. Most developers who face using WebSockets usually try to find a library that helps them deal with the most common things such as handling of reconnects, fallbacks to polling, and helping them with some of the intricacies of the protocol. Let’s start with what tutorials show vs what actually is needed.</p><p>This is the most common explanation of how one uses WebSocket API:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZWUIljxI5EiGPEW10Eh8Hw.png" /><figcaption>Most common example of how a WebSocket connects client to server.</figcaption></figure><p>However, there is a stark difference with reality. Systems that use the WebSocket API tend to try to use a fewer number of connections per client, thus they start to pipe different APIs and services through one (or few) connection. Eventually, the server that maintains the connections with the end users won’t be able to do much more than just that and if any business logic was somehow added there it would end up getting extracted into other services and/or APIs. Otherwise, the challenges with scaling would be disproportional to the load experienced by the system.</p><p>If the system is optimized for a number of connections per user (one of the most common cases), data is going to be piped from different services through the same server. Handling this doesn’t require a lot of resources. However, the number of connection a server could hold concurrently regardless of the amounts of available CPU and/or RAM could be exhausted very quickly. The server that maintains the WebSocket connections is probably getting replicated to handle more and more load from the clients, which means a load balancer is required to spread the load.</p><p>The picture above quickly transforms into something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8FVq7BuT-kDEu0DfJFp05A.png" /><figcaption>A more representative view of a typical system, while omiting a lot of detail.</figcaption></figure><p>Note that on the figure above there is no notion of: data centers, geographical locations, scaling of the back-end services, CDNs, WAFs, service discovery, clustering, etc., which all bring additional complexity omitted for the sake of clarity.</p><h3>State Management</h3><p>Once there is an open connection maintained between two entities, there is state. This state alone is not very interesting as it is the bare minimum required for the connection to exist. What is more interesting, is how data flowing through the connection is organized. Something quite common over the Internet (especially mobile) is for a connection to drop. If dropped, should the client connect to the same server instance to complete their transaction? Is the system implementation depending on the so-called “sticky session?”</p><p>Maintaining state information for each connected client can be resource-intensive, since the server needs to store and process this information. This can impact the performance and scalability of the system. When scaling WebSockets across multiple servers, it can be difficult to keep the state information for each client synchronized across all of the servers.</p><p>If the data delivered to the client is personalized, a way to optimize delivery is to employ caching on the server, so clients’ requests could be answered faster. In that case, the client would be required to always hit the same server where the data is already cached. This is called session persistence, a.k.a. sticky session. It involves directing a client’s requests to one application or backend web server for the duration of a “session.” The session is the time it takes a client to complete a transaction or task that might include multiple requests.</p><p>Utilizing sticky sessions could inadvertently lead to having them in multiple places:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ruc_V77fmKat9IH25a9DOw.png" /></figure><p>The impact of interrupting the session would be different for the different systems. Possibly, it could mean a user might get logged out, or a transaction of some kind might be interrupted and reverted and the user would have to start over.</p><p>Sticky sessions limit the ability of a system to:</p><ul><li>recover, in case of system failure — clients will have to start their transactions from scratch if they get interrupted. There would be some inherent complexity on the backend related to that possibility — revert, timeout, or throw away mechanism would need to handle that along the chain of services/APIs.</li><li>rolling deployments and restarts — clients’ transactions have to be completed before closing the connection and stopping the server. In some cases, there is no way to know how long that could take.</li><li>scale, load balance and re-balance — expanded upon in the next chapter.</li><li>scale based on resource utilization — if a sticky session requires too many resources and the load balancer needs to move some of the clients to another server, this would have the same implications as with recovery in cases of system failure and rolling deployments.</li></ul><p>Persisting clients’ progress for long-lived sessions could be moved into an external store like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6gMioTM__AizbvIjm0aQDw.png" /><figcaption>Persisting clients’ progress in a shared resource.</figcaption></figure><p>But now there is yet another component that could become a:</p><ul><li>New bottleneck;</li><li>Single point of failure;</li><li>Scaling issue;</li><li>Complication with data sharding;</li><li>Source of latency;</li></ul><h3>Load Balancing Long-Lived Connections</h3><p>WebSockets are long-lived connections and as such, they bring a lot of interesting challenges with load balancing. Assuming clients’ session persistence is solved, the next item on the list is to properly load balance these sessions across the infrastructure. Regardless of the type of load-balancer or the technique used (round-robin, random, etc.), there are precautions to put in place.</p><p>Starting from the example of a perfectly balanced system load:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oTOsntd35INxZCz64PCMqg.png" /><figcaption>Balanced system</figcaption></figure><p>If a WebSocket-based app is running on a few servers and the load balancer is distributing connections evenly across them, it won’t redistribute any of the already existing traffic, in case of re-sizing the system. If a server dies, the load balancer is going to spread its traffic onto the other servers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bclruARkFgguZDpodJoVCQ.png" /><figcaption>Event that forces clients to move to a different instance of the server.</figcaption></figure><p>However, once a new server instance is added back to the load balancer, there would be no reason for it to redistribute the traffic. The system load could start to look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jSfLpRLIAColnGpMTM140w.png" /><figcaption>New server comes online but there is no reason to re-balance the system as connections are not idle.</figcaption></figure><p>Depending on the load balancing technique, there are a variety of ways how new and returning traffic could be distributed in the example system above. The important thing to note is there is no reasonable way to move connections from one server to another to re-balance the system! Servers might not even be in the same physical location…</p><p>Because, by their very nature, WebSocket connections are persistent, they remain on whatever node they were first established. Some of the reasons to force re-balancing of the system are:</p><ul><li>Servers were overloaded and restarted</li><li>Rolling deployments</li><li>Infrastructure maintenance</li><li>System failure</li><li>Scale out</li></ul><h4>How to return to proper load balancing again?</h4><p>The nature of the business domain could give clues about what is reasonable to do or not to do. However, there is a way to make this process predictable and manageable. To load balance long-lived connections, you should cut them off periodically and force the clients to reconnect. Don’t use WebSockets without planned periodic interruptions at the gateway level! The rest is used for varying levels of optimizations that could be done to improve the UX.</p><p>The client applications must be aware of the cutoff and apply a variety of mechanisms, such as:</p><ul><li>reconnecting and retrying mechanism with circuit breaker and timeout</li><li>randomized reconnect strategy</li><li>incremental back-off</li><li>data validation and recovery</li><li>state reconciliation</li></ul><p>Making periodic interruptions part of the design of the system would force the clients to handle it gracefully.</p><h3>Backpressure</h3><p>Backpressure is a term borrowed in software engineering from fluid dynamics. From <a href="https://en.wikipedia.org/wiki/Back_pressure">Wikipedia:</a> <em>backpressure is a resistance or force opposing the desired flow of fluid through pipes, leading to friction loss and pressure drop</em>. In software systems we call backpressure the situation when producers of information overwhelm consumers with messages (or: queries, requests, data) they are not able to process. Eventually, people also apply this term as the mechanism to control and handle it.</p><p>When using Service-Oriented Architecture (SOA) and WebSockets, backpressure can occur when the system is not able to handle the volume of requests or data it is receiving. But backpressure could also happen on the network, based on CWND (congestion window) and RWND (receive window) which could lead to TCP send/receive buffers overflow.</p><p>When using push over WebSocket APIs there are a few different scenarios that could arise:</p><ul><li>A client could be sent too many messages and can’t process them, and the TCP receive buffer will overflow</li><li>Client may advertise a smaller receive window than the amount of data you want to push so the sender’s TCP send buffer will overflow</li><li>An unreliable network won’t be able to deliver all packets you are trying to send and retransmission may lead to TCP send buffer overflow</li><li>In client’s case the resources of the device could get exhausted (RAM, CPU, storage).</li><li>In the network’s case the server that is trying to send the messages will eventually exhaust it’s packet’s queue. [7]</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NEbm6ZXF-jJSWT9iDVZYtw.png" /><figcaption>Inability for client to handle or even receive the traffic could force a system to overflow its TCP send buffer.</figcaption></figure><p>Whenever backpressure occurs, it is usually a symptom of inefficient communication protocol. In that case, the problem is not the WebSocket API per say, but what is being sent over it and with what frequency. There are a few ways to deal with backpressure. Some would require design up-front, some could be automated as a defensive technique.</p><p>The results of backpressure could be:</p><ul><li>Performance degradation: When a system is under backpressure, it may not be able to process requests or data as quickly as it normally would, which can lead to performance degradation.</li><li>System failures: If a system is under severe backpressure, it may be unable to process requests or data at all, which can lead to system failures.</li><li>Data loss: If a system is under backpressure, it may be unable to process incoming requests or data in a timely manner, which can lead to data loss.</li></ul><h4>Compaction</h4><p>In telecommunication, data compaction is the reduction of the number of data elements, bandwidth, cost, and time for the generation, transmission, and storage of data without loss of information by eliminating unnecessary redundancy, removing irrelevancy, or using special coding. [8] A good example of data compaction could be using acronyms while chatting: “ASAP” vs “as soon as possible”, and “LOL” vs “laughing out loud”.</p><p>Compaction is a technique that can be used to mitigate backpressure in a system by reducing the amount of data that needs to be processed. Compaction involves removing or consolidating duplicate or unnecessary data from the system, which can help reduce the load on the system and improve its performance.</p><p>There are several compaction strategies that can be used to mitigate backpressure, including:</p><ul><li>Data deduplication</li><li>Data compression</li><li>Data aggregation</li><li>Data pruning</li></ul><p>Compaction also has trade-offs, such as increased complexity or reduced data accuracy. It is important to carefully evaluate the benefits and drawbacks of each strategy and choose the one that is most appropriate for the specific system and use case.</p><p>A few considerations for using data compaction are:</p><ul><li>Precision → would the client application make sense of number changes after a certain digit behind the decimal point — 2.150000001 vs 2.150000075 vs 2.15. Does it make sense to distribute such changes over the wire?</li><li>Frequency of change vs delivery time→ if updates are produced every 10 ms. but the data is sent over the Internet where the average latency is about 150 ms, maybe some could be skipped and fewer changes could be sent within a certain threshold.</li><li>Criticality of the outdated data → this really depends on what the client application is doing with the data. Is it calculating trends based on multiple changes so it needs all of them, or it is just displaying the current price for a user to pick? Is that update needed for the shopping cart before the customer submits an order? Or is it just a price list displaying many items at the same time? Questions like these could help determine the frequency of updates required based on use-case.</li><li>Usefulness of the data → maybe this is read by a human and they won’t be able to (or shouldn’t) react on every change, or maybe the screen they are watching can’t display everything at once.</li></ul><p>Effective techniques for compaction:</p><ul><li>Sparse contracts → sending only differences rather than whole entities.</li><li>Bulks → bulking the data to reduce frequency of updates and optimize network transport. See “Buffering” below.</li></ul><h4><strong>Compression</strong></h4><p>Compression removes strings of repeated characters, thereby reducing the amount of data that must be sent over the communication channel. Data compression is a useful mechanism to reduce the payload size while retaining the amount of content sent. The WebSocket API has a compression extension [9]. This compression could be a quick way to optimize the size of the generated traffic. However, it is not as efficient as some other ways, such as using a binary protocol like ProtoBuf instead of the text-based JSON.</p><h4><strong>Control over the producer</strong></h4><p>Controlling the producer of the messages, when it is actually possible, is one very good option for handling backpressure. Unfortunately, that is rarely possible. Feeds of events on the stock market, sporting events, multiplayer games, and other’s rate of generating messages are rarely controllable by the client. In fact, many of them are optimized to deliver changes as soon as they occur and the client software must be designed to handle that.</p><p>A good example of limiting the producer is when, in video games, there is a VSYNC option to limit the frame rate so that the hardware won’t generate frames that the display cannot show.</p><h4><strong>Buffering</strong></h4><p>Buffering is when data is loaded into memory with the intent of offloading its processing for a later stage. It could be dangerous if left unbounded — meaning the buffer must have a size limit. Unbounded buffers are a common source of memory crashes in servers. When used, a good question to ask is: is it possible that the rate the buffer grows will ever exceed the rate it is drained for any appreciable amount of time?</p><h4><strong>Dropping</strong></h4><p>Dropping is the last strategy, and as mentioned it’s often combined with buffering too. Doing this could help prevent the system from becoming overwhelmed and allow it to continue operating efficiently. When this process is not controlled, the consequence of it happening are hard to predict and may lead to unexpected system states. The impact of the dropped messages must be considered and must ensure that what was lost was not critical to the system’s operation. As an example, the system might be dropping messages older than X amount of time, or overwrite messages on the same topic.</p><p>Where possible the client application should be able to subscribe to only what it needs and be delivered over an efficient protocol.</p><h3>Protocol Efficiency</h3><p>There are several issues that can arise with protocol efficiency when using WebSocket APIs, including:</p><ul><li>Bandwidth consumption: WebSocket protocols can consume a significant amount of bandwidth, which can be a problem for systems that have limited bandwidth or that are experiencing high levels of traffic.</li><li>Latency: WebSocket protocols can introduce latency into the communication process, as the data needs to be transmitted and processed by the server before a response can be sent back to the client.</li><li>Overhead: WebSocket protocols can add overhead to the communication process, as they require additional data to be transmitted and processed to establish and maintain the connection.</li></ul><p>It is tempting to say that WebSocket abstracts communication to make it easy, but one of the most commonly leaking abstractions is the underlying network protocol — TCP/IP. WebSockets are a frame-based protocol. By utilizing frames and having a message-based protocol (vs. a stream based protocol) it is easier to write web focused applications. Common actions, such as sending JSON data, become easier and don’t require every application to implement a network-data buffering/caching layer for fragmented messages.</p><p>This Juicy JSON in the figure below cannot be sent as one TCP/IP packet although it is sent as one WebSocket frame. The WebSocket protocol allows each message to be fragmented across multiple TCP/IP packets (according to the congestion control mechanisms). This allows a 4Gb message to be split into thousands of TCP/IP packets during transmission.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fibbKk-SnDFtO7c1RByQFQ.png" /><figcaption>The reality of abstraction over abstraction over abstraction is that when details leak through the layers the application code must be able to handle it gracefully.</figcaption></figure><p>It is a message-based protocol built on top of a streaming protocol and with that there are some things to keep in mind:</p><ul><li>The TCP/IP layer guarantees network packet delivery and ordering, but has no concept of data length. It is a streaming protocol and it promises that the stream will arrive in order, no more than that. Smaller messages would be proffered in that situation as any lost packet is going to delay parsing of the message with hundreds of milliseconds to seconds until it gets re-transmitted.</li><li>Bandwidth is finite, so trying to stream large amounts of data is going to saturate the link until it starts getting packet loss and then back-off to leave some bandwidth. The problem is that data cannot be partially parsed if it is in JSON format as in the example above. All of it should be buffered and parsed together which in turn starts filling the TCP receive buffers — maybe the device can handle it, maybe not. The service generating the push has no information or control over the clients.</li><li>Sending data from multiple backend services to the client could also be effected by the link saturation. It doesn’t matter if the content is multiplexed and packages are spread ideally. Sending 4Gb over 4G network will on average need 30–40 minutes to download (favorable conditions). Mixing a few data streams with the big one is going to delay all data streams.</li></ul><p>Ideally, an efficient communication protocol would help developers avoid such problems. When large amounts of data have to be streamed they should be split in the smallest possible digestible unit. A well designed and efficient protocol could help with backpressure too. If large single files are being delivered, streaming is most probably not a good option. A CDN should be considered, or a protocol that was designed to handle copying files in a secure manner.</p><h3>Network and Application Security</h3><p>Securing a network that uses WebSockets involves implementing measures to protect the communication between clients and servers from unauthorized access or manipulation. There are several steps that can be taken to improve the security of a network that uses WebSockets, including:</p><ul><li>Encryption: WebSockets supports encryption, which can be used to secure the communication between clients and servers. This can help to prevent man-in-the-middle attacks and protect against the interception of sensitive data.</li><li>Authentication: Prevent unauthorized access to the network by requiring clients to provide valid credentials before they are allowed to connect.</li><li>Access controls: Firewall rules or access control lists can help limit access to the network and protect against unauthorized access or attacks.</li><li>Monitor network activity: Identify and prevent security threats, such as DoS attacks or malicious traffic.</li><li>Secure protocol: wss:// (WebSocket Secure) for connection establishment and enabled Transport Layer Security (TLS) or Secure Sockets Layer (SSL) can help secure communication between clients and servers.</li></ul><p>At Draftkings Inc. we use JWT to authenticate the user at the gateway level, which strips the JWT and applies a collection of headers to the requests it is passing forth. This allows for the requests to be “signed” by variety of identifiers which are understood and re-verified by the services behind the gateway.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q6dHKCYSwm-SjN8oTTbDsg.png" /><figcaption>Note that this scenario might not be suitable for sensitive information or money transactions which could require the system to use additional verification methods such as MFA and/or others.</figcaption></figure><p>It is important to note that web application firewalls (WAFs) and gateways may not be able to understand WebSocket traffic by default. A WAF is a security tool that protects web applications from attacks by monitoring and filtering inbound traffic. To secure WebSockets, WAFs can apply rules and filters to incoming WebSocket traffic to identify and block malicious entities. However, once a connection is established successfully, the traffic becomes opaque to the WAFs. Thus a layer of security measures must be implemented on the server that handles the WebSocket connections such as:</p><ul><li>Authorization — even if an entity is allowed through the gateway when connected, verify if access to any resources should be allowed before sending any data.</li><li>User input validation — ensure data format correctness and meets the requirements of the application.</li><li>Sanitization — remove or encode potentially harmful characters from user input. This is especially important to prevent XSS and other injection attacks.</li><li>Rate limiting — it could be limiting of the number of messages sent from the client application to the server, timeouts and/or throttling of the processing of the messages.</li><li>Limit the number of opened connections per IP, client, token, etc.</li><li>Monitoring and alerting — errors in parsing messages, number, size and frequency of messages, and others.</li></ul><h3>Application Performance Monitoring (APM)</h3><p>Application performance monitoring for streaming connections involves monitoring various metrics related to the performance of the streaming connection, such as the connection’s latency, jitter, and packet loss. This can be done by using specialized tools or software that can track and analyze these metrics in real-time. Additionally, it may also include monitoring the overall health and status of the streaming service, including the number of active connections and any errors or issues that may be impacting performance. The goal of application performance monitoring for streaming connections is to ensure that the service is providing a high-quality, reliable streaming experience for users.</p><p>The majority of the available tools sample HTTP traffic by default, as it is a standard protocol with fixed framing and headers. To be able to sample WebSocket traffic, it often requires development on the client and server that exchange messages to enrich them with metadata which gets reported before send and on receive.</p><p>Another tricky aspect of performance with streaming protocols is that it varies based on the conditions. One source of traffic over a stable TCP/IP connection could yield flawless results. In a real-life scenario, an users may be on the move, causing their device to constantly switch network cells and compete for bandwidth with other devices in the areas they traverse. Consider also the scenario when a device is switching between WiFi and the mobile network. To get ahead on this it is good to track at least:</p><ul><li>Messages sent vs messages received — Are there any messages missing? Were they important? With what rate were they missed?</li><li>Bandwidth consumption — Is the amount of data growing? Should the protocol be changed to counter? Is there a need for compaction?</li><li>Connection latency and jitter — Are the messages received fast enough? How big is the variation between the different messages?</li><li>Packet loss — How often messages have to be re-transmit?</li><li>Error and disconnect rates — How often the connections get unplanned interruptions?</li><li>Memory and CPU usage — How much resources go into maintaining connections? How much is in encoding/decoding/parsing messages?</li></ul><h3>Honorable mentions</h3><p>This section contains some interesting issues that are good to be aware of:</p><ul><li><strong>HoL blocking</strong> → Head-of-line blocking (HOL blocking) in computer networking is a performance-limiting phenomenon that occurs when a line of packets is held up in a queue by a first packet. Examples include input buffered network switches, out-of-order delivery and multiple requests in HTTP pipelining. [2]</li><li><strong>No caching</strong> → Data going over WebSocket connections cannot be cached by gateways and CDNs to distribute to multiple clients. You also can’t leverage the browser-caching mechanisms for HTTP requests. If an application could benefit from such mechanism, maybe some of the its flows should go via plain HTTPS calls or have the engineers implement a custom solution for it.</li><li><strong>Connection fanout</strong> → Number of opened connections is limited. There may be a situation where one user’s connection may lead to 10s connections to the BE services. This could potentially limit the amount of users that could be served given a one-to-one connection ratio. This is why earlier in the article it is suggested that scaling based on number of open connections is a better determinator for the need of new instances.</li><li><strong>CSWH </strong>→ CSWH is an attack similar to CSRF (Cross-Site Request Forgery). Both attacks are possible when the server relies only on session authentication data (cookies) to perform an authenticated action.</li></ul><h3>Summary</h3><p>The WebSocket API is a messaging protocol built on top of a streaming one. As such it hides some of the complexity of dealing with streaming but it is still susceptible to all of its pitfalls.</p><p>There is a very important question to answer before jumping in → <strong>Do I really need real-time bi-directional communication?</strong></p><p>It is not enough to simply decide to use WebSockets. The backend system must be designed to accommodate such communication in a secure and safe manner. To be efficient, it is necessary to build a custom protocol and communication pattern. Remote procedure calls (RPC) and JSON are both options, but the choice ultimately depends on the specific requirements of the application.</p><h3>APPENDIX: The alternatives</h3><p>WebSocket APIs are a useful tool for applications that require real-time, low-latency communication, but they may not be the best choice for all applications. Other alternatives, such as Server-Sent Events, WebRTC or Long-Polling, may be more suitable for certain use cases.</p><p><strong>Good’ol Polling</strong></p><p>This is the software implementation equivalent of the annoying child sitting in the back seat of your car constantly asking “Are we there yet?”. In short, the client asks the server for data. If the server has data, it responds with it. If not, it waits for a predefined amount of time and if nothing useful for the client comes up, it closes the connection. The client could decide to open another request if needed. This is usually the fallback strategy once everything else fails.</p><ul><li>Connection: Long polling involves opening and closing an HTTP connection for each request-response cycle, while WebSockets maintain an open connection for bidirectional communication.</li><li>Latency: Long polling has higher latency than WebSockets, as it requires a new connection to be established for each request-response cycle. WebSockets, on the other hand, maintain an open connection, allowing data to be sent as soon as it is available.</li><li>Bandwidth: Long polling can consume more bandwidth than WebSockets, as it requires opening and closing many connections.</li><li>Timeouts: There are various timeouts to be aware of. By default a fetch() request times out at the time setup by the browser. In Chrome, at the time of writing, this setting equals 300 seconds. However, in NodeJS, by default, this timeout is 120 seconds.</li><li>Use cases: Long polling is typically used for simple, low-frequency updates, such as chat or notifications, while WebSockets are better suited for more complex, high-frequency updates, such as online gaming or financial data streaming.</li></ul><p><strong>Server-Sent Events (SSE)’</strong></p><p>SSE is a server push technology enabling a client to receive automatic updates from a server via an HTTP connection, and describes how servers can initiate data transmission towards clients, once an initial client connection has been established. They are commonly used to send message updates or continuous data streams to a browser client and designed to enhance native, cross-browser streaming through a JavaScript API called EventSource, through which a client requests a particular URL in order to receive an event stream. [6]</p><p>There are some fundamental differences between them and the WebSocket API:</p><ul><li>One-way vs. bidirectional: SSE is a one-way communication protocol, where the server sends data to the client, but the client cannot send data to the server. WebSockets, on the other hand, provide bidirectional communication, where both the client and the server can send and receive data.</li><li>Protocols: SSE is based on the HTTP/HTTPS protocol, while WebSockets use a different protocol, which is built on top of TCP.</li><li>Browser support: SSE is supported in most modern browsers, while WebSockets are supported in all modern browsers.</li><li>Latency: WebSockets have a lower latency compared to SSE, as it maintains an open connection between the client and the server, allowing data to be sent as soon as it is available.</li><li>Use cases: SSE is typically used for simple, one-way data streaming, such as updating a stock ticker or a chat room, WebSockets are better suited for more complex, real-time applications, such as online gaming or video conferencing.</li></ul><p><strong>WebRTC</strong></p><p>With WebRTC, you can add real-time communication capabilities to your application that works on top of an open standard. It supports video, voice, and generic data to be sent between peers, allowing developers to build powerful voice- and video-communication solutions. The technology is available on all modern browsers as well as on native clients for all major platforms. The technologies behind WebRTC are implemented as an open web standard and available as regular JavaScript APIs in all major browsers. For native clients, like Android and iOS applications, a library is available that provides the same functionality. The WebRTC project is open-source and supported by Apple, Google, Microsoft, and Mozilla, amongst others. [5]</p><p>WebRTC (Web Real-Time Communication) and WebSockets are both technologies used for real-time communication, but they have different use cases and functionalities:</p><ul><li>Media vs. data: WebRTC is mainly used for real-time communication of audio, video, and other multimedia data, while WebSockets are mainly used for real-time communication of data such as text or JSON.</li><li>P2P vs. server-client: WebRTC enables peer-to-peer communication between two or more clients, while WebSockets typically involve a client connecting to a server.</li><li>Protocols: WebRTC uses a variety of protocols such as RTP, SRTP, and DTLS, while WebSockets use the WebSocket protocol.</li><li>Security: WebRTC’s communication is typically encrypted using DTLS-SRTP, while WebSockets can use different encryption methods, such as SSL/TLS.</li><li>Use cases: WebRTC is mainly used for real-time multimedia applications such as video conferencing, online gaming, and live streaming, while WebSockets are used for more generic real-time data communication, such as chat applications and online collaboration tools.</li></ul><p>Want to learn more about DraftKings’ global Engineering team and culture? Check out our <a href="https://careers.draftkings.com/draftkings-life-blog/engineering/?utm_source=Medium&amp;utm_medium=Blog&amp;utm_campaign=DK+Blog&amp;utm_content=Medium+DK+Life+Engineering">Engineer Spotlights</a> and <a href="https://careers.draftkings.com/jobs/?utm_source=Medium&amp;utm_medium=Blog&amp;utm_campaign=Job+Openings&amp;utm_content=Medium+Open+Jobs">current openings</a>!</p><h3>Resources</h3><ul><li>[1] <a href="https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API">https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API</a></li><li><a href="https://www.smashingmagazine.com/2018/02/sse-websockets-data-flow-http2/">SSE vs WebSocket vs Polling</a></li><li>[2] <a href="https://en.wikipedia.org/wiki/Head-of-line_blocking">https://en.wikipedia.org/wiki/Head-of-line_blocking</a></li><li>[3] <a href="https://www.rfc-editor.org/rfc/rfc6202#section-2.2">https://www.rfc-editor.org/rfc/rfc6202#section-2.2</a></li><li>[4] <a href="https://github.com/Yaffle/EventSource/blob/master/src/eventsource.js#L533">https://github.com/Yaffle/EventSource/blob/master/src/eventsource.js#L533</a></li><li>[5] <a href="https://webrtc.org/">https://webrtc.org/</a></li><li>[6] <a href="https://en.wikipedia.org/wiki/Server-sent_events">https://en.wikipedia.org/wiki/Server-sent_events</a></li><li>[7] <a href="https://www.alibabacloud.com/blog/tcp-syn-queue-and-accept-queue-overflow-explained_599203">https://www.alibabacloud.com/blog/tcp-syn-queue-and-accept-queue-overflow-explained_599203</a></li><li>[8] <a href="https://en.wikipedia.org/wiki/Data_compaction">https://en.wikipedia.org/wiki/Data_compaction</a></li><li>[9] <a href="https://www.rfc-editor.org/rfc/rfc7692">https://www.rfc-editor.org/rfc/rfc7692</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=604617a54cdb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/draftkings-engineering/lessons-learned-websocketapi-at-scale-604617a54cdb">Lessons Learned: WebSocketAPI at scale</a> was originally published in <a href="https://medium.com/draftkings-engineering">DraftKings Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>