<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Harsha Reddy on Medium]]></title>
        <description><![CDATA[Stories by Harsha Reddy on Medium]]></description>
        <link>https://medium.com/@harsha_reddy?source=rss-e742dd2851------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3H52b8-8YLM5ZuYab46jhg.jpeg</url>
            <title>Stories by Harsha Reddy on Medium</title>
            <link>https://medium.com/@harsha_reddy?source=rss-e742dd2851------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:44:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@harsha_reddy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Intro to Code Profiling]]></title>
            <link>https://klaviyo.tech/intro-to-code-profiling-c7efc893bdc4?source=rss-e742dd2851------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7efc893bdc4</guid>
            <category><![CDATA[code-profiling]]></category>
            <category><![CDATA[code-optimization]]></category>
            <category><![CDATA[instrumentation]]></category>
            <dc:creator><![CDATA[Harsha Reddy]]></dc:creator>
            <pubDate>Mon, 01 Apr 2024 15:22:40 GMT</pubDate>
            <atom:updated>2024-04-01T15:22:40.899Z</atom:updated>
            <content:encoded><![CDATA[<p>Code profiling is an essential skill in an engineer’s toolbox. In this post, intended for engineers new to optimizing code, I describe the tools and techniques we frequently use here at Klaviyo.</p><p>Good system performance contributes to better user experience,<a href="https://developer.mozilla.org/en-US/docs/Learn/Performance/why_web_performance"> increased user retention, resource efficiency, and lower bounce rates</a>.</p><p>At the core of improving system performance is the ability to accurately identify and measure the problem. The best practice is to use observability frameworks to achieve this, including production time series metrics, OpenTelemetry tracing, and structured logging. These tools enable engineers to formulate hypotheses and pinpoint inefficiencies within the system. Profiling tools and query instrumentation then become invaluable in identifying the root cause of performance issues. Approaching optimization logically allows for quick identification of problem areas. Lack of discipline can lead to wasted time on unnecessary code modifications that fail to enhance the customer experience.</p><p>Let’s dive into the preferred tools and methodologies for code optimization, and how you can take a structured and measured approach to diagnosing and resolving performance bottlenecks.</p><h3>OpenTelemetry</h3><p><a href="https://opentelemetry.io/docs/what-is-opentelemetry/">OpenTelemetry</a> is a broad standard to improve observability in systems. OpenTelemetry frameworks provide tools and APIs designed to collect, process, and export telemetry data like metrics, logs, and traces from software applications. It aims to provide a standardized way to gather insights into software performance and behavior, helping developers and operators better understand and optimize their systems.</p><figure><img alt="OpenTelemetry logo" src="https://cdn-images-1.medium.com/max/128/0*8XiBP0FtFuN8MD9_" /><figcaption>OpenTelemetry is an <a href="https://opentelemetry.io/docs/concepts/observability-primer/#what-is-observability">observability</a> framework and toolkit designed to create and manage telemetry data such as <a href="https://opentelemetry.io/docs/concepts/signals/traces/">traces</a>, <a href="https://opentelemetry.io/docs/concepts/signals/metrics/">metrics</a>, and <a href="https://opentelemetry.io/docs/concepts/signals/logs/">logs</a>. Crucially, OpenTelemetry is vendor- and tool-agnostic.</figcaption></figure><p>OpenTelemetry is excellent as a precursor to code profiling. It reduces the need for an engineer to take guesses about performance. Use it to generate hypotheses before diving into profiling code.</p><p>At Klaviyo, we instrument all the Django view endpoints with OpenTelemetry. We also provide Python decorators for methods to record spans for them in OpenTelemetry traces.</p><pre>@start_span(&quot;_validate&quot;) # OpenTelemetry span decorator<br>def _validate(self, thing_to_validate: str) -&gt; bool:<br>    ...</pre><p>The data collected:</p><ul><li><strong>Traces:</strong> In OpenTelemetry, a trace represents a single, logical operation like a transaction or a request. It’s composed of multiple components that together provide a detailed understanding of the operation. The operation could span across multiple software components (e.g. code, database), and services (internal and external). Although traces do resemble the output of code profiling, there are important differences, which I’ll get into below.</li><li><strong>Metrics:</strong> Quantitative measurements, such as response times or error rates. This can be fed into a tool like Grafana for further analysis and creating dashboards.</li><li><strong>Logs:</strong> Structured logs are tied to each span (the leaf nodes in a trace), providing insight into the runtime behavior and status of the request.</li></ul><p>OpenTelemetry provides real performance data across systems for millions of user requests. If you see major performance issues in these traces, you can be guaranteed that fixing them will improve the user experience. You can capture the before and after state of the operation, confirming your fix. (And bonus: these before and after plots make great visuals.)</p><p><em>Given the sheer amount of data being produced, OpenTelemetry instrumentation is not intended to capture every single user request and every single line of code, due to performance implications and infrastructure costs.</em></p><p>Here’s a screen capture of an OpenTelemetry trace from a single request to a service at Klaviyo. A service <em>app-flows</em> (in red) executes a task to perform batch operations. The task runs a bunch of data queries on a Redis data source to start and makes a couple of requests to the <em>app-event-query</em> (in blue) service to gather the data needed for completion.</p><figure><img alt="Grafana opentelemetry trace" src="https://cdn-images-1.medium.com/max/1024/0*kR5Q8d3czKZw1ynr" /><figcaption>Example of an OpenTelemetry trace</figcaption></figure><p>Here’s an example of a Grafana dashboard showing metrics captured by our OpenTelemetry framework.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R_O7nr-yauHKUAKaw2JteQ.png" /><figcaption>Example of OpenTelemetry performance metrics</figcaption></figure><h3>Database query performance instrumentation</h3><p>All applications utilize some data backend like a database, cache, or files. It’s essential to have a good understanding of how to analyze the performance of your data sources and the impact your app has on them.</p><p>At Klaviyo we like Django! Django’s ORM system provides a Pythonic way to run SQL using<a href="https://docs.djangoproject.com/en/4.2/topics/db/"> models</a>. This abstraction sometimes leads to non-performant SQL queries sneaking their way into production.</p><p>We forked <a href="https://github.com/dobarkod/django-queryinspect">Django Query Inspector</a> internally to help engineers quickly identify SQL queries that could result in performance regressions.</p><p>This tooling helps them:</p><ul><li>Identify slow SQL queries during the development phase</li><li>Identify <a href="https://planetscale.com/blog/what-is-n-1-query-problem-and-how-to-solve-it">n+1 problems</a> with SQL queries</li><li>Identify any repeated SQL queries across different modules</li></ul><p>The internal fork adds a few more features on top, that are helpful for engineers at Klaviyo:</p><ul><li>DB Replica support and total SQL time per replica as well as global SQL time for a Django request</li><li>Console colors (if the terminal supports it)</li><li>Optimizations to the module when capturing hundreds of SQL statements</li><li>Scrubs sensitive data from being rendered to the console</li></ul><p>It is enabled in Django’s middleware settings with a single line of code</p><pre>MIDDLEWARE = (<br>   &quot;app.tracing.middleware.TracingMiddleware&quot;,<br>   ...<br>   &quot;middleware.DBQueryPerfMiddleware&quot;,<br>   ...<br>)</pre><p>Sample output (with some redaction) from the tool is shown below. <br>The first line is a summary of the SQL queries on the request and as you can see there are a total of 5 queries over 37ms and there is 1 duplicate query. The detailed output shows the offending query, which was repeated 3 times, and, could be optimized further.</p><pre>[SQL] 5 queries (1 duplicates), 37 ms SQL time<br> - - -<br><br>Repeated query (3x): [DB] CoreDB: [SQL] SELECT `django_session`.`session_key`, <br>`django_session`.`session_data`, `django_session`.`expire_date`<br>FROM `django_session` <br>WHERE (`django_session`.`expire_date` &gt; &#39;?&#39;<br>AND `django_session`.`session_key` = ?) <br>LIMIT 20<br><br>Repeated query (1x): [DB] CoreDB: [SQL] SELECT `auth_user`.`id`, <br>`auth_user`.`password`, `auth_user`.`last_login`, `auth_user`.`is_superuser`,<br>`auth_user`.`username`, `auth_user`.`first_name`, `auth_user`.`last_name`, <br>`auth_user`.`email`, `auth_user`.`is_staff`,`auth_user`.`is_active`, <br>`auth_user`.`date_joined` <br>FROM `auth_user` <br>WHERE `auth_user`.`id` = ? <br>LIMIT 20<br><br>Repeated query (1x): [DB] AuditDB: [SQL] INSERT INTO `audit` <br>(`created`, `auth_id`, `view_name`, `view_resource_kwarg`, `request_method`,<br>`client_ip`, `url`, `role`) <br>VALUES (?, ?, ?, ?, ?, ?, ?, ?)</pre><h3>Profiling tools</h3><p>Now we get into code profiling tools. Code profiling tools are essential instruments for developers, offering deeper insights into performance by identifying bottlenecks, resource usage, and execution hotspots. Armed with the hypotheses, you can rely on code profiling tools to dig deep.</p><p>Here’s sample output from a code profiler:</p><figure><img alt="A flamechart style visualization of a code profile" src="https://cdn-images-1.medium.com/max/1024/0*YTdk2QU5R21t8KUm" /><figcaption>Source: <a href="https://jiffyclub.github.io/snakeviz/">Snakeviz</a></figcaption></figure><p>Before we showcase a profiling tool, there is one item to be wary of: the choice of profiler. There are two kinds — statistical and deterministic.</p><p><strong>Statistical profilers</strong> will capture the state of your application based on a preset sampling rate (e.g. 1ms). At Klaviyo, we prefer statistical profilers for most optimization work. They have a few advantages:</p><ol><li>Vastly reduces the amount of noise in the traces by capturing the state of the program at a predefined sampling rate.</li><li>The program’s performance is not massively impacted. Statistical profilers like <a href="https://github.com/benfred/py-spy">py-spy</a> cause less program slowdown during a profiling session than deterministic options like <a href="https://github.com/vpelletier/pprofile">pprofile (determistic)</a>. (See chart below.)</li><li>We rarely need extreme precision on tasks when assessing the performance of the program.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dtcerCIWNwXS9tCmh-u2_Q.png" /><figcaption>Source: PyCon US 2021 <a href="https://youtu.be/5iEf-_7mM1k?feature=shared&amp;t=202">Scalene talk</a></figcaption></figure><p>On the flip side, <strong>deterministic profilers</strong> systematically record every function call, the time spent in each function, and other events that occur during the execution of a program. This is incredibly useful when you need to understand specific code paths that resulted in your system’s poor performance. Wall clock accuracy should not be a concern when using deterministic profilers.</p><p>As mentioned earlier, for most scenarios at Klaviyo, we prefer statistical (sampling) profilers: <a href="https://github.com/benfred/py-spy">py-spy</a> and <a href="https://github.com/joerick/pyinstrument">pyinstrument</a>.</p><p><a href="https://github.com/benfred/py-spy">Py-spy</a> is great for attaching to running processes. For example, it can be attached to a runaway process to quickly figure out what’s going on. Attaching py-spy to a Django process goes as follows:</p><pre># Find the Process ID<br>ps aux | grep django<br><br># Monitor<br>sudo /home/$USER/.local/bin/py-spy top - pid {pid}<br><br># Bonus: you can record a Speedscope visualization<br>sudo /home/$USER/.local/bin/py-spy record - pid {pid} \<br>-f speedscope -o profile.json</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5-RDUF6OzToSGxq5" /><figcaption>Source: py-spy <a href="https://github.com/benfred/py-spy#record">README</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4emkAby6m8LtDLI5" /><figcaption>Source: py-spy <a href="https://github.com/benfred/py-spy#record">README</a></figcaption></figure><p>We also make <a href="https://pyinstrument.readthedocs.io/en/latest/home.html">pyInstrument</a> available, which has excellent integration with the Django framework. It is as straightforward as adding the following line to the middleware stack:</p><pre>MIDDLEWARE = (<br>   &quot;app.tracing.middleware.TracingMiddleware&quot;,<br>   &quot;middleware.DBQueryPerfMiddleware&quot;<br>   ...<br>   &quot;pyinstrument.middleware.ProfilerMiddleware&quot;<br>   ...<br>)</pre><p>Appending <em>?profile</em> query param to a web request, generates a code profile on the running Django server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o2UwpX9gyP_rARQP.jpg" /><figcaption>Source: pyInstrument <a href="https://pyinstrument.readthedocs.io/en/latest/home.html">documentation</a></figcaption></figure><p>The ability to profile in any environment from local to production (with safeguards) is incredibly clutch. At Klaviyo we pride ourselves on making code profiling available everywhere from local development, docker images, development, and production environments. We do this to improve our engineers’ confidence in optimizing and triaging issues.</p><h3>Things to ask yourself before optimizing the code</h3><p>At this stage, let’s assume you have identified something slow in your codebase, or someone asked you to improve the performance of a component. Do a few sanity checks before you proceed:</p><h4>Will it matter?</h4><ul><li>Make sure it has customer impact. (There’s an art to dissecting the customer feedback to get at the root of what’s being reported. Sometimes speed is the culprit. Sometimes it isn’t.)</li><li>If you have a long list of items that can be made faster, start with the ones hit more often by more users.</li><li>Seek customer feedback when major optimizations are rolled out. (Read about <a href="https://www.smashingmagazine.com/2015/11/why-performance-matters-part-2-perception-management/">perception management</a> for some examples.)</li></ul><h4>Is your code improvement resulting in performance (or correctness) regressions elsewhere?</h4><ul><li>Utilize OpenTelemetry to have a cohesive view of all upstream and downstream systems of your service/code post-optimization rollout.</li><li>Avoid adding to tech debt with substandard changes. These solve the problem short term but increase maintenance headaches for your team in the future.</li><li>Ensure you add good functional and non-functional tests in your system to identify regressions.</li></ul><h4>Stay focused and avoid <a href="https://americanexpress.io/yak-shaving/">yak shaving</a>.</h4><ul><li>Avoid <a href="https://americanexpress.io/yak-shaving/">yak shaving</a> by trying to improve unrelated aspects of your system. Stay focused on the problem at hand.</li><li>Limit yourself to your domain and any external changes ought to be delegated to the responsible teams.</li></ul><h4>Are you collaborating with your management?</h4><ul><li>Keep visibility high so that everyone is aware of the outcomes.</li><li>Any deviation from expected outcomes must be communicated clearly to assist with customer outreach.</li></ul><h4>Are you able to measure performance before and after the optimization?</h4><ul><li>Add a counter, time, or gauge metric (e.g. <a href="https://github.com/statsd/statsd">statsd</a>, <a href="https://www.influxdata.com/">influxdb</a>) around the non-performant code to produce mean, median, 95th, and 99th metric percentiles.</li><li>Perform as many code profiles as needed to reproduce the performance numbers before and after the improvement. We recommend capturing and comparing mean, median, 95th, and 99th percentiles in this case too. And again, remember that your CPU and I/O on your local are different from production systems.</li><li>If you cannot showcase clear numbers from the above, consider it a risk to roll the improvement to your customers.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dwu1NNEDN1aQuaLP13fAzg.png" /><figcaption>Funny but serious rules from <a href="https://wiki.c2.com/?RulesOfOptimization">Programming Pearls</a></figcaption></figure><h4>Code profiling does not measure accurate performance</h4><p>Never rely on code profiling to produce accurate performance measurements. Tools like py-spy can get close but you ought to rely on other instrumentation to provide accuracy rather than code profiling.</p><p>On production, we recommend utilizing <a href="https://opentelemetry.io/docs/specs/otel/metrics/">OpenTelemetry trace metrics</a>, <a href="https://github.com/statsd/statsd">StatsD</a>, <a href="https://www.influxdata.com/">InfluxDB</a>, or any such time series ecosystem to record counters, timers, and gauges. They are much more efficient at reducing the overhead when recording such data.</p><p>On your local code profiling session, if you want to avoid overhead, we recommend starting with statistical profilers (e.g. py-spy) to find the unoptimized code. Then, wrap that code around a simpler module like <a href="https://docs.python.org/3/library/timeit.html">timeit</a> to capture accurate measurements across thousands of iterations.</p><pre>$ python -m timeit &quot;&#39;-&#39;.join(str(n) for n in range(100))&quot;<br>10000 loops, best of 5: 30.2 usec per loop<br><br>$ python -m timeit &quot;&#39;-&#39;.join([str(n) for n in range(100)])&quot;<br>10000 loops, best of 5: 27.5 usec per loop<br><br>$ python -m timeit &quot;&#39;-&#39;.join(map(str, range(100)))&quot;<br>10000 loops, best of 5: 23.2 usec per loop</pre><h3>Fix the identified performance issue</h3><p>Profiling is very effective in deep diving into specific targets and testing your hypotheses. Delving into the finer nuances of our code enables us to pinpoint the specific segments that require optimization. This granular approach is key in efficiently targeting and resolving performance issues. You will only get such detailed information from code profiling.</p><p>The next challenge is to find out why the code in question is slow.</p><p>Code profiling will, in most cases, not show you the exact changes to make to optimize.</p><p>For example, running a code profile could result in the following trace. We can eyeball this to understand that one of the SQL queries is slow. The profile does not tell you how to optimize that SQL query.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/0*xfHGUuHoa6uk3XqO" /></figure><p>In such a case, with SQL, you can use tools like SQL <a href="https://www.postgresql.org/docs/current/sql-explain.html">EXPLAIN</a> to understand the inner workings of the query.</p><p>As you run more code profiles, you gain experience choosing and working with various domain-specific tools that make it easier to answer the “Why is the code slow?” question.</p><h3>🎉 Case study: Optimizing SSO logins</h3><p>I want to share an example of using the tools and techniques described above to identify the cause of slowness and implement a simple fix that resulted in a big improvement in user experience.</p><p>Our customers shared that a subset of user logins was much slower when they used single sign-on (SSO) vs traditional username and password. Our metrics (in OpenTelemetry and StatsD) confirmed this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_35fM18Y7O8ibt_e1m3RQw.png" /></figure><p>Next, we generated a hypothesis as to why SSO was slower. We spun up the pyInstrument profiler and saw that there was a significant slowdown in the user lookup method for SSO logins, and that most of the extra time was spent in SQL queries. We locked in on this hypothesis.</p><p><em>The traces below are simplified for demonstration purposes.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bmZ5RBWAp6CG-zxE" /><figcaption>SSO login trace (simplified)</figcaption></figure><p>Compare the above to the traditional non-SSO login.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/0*1XI1DEgZn22TfRDN" /><figcaption>Non-SSO login trace (simplified)</figcaption></figure><p>The user lookup from the code profile, among other things, ran a select query on the User table. To find the query being run, we spun up DB QueryInspector and replayed the requests. This confirmed that the slowness was due to a specific SQL query. We were getting closer.</p><p>We then ran a <a href="https://dev.mysql.com/doc/refman/8.0/en/using-explain.html#:~:text=The%20EXPLAIN%20statement%20provides%20information,about%20the%20statement%20execution%20plan.">SQL EXPLAIN</a> on the query and found that the Email field in the query to look up a user was not indexed on the table, whereas, the Username field was indexed and unique.</p><p>A simple one-line change to use the indexed Username column instead of Email resulted in a massive performance boost.</p><pre># Removed (simplified code)<br>- return User.objects.get(email=email)<br><br># Added (simplified code)<br>+ return User.objects.get(username=convert_email_to_username(email))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*UoM--IdEAW_IWVDzF-ct3g.png" /></figure><p>Subsequently, we used similar profiling techniques to get the non-SSO login to match SSO for performance with further code and database improvements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qMlKGHooBG6o-9bdzlIFvA.png" /></figure><h3>Useful links</h3><ul><li><a href="https://developer.mozilla.org/en-US/docs/Learn/Performance/why_web_performance">The “why” of web performance</a></li><li><a href="https://www.youtube.com/watch?v=5iEf-_7mM1k">Scalene: a high-performance, high-precision CPU+GPU+memory profiler for Python (PyCon US 2021)</a></li><li><a href="https://pyinstrument.readthedocs.io/en/latest/">PyInstrument</a> and <a href="https://github.com/benfred/py-spy">Py-spy</a></li><li><a href="https://www.youtube.com/watch?v=qE1ggEmvz2Y">OpenTelemetry: The Vision, Reality, and How to Get Started — Dotan Horovits, Logz.io</a></li><li><a href="https://dev.mysql.com/doc/refman/8.0/en/using-explain.html">Optimizing SQL Queries with EXPLAIN</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7efc893bdc4" width="1" height="1" alt=""><hr><p><a href="https://klaviyo.tech/intro-to-code-profiling-c7efc893bdc4">Intro to Code Profiling</a> was originally published in <a href="https://klaviyo.tech/">Klaviyo Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Load testing Zitadel using K6]]></title>
            <link>https://medium.com/@harsha_reddy/load-testing-zitadel-using-k6-a68328c89b33?source=rss-e742dd2851------2</link>
            <guid isPermaLink="false">https://medium.com/p/a68328c89b33</guid>
            <category><![CDATA[load-testing]]></category>
            <category><![CDATA[zitadel]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[k6]]></category>
            <category><![CDATA[authentication]]></category>
            <dc:creator><![CDATA[Harsha Reddy]]></dc:creator>
            <pubDate>Wed, 27 Mar 2024 15:59:54 GMT</pubDate>
            <atom:updated>2024-03-27T16:12:36.899Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*RR4FNY3QTW2H5XJW" /><figcaption>Image Credit: <a href="https://unsplash.com/@cinusek">Marcin Simonides</a>, Unsplash</figcaption></figure><p>I wrote this article to share the <a href="https://k6.io/">https://k6.io</a> load testing setup for Zitadel which I use regularly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/1*fuS_FI8nxWvTjXW0m9-DRQ.png" /></figure><p>Once you complete this article, you’ll be able to get a nice Grafana dashboard for your load test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FNG5ZE9sh5BQI1QO" /></figure><p><a href="https://www.zitadel.com/">Zitadel</a> provides secure authentication management for your application. Customize as you grow, with easy APIs and programmable workflows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*GcMupfxWC9SdS5-uwmTr7Q.png" /></figure><h3>Purpose</h3><p>Load testing is performed to assess the performance and scalability of the Zitadel application (or any application for that matter) under various simulated user loads.</p><h3>Prerequisites</h3><ul><li>Zitadel running with an accessible domain (<a href="https://zitadel.com/docs/self-hosting/deploy/overview">https://zitadel.com/docs/self-hosting/deploy/overview</a>)</li><li>Create a service user in Zitadel <a href="https://zitadel.com/docs/guides/integrate/service-users/personal-access-token#call-zitadel-api-with-pat">https://zitadel.com/docs/guides/integrate/service-users/personal-access-token#call-zitadel-api-with-pat</a></li><li>Docker is installed on your machine (<a href="https://docs.docker.com/desktop/">https://docs.docker.com/desktop/</a>)</li></ul><h3>Setup</h3><h4>Create a new directory</h4><pre>mkdir loadtest &amp;&amp; cd loadtest</pre><h4>Create a `.env` file</h4><p>Create a basic .env file with the following contents</p><pre>ZITADEL_URL=&lt;Your Zitadel URL e.g. https://my-zitadel-domain.com&gt;<br>ZITADEL_PERSONAL_ACCESS_TOKEN=&lt;Paste the token you created above&gt;</pre><h4>Create a Dockerfile for k6</h4><p>We need to build a custom k6 image that embeds the K<a href="https://github.com/grafana/xk6-output-prometheus-remote/tree/main">6 Prometheus extension</a></p><p><a href="https://github.com/prometheus"><em>Prometheus</em></a><em> is an open-source systems monitoring and alerting toolkit.</em></p><pre>touch Dockerfile</pre><p>Paste the following into that</p><pre><br>FROM golang:1.20-alpine3.18 as builder<br>WORKDIR $GOPATH/src/go.k6.io/k6<br>COPY . .<br>RUN apk --no-cache add git=~2<br>RUN CGO_ENABLED=0 go install go.k6.io/xk6/cmd/xk6@latest<br>RUN CGO_ENABLED=0 xk6 build \<br>    --with github.com/grafana/xk6-output-prometheus-remote@latest \<br>    --output /tmp/k6<br><br>FROM alpine:3.18<br><br>RUN apk add --no-cache ca-certificates &amp;&amp; \<br>    adduser -D -u 12345 -g 12345 k6<br><br>COPY --from=builder /tmp/k6 /usr/bin/k6<br><br>USER 12345<br>WORKDIR /home/k6<br><br>ENTRYPOINT [&quot;sh&quot;]</pre><h4>Create a docker compose services file</h4><pre>touch docker-compose.yaml</pre><p>Paste the following into that</p><pre>version: &#39;3.7&#39;<br><br>volumes:<br>  prometheus-data:<br>  grafana-data-storage:<br><br>services:<br>  k6-with-prometheus:<br>    build: .<br>    depends_on:<br>      - grafana<br>    volumes:<br>      - .:/app:delegated<br>    environment:<br>      - K6_NO_USAGE_REPORT=false<br>      - K6_PROMETHEUS_RW_SERVER_URL=http://prometheus:9090/api/v1/write<br>      - ZITADEL_PERSONAL_ACCESS_TOKEN=${ZITADEL_PERSONAL_ACCESS_TOKEN}<br>      - ZITADEL_URL=${ZITADEL_URL}<br>    command: /app/run_tests.sh<br><br>  prometheus:<br>    image: prom/prometheus:latest<br>    depends_on:<br>      - grafana<br>    ports:<br>      - 9000:9090<br>    volumes:<br>      - ./prometheus.yaml:/etc/prometheus/prometheus.yaml<br>      - prometheus-data:/prometheus<br>    command: --enable-feature=remote-write-receiver --config.file=/etc/prometheus/prometheus.yaml<br><br>  grafana:<br>    image: grafana/grafana:latest<br>    volumes:<br>      - grafana-data-storage:/var/lib/grafana<br>    ports:<br>      - 4000:4000<br>    environment:<br>      - GF_SERVER_HTTP_PORT=4000</pre><h4>Create a Prometheus config file</h4><pre>touch prometheus.yaml</pre><p>Paste the following into that</p><pre>global:<br>  scrape_interval: 30s<br>  scrape_timeout: 10s<br>scrape_configs:<br>  - job_name: services<br>    metrics_path: /metrics<br>    static_configs:<br>      - targets:<br>          - &#39;prometheus:9090&#39;</pre><h3>Create a basic k6 load test</h3><pre>touch run_tests.sh &amp;&amp; mkdir tests &amp;&amp; cd tests &amp;&amp; touch basic_k6_test.js</pre><p>Pase the following into <em>basic_k6_test.js</em></p><pre>/**<br> * This script is used for load testing the Zitadel authentication API.<br> * It sends HTTP requests to the &#39;/auth/v1/users/me&#39; endpoint with a constant arrival rate of 30 requests per second.<br> *<br> * @author Harsha Reddy<br> * @version 1.0<br> * @date 2024-02-10<br> * @see {@link https://grafana.com/docs/k6/latest/examples/get-started-with-k6/}<br> */<br><br>import http from &#39;k6/http&#39;;<br>import { check } from &#39;k6&#39;;<br>import { Trend, Counter } from &#39;k6/metrics&#39;;<br><br>// Define metrics for different response codes<br>let responseCodes = {<br>  &#39;200&#39;: new Counter(&#39;status_200&#39;),<br>};<br><br><br>export const options = {<br>  tags: {<br>    testid: &quot;zitadel_pat_k6_test&quot;,<br>  },<br>  discardResponseBodies: true,<br>  scenarios: {<br>    // Up to 100.00 iterations/s for 5m0s over 3 stages (maxVUs: 50-100)<br>    ramp_up_test: {<br>      executor: &#39;ramping-arrival-rate&#39;,<br>      startRate: 30, // we start at 30 iterations per second<br>      timeUnit: &#39;1s&#39;,<br>      preAllocatedVUs: 150, // how large the initial pool of VUs would be<br>      maxVUs: 250, // if the preAllocatedVUs are not enough, we can initialize more<br>      stages: [<br>        { duration: &#39;1m&#39;, target: 100 }, // traffic ramp-up from startRate of 30 to 100 users.<br>        { duration: &#39;3m30s&#39;, target: 150 }, // stay at 100 users for 3 min<br>        { duration: &#39;1m30s&#39;, target: 0 }, // ramp-down to 0 users<br>      ],<br>    },<br>    large_rps_test: {<br>      executor: &#39;constant-arrival-rate&#39;,<br>      rate: 200,<br>      startTime: &#39;7m&#39;,<br>      timeUnit: &#39;1s&#39;,<br>      duration: &#39;1m&#39;,<br>      preAllocatedVUs: 150,<br>      maxVUs: 300,<br>    },<br>  },<br>  thresholds: {<br>    &#39;status_200{scenario:ramp_up_test}&#39;: [<br>      &#39;rate &gt;= 90&#39;<br>    ],<br>    &#39;status_200{scenario:large_rps_test}&#39;: [<br>      &#39;rate &gt;= 200&#39;<br>    ],<br>    &#39;http_req_duration{scenario:ramp_up_test}&#39;: [&#39;p(99)&lt;300&#39;],<br>    &#39;http_req_duration{scenario:large_rps_test}&#39;: [&#39;p(99)&lt;500&#39;],<br>  },<br>};<br><br>// The function that defines VU logic.<br>//<br>// See https://grafana.com/docs/k6/latest/examples/get-started-with-k6/ to learn more<br>// about authoring k6 scripts.<br>//<br>export default function() {<br>  const url = `${__ENV.ZITADEL_URL}/auth/v1/users/me`;<br><br>  const params = {<br>    headers: {<br>      &#39;Content-Type&#39;: &#39;application/json&#39;,<br>      &#39;Authorization&#39;: `Bearer ${__ENV.ZITADEL_PERSONAL_ACCESS_TOKEN}`<br>    },<br>  };<br><br>  const res = http.get(url, params);<br><br>  let statusCode = res.status.toString();<br><br>  // Check if the response code is one that we&#39;re tracking<br>  if (responseCodes[statusCode]) {<br>    responseCodes[statusCode].add(1);<br>  }<br><br>  check(res, {<br>    &#39;status was 200&#39;: (r) =&gt; r.status == 200<br>  })<br>}</pre><p>Then paste the following into run_tests.sh</p><pre>echo &quot;Running the tests&quot;<br><br>echo &quot;**********************************&quot;<br>echo &quot;Running the basic k6 test&quot;<br>k6 run -o xk6-prometheus-rw ../tests/basic_k6_test.js<br>echo &quot;**********************************&quot;</pre><h3>Run your first load test</h3><pre>docker compose -f docker-compose.yaml up -d</pre><p>You can check the container logs to see the test in action</p><pre>docker compose -f docker-compose.yaml logs k6-with-prometheus</pre><p>Grafana and Prometheus will be accessible even after your load test is complete.</p><p>To re-run your load test, you can run the following</p><pre>docker compose -f docker-compose.yaml start k6-with-prometheus</pre><h3>Configure Grafana</h3><p>Next, we’ll set up the Grafana dashboard</p><p>If you haven’t already run the following, please do so now. This will spin up Grafana and Prometheus along with your load test.</p><pre>docker compose -f docker-compose.yaml up -d</pre><ul><li>Open Grafana <a href="http://localhost:4000/?orgId=1](http://localhost:4000/?orgId=1)">http://localhost:4000/?orgId=1</a></li><li>In the Search box at the top of the UI, type data sources and press enter.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*isAL4-IEnIpWExArWjis-Q.png" /></figure><ul><li>Click Add new data source and select <em>Prometheus</em>.</li><li>Input data as shown below and click <em>Save &amp; Test</em> at the bottom of the page.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YPxCFzpWMdpmzQaBXWm60w.png" /></figure><ul><li>Next, in the search bar type <em>New Dashboard</em> and then, click <em>Import Dashboard</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZJXADmMFWKI-97QWchD2rA.png" /></figure><ul><li>Enter the following URL `<a href="https://grafana.com/grafana/dashboards/19665-k6-prometheus/%60">https://grafana.com/grafana/dashboards/19665-k6-prometheus/`</a> and click `Load`</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*5o3FNjKkQQIvIwrXBk4gIA.png" /></figure><ul><li>You’ll be taken to the dashboard page like so with a pre-populated test</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ki_FdrRKSnc0G3X6lJ-t_Q.png" /></figure><p>In case you don’t see anything, re-run the load test:</p><pre>docker compose -f docker-compose.yaml start k6-with-prometheus</pre><h3>Shutdown</h3><pre>docker compose -f docker-compose.yaml down</pre><p><em>Your data is persisted to disk, this is a safe command to run.</em></p><p>Next time you re-run the command <em>docker compose -f docker-compose.yaml up -d</em>, all the Grafana setup will be revived.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a68328c89b33" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>