<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Locust.io experiments - Medium]]></title>
        <description><![CDATA[Locust.io is Python based load testing tool. The authors claim that it’s very easy to use and to extend Locust. Having no experience in Locust I decided to give it a quick try and attempt to address potential needs/problems with series of quick experiments. - Medium]]></description>
        <link>https://medium.com/locust-io-experiments?source=rss----84c32fc5c3de---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Locust.io experiments - Medium</title>
            <link>https://medium.com/locust-io-experiments?source=rss----84c32fc5c3de---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 20:17:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/locust-io-experiments" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Locust.io experiments — Emitting results to external DB]]></title>
            <link>https://medium.com/locust-io-experiments/locust-io-experiments-emitting-results-to-external-db-bd3de1994259?source=rss----84c32fc5c3de---4</link>
            <guid isPermaLink="false">https://medium.com/p/bd3de1994259</guid>
            <category><![CDATA[locust]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[experiment]]></category>
            <category><![CDATA[load-testing]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <dc:creator><![CDATA[Karol Brejna]]></dc:creator>
            <pubDate>Thu, 17 Oct 2019 14:26:48 GMT</pubDate>
            <atom:updated>2019-10-18T09:48:29.540Z</atom:updated>
            <content:encoded><![CDATA[<h3>Locust.io experiments — Emitting results to external DB</h3><p>One of the things that Locust does for you when used as your testing tool is collecting request/response data.</p><p>By default Locust periodically dumps test results and computed aggregated values like median, average response time, number of requests per second, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WnLKYCMtxyo8gTee" /><figcaption>Source: <a href="https://asiatrend.org/lifestyle/10-plagues-egypt-ai-era/">https://asiatrend.org/lifestyle/10-plagues-egypt-ai-era/</a></figcaption></figure><p>They are presented in the UI, printed in the logs or can be persisted to a file (using — csv option).</p><p>Basically, you have access to request-specific distribution pre-aggregated data, as illustrated in the following picture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fB2R8ezMHNnFR47I586-IQ.png" /><figcaption>Statistics screen from Locust 0.11.0</figcaption></figure><p>Most of the time, the information presented in such a way is good enough.</p><p>In this article, I’ll try to show how to deal with cases where you would like to do more with the results — correlating with some CPU/mem/IO metrics, archiving for future analysis, drawing charts, etc. — which would require storing them in a database.</p><p>Moreover, let’s assume that having only aggregated data is not enough (for example you want to be able to analyze/drill the results down to a single request).</p><p>In one of the previous installments of this series, Locust.io experiments — enriching results (https://medium.com/locust-io-experiments/locust-io-experiments-enriching-results-183d2ae4a4c2), I’ve touched on the topic of capturing atomic results.</p><p>Now, building upon the information on how to access individual requests data, I will introduce the following elements:</p><ul><li>programmatically accessing aggregated request data,</li><li>sending data to an external database,</li><li>making sure that communication doesn’t affect test performance.</li></ul><h3>Overall solution description</h3><p>The logical architecture of the solution could look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/1*P3RSRtl22POFVXT8XLVh8A.png" /><figcaption>Logical architecture</figcaption></figure><p>Assuming there is a “System Under Test” (SUT), we use Locust cluster (master and slaves) to stress the system.</p><p>Internal mechanics of Locust determine at which points we are able to collect (and then emit to an external database) test results:</p><ul><li>atomic results (request by requests) are handled by the workers (success handler and failure handler)</li><li>aggregated data are collected by the master from the workers</li></ul><p>To persist atomic data, we’ll need to write some code for the workers. To persist pre-aggregated data — some code running on the master is required.</p><p>In other words:</p><ul><li>SUT is our system under testing</li><li>Locust will do the stress/load testing</li><li>Locust will send the results to Elasticsearch</li></ul><h3>Elasticsearch</h3><p>For this experiment, I am going to use Elasticsearch as the backing database.</p><p>Depending on the technology stack you have in your production or development environment, experience and preferences, you could probably go for some time-series database (InfluxDB, Graphite — you name it) or monitoring solution (Prometheus, Amazon CloudWatch, etc.).</p><p>My choice was purely subjective: I had good experiences using ELK stack before, it’s quite popular, it has simple API and a new version has been just released so I will have a chance to play with it!</p><p>The code I am showing here can be easily adjusted to support the backend of your choice if needed.</p><h4>Elasticsearch basics</h4><p>This article is not about Elasticsearch (ES) itself — it focuses on the mechanics of collecting data by Locust and persisting them in a DB — but let’s spend a few paragraphs on the product so the terminology and ideas used here are clear.</p><p>ES is an open-source tool that started as a full-text search engine based on Apache Lucene. (I am saying “started” because right now it includes many features useful for data analysis, graph traversals or machine learning.) Since its release (in 2010), it has quickly become the most popular search engine (for example, number 1 in search engine category, and first 10 in database engines in general according to <a href="https://db-engines.com/en/ranking">https://db-engines.com/en/ranking</a> as for May 2019).</p><p>Let’s take a look at some concepts.</p><p><strong>Index, document</strong></p><p>Elasticsearch stores data (<em>documents</em>) in <em>indices.</em></p><p>An <strong>index </strong>is a named collection of documents. You could look at an index as something similar to a SQL table. You could have for example an index for users, another one for products, orders, whatever.</p><p>In a single cluster, there can be as many indexes as needed.</p><p><strong>Documents</strong> in Elasticsearch are represented in JSON format. Hence, to store a document you send a JSON object to ES and when retrieving a document, you’ll get JSON object as a response.</p><p>When stored in some <em>index, </em>it’s given a unique ID. Within an index, you can store as many documents as you want.</p><p>The following API call stores a document in <em>myindex </em>under id = 1:</p><pre>curl -X PUT &quot;localhost:9200/myindex/_doc/1&quot; \ <br>     -H &#39;Content-Type: application/json&#39; -d&#39;<br>{<br>  &quot;metric&quot;: &quot;average&quot;,<br>  &quot;value&quot;: 7<br>}&#39;</pre><p><strong>Scalability and high availability</strong></p><p>ES gives the ability to manipulate large datasets. It does that by providing the ability to partition any index into multiple smaller pieces called <strong>shards</strong>. Each shard is, in fact, a fully-functional index that can be hosted on any node in the cluster.</p><p>Elasticsearch handles shards smartly, so you can<strong> horizontally scale</strong> the cluster and have better <strong>performance </strong>when searching — searches can be distributed and parallelized across shards.</p><p>Another positive aspect of Elasticsearch is <strong>high availability</strong>. It allows you to make one or more copies of your index’s shards (called <strong>replicas</strong>) to store them on different nodes. When a node fails, one replica will take over the role of the primary shard.</p><p><strong>Indexing, mapping</strong></p><p>When you store your document in Elasticsearch, it processes it using <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html"><strong>mapping</strong></a><strong> </strong>to be able to search it or analyze it more effectively. As a user, you can decide which fields of the document need to be stored, which don’t, what type they will have (string, date, number, …), which will be subject of full-text search, etc.</p><p>Moreover, the text undergoes a process called <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis.html"><strong>analysis</strong></a>. This includes breaking down a text into smaller portions (tokens, terms), removing stopwords (frequent words as “and”, “the”, etc.) and r<a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/stemming.html">educing words to their root form</a> (removing the difference between singular and plural form, the difference between lowercase and uppercase, difference between tenses, etc.)</p><p>This way Elasticsearch prepares some look-up indexes for itself for fast and accurate searches.</p><p><strong>Querying</strong></p><p>Retrieving a document, e.g. the one we’ve indexed a few paragraphs before — when you know in which index it resides and the id of the document — can be done with a call like this:</p><pre>curl -X GET &quot;localhost:9200/myindex/_doc/1?pretty&quot; \<br>       -H &#39;Content-Type: application/json&#39;<br>{<br>  &quot;_index&quot; : &quot;myindex&quot;,<br>  &quot;_type&quot; : &quot;_doc&quot;,<br>  &quot;_id&quot; : &quot;1&quot;,<br>  &quot;_version&quot; : 1,<br>  &quot;_seq_no&quot; : 0,<br>  &quot;_primary_term&quot; : 1,<br>  &quot;found&quot; : true,<br>  &quot;_source&quot; : {<br>    &quot;metric&quot; : &quot;average&quot;,<br>    &quot;value&quot; : 7<br>  }<br>}</pre><p>But more interesting usage patterns can be delivered with <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html">Query DSL</a>. It allows for combining multiple search/filter condition, including:</p><ul><li>match the exact value, match value greater/smaller than a given one,</li><li>test existence of a value (match documents that have a given attribute set/non-empty),</li><li>text search related (match phrase, match the exact text, regex match, match word similar to a given one, etc.),</li><li>geo match (find objects that are located in a given area, find an object within given distance).</li></ul><p>Which is more, the Elasticsearch API allows for different kinds of aggregations. I won’t spend time describing them here, we’ll use some of them to get some insights into the results later on.</p><p><strong>More info</strong></p><p>I’ve just only mentioned some of the terms relevant to Elasticsearch discussion. They are probably obvious things to some of you, but if you are new to Elasticsearch, I’d suggest some good readings from the producer:</p><ul><li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-concepts.html">Basic Concepts | Elasticsearch Reference [7.0] | Elastic</a></li><li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html">Query DSL | Elasticsearch Reference [7.6] | Elastic</a></li><li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html">Aggregations</a></li></ul><h3>Project code walkthrough</h3><p>After this short interlude, let’s go back straight to business.</p><p>The git project accompanying this article contains that addresses these areas:</p><ul><li>test logic — example test code that is producing some results (so we can collect them and send to Elasticsearch, obviously),</li><li>intercepting test results — code for collecting the results (both pre-aggregated and individual),</li><li>storing data in Elasticsearch — code for sending the results to Elasticsearch,</li><li>dealing with communication costs — recipes for making database communication a bit more efficient</li></ul><p>The following sections try to explain some implementation details.</p><h4>Test logic</h4><p>To make the results a bit more interesting, I decided to use <em>httpbin </em>server as the system under test (SUT). It is a very useful development tool that is able, among other things, to:</p><ul><li>Receive different HTTP requests (GET, POST, etc.),</li><li>Respond with different payloads (JSON, XML, etc.),</li><li>Simulate different response lengths and times.</li></ul><p>The following test code will produce numbers that allow for a demonstration of some analysis (for example: find the slowest requests — which wouldn’t make much sense if all the responses would look almost identical).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7bb41f5adaa816901a61cfbb2351e6ea/href">https://medium.com/media/7bb41f5adaa816901a61cfbb2351e6ea/href</a></iframe><p>You see that few different HTTP methods being used (again, to have more diverse results) and some error response produced intentionally (<em>/status/404</em> endpoint). One part to be noted is line 4.</p><p>First of all, it uses <em>/delay/</em> endpoint with a random wait value. This will cause <em>httpbin </em>to “process” the request for the given time length.</p><p>Secondly, gevent.spawn is used, so Locust doesn’t block on the call and go on with other requests instead (so we can have many more requests during the same time).</p><h4>Intercepting test results</h4><p>As we planned at the beginning, we’ll try to collect both individual and aggregated results from the tests.</p><p>In Locust, <strong>individual results</strong> are produced every time some request is issued. You can intercept them using <a href="https://docs.locust.io/en/stable/api.html#locust.events.request_success">request_success</a> and <a href="https://docs.locust.io/en/stable/api.html#locust.events.request_failure">request_failure</a> event hooks. The mechanics were already described in <a href="https://medium.com/locust-io-experiments/locust-io-experiments-enriching-results-183d2ae4a4c2">Locust.io experiments — Enriching results</a>, so I won’t repeat myself.</p><p>Take a quick look at the code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/457aa22176cef5ce33eb640bd6b5a324/href">https://medium.com/media/457aa22176cef5ce33eb640bd6b5a324/href</a></iframe><p>This method will be added as an additional <em>success_handler</em>. Its job will be to prepare a message to be stored in an external database and use a forwarder object to remember it (soon, I’ll tell more about the forwarder), and the forwarder will take care of the actual sending.</p><p>As for <strong>aggregated results</strong>, Locust slaves collect atomic results and combine them to compute aggregations (relevant Locust code is located in <a href="https://github.com/locustio/locust/blob/0.10.0/locust/stats.py">stats.py</a> if you are interested) and sent to the master.</p><p>You can leverage <a href="https://docs.locust.io/en/stable/api.html#locust.events.slave_report">slave_report</a> event hook to get access to the data. In our case, we have <a href="https://github.com/karol-brejna-i/locust-experiments/blob/emitting-results/sending-results/locust-scripts/locustfile.py#L30-L43">report_data_producer</a> method added as the <em>slave_report</em> <em>event </em>listener:</p><pre>locust.events.slave_report += report_data_producer</pre><p>For the sake of the complicity, the code sends some portion of the pre-aggregated results to Elasticsearch. On the other hand, this is not that interesting from the point of storing request data in some external database and being able to analyze them.</p><p>Some details on <em>slave_report</em> payload are explained in the companion git repository. Let’s proceed with the more important stuff.</p><h4>Storing data in Elasticsearch</h4><p>I decided to use the <a href="https://pypi.org/project/elasticsearch/">official Python client</a>. Please, take a look at the example from <a href="https://github.com/elastic/elasticsearch-py/blob/7.0.1/docs/index.rst#example-usage">the documentation of that package</a>.</p><p>In order to save a document in Elasticsearch, you need to connect to the database first (es = Elasticsearch()) and then simply index the document (es.index(index=”my-index”, id=42, body={“any”: “data”, “timestamp”: datetime.now()}). You can find the relevant code in <a href="https://github.com/karol-brejna-i/locust-experiments/blob/emitting-results/sending-results/locust-scripts/tools/elastic.py">https://github.com/karol-brejna-i/locust-experiments/blob/emitting-results/sending-results/locust-scripts/tools/elastic.py</a>.</p><p>Note, that indexing operation (see: “Index, document” section) is actually translated to an HTTP call. Such a call takes time…</p><p>Let’s see what we can do about smoothing out the costs.</p><h4>Dealing with communication costs</h4><p>As communication with the external database is expensive, we’ll try to minimize the impact of sending the information about every single request.</p><p>Here is the proposed solution to this problem.</p><p>First, we’ll use an <strong>internal buffer</strong> to store the information. While collecting the results we will only do a write to this buffer. This will be fast.</p><p>Then we can have a <strong>dedicated routine</strong> that will read data from that buffer and<strong> take care of the actual sending</strong>. The code can run in the background (namely in its own greenlet) — at its own pace. This way the communication won’t disturb the execution of the test scripts.</p><p>The piece of code responsible for that is a DBForwarder class.</p><h4>Dealing with different backend systems</h4><p>In this article, we’re storing the results in Elasticsearch database. Underneath, we are sending JSON objects over HTTP. In the future maybe we’ll want to switch to some other DB (or simply add yet another). Then the format of the exchanged data and/or the communication channel can be completely different.</p><p>Please, take a look at <a href="https://github.com/karol-brejna-i/locust-experiments/blob/master/sending-results/locust-scripts/tools/backend_base.py">the following code</a> that implements the features discussed above.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/535c6566372b2d5d18bbe8bc112f3418/href">https://medium.com/media/535c6566372b2d5d18bbe8bc112f3418/href</a></iframe><p>Lines 9 t o17 hold the definition of a base backend adapter. The class declares methods that a concrete database adapter (the object that is responsible for sending data to a given database) should implement.</p><p>Actually, only one method is responsible for the actual work: send is supposed to do just what the name suggests — send some data to the database.</p><p>Then goes the DBForwarder class definition (lines 23–48):</p><ul><li><em>add_backend </em>and <em>remove_backend</em> are used to modify backend lists. For now, the forwarder will send data only to Elasticsearch backend, but in the future we may want to forward results to other databases as well. Then we’ll just add another backend to the forwarder</li><li><em>add</em> is used to queue new data to be sent</li><li><em>run</em> does the heavy lifting: in an infinite loop it takes the data that are stored in the internal queue (line 45) and pushes them to any backend defined in the forwarder (line 48)</li></ul><p>That’s pretty much it.</p><p>What is left is initializing and starting the forwarder “in the background” on master and workers. The following code does that:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee93318a652a2d7173d6456d447be7f4/href">https://medium.com/media/ee93318a652a2d7173d6456d447be7f4/href</a></iframe><h3>​Running the experiment</h3><p>Presented experiments can be run on any platform of choice: bare metal, VM, cloud, Kubernetes, etc. I’ve prepared a set of docker and docker composer files to get the advantage of the simplicity of development with docker.</p><p>The next few sections explain setting up different components of the experiment.</p><h4>Starting Elasticsearch and Kibana</h4><p>Elasticsearch will collect the test results and Kibana can be used for their visualization.</p><p>There are two “flavors” of these components prepared here.</p><p>One defines full 3-node Elasticsearch cluster. You can start it using <em>docker-compose-elastic.yml</em>:</p><pre>docker-compose -f docker-compose-elastic.yml up -d</pre><p>The other one is defined in <em>docker-compose-mini.yml</em>. The following command:</p><pre>docker-compose -f docker-compose-mini.yml up -d</pre><p>will start Elasticsearch in a single-node “development” mode (to simplify things). If you want to have everything “clean” there, you should update the default replication factor for the indices:</p><pre>curl -H &quot;Content-Type: application/json&quot; -XPUT &quot;http://localhost:9200/_template/dev&quot; -d &#39;<br>{<br>  &quot;order&quot;: 0,<br>  &quot;index_patterns&quot;: &quot;locust*&quot;,<br>  &quot;settings&quot;: {<br>    &quot;number_of_replicas&quot;: 0<br>  },<br>  &quot;mappings&quot;: {<br>      &quot;numeric_detection&quot;: true<br>    }<br>}&#39;</pre><p>Otherwise, the indices you create will be under-replicated (the default replication factor is 1 — meaning Elasticserach is trying to create one “copy” of each index — and since we have only one node, there is no way to create the copy) and the cluster will appear unhealthy.</p><p><strong>Validating Elasticsearch deployment</strong><br>If Elasticsearch is up and running, it provides an HTTP-based API for controlling the cluster and accessing the data.</p><p>The simplest way to validate if Elasticsearch is running is by issuing the following request:<br>curl -sS <a href="http://localhost:9200```">http://localhost:9200</a></p><p>After this, you should see something similar to:</p><pre>{<br>  &quot;name&quot; : &quot;elasticsearch1&quot;,<br>  &quot;cluster_name&quot; : &quot;docker-cluster&quot;,<br>  &quot;cluster_uuid&quot; : &quot;SzrjlU8PQ6aS3YK5nD856g&quot;,<br>  &quot;version&quot; : {<br>    &quot;number&quot; : &quot;7.0.1&quot;,<br>    &quot;build_flavor&quot; : &quot;default&quot;,<br>    &quot;build_type&quot; : &quot;docker&quot;,<br>    &quot;build_hash&quot; : &quot;e4efcb5&quot;,<br>    &quot;build_date&quot; : &quot;2019-04-29T12:56:03.145736Z&quot;,<br>    &quot;build_snapshot&quot; : false,<br>    &quot;lucene_version&quot; : &quot;8.0.0&quot;,<br>    &quot;minimum_wire_compatibility_version&quot; : &quot;6.7.0&quot;,<br>    &quot;minimum_index_compatibility_version&quot; : &quot;6.0.0-beta1&quot;<br>  },<br>  &quot;tagline&quot; : &quot;You Know, for Search&quot;<br>}</pre><p>Docker compose scripts should also bring up Kibana. To verify if it’s running, you could open http://localhost:5601/app/kibana in a browser.</p><p>This should display Kibana UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*KjcyoOu5uYQuRzxDqq17fQ.png" /></figure><h4>Starting the tests</h4><p>To start the tests, you could use either <em>docker-compose.yml</em> or <em>docker-compose-headless.yml</em>.</p><p>These docker compose files include the following services:</p><ul><li>master, slave — for creating a cluster</li><li>standalone — single node Locust setup</li><li>httpbin, sut — for emulation of a system under test and request logging</li></ul><p>The first option kicks off Locust with the UI. You’ll be able to see the statistics charts there and start and stop the test manually.</p><p>The second option starts locust without the UI and runs the test automatically for a given period.</p><p>To start the test you need to bring the <strong>test endpoint </strong>(sut, httpbin) and <strong>Locust</strong> setup of your choice.</p><p>For a headless setup you could do:</p><pre>docker-compose -f docker-compose-headless.yml up httpbin sut master slave</pre><p>where you control starting individual services, or start them all together:</p><pre>docker-compose -f docker-compose-headless.yml up</pre><h4>Inspecting the results</h4><p>After running the test for some time, Elasticsearch will hold the results.</p><p>From this moment, you could do whatever analysis you want. Let’s examine a few examples.</p><p>There are several ways of accessing/analyzing the results:</p><ul><li>simple HTTP requests</li><li>programmatically — with a client, like the one we used to send the results (<a href="https://pypi.org/project/elasticsearch/">https://pypi.org/project/elasticsearch/</a>)</li><li>dedicated visualization tool</li></ul><p>Let’s start with a few simple queries (using <em>curl</em>) then turn to Kibana (which is a tool from Elastic that lets you “visualize your Elasticsearch data and navigate the Elastic Stack so you can do anything from tracking query load to understanding the way requests flow through your apps”).</p><h4>Basic statistics</h4><p>Let’s get a few results to see the format it is stored in:</p><pre>curl -Ss -H &quot;Content-Type: application/json&quot; localhost:9200/locust/_search?pretty -d &#39;<br>{<br>  &quot;size&quot;:5,<br>  &quot;query&quot;: {<br>    &quot;match_all&quot;: {}<br>  }<br>}&#39;</pre><p>It will return a JSON object, something like:</p><pre>{<br>  &quot;took&quot; : 3,<br>  &quot;timed_out&quot; : false,<br>  &quot;_shards&quot; : {<br>    &quot;total&quot; : 1,<br>    &quot;successful&quot; : 1,<br>    &quot;skipped&quot; : 0,<br>    &quot;failed&quot; : 0<br>  },<br>  &quot;hits&quot; : {<br>    &quot;total&quot; : {<br>      &quot;value&quot; : 1809,<br>      &quot;relation&quot; : &quot;eq&quot;<br>    },<br>    &quot;max_score&quot; : 1.0,<br>    &quot;hits&quot; : [<br>      {<br>        &quot;_index&quot; : &quot;locust&quot;,<br>        &quot;_type&quot; : &quot;_doc&quot;,<br>        &quot;_id&quot; : &quot;Qz4IZ20BfogUoBnF3JQf&quot;,<br>        &quot;_score&quot; : 1.0,<br>        &quot;_source&quot; : {<br>          &quot;type&quot; : &quot;success&quot;,<br>          &quot;payload&quot; : {<br>            &quot;request_type&quot; : &quot;GET&quot;,<br>            &quot;name&quot; : &quot;/get&quot;,<br>            &quot;result&quot; : &quot;OK&quot;,<br>            &quot;response_time&quot; : 5.785226821899414,<br>            &quot;response_length&quot; : 255,<br>            &quot;other&quot; : { }<br>          }<br>        }<br>      },<br>      ...<br>    ]<br>}</pre><p>We see there is information on if the request was successful or note (<em>type</em> field), and the request details (stored in <em>payload</em> object) as request type, size, name of the endpoint, response time and size.</p><p>Let’s get <strong>five slowest requests</strong>:</p><pre>curl -Ss -H &quot;Content-Type: application/json&quot; localhost:9200/locust/_search -d &#39;<br>{<br>  &quot;query&quot;: {<br>    &quot;match&quot;: {<br>      &quot;type&quot;: &quot;success&quot;<br>    }<br>  },<br>  &quot;sort&quot;: {<br>    &quot;payload.response_time&quot;: &quot;desc&quot;<br>  },<br>  &quot;size&quot;: 5<br>}&#39; | jq &#39;.hits.hits[]._source.payload.response_time&#39;</pre><p>(Please, note, I am using <a href="https://stedolan.github.io/jq/">jq</a> to manipulate/shorten the presented results. You can skip it to see the whole response.)</p><p>It’s time for something more complicated. Let’s inspect <strong>what are different types of results</strong>, different <strong>endpoints and how many results there are</strong> for them:</p><pre>curl -Ss -H &quot;Content-Type: application/json&quot; localhost:9200/locust/_search -d &#39;<br>{<br>  &quot;query&quot;:{<br>    &quot;match_all&quot;: {}<br>  },<br>  &quot;size&quot;:0,<br>  &quot;aggs&quot;:{<br>    &quot;grouping_by_type&quot;:{<br>      &quot;terms&quot;:{<br>        &quot;field&quot;:&quot;type.keyword&quot;<br>      },<br>      &quot;aggs&quot;:{<br>        &quot;grouping_by_name&quot;:{<br>          &quot;terms&quot;:{<br>            &quot;field&quot;:&quot;payload.name.keyword&quot;<br>          }<br>        }<br>      }<br>    }<br>  }<br>}&#39; | jq &#39;.aggregations&#39;</pre><pre>curl -Ss -H &quot;Content-Type: application/json&quot; localhost:9200/locust/_search -d &#39;<br>{<br>  &quot;query&quot;:{<br>    &quot;match&quot;:{<br>      &quot;type&quot;:&quot;success&quot;<br>    }<br>  },<br>  &quot;size&quot;:0,<br>  &quot;aggs&quot;:{<br>    &quot;grouping_by_name&quot;:{<br>      &quot;terms&quot;:{<br>        &quot;field&quot;:&quot;payload.name.keyword&quot;<br>      },<br>      &quot;aggs&quot;:{<br>        &quot;avg_response_time&quot;:{<br>          &quot;avg&quot;:{<br>            &quot;field&quot;:&quot;payload.response_time&quot;<br>          }<br>        }<br>      }<br>    }<br>  }<br>}&#39; | jq &#39;.aggregations&#39;</pre><p>I pick a piece of the whole response to explain how to read the results:</p><pre>{<br>        &quot;key&quot;: &quot;error&quot;,<br>        &quot;doc_count&quot;: 441,<br>        &quot;grouping_by_name&quot;: {<br>          &quot;doc_count_error_upper_bound&quot;: 0,<br>          &quot;sum_other_doc_count&quot;: 0,<br>          &quot;buckets&quot;: [<br>            {<br>              &quot;key&quot;: &quot;/status/404&quot;,<br>              &quot;doc_count&quot;: 441<br>            }<br>          ]<br>        }<br>      }</pre><p>The type of the result is “error” (recorded for request that finished in error), there are 441 docs of that type (<em>doc_count</em>), and there are 441 results for URL <em>/status/404</em>.</p><p>Now, check the <strong>average response times by URL</strong>:</p><pre>curl -Ss -H &quot;Content-Type: application/json&quot; localhost:9200/locust/_search -d &#39;<br>{<br>  &quot;query&quot;:{<br>    &quot;match&quot;:{<br>      &quot;type&quot;:&quot;success&quot;<br>    }<br>  },<br>  &quot;size&quot;:0,<br>  &quot;aggs&quot;:{<br>    &quot;grouping_by_name&quot;:{<br>      &quot;terms&quot;:{<br>        &quot;field&quot;:&quot;payload.name.keyword&quot;<br>      },<br>      &quot;aggs&quot;:{<br>        &quot;avg_response_time&quot;:{<br>          &quot;avg&quot;:{<br>            &quot;field&quot;:&quot;payload.response_time&quot;<br>          }<br>        }<br>      }<br>    }<br>  }<br>}&#39; | jq &#39;.aggregations&#39;</pre><p>With the following results:</p><pre>{<br>  &quot;grouping_by_name&quot;: {<br>    &quot;doc_count_error_upper_bound&quot;: 0,<br>    &quot;sum_other_doc_count&quot;: 0,<br>    &quot;buckets&quot;: [<br>      {<br>        &quot;key&quot;: &quot;/post&quot;,<br>        &quot;doc_count&quot;: 476,<br>        &quot;avg_response_time&quot;: {<br>          &quot;value&quot;: 6.643738065447126<br>        }<br>      },<br>      {<br>        &quot;key&quot;: &quot;/get&quot;,<br>        &quot;doc_count&quot;: 456,<br>        &quot;avg_response_time&quot;: {<br>          &quot;value&quot;: 6.635955547031603<br>        }<br>      },<br>      {<br>        &quot;key&quot;: &quot;/delayed&quot;,<br>        &quot;doc_count&quot;: 400,<br>        &quot;avg_response_time&quot;: {<br>          &quot;value&quot;: 4090.4237358283995<br>        }<br>      }<br>    ]<br>  }<br>}</pre><p>Let’s see some <strong>percentiles</strong>. Send the following request:</p><pre>curl -Ss -H &quot;Content-Type: application/json&quot; localhost:9200/locust/_search -d &#39;<br>{<br>  &quot;query&quot;:{<br>    &quot;match&quot;:{<br>      &quot;type&quot;:&quot;success&quot;<br>    }<br>  },<br>  &quot;size&quot;:0,<br>  &quot;aggs&quot;:{<br>    &quot;load_time_outlier&quot;:{<br>      &quot;percentiles&quot;:{<br>        &quot;field&quot;:&quot;payload.response_time&quot;<br>      }<br>    }<br>  }<br>}&#39; | jq &#39;.aggregations&#39;</pre><p>which produces:</p><pre>{<br>  &quot;load_time_outlier&quot;: {<br>    &quot;values&quot;: {<br>      &quot;1.0&quot;: 4.651503562927246,<br>      &quot;5.0&quot;: 4.971170425415039,<br>      &quot;25.0&quot;: 5.747049384646946,<br>      &quot;50.0&quot;: 6.653642654418945,<br>      &quot;75.0&quot;: 1006.5738342285156,<br>      &quot;95.0&quot;: 8006.144401041666,<br>      &quot;99.0&quot;: 9007.9398046875<br>    }<br>  }<br>}</pre><p>And some even more stats for successful requests to <em>/delayed</em> endpoint:</p><pre>curl -Ss -H &quot;Content-Type: application/json&quot; localhost:9200/locust/_search -d &#39;<br>{<br>  &quot;query&quot;: {<br>    &quot;bool&quot;: {<br>      &quot;filter&quot;: [<br>        { &quot;term&quot;: { &quot;payload.name.keyword&quot;: &quot;/delayed&quot; } },<br>        { &quot;term&quot;: { &quot;type.keyword&quot;: &quot;success&quot; } }<br>      ]<br>    }<br>  },<br>  &quot;size&quot;: 0,<br>  &quot;aggs&quot;: {<br>    &quot;time_stats&quot;: {<br>      &quot;extended_stats&quot;: {<br>        &quot;field&quot;: &quot;payload.response_time&quot;<br>      }<br>    }<br>  }<br>}&#39; | jq &#39;.aggregations.time_stats&#39;</pre><pre>{<br>  &quot;count&quot;: 400,<br>  &quot;min&quot;: 4.570722579956055,<br>  &quot;max&quot;: 9012.703125,<br>  &quot;avg&quot;: 4090.4237358283995,<br>  &quot;sum&quot;: 1636169.4943313599,<br>  &quot;sum_of_squares&quot;: 10237234378.891014,<br>  &quot;variance&quot;: 8861519.608599177,<br>  &quot;std_deviation&quot;: 2976.8304635298223,<br>  &quot;std_deviation_bounds&quot;: {<br>    &quot;upper&quot;: 10044.084662888044,<br>    &quot;lower&quot;: -1863.237191231245<br>  }<br>}</pre><p>The type of the result is “error” (recorded for request that finished in error), there are 441 docs of that type (<em>doc_count</em>), and there are 441 results for URL <em>/status/404</em>.</p><p>Now, check the <strong>average response times by URL</strong>:</p><pre>curl -Ss -H &quot;Content-Type: application/json&quot; localhost:9200/locust/_search -d &#39;<br>{<br>  &quot;query&quot;:{<br>    &quot;match&quot;:{<br>      &quot;type&quot;:&quot;success&quot;<br>    }<br>  },<br>  &quot;size&quot;:0,<br>  &quot;aggs&quot;:{<br>    &quot;grouping_by_name&quot;:{<br>      &quot;terms&quot;:{<br>        &quot;field&quot;:&quot;payload.name.keyword&quot;<br>      },<br>      &quot;aggs&quot;:{<br>        &quot;avg_response_time&quot;:{<br>          &quot;avg&quot;:{<br>            &quot;field&quot;:&quot;payload.response_time&quot;<br>          }<br>        }<br>      }<br>    }<br>  }<br>}&#39; | jq &#39;.aggregations&#39;</pre><p>With the following results:</p><pre>{<br>  &quot;grouping_by_name&quot;: {<br>    &quot;doc_count_error_upper_bound&quot;: 0,<br>    &quot;sum_other_doc_count&quot;: 0,<br>    &quot;buckets&quot;: [<br>      {<br>        &quot;key&quot;: &quot;/post&quot;,<br>        &quot;doc_count&quot;: 476,<br>        &quot;avg_response_time&quot;: {<br>          &quot;value&quot;: 6.643738065447126<br>        }<br>      },<br>      {<br>        &quot;key&quot;: &quot;/get&quot;,<br>        &quot;doc_count&quot;: 456,<br>        &quot;avg_response_time&quot;: {<br>          &quot;value&quot;: 6.635955547031603<br>        }<br>      },<br>      {<br>        &quot;key&quot;: &quot;/delayed&quot;,<br>        &quot;doc_count&quot;: 400,<br>        &quot;avg_response_time&quot;: {<br>          &quot;value&quot;: 4090.4237358283995<br>        }<br>      }<br>    ]<br>  }<br>}</pre><p>Let’s see some <strong>percentiles</strong>. Send the following request:</p><pre>curl -Ss -H &quot;Content-Type: application/json&quot; localhost:9200/locust/_search -d &#39;<br>{<br>  &quot;query&quot;:{<br>    &quot;match&quot;:{<br>      &quot;type&quot;:&quot;success&quot;<br>    }<br>  },<br>  &quot;size&quot;:0,<br>  &quot;aggs&quot;:{<br>    &quot;load_time_outlier&quot;:{<br>      &quot;percentiles&quot;:{<br>        &quot;field&quot;:&quot;payload.response_time&quot;<br>      }<br>    }<br>  }<br>}&#39; | jq &#39;.aggregations&#39;</pre><p>which produces:</p><pre>{<br>  &quot;load_time_outlier&quot;: {<br>    &quot;values&quot;: {<br>      &quot;1.0&quot;: 4.651503562927246,<br>      &quot;5.0&quot;: 4.971170425415039,<br>      &quot;25.0&quot;: 5.747049384646946,<br>      &quot;50.0&quot;: 6.653642654418945,<br>      &quot;75.0&quot;: 1006.5738342285156,<br>      &quot;95.0&quot;: 8006.144401041666,<br>      &quot;99.0&quot;: 9007.9398046875<br>    }<br>  }<br>}</pre><p>And some even more stats for successful requests to <em>/delayed</em> endpoint:</p><pre>curl -Ss -H &quot;Content-Type: application/json&quot; localhost:9200/locust/_search -d &#39;<br>{<br>  &quot;query&quot;: {<br>    &quot;bool&quot;: {<br>      &quot;filter&quot;: [<br>        { &quot;term&quot;: { &quot;payload.name.keyword&quot;: &quot;/delayed&quot; } },<br>        { &quot;term&quot;: { &quot;type.keyword&quot;: &quot;success&quot; } }<br>      ]<br>    }<br>  },<br>  &quot;size&quot;: 0,<br>  &quot;aggs&quot;: {<br>    &quot;time_stats&quot;: {<br>      &quot;extended_stats&quot;: {<br>        &quot;field&quot;: &quot;payload.response_time&quot;<br>      }<br>    }<br>  }<br>}&#39; | jq &#39;.aggregations.time_stats&#39;</pre><pre>{<br>  &quot;count&quot;: 400,<br>  &quot;min&quot;: 4.570722579956055,<br>  &quot;max&quot;: 9012.703125,<br>  &quot;avg&quot;: 4090.4237358283995,<br>  &quot;sum&quot;: 1636169.4943313599,<br>  &quot;sum_of_squares&quot;: 10237234378.891014,<br>  &quot;variance&quot;: 8861519.608599177,<br>  &quot;std_deviation&quot;: 2976.8304635298223,<br>  &quot;std_deviation_bounds&quot;: {<br>    &quot;upper&quot;: 10044.084662888044,<br>    &quot;lower&quot;: -1863.237191231245<br>  }<br>}</pre><h3>Summary</h3><p>In this article, I wanted to demonstrate, that sending results to an external database could be easy. Storing data in the DB gives us some additional options. These include:</p><ul><li>being able to search or drill down the results,</li><li>filter or aggregate the data in the way we need,</li><li>compare results among different test runs,</li><li>have some alerting in place (for example, publishing warnings on a slack channel if there are some errors),</li><li>etc. etc.</li></ul><p>In the implementation, I tried to make it easy to implement alternative backend (so data are sent to some other database than Elasticsearch) or additional backend (the data are sent to multiple databases).</p><p>The code itself is “demonstrate-the-problem quality”. I wanted it to be simple. There is certainly room for improvement. For example, Elasticsearch has the batching capability — in other words, you can send a whole bulk of data in one request. This could improve performance greatly.</p><p>Thank You for your patience.</p><p>As usual, the sources mentioned in this article are stored in <a href="https://github.com/karol-brejna-i/locust-experiments">https://github.com/karol-brejna-i/locust-experiments</a>. <em>sending-results</em> folder holds locus files, docker compose files, etc. (see the readme for details).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd3de1994259" width="1" height="1" alt=""><hr><p><a href="https://medium.com/locust-io-experiments/locust-io-experiments-emitting-results-to-external-db-bd3de1994259">Locust.io experiments — Emitting results to external DB</a> was originally published in <a href="https://medium.com/locust-io-experiments">Locust.io experiments</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Locust Experiments — Docker image updates]]></title>
            <link>https://medium.com/locust-io-experiments/locust-experiments-docker-image-updates-c7576e410efa?source=rss----84c32fc5c3de---4</link>
            <guid isPermaLink="false">https://medium.com/p/c7576e410efa</guid>
            <category><![CDATA[experiment]]></category>
            <category><![CDATA[locust]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[load-testing]]></category>
            <dc:creator><![CDATA[Karol Brejna]]></dc:creator>
            <pubDate>Wed, 10 Apr 2019 06:13:04 GMT</pubDate>
            <atom:updated>2019-04-09T12:21:00.730Z</atom:updated>
            <content:encoded><![CDATA[<h3>Locust Experiments — Docker image updates</h3><p>Some time ago I did an experiment on dockerizing Locust.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-7Cclq-G7tiplEwn" /><figcaption>Photo by <a href="https://unsplash.com/@fancycrave?utm_source=medium&amp;utm_medium=referral">Fancycrave</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The results are stored in the following repo: <a href="https://github.com/karol-brejna-i/locust-experiments/tree/master/docker-image">https://github.com/karol-brejna-i/locust-experiments/tree/master/docker-image</a>. The image itself is hosted on docker hub ( <a href="https://hub.docker.com/r/grubykarol/locust">https://hub.docker.com/r/grubykarol/locust</a>).</p><p>I’ve been using this image in Kubernetes clusters and in my Locust experiments with success.</p><p>Recently a new version of Locust was released so I decided to use this as an excuse to refresh the docker image and show information on how to use it.</p><h3>GitHub repository</h3><p>I decided to create a dedicated repository for the project. If you visit <a href="https://github.com/karol-brejna-i/docker-locust">https://github.com/karol-brejna-i/docker-locust</a>, you’ll notice that the repo is organized so it can maintain many versions of the image.</p><p>Tagging “system” was chosen to easily distinguish between the versions. It reflects the directory structure and contains information about all the components of the image: Locust version, Python version, and the OS version.</p><p>Folder structure looks like this:</p><pre>   .<br>   |-- locust version a<br>   |   |--- python version x<br>   |   |    |--  OS version 1 <br>   |   |    `--  OS version 2 <br>   |   `--- python version y<br>   |        |---  OS version 1 <br>   |        `---  OS version 2 <br>   |-- locust version b<br>   ...</pre><p>Where:</p><ul><li>locust version is a specific version of Locust (i.e. 0.9.0)</li><li>python version is a specific version of Python (i.e. 3.6)</li><li>OS version 1 is a specific version of the operating system (i.e. alpine3.9)</li></ul><p>For example, for locust 0.10.0 that runs on Python 3.6 on Alpine 3.9 the dockerfile is placed in:</p><pre>\<br>   |--- 0.10.0<br>   | `-- python3.6<br>   |   `-- alpine3.9</pre><p>The tag for this image will be:</p><blockquote>grubykarol/locust:0.10.0-python3.6-alpine3.9</blockquote><h3>Usage</h3><p>The image does not include locust scripts during a build. It expects that the scripts will be supplied on runtime by mounting a volume (to /locust path).</p><p>This gives the ability to use the exact same image for different deployments. There is no need to build your image that would inherit from this one and only include test scripts (although it’s also possible).</p><h4>Pulling the image</h4><p>Pull the latest stable version:</p><pre>docker pull grubykarol/locust</pre><p>Or choose Locust, Python and OS (Operating System) version you want and pull and the image that is tagged accordingly:</p><pre>docker pull grubykarol/locust:0.10.0-python3.6-alpine3.9</pre><h4>Running the image</h4><p>The image uses the following environment variables to configure its behavior:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ecc3e5a899898c7a47a1c2d0c6216899/href">https://medium.com/media/ecc3e5a899898c7a47a1c2d0c6216899/href</a></iframe><p>In order to run your tests with Locust, you need to provide it with test sources. The image doesn’t any. It assumes the sources will be delivered, for example by mounting as a volume (/locust) during runtime. So, in the simplest case, for a <strong>standalone node (</strong>with a <em>/usr/scirpts</em> host folder holding locustfile.py), a basic run command could look like this:</p><pre>docker run --rm --name standalone \<br>  --hostname standalone \<br>  -e ATTACKED_HOST=http://standalone:8089 \<br>  -p 8089:8089 -d \<br>  <strong>-v /usr/scirpts</strong><strong>:/locust</strong> grubykarol/locust</pre><p>or, with additional runtime options to run without the UI:</p><pre>docker run --rm --name standalone \<br>  --hostname standalone \<br>  -e ATTACKED_HOST=http://example.com \<br>  -e &quot;LOCUST_OPTS=--no-web&quot; \<br>  -v /usr/scirpts:/locust grubykarol/locust</pre><p>For <strong>distributed mode</strong>, you could run the <strong>master </strong>with:</p><pre>docker run --name master --hostname master \<br> -p 8089:8089 -p 5557:5557 -p 5558:5558 \<br> -v /usr/scirpts:/locust \<br> -e ATTACKED_HOST=http://master:8089 \<br> -e LOCUST_MODE=master \<br> --rm -d grubykarol/locust</pre><p>and then some <strong>slaves</strong>:</p><pre>docker run --name slave0 \<br> --link master --env NO_PROXY=master \<br> -v /usr/scirpts:/locust \<br> -e ATTACKED_HOST=http://master:8089 \<br> -e LOCUST_MODE=slave \<br> -e LOCUST_MASTER=master \<br> --rm -d grubykarol/locust</pre><pre>docker run --name slave1 \<br> --link master --env NO_PROXY=master \<br> -v /usr/scirpts:/locust \<br> -e ATTACKED_HOST=http://master:8089 \<br> -e LOCUST_MODE=slave \<br> -e LOCUST_MASTER=master \<br> --rm -d grubykarol/locust</pre><h4>Extending the image</h4><p>In some cases, it would probably be convenient to build a dedicated image. For example, to following dockerfile adds locust scripts and installs some Python dependencies:</p><pre>FROM grubykarol/locust:0.11.0-python3.6-alpine3.9<br>COPY locust-scripts /locust<br>ENV ATTACKED_HOST http://example.com<br>RUN  pip install --no-cache-dir elasticsearch&gt;=6.3.1 redis</pre><p>After building the image (docker build -t locust/mytests .) you could run it with:</p><pre>docker run --rm -d -p 8089:8089 locust/mytests</pre><p>No more need for mounting a volume (the scripts had been added while building the image) and no need to set ATTACKED_HOST (it’s done already in the dockerfile).</p><h3>Examples</h3><p>There is a <a href="https://github.com/karol-brejna-i/docker-locust/blob/master/examples">folder in the GitHub repository</a> that contains a few examples of running locust docker images (using Docker Compose). Their goal is to help understand how to use specific settings (environment variables).</p><p>They include configurations for:</p><ul><li>running a standalone node of Locust</li><li>running distributed Locust cluster</li><li>running distributed Locust cluster without web UI with timed execution</li></ul><h3>Conclusions</h3><p>Locust is a great load/stress testing tool. One of the greatest advantages is that is easy to use. Running it with Docker makes things even easier.</p><p>The images that I prepared should work for most (hopefully: all) use cases. Please, let me know if they work/don’t work for you?</p><p>Any feedback can help to improve them. And making someone’s life easier, at the same time ;-)</p><p>Thank you for our attention!</p><p>P.S.</p><p>The sources for this article are stored in the following GitHub repo: <a href="https://github.com/karol-brejna-i/docker-locust">https://github.com/karol-brejna-i/docker-locust</a>.</p><p>Docker images are hosted on docker hub: <a href="https://hub.docker.com/r/grubykarol/locust">https://hub.docker.com/r/grubykarol/locust</a>.</p><p>ATTOW they include images for Locust 0.8.1, 0.9.0, 0.10.0, 0.11.0 (although I am not sure which versions were officially released ;-) ) for Python 3.6 and Alpine 3.9.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7576e410efa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/locust-io-experiments/locust-experiments-docker-image-updates-c7576e410efa">Locust Experiments — Docker image updates</a> was originally published in <a href="https://medium.com/locust-io-experiments">Locust.io experiments</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Locust experiments — Feeding the locusts]]></title>
            <link>https://medium.com/locust-io-experiments/locust-experiments-feeding-the-locusts-cf09e0f65897?source=rss----84c32fc5c3de---4</link>
            <guid isPermaLink="false">https://medium.com/p/cf09e0f65897</guid>
            <category><![CDATA[experiment]]></category>
            <category><![CDATA[locust]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[load-testing]]></category>
            <dc:creator><![CDATA[Karol Brejna]]></dc:creator>
            <pubDate>Mon, 08 Apr 2019 11:34:05 GMT</pubDate>
            <atom:updated>2019-04-09T06:56:40.568Z</atom:updated>
            <content:encoded><![CDATA[<h3>Locust experiments — Feeding the locusts</h3><p>There are times when you want your Locust scripts to use some specific input data in order to perform the tests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-aPLYhFiPiif7FtZ" /><figcaption>“Closeup of eyes and antennae on a grasshopper” by <a href="https://unsplash.com/@borisworkshop?utm_source=medium&amp;utm_medium=referral">Boris Smokrovic</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>For example:</p><ul><li>you want to use a list of users and their credentials in order to log in to some service,</li><li>you have a set of form values that need to be submitted,</li><li>you need to upload specific files as a part of requests to your service,</li><li>you want to have repeatable/comparable test conditions (by providing the exact same input data),</li><li>you have some data that exploits the service’s corner cases (validation, special values, special characters, etc.).</li></ul><p>In this “episode”, I’ll try to examine one of the ways of addressing such problems.</p><p>Along the way, the following “techniques” will be used here, which can also be useful in other Locust tests developments:</p><ul><li>determining if the code runs on master or slave,</li><li>doing some preparations before the tests start,</li><li>sending some data from the master to workers.</li></ul><h3>The use-case</h3><p>Let’s write down our requirements.</p><p>Assume, that we are testing a web service’s endpoint that accepts<strong> POST requests</strong> with JSON payload.</p><p>For this, we want to use a <strong>specific data set</strong>. Let’s say, we have a <strong>CSV file </strong>where every row represents a single message to be sent. We want to be able to read the content <strong>before the tests start</strong>.</p><p>Furthermore, we require being able to run the test in a <strong>distributed </strong>manner — running multiple slave workers that will issue the POST request.</p><p>Moreover, we don’t want to have <strong>any duplicates </strong>(sending the same value more than once).</p><h3>The plan</h3><p>So, we want Locust to be able to read the file and its workers to use the data to test the service, but each record should be utilized just once.</p><p>It seems that in a naive approach — each worker reads the file on its own — it’s hard to make sure that one worker doesn’t use the record that some other worker had already processed.</p><p>When dealing with the problem in a standalone setup (single locust node, no additional workers) there is no problem with coordination among the tasks — they all run in the same “place”. We’d probably build a <em>queue </em>containing the data upfront (before tests start) and then the tasks would consume the records one by one.</p><p>How we could achieve that for a distributed setup?</p><p>In general, in Locust, it’s the master who is in charge of things like spawning new locusts (users), collecting and presenting results, running some preparatory tasks, etc. So, it appears like we need some code that runs on the master and will:</p><ul><li>get the data from the disk</li><li>communicate with the workers, to exchange data</li></ul><h3>Preparations</h3><p>Let’s see what and how we need to do and if we have “tools” for that.</p><h4>Running pre-test actions</h4><p>There are some points in Locust cluster’s or locust test session’s lifecycle that you could use to run some required actions — setting up some resources before tests or cleaning things up afterward. A few of these “moments” are:</p><ul><li>when the cluster starts</li><li>when the tests start (for example, a user presses “Start swarming” button on the UI)</li><li>when all the locusts had hatched (the process of creating new test users had finished)</li></ul><p>The topic itself probably deserves a dedicated article, as it is very interesting and a bit confusing at the same time. Let’s focus for now on what we need to do here.</p><p>So, we need to load the input data. As it is a static CSV file, doing this once is enough (there is no point in rereading the dataset every time the test starts — it’s a waste of resources).</p><p>Then, we want to begin distributing the data every time the tests are started.</p><p><strong>Running code at the startup time</strong></p><p>This case is pretty simple. It’s enough to put the code in the global scope of your locus file. The following will execute <em>do_some_stuff()</em> when the cluster is coming up.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b3a6a523f05b6aa241d7167d77b514f6/href">https://medium.com/media/b3a6a523f05b6aa241d7167d77b514f6/href</a></iframe><p>Please note, that due to GIL nature (and if you won’t take care of it explicitly), if the code is time-consuming, it will “stop the world” — the rest of the code needs to wait for this one to finish.</p><p><strong>Running code before tests</strong></p><p>There are at least a few options here. Without going into too much detail (as I said, the topic is probably in line for an article of his own), I choose to use master_start_hatching event hook for that.</p><pre><strong>def </strong>on_master_start_hatching(**kw):<br>    print(<strong>f&quot;Do some work when the tests are started&quot;</strong>)</pre><pre>events.master_start_hatching += on_master_start_hatching</pre><p>Using the preceding code in the locust file does the required job:</p><ul><li>defines a handler (a function that will do the actual work) for the event</li><li>registers the handler</li></ul><p>(It turns out that master_start_hatching is not fired in standalone mode, so the code will only run properly in distributed mode).</p><p>It looks like we have the topic covered.</p><h4>Detecting the master</h4><p>According to the <a href="https://docs.locust.io/en/0.9.0/running-locust-distributed.html">documentation</a> (ATTOW the latest version is 0.9), as I understand it, master and workers use the same test script file:</p><blockquote>Both the master and each slave machine, must have a copy of the locust test scripts when running Locust distributed.</blockquote><p>Because we are going to run some code on the master exclusively (not on the slaves), we need to find a way to detect the current running context.</p><p><strong>The context of running tests</strong></p><p>If you take a look into Locust code, you’ll find <a href="https://github.com/locustio/locust/blob/0.9.0/locust/runners.py">runners</a> module with classes responsible for managing locusts lifecycle. There are subclasses dedicated to master, slave and standalone nodes, that can be utilized like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3f1d59d1b37d05bf3215570c6b2991c2/href">https://medium.com/media/3f1d59d1b37d05bf3215570c6b2991c2/href</a></iframe><p>Please remember, that when running in standalone mode, the node plays a role of both master and slave, so you may want to consider this if you want your code to be portable between standalone and distributed mode.</p><p><strong>The context of locust file code</strong></p><p>If there is some code running in the global scope of your locus file, the following can be used to detect where it runs:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bb2388e0cd4fae97ed6930e0b5bef970/href">https://medium.com/media/bb2388e0cd4fae97ed6930e0b5bef970/href</a></iframe><h4>Reading CSV files</h4><p>I used <em>csv</em> package here to create a simple object which sole responsibility is to read the file and return an array of dictionary objects (so attributes can be accessible by CSV column name):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9a24825da6bf069ae85e5dd7074adb85/href">https://medium.com/media/9a24825da6bf069ae85e5dd7074adb85/href</a></iframe><h4>Master-slave communication</h4><p>First of all, let’s figure out how the master could distribute some information among its slaves.</p><p>There could be many approaches here, but for this experiment, let’s choose one that:</p><ul><li>is simple enough to illustrate the principle,</li><li>doesn’t introduce new dependencies (new external components, frameworks).</li></ul><p>Well, Locust comes with <strong>ZeroMQ </strong>as its communication “backend”. Browsing quickly through the library’s documentation it appears it covers some messaging patterns including the one that looks of particular interest to us:</p><p><a href="http://zguide.zeromq.org/page:all#Divide-and-Conquer">ØMQ - The Guide - ØMQ - The Guide</a></p><p>Take a look at the following diagram (taken from 0mq docs)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*i4aCFLAyrr_O4eDpeooeLQ.png" /><figcaption>source: <a href="https://github.com/imatix/zguide/raw/master/images/fig5.png">https://github.com/imatix/zguide/raw/master/images/fig5.png</a></figcaption></figure><p>In short: The ventilator pushes tasks to the workers. Workers pull the data and do their job. The workers send their results to a sink.</p><p>In our case:</p><ul><li>Maser would play the role of the ventilator</li><li>Tasks running on slaves will act as workers</li><li>we are not interested in sending any results from the workers (no pushing to Sink is required)</li></ul><p>The steps cover the scenario of input data distribution that we’ve planned and ZeroMQ would be more than enough for this experiment, then.</p><h3>The execution</h3><p>OK. Here goes the spoiler: ventilator-workers pattern looked really great, but it doesn’t work so great for me. (Of course, I know that only after writing all the code already.)</p><h4>Disappointment</h4><p>After running some tests (with different numbers of workers, different hatching rates) I saw data distribution acted really weird.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgfycat.com%2Fifr%2Flastingneedyborer&amp;url=https%3A%2F%2Fgfycat.com%2Flastingneedyborer&amp;image=https%3A%2F%2Fthumbs.gfycat.com%2FLastingNeedyBorer-size_restricted.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=gfycat" width="178" height="100" frameborder="0" scrolling="no"><a href="https://medium.com/media/f9be042597365149d2598f2bb6dcf107/href">https://medium.com/media/f9be042597365149d2598f2bb6dcf107/href</a></iframe><p>The messages’ order was greatly disturbed and the timing was, let’s call it, less than optimal.</p><p>I don’t have a definite explanation for this yet, only a working theory.</p><p><a href="http://api.zeromq.org/4-2:zmq-socket#toc15">The docs</a> say:</p><blockquote>A socket of type <em>ZMQ_PUSH</em> is used by a pipeline <em>node</em> to send messages to downstream pipeline <em>nodes</em>. Messages are round-robined to all connected downstream <em>nodes</em>.</blockquote><p>If the receivers are chosen with a simple round-robin, without knowing if they are able to consume the message, there is no real load balancing going on there. I think the following happens:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*_XTp9zutyWvIYNV6Hv8vJw.png" /><figcaption>Push-pull based distribution</figcaption></figure><p>On the left, we have an outline of an ideal situation. On the right, we have a worker that is about to receive a message, but it’s busy. I suspect that sending blocks (and stays blocked until the worker is finally ready for the message), messing up the entire flow.</p><h4>New hope</h4><p>I changed the approach for data distribution for the one that allows the workers for “volunteering” to receive a message when they are ready for it. In case the worker is busy, it will simply not take part in the data exchange not preventing other workers from consuming the messages:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*Qjffwi9NiptHBKl6WTkHoA.png" /><figcaption>Request-response based distribution</figcaption></figure><p>I’ve kept the problematic code in <a href="https://github.com/karol-brejna-i/locust-experiments/blob/master/feeding-locusts/locust-scripts/locustfile_pushpull.py">locust-scripts/locustfile_pushpull.py</a> for further investigation or for reader’s inquiring mind and I am focusing on the new solution.</p><p>The key elements here are:</p><ul><li>code for sending the data to workers</li><li>code requesting new data</li><li>code initializing the process</li></ul><p>Let’s quickly go through them…</p><p>Sending logic is quite simple:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/44158a6ae1b7b3a1ade171348d109ec3/href">https://medium.com/media/44158a6ae1b7b3a1ade171348d109ec3/href</a></iframe><p>The class uses an internal queue to store the input data (data to be sent). It populates the queue on init (lines 3–4).</p><p>The <em>run</em> method loops forever and waits (blocks) on a new message to be received (line 15). If the message text is “available”, it tries to get a portion of data from the queue. If the queue is empty it sends a special message back (empty JSON object, line 25) to inform the client that there is nothing more to do, otherwise, it sends proper data to the worker (line 20).</p><p>The feeder object can be initialized like this:</p><pre><strong>def </strong>init_feeder():<br>    sender = ZMQFeeder(INPUT_DATA, <br>                <strong>f&quot;tcp://0.0.0.0:{</strong>FEEDER_BIND_PORT<strong>}&quot;</strong>)<br>    sender.run()</pre><pre><strong>def </strong>on_master_start_hatching():<br><em>    </em>gevent.spawn(init_feeder)</pre><pre>events.master_start_hatching += on_master_start_hatching</pre><p>It will be executed on the master when the tests will be started. An important piece of information here is, that the feeder will work in its own greenlet and will not block other code running on the master.</p><p>Receiving part is even simpler:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/58c3c86272a21bf5a897841a5a95b505/href">https://medium.com/media/58c3c86272a21bf5a897841a5a95b505/href</a></iframe><p><em>await_data</em> method’s job is to ask the master for new data (l. 11) and receive and return it when available. Then, the test code that utilizes the class will run on the slaves and could look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a11a9db6cb03a065f8a782a98b6f15e1/href">https://medium.com/media/a11a9db6cb03a065f8a782a98b6f15e1/href</a></iframe><p>On start, it initializes the requester (line 3). The actual work is happening in <em>task1 </em>method: it waits for new data to be delivered (line 6) and uses it for making the request (l. 11) if something meaningful was sent.</p><p>Now, that we have all the code in place (see: locust-scripts/locustfile.py), let’s try it out.</p><h3>Testing</h3><p>For testing the solution we need to run Locust in distributed mode. You could do this by manually running master and slaves yourself from the command line. Another option here is doing it with kubernetes:</p><p><a href="https://medium.com/locust-io-experiments/locust-io-experiments-running-in-kubernetes-95447571a550">Locust.io experiments — running in Kubernetes</a></p><p>This time, on the other hand, I’ll use Docker Compose (my Kubernetes doesn’t want to get up after last Windows update…).</p><h4>Setting up the cluster</h4><p>For setting up the cluster two versions of docker compose configuration were created: one for running Locust headless (without web UI) and one for running the UI, too.</p><p>Docker compose files include the definition of services for Locust master and locust slave with all required environment variables set and a volume containing test code configured. Take a look at docker-compose-headles.yml, for example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d2d7956724ea84198a314782a81da01/href">https://medium.com/media/7d2d7956724ea84198a314782a81da01/href</a></iframe><p>You will notice, that the test will run for 10 seconds (-t 10s) with no GUI (--no-web). You will also notice the settings for the feeder (host, port), the delay between the tasks (TASK_DELAY) and the address of web service we are testing (ATTACKED_HOST).</p><h4>Tested service mock</h4><p>The docker compose file also contains a small section for simulating the system under test was included.</p><p>The whole use case (to remind you) is about reading some input data and using them to form POST requests. In order not to write anything here, I quickly googled out docker image that will play the role of the tested system.</p><p>The service is able to accept different requests, including POST, and it will simply log the request (with the body) so we can confirm which data was used.</p><p>Maybe mocking the system under test could be easier. For me, using “kennethreitz/httpbin” and “lucascimon/nginx-logging-proxy” docker images was the quickest.</p><h4>Test data</h4><p>This experiment assumes there is SourceDataReader class that reads data.csv file. The code “doesn’t care” about the format of data it uses. It is written in such a way that after reading the CSV file, it returns an array of dictionary objects (so attributes can be accessible by CSV column name).</p><p>The test data I used is the list of top-ranked chess players that contains 1000 records. I had obtained the test data with data/obtain_elo_ranks.py. See data/README.md for the details.</p><p>If you want to play with your own data, make sure it is a comma-separated CSV file (with a header).</p><h4>Results</h4><p>Let’s start the tests:</p><p>docker-compose -f docker-compose-headless.yml up</p><p>This will bring up all the containers — Locust master, slave, tested service — and start logging in the console.</p><p>The logs will represent the order of events: starting the nodes, reading the input data by the master, starting the “feeder” and then requesting new data by the slave and using them to issue POST request:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c67486bfa2782a5c0c9b2ff6c508a02/href">https://medium.com/media/3c67486bfa2782a5c0c9b2ff6c508a02/href</a></iframe><p>It will probably take some time for you to find some order in this.</p><p>In the real code (see the associated GitHub project), I had also added appending the data received by slaves to a text file (with timestamps) — <em>received.txt</em>. This way it is easier to track the data distribution.</p><h3>Conclusions</h3><p>It looks like the presented way of distributing data to Locust slaves works quite well. Writing the communication code using ZeroMQ was quite easy. On the other hand, quite a thorough understanding of ZeroMQ internals was required.</p><p>The implementation is probably enough for most of the simple use-cases.</p><p>There are topics here, though, that would require more thoughts. For example, should we stop the tests when we ran out of input data? If yes, then how? Expanding on the topic would give some possibilities for more sophisticated testing. Probably a topic for some future experiment…</p><p>Anyhow, I hope you can find the article helpful and informative.</p><p>Thank You!</p><p>Few last formalities:</p><p>As usual, the sources mentioned in this article are stored in <a href="https://github.com/karol-brejna-i/locust-experiments">https://github.com/karol-brejna-i/locust-experiments</a>. <em>feeding-locusts </em>folder holds locus files, docker compose files, etc. (see the readme for details).</p><p>A docker image for running Locust 0.9.0 on Python 3.6 was used ( source: <a href="https://github.com/karol-brejna-i/docker-locust">https://github.com/karol-brejna-i/docker-locust</a>)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf09e0f65897" width="1" height="1" alt=""><hr><p><a href="https://medium.com/locust-io-experiments/locust-experiments-feeding-the-locusts-cf09e0f65897">Locust experiments — Feeding the locusts</a> was originally published in <a href="https://medium.com/locust-io-experiments">Locust.io experiments</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Locust experiments — Locust meets Kafka]]></title>
            <link>https://medium.com/locust-io-experiments/locust-experiments-locust-meets-kafka-b9d9e6e49537?source=rss----84c32fc5c3de---4</link>
            <guid isPermaLink="false">https://medium.com/p/b9d9e6e49537</guid>
            <category><![CDATA[load-testing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[locust]]></category>
            <dc:creator><![CDATA[Karol Brejna]]></dc:creator>
            <pubDate>Mon, 21 May 2018 17:09:55 GMT</pubDate>
            <atom:updated>2019-10-29T12:03:35.425Z</atom:updated>
            <content:encoded><![CDATA[<p>This time, let’s teach Locust how to test systems using Apache Kafka.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*p4GKB3WpQ9u9npE43dhPHg.jpeg" /><figcaption>from: <a href="https://bookquotemonster.files.wordpress.com/2014/09/vermin.jpg">https://bookquotemonster.files.wordpress.com/2014/09/vermin.jpg</a></figcaption></figure><p>Last time <a href="https://medium.com/locust-io-experiments/locust-io-experiments-enriching-results-183d2ae4a4c2#9ba1">I created a custom client for Locust</a>. I think it will be beneficial to continue with the topic. Firstly, the client was a mock — all it did was to emulate requests/responses for some artificial system. Secondly, it worked in a synchronous manner: the requests code blocked until the response arrived. This is exactly how the default HTTP client works.</p><p>The next step would be to verify how to test systems that provide some kind of non-blocking API and to create a REAL client.</p><p>Apache Kafka is quite a popular piece of software that allows for exchange of data/messages between systems (an up-to-date slogan now seems to be: <a href="https://en.wikipedia.org/w/index.php?title=Apache_Kafka&amp;oldid=830231292"><em>open-source stream processing software platform</em></a>). As you probably already know (or will discover soon, if trying to find out more) it has <a href="https://kafka.apache.org/uses">many uses</a>, including some <a href="https://engineering.linkedin.com/blog/2016/04/kafka-ecosystem-at-linkedin">“serious” applications</a>. It is very performant: it is supposed to handle <a href="https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines">millions messages per second on commodity hardware.</a> It also happens that Kafka producer (a component that is responsible for sending messages to Kafka) provides non-blocking API by default (see: <a href="https://kafka.apache.org/10/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html">https://kafka.apache.org/10/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html</a>).</p><p>So, creating a Kafka client for Locust could be a time well invested!</p><h3>The theory</h3><p>The only programmatic client I used ‘till recently is official Java client — very mature, stable, feature reach. Having little experience with Python clients, I’ll need to use my luck and find a proper one for our purposes. Ideally, of similar qualities (including thread safety, asynchronicity and ability to <a href="https://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html#send(org.apache.kafka.clients.producer.ProducerRecord,%20org.apache.kafka.clients.producer.Callback)">register a callback for message delivery</a>).</p><p>This way I could write a custom Locust client that:</p><ul><li>Could be shared amongst users</li><li>Would send messages to Kafka in fire-and-forget manner</li><li>Use the callbacks to report the requests as successful or failed (measure time and collect metadata)</li></ul><p>Ok, let’s give google a chance…</p><p>The first three hits for “python Kafka client” for me were: <a href="https://github.com/dpkp/kafka-python">kafka-python</a>, <a href="https://github.com/confluentinc/confluent-kafka-python/">confluent-kafka-python</a> and <a href="https://github.com/Parsely/pykafka">pykafka</a>.</p><p>Let’s go through them quickly.</p><h4>pykafka</h4><p>Checking the code, it looks like <strong>pykafka </strong>doesn’t include callback parameter in <a href="https://github.com/Parsely/pykafka/blob/2.7.0/pykafka/producer.py#L304">the code responsible for sending the messages</a>:</p><pre>def produce(self, message, partition_key=None, timestamp=None)</pre><p>Instead, if you are interested in the results, you are responsible for calling a <a href="https://github.com/Parsely/pykafka/blob/2.7.0/pykafka/producer.py#L358">dedicated method</a> for this:</p><pre>def get_delivery_report(self, block=True, timeout=None):</pre><p>In the last resort, acceptable. Let’s examine other alternatives first.</p><h4>confluent-kafka-python</h4><p>This one, I like a lot. Good docs, tests, <a href="https://github.com/confluentinc/confluent-kafka-python/blob/v3.0.0/examples/integration_test.py#L92">callback mechanics in place</a>:</p><pre>mydr = MyTestDr()    <br>p.produce(&#39;test&#39;, value=&#39;This one has a dr callback&#39;, <br>    callback=mydr.delivery)    <br>p.produce(&#39;test&#39;, value=&#39;This one has a lambda&#39;, <br>    callback=lambda err, msg: MyTestDr._delivery(err, msg))    </pre><p>The only problem I have with it for now is its dependency on <a href="https://github.com/edenhill/librdkafka">librdkafka</a>. Its resolution is “a bit” problematic on my Windows development machine. Although it looks like a great candidate, I’ll examine the next alternative (at least until I figure out how to handle the dependency on different platforms).</p><h4>kafka-python</h4><p><a href="https://github.com/dpkp/kafka-python/blob/1.4.1/kafka/producer/kafka.py">kafka-python producer</a> is thread safe, as the developers declare. It can be pure python (no dependencies). There is also this feature (which must be <strong>very embarrassing</strong>, because the developers are very shy to admit it; it’s very hard to find any information on the subject) — you can declare handlers for <a href="https://github.com/dpkp/kafka-python/blob/1.4.1/kafka/future.py#L48">success</a> and <a href="https://github.com/dpkp/kafka-python/blob/1.4.1/kafka/future.py#L57">failure</a>.</p><p>For starters, I decided to go with kafka-python for the initial attempt.</p><h3>The preparations</h3><p>OK, I have something that starts looking like a plan. I know what I want to do, I know what components (libraries) I want to use.</p><p>In order to be able to develop the Kafka client and test it, I need an environment with a working Kafka cluster and Locust cluster.</p><p>If you are replaying my experiment, you can achieve it in many ways:</p><ul><li>you could connect to an existing Kafka cluster,</li><li>“bare metal” Kafka deployment — in *nix world it’s quite easy. Also, there are some daredevils that are able to <a href="https://medium.com/@shaaslam/installing-apache-kafka-on-windows-495f6f2fd3c8">run Kafka+Zookeeper on Windows machines</a>,</li><li>docker — there are many docker images that allow for running Kafka in Docker (<a href="https://github.com/wurstmeister/kafka-docker">https://github.com/wurstmeister/kafka-docker</a> being one of the most popular),</li><li>kubernetes — this is the option I opted for. First of all, I have already <a href="https://medium.com/locust-io-experiments/locust-io-experiments-running-in-kubernetes-95447571a550">tested Locust cluster creation with K8s</a>. Additionally, I have a good experience with <a href="http://debezium.io/">Debezium</a> images of Kafka deployment on Kubernetes.</li></ul><p>For me — personally — the easiest approach is to go with K8s. It’s for development purposes, and this choice doesn’t affect the heart of the matter, so let’s not ponder too much here. Your development setup can be anything you want — no changes in the code should be required.</p><p>If you would like to reproduce the thing I created, take a look at the <a href="https://github.com/karol-brejna-i/locust-experiments/tree/kafka-client/kafka-client#development-environment-setup">readme file’s “setup” section</a>. It all boils down to building Locust image (added installation of dependencies) and running two Kubernetes commands. All required sources are in an <a href="https://github.com/karol-brejna-i/locust-experiments/tree/kafka-client/kafka-client">accompanying GitHub repo</a>.</p><p>Now, I assume you have your environment prepared and we can start coding.</p><h4>Creating Kafka Client</h4><p>If we want have Locust stress testing some application that is exposed via Kafka topics, we need a client that is able to produce some messages. “Real” Kafka producer can be quite complicated (many configuration options), but — as usual — I’d like to start with something simple. Let our Kafka client to fulfill the following contract:</p><pre><strong>def </strong>send(self, topic, key=<strong>None</strong>, message=<strong>None</strong>):<br>    pass</pre><p>If it works, we can go for something more sophisticated…</p><p>But for now, let’s see what needs to be done in order for it to work.</p><p>Here’s my attempt:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6df0f45cac16f77c87a1e71a42fe3adf/href">https://medium.com/media/6df0f45cac16f77c87a1e71a42fe3adf/href</a></iframe><p>Let’s go step by step through the code:</p><ul><li>initialization (line 9) — when creating new object of KafkaClient, the user can supply a list of ‘host[:port]’ strings representing kafka nodes that the producer should contact to bootstrap initial cluster metadata (see <a href="https://github.com/dpkp/kafka-python/blob/1.4.1/kafka/producer/kafka.py">https://github.com/dpkp/kafka-python/blob/1.4.1/kafka/producer/kafka.py</a> for details)</li><li>send message logic (lines 48–53) consists of “sending” the message by the producer (actually: adding a record to a buffer of pending records to be sent; this allows the producer to batch together individual records for efficiency) and registering callback for success and failure (please, note, it provides the handlers with all the data required for computing result metadata, like time, size, topic name)</li><li>success handler (lines 16–30) is quite interesting: it gets a <a href="https://github.com/dpkp/kafka-python/blob/1.4.1/kafka/producer/future.py">producer’s future </a>as an argument and start timestamp. The <a href="https://github.com/dpkp/kafka-python/blob/1.4.1/kafka/producer/future.py#L31">future</a> resolves to RecordMetadata with in turn holds the attributes that may be interesting from result metadata collecting point of view (i.e. topic, partition, serialized key size, serialized value_size). Here we measure time (lines 17–18), then result metadata record is constructed (20–25) and Locust gets notified on successful request (27)</li><li>failure handler (32–40) works similarly. One difference here — in case of failure, the future doesn’t get resolved to RecordMetad and required information needs to be delivered when declaring the callback. Fortunately, all we need here is topic name and start time, hence: add_errback(self.__handle_failure, <strong>start_time=start_time, topic=topic</strong>) (see: line 53) is enough. Additionally, the exception (failure reason) is provided as the first positional argument (arguments[0])</li></ul><p>That should be it.</p><h4>Test code</h4><p>Let’s test the clients basic features. Here is a Locust test that will generate a message (variable length string, started with a timestamp) and send it to a Kafka topic (“test-topic”):</p><pre>TASK_DELAY = os.getenv(<strong>&quot;TASK_DELAY&quot;</strong>, 0)</pre><pre>class KafkaBehaviour(TaskSet):<br>    <strong>def </strong>timestamped_message(self):<br>        <strong>return </strong>f&quot;{time.time() * 1000}:&quot; \<br>               + (&quot;kafka&quot; * 24)[:random.randint(32, 128)]</pre><pre>    @task<br>    <strong>def </strong>task1(self):<br>        self.client.send(&quot;test-topic&quot;,<br>                         message=self.timestamped_message())<br>    @task<br>    <strong>def </strong>task2(self):<br>        self.client.send(&quot;test-topic&quot;,<br>                         message=self.timestamped_message(), <br>                         key=&quot;key&quot;)</pre><pre><strong>class </strong>KafkaUser(KafkaLocust):<br><em>    </em>task_set = KafkaBehaviour<br>    min_wait = TASK_DELAY<br>    max_wait = TASK_DELAY</pre><p>TASK_DELAY is an environment variable that allows for controlling the pace that KafkaUser is producing the messages (using Locust’s min/max_wait attributes).</p><h3>Results</h3><p>Now that everything is in place, it’s time to start some tests.</p><p>I want to see, how the client performs for different slave numbers, number of users and load intensity (TASK_DELAY).</p><h4>Moderate intensity</h4><p>Setting up the delay between tasks (=message sends) allows for simulating less restrictive load. Let us start with 10 ms delay. In this case, we would expect a single client being able to perform 100 tasks per second (which in turn would translate to sending <strong>100 messages per second</strong>).</p><p>I plan starting with a single worker deployment and then go up with number of users and slaves to see how the changes influence the results (RPS, response time).</p><p>On my environment, I’ll scale down Locust cluster with:</p><pre>kubectl scale --replicas=1 deployment/locust-slave</pre><p>Due to Locust bug (feature?) restart is required (otherwise the master doesn’t see the workers are gone; see the <a href="https://medium.com/locust-io-experiments/locust-io-experiments-extending-the-ui-6ee809ac91e1">proof</a>):</p><p>kubectl delete pods -l &#39;role in (locust-master, locust-slave)&#39;</p><p>Now, we are ready. So, let’s start tests with one user:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*Xz0pO-ARKMPjRIVjRVI9YQ.png" /></figure><p><strong>One user </strong>is doing ~90 RPS (almost as expected). Response times are very good with median around 1 ms. Kafka Client has plenty of “space” and should scale well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1n5vqBy9I27NGcRheBCbWw.png" /><figcaption>Moderate intensity, 1 slave, 1 user vs 2 users</figcaption></figure><p>Running the tests for<strong> two users</strong> gives hope for linear scalability. It shows twice as many RPS and similar response times. Until we let Kafka Client work in its <em>comfort zone </em>it should be that way. For example, <strong>10 users</strong> test proves it (times almost the same, RPS increased tenfold).</p><p>So, for this kind of load, Kafka Client should scale with the number of users and the number of slaves.</p><p>Let’s see how much “juice” we can squeeze from it…</p><h4>Max intensity — single slave</h4><p>For this purpose, we are going to run the tests with TASK_DELAY=0 (meaning: do as many requests as possible, don’t get any “rest” in between).</p><p>For starters, update TASK_DELAY environment variable (kubectl get cm locust-cm -o yaml | sed &#39;s/&quot;10&quot;/&quot;0&quot;/g&#39; | kubectl replace cm -f -in my case) and get rid of all but 1 slave (kubectl scale --replicas=1 deployment/locust-slave; remember the unfortunate restart).</p><p>After starting the test with one user, we observe the following results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*wyV6152jupSGto10w7o5GQ.png" /><figcaption>1 slave, 1 user, TIME_DELAY=0</figcaption></figure><p>It looks like when 1 user is doing its best, it can go about 4.9k requests per second with mean response times around 50 ms.</p><p>Let’s see if increasing the number of users helps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*9-cke8N-ZRFrgkwHCFwiwg.png" /><figcaption>1 slave, 4 users, TASK_DELAY=0</figcaption></figure><p>4 users (on 1 slave) — no gain here. RPS around 4.9k, response times ~50ms. Probably these numbers are the best we could squeeze from a single slave (remember: my setup is Locust running in docker, running in Kubernetes, running on Windows with minikube; many “opportunities” for ineffectiveness) and we hit Locust limits.</p><p>Telling the truth, 5k requests per second per slave doesn’t look too impressive. I expected something better. I’ll restrain from drawing early conclusions, for now.</p><p>Certainly, the bumping up the number of slaves should help.</p><h4>Gradually increasing the number of clients</h4><p>We have tested that for moderate load the Kafka Client shows a promise of scalability. We have also tried the maximum load.</p><p>This time, let’s inspect how the client behaves when the load is steadily increasing.</p><p>For this, we’ll go back to “moderate mode” (TASK_DELAY=10), scale up the cluster to 4 slaves:</p><pre>kubectl get cm locust-cm -o yaml | sed &#39;s/&quot;0&quot;/&quot;10&quot;/g&#39; \<br>   | kubectl replace cm -f -<br>kubectl scale --replicas=4 deployment/locust-slave<br>kubectl delete pods -l &#39;role in (locust-master, locust-slave)&#39;</pre><p>For this setup a test with 400 users will be run (with hatch rate of 4 users per second).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*9gFqy1JaC-LPUElhukOkOQ.png" /><figcaption>4 slaves, 400 users, TASK_DELAY=10</figcaption></figure><p>For a small number of users (1:43:36 to 1:44:37) the response times are very good (with median &lt; 10ms). They increase gradually but are quite ok (~50ms).</p><p>So, it looks like with this client we could use Locust to test applications that are consuming messages from Kafka. The has very basic feature set, but it’s able to scale quite well.</p><h3>Conclusions</h3><p>In this experiment we managed to:</p><ul><li>choose a Python library to connect to Kafka cluster</li><li>create a Kafka client — although very basic one — that uses asynchronous API</li><li>scale the client well with the number of workers</li></ul><p>The client is ready to use as is, but there is a room for improvement (probably necessary for “serious” tests), some of the topics include:</p><ul><li>Producer configurability — there is plenty of settings that you can apply to fine-tune your producer:, like batch size, buffer size and mang many more (see: <a href="https://kafka.apache.org/documentation/#producerconfigs">https://kafka.apache.org/documentation/#producerconfigs</a>).</li><li>message serializers/deserializers definition— allow the client do define how messages should be “translated” into Python values/objects</li><li>ability to target specific topic’s partitions</li></ul><p>Another thing is: While I was writing this article, a new release of kafka-python was published (1.4.2). It would be interesting to see how this version affects the performance, or even try something completely different like confluent-kafka-python library).</p><p>Hope someone finds this helpful ;-)</p><p>Thank You!</p><p>Few last formalities:</p><p>As usual, the sources mentioned in this article are stored in <a href="https://github.com/karol-brejna-i/locust-experiments">https://github.com/karol-brejna-i/locust-experiments</a>. <em>kafka-client </em>folder holds: docker image for Locust (that includes kafka dependency), kubernetes descriptors for setting up Kafka and Locust clusters, and example locustfiles.</p><p>Locust 0.8.1 was used.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9d9e6e49537" width="1" height="1" alt=""><hr><p><a href="https://medium.com/locust-io-experiments/locust-experiments-locust-meets-kafka-b9d9e6e49537">Locust experiments — Locust meets Kafka</a> was originally published in <a href="https://medium.com/locust-io-experiments">Locust.io experiments</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Locust.io experiments — enriching results]]></title>
            <link>https://medium.com/locust-io-experiments/locust-io-experiments-enriching-results-183d2ae4a4c2?source=rss----84c32fc5c3de---4</link>
            <guid isPermaLink="false">https://medium.com/p/183d2ae4a4c2</guid>
            <category><![CDATA[load-testing]]></category>
            <category><![CDATA[locust]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[experiment]]></category>
            <dc:creator><![CDATA[Karol Brejna]]></dc:creator>
            <pubDate>Thu, 08 Mar 2018 11:01:06 GMT</pubDate>
            <atom:updated>2018-04-09T09:42:23.522Z</atom:updated>
            <content:encoded><![CDATA[<p>One of the good things about Locust is its event-based nature. This time I’ll try to exploit this to extend the system capabilities with features like custom (non-HTTP) client and results enrichment.</p><h3>The rationale</h3><p>Before going into technicalities, let me explain why it could be even of any interest to you.</p><p>When doing performance tests it’s often important to be able to introduce some additional metadata to the “standard” results.</p><p>You may want to analyze the data by grouping some requests together, filtering others. In this case, it would be good if you were able to “tag” specific request types, for example, by service/component name, criticality (these requests should never fail, these are less important) or any other criteria. Going further, having these kind of “tagging”, you’d be able to investigate things like: “which component fail most frequently under the stress“.</p><p>Having custom (non-HTTP) system under testing is also a good use case for result data shaping. For example, <em>request URL</em> makes perfect sense for a web service, but if you are stress-testing a service exposed via Kafka topics, you’d probably want to collect some other information instead.</p><p>Even in simple HTTP case, there is a room for this. For example, Locust keeps track of <a href="https://github.com/locustio/locust/blob/a4c377efc7914e8d90701d9226a5cd843859b790/locust/clients.py#L144">the following metadata</a> for the requests it produces: request type, name, response time, <strong>response </strong>length. When you are testing a file upload (via POST), it’s more important how big the <strong>request</strong> is and this is the value you want to collect.</p><h3>The status quo</h3><p>Let’s see what what is under our disposal in Locust.</p><h4>Events and handlers</h4><p>Internally, Locust uses events and event listeners to communicate the state the system all over the cluster.</p><p>Every time something interesting is happening, an event is triggered (or “fired”).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3cc0f47fba0b7ad620c92b7b368520fb/href">https://medium.com/media/3cc0f47fba0b7ad620c92b7b368520fb/href</a></iframe><p>When some action needs to be performed in response to given event, it’s put into a handler and the handler gets hooked to that event.</p><p>For example, when you want to do something special when tests start to fail, you’d use <em>request_failure</em> event. The following pseudo-code would work:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/14bc7f8eaea746c89cbda56f2f7de175/href">https://medium.com/media/14bc7f8eaea746c89cbda56f2f7de175/href</a></iframe><p>The logic is defined on the module level (line 3) here and then it is added to <em>request_failure</em> handlers (line 6). Now, if <em>send_message_to_slack</em> would really exist, you’d be notified every time some request fails.</p><p>On the other hand, if — for some reason — you wanted to produce an event, you’d use something like this: events.request_failure.fire(request_type, name, response_time, exception)</p><p>And that’s it. Quite easy and smart.</p><h4>Default request success/failure handlers</h4><p>Now, that we know Locust communication is based on events, there is no surprise that the mechanics are also used to handle information about requests.</p><p>Locust encapsulates HTTP communication in HttpSession class (clients.py). Take a look at <a href="https://github.com/locustio/locust/blob/a4c377efc7914e8d90701d9226a5cd843859b790/locust/clients.py#L143-L148">this fragment</a> of <a href="https://github.com/locustio/locust/blob/a4c377efc7914e8d90701d9226a5cd843859b790/locust/clients.py#L74"><em>request </em>method</a> for notification on successful request:</p><pre>events.request_success.fire(<br>    request_type=request_meta[&quot;method&quot;],<br>    name=request_meta[&quot;name&quot;],<br>    response_time=request_meta[&quot;response_time&quot;],<br>    response_length=request_meta[&quot;content_size&quot;],<br>)</pre><p>And the <a href="https://github.com/locustio/locust/blob/master/locust/stats.py#L557">corresponding handler</a> is defined in <em>stats.py:</em></p><pre>def on_request_success(request_type, name, response_time, response_length):<br>    global_stats.log_request(request_type, name, response_time, response_length)</pre><p>Similarly, <a href="https://github.com/locustio/locust/blob/master/locust/stats.py#L560">failure handler</a> looks like this:</p><pre>def on_request_failure(request_type, name, response_time, exception):<br>    global_stats.log_error(request_type, name, exception)</pre><h3>The Execution</h3><p>Initially, I wanted to change the way “standard” requests data is collected. But then I thought I could kill two birds with one stone and illustrate data result enrichment while introducing an example of custom client at the same time.</p><p>The principles of operation would be very similar, so instead of experimenting with one thing I’ll have a chance to verify more stuff at once.</p><h4>Capturing atomic results</h4><p>By default Locust periodically dumps test results and computes aggregated values like median, average response time, number of requests per second, etc.</p><p>Now, I’ll introduce individual (not aggregated) results logging. In my case, I just want to make sure additional metadata I am adding to the results are really there, but this feature could be particularly useful if you plan to <strong>send the results to some external store</strong> and/or calculate the aggregates yourself.</p><p>I’ll hook to request_success and request_failure events, so in response to them I can output a particular result.</p><p>Short document on <a href="https://docs.locust.io/en/latest/extending-locust.html">extending locust</a> gives the following information and advice:</p><pre>from locust import events<br><br>def my_success_handler(request_type, name, response_time, response_length, **kw):<br>    print &quot;Successfully fetched: %s&quot; % (name)<br><br>events.request_success += my_success_handler</pre><blockquote>It’s highly recommended that you add a wildcard keyword argument in your listeners (the **kw in the code above), to prevent your code from breaking if new arguments are added in a future version.</blockquote><p>Let’s do this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f3f8b935717b115751a4ffa7520acafb/href">https://medium.com/media/f3f8b935717b115751a4ffa7520acafb/href</a></iframe><p>The first piece of code defines handlers that print results in json format to the console. In addition to the default parameters, the handlers accept arbitrary number of keyword arguments. For this implementation, all the additional data will go into “other” field.</p><h4>Custom client</h4><p>I will start with a client (for an imaginary queuing system) with the following contract:</p><pre><strong>class </strong>CustomClient:<br>    <strong>def </strong>__init__(self, host, port):<br>       &quot;&quot;&quot; Initialize the connection &quot;&quot;&quot; <br>       pass<br><br>    <strong>def </strong>push(self, topic):<br>        <em>&quot;&quot;&quot; push message using custom protocol &quot;&quot;&quot;<br>        </em>pass<br><br>    <strong>def </strong>pull(self, topic):<br>        <em>&quot;&quot;&quot; pull message using custom protocol &quot;&quot;&quot;<br>        pass</em></pre><p>On object creation it will “connect” to the system under testing and provide two operations: <em>push</em> and <em>pull.</em></p><p>Corresponding methods, should contain the actual logic (communication with the system) — now it’s only a simulation. For the results produced by this client to be more “realistic” I am introducing random “execution” time and force sporadic failures (see _sleep and _decide_the_fate).</p><p>Custom client is responsible for informing Locust about the outcome of the requests it produces, so <em>push </em>and <em>pull</em> need to handle:</p><ul><li>measuring the time,</li><li>collecting other metadata (request type, success/failure, response time),</li><li>firing proper event (request_success or request_failure).</li></ul><p>In my artificial case, the logic for this is going to be exactly the same for both methods. I decided to put it into custom_timer function wrapper and use it for push and pull.</p><p>As the example assumes blocking API, time measurement will be simple: the wrapper will call actual (wrapped) method while checking time before and after the invocation. (Let me attack the problem of asynchronous client in the near future.)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/16361a93e5020766025e83f90b9d7985/href">https://medium.com/media/16361a93e5020766025e83f90b9d7985/href</a></iframe><p>If you take a look at triggering the events (lines 23 and 27) you’ll notice that besides default parameters (see earlier section: “Default request success/failure handlers”) new one is being used: <em>tag.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2e7a7cfc97b31037c60a578adb37209/href">https://medium.com/media/e2e7a7cfc97b31037c60a578adb37209/href</a></iframe><p>There is a little snack smuggled into the code. The wrapper (lines 14-15) extracts the name of the function where push/pull was invoked. This way we are able to <strong>automatically tag the result with test method name</strong>. So, for the user defined in the code above, <em>test1</em> and <em>test2</em> will be used as values for <em>tag.</em></p><p>Please, don’t run the code yet.</p><h4>Multiple users</h4><p>Back then, when I started with Locust I didn’t even know if having multiple (different) user behaviour defined is possible. I promised myself to check. Now, of course, I know it is easily doable (at least in the most basic form — without the control on the order of test invocation, for example), but let me show it anyways. Maybe somebody who starts his adventure with Locust will find the example useful.</p><p>Here’s the extract from locustifle.py:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1b90947e5935ccb2b0600b9fd92a9a18/href">https://medium.com/media/1b90947e5935ccb2b0600b9fd92a9a18/href</a></iframe><p>It defines “normal” HTTP user and our new custom user and their tests.</p><h3>The results</h3><p>It’s time to run some code…</p><p>All the sources for this experiments are stored in <a href="https://github.com/karol-brejna-i/locust-experiments/tree/master/enriching-results">/enriching-results</a> folder of an accompanying GitHub repo.</p><p>Let’s clone the repo and start Locust in standalone mode using docker (and of course my favourite OS). Start powershell and use the following commands:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0cb5f77e2ba840216d2c3451127bb420/href">https://medium.com/media/0cb5f77e2ba840216d2c3451127bb420/href</a></iframe><p>Locust has started. Everything should be fine, so open the browser and start the tests.</p><h4>Small problem</h4><p>Well, everything is not fine. If you run the code against the current version (Locust 0.8.1), you get error like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b736f662122601e90e1299d1ded5d448/href">https://medium.com/media/b736f662122601e90e1299d1ded5d448/href</a></iframe><p>What happened?</p><p>In this experiments we created a custom client that produces some additional arguments for request_success (and failure) events. We also introduced dedicated handlers that are capable of consuming extra data — thanks to the advice given in Locust docs (see: <em>Capturing atomic results</em> section) — by using **kwargs.</p><p>When you take a second look at the default handlers implementation, you’ll notice that the Locust creators hadn’t took their advice seriously ;-).</p><h4>Small fix</h4><p>Because <em>on_request_success</em> and <em>on_request_failure </em>don’t declare **kwargs, Locust fails when invoking them. There is an easy fix for this. I proposed <a href="https://github.com/locustio/locust/pull/746">the following change</a> to the Locust sources:</p><pre>diff --git a/locust/stats.py b/locust/stats.py<br>index dab8bc7..1fdcb5d 100644<br>--- a/locust/stats.py<br>+++ b/locust/stats.py<br>@@ -554,10 +554,10 @@ global_stats = RequestStats()<br> A global instance for holding the statistics. Should be removed eventually.<br> &quot;&quot;&quot;<br><br>-def on_request_success(request_type, name, response_time, response_length):<br>+def on_request_success(request_type, name, response_time, response_length, **kwargs):<br>     global_stats.log_request(request_type, name, response_time, response_length)<br><br>-def on_request_failure(request_type, name, response_time, exception):<br>+def on_request_failure(request_type, name, response_time, exception, **kwargs):<br>     global_stats.log_error(request_type, name, exception)<br><br> def on_report_to_master(client_id, data):</pre><p>To continue the experiment, please go to <a href="https://github.com/karol-brejna-i/locust-experiments/tree/master/docker-handler-fix"><em>/docker-handler-fix</em></a><em> </em>folder and build patched version of Locust:</p><pre>docker build -t grubykarol/locust:0.8.1-py3.6-patch -t grubykarol/locust:latest .</pre><h4>Big success</h4><p>After re-running the docker image (please, note the label used here, it forces patched version of Locust):</p><pre>docker run --rm -it --name standalone --hostname standalone -e ATTACKED_HOST=<a href="http://standalone:8089/">http://standalone:8089</a> -p 8089:8089 -v c:\experiments\enriching-results\locust-scripts:/locust grubykarol/locust:0.8.1-py3.6-patch</pre><p>and starting the tests you will notice custom requests being logged:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e5f195d53a39a8779c5cdabab2aefc24/href">https://medium.com/media/e5f195d53a39a8779c5cdabab2aefc24/href</a></iframe><p>Take a look at example <em>Making a virtual data push</em> entry (line 4) and the following line containing JSON object:</p><pre>{<br>  &quot;request_type&quot;:&quot;CUSTOM&quot;,<br>  &quot;name&quot;:&quot;push /metrics/&quot;,<br>  &quot;result&quot;:&quot;OK&quot;,<br>  &quot;response_time&quot;:&quot;64&quot;,<br>  &quot;response_length&quot;:&quot;0&quot;,<br> <strong> &quot;other&quot;:{<br>    &quot;tag&quot;:&quot;task1&quot;<br>  }</strong><br>}</pre><p>As expected, the object contains “other” field that holds our tag. In this case tag’s value is <em>task1</em>, which — as we wanted — happens to be the test method name.</p><p>Thanks to fixed default handlers, our custom requests can be treated by Locust as any other request, so they are included in the statistics and the UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*ubZ9C7nK5_518ZRC2gDa5w.png" /><figcaption>Custom client results</figcaption></figure><h3>Conclusions</h3><p>In this experiment we managed to:</p><ul><li>create a custom client — here it’s just a mock, but it shows it’s possible to implement something sophisticated (create RPC-client, deal with other protocols, like TCP, create Kafka client, etc.)</li><li>log individual results (not aggregated, for further “consumption”)</li><li>define multiple users (one for web tests, one for custom client)</li><li>make custom requests visible in Locust as any other requests</li></ul><p>Thanks to even-based nature of Locust it was quite an easy exercise.</p><p>That’s wraps it up.</p><p>Thank You!</p><p>Few last formalities:</p><p>As usual, the sources mentioned in this article are stored in <a href="https://github.com/karol-brejna-i/locust-experiments">https://github.com/karol-brejna-i/locust-experiments</a>. <em>enriching-results </em>folder holds locustfiles, <em>docker-handler-fix </em>has docker image definition that includes Locust fix (it’s going to be obsolete if/when <a href="https://github.com/locustio/locust/pull/746">https://github.com/locustio/locust/pull/746</a> is merged).</p><p>Locust 0.8.1 was used.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=183d2ae4a4c2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/locust-io-experiments/locust-io-experiments-enriching-results-183d2ae4a4c2">Locust.io experiments — enriching results</a> was originally published in <a href="https://medium.com/locust-io-experiments">Locust.io experiments</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Locust.io experiments — extending the UI]]></title>
            <link>https://medium.com/locust-io-experiments/locust-io-experiments-extending-the-ui-6ee809ac91e1?source=rss----84c32fc5c3de---4</link>
            <guid isPermaLink="false">https://medium.com/p/6ee809ac91e1</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[experiment]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[locust]]></category>
            <category><![CDATA[scaling]]></category>
            <dc:creator><![CDATA[Karol Brejna]]></dc:creator>
            <pubDate>Sat, 24 Feb 2018 11:26:50 GMT</pubDate>
            <atom:updated>2018-02-25T16:47:20.847Z</atom:updated>
            <content:encoded><![CDATA[<p>Locust provides a small HTTP API and a Web UI to control the execution of the tests and browsing through the results. One can use the API, for example, to automatically trigger stress tests as a part of build process.</p><p>What if we want to extend the UI by adding some customized result views, new actions?</p><p>According to <a href="https://docs.locust.io/en/latest/extending-locust.html">the docs</a> it should be fairly easy.</p><h3>The goal</h3><p>Now, that we are able to <a href="https://medium.com/locust-io-experiments/locust-io-experiments-running-in-kubernetes-95447571a550">run Locust in Kubernetes</a> it (K8s) will allow us easily <strong>rescale the cluster.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SqWqsG69YN6yrNQf." /><figcaption>Photo by <a href="https://unsplash.com/@mrmrs?utm_source=medium&amp;utm_medium=referral">adam morse</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This simple command will do:</p><pre>&gt;  kubectl scale --replicas=3 deployment/locust-slave</pre><p>Let’s be nice to the users and not force them to use command line tools (use a terminal connected to K8s cluster) and <strong>prepare some graphical user interface to perform this action</strong>, instead.</p><h3>Components of the solution</h3><p>What we’ll need (the minimal version) is:</p><ul><li>code that will hold the logic (communicate with K8s and perform rescale)</li><li>HTTP endpoint that exposes the functionality</li><li>a web form where new number of slaves can be entered</li></ul><h4>The logic</h4><p>In our code, we want to repeat the same thing that kubectl scale command is doing. Let’s sniff the traffic that kubectl generates so we can see what APIs are used there:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2e680c5fdc8b8ee948d15353af4fd339/href">https://medium.com/media/2e680c5fdc8b8ee948d15353af4fd339/href</a></iframe><p>It looks like it GETs current deployment definition, updates replicas count and PUTs modified deployment back. (Ok, I cheated. Actually, I used--v=8 switch which shows not only HTTP operations, but also request/response bodies. Presenting this output would overflow the boundaries of this article…)</p><p>Digging a bit deeper, it turns out that we can be even smarter and do the change in a single PATCH request. Take a look at this pseudo-code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d9a45b4b3ab766a100325aaa523824bb/href">https://medium.com/media/d9a45b4b3ab766a100325aaa523824bb/href</a></iframe><p>The code performing required action is collected in <a href="https://github.com/karol-brejna-i/locust-experiments/blob/extend-web-ui/extend-web-ui/locust-scripts/kubernetes.py">kubernetes.py</a>. (As usual, you’ll find all the sources in the accompanying GitHub repo: <a href="https://github.com/karol-brejna-i/locust-experiments">https://github.com/karol-brejna-i/locust-experiments</a>).</p><p>One last thing to mention here is what needs to be done in order to access K8s API from the pod. I chose to obtain an authentication token a do the HTTP request myself. According to the official documentation, the preferred way is either using kube-proxy or a programmatic client (see <a href="https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod">this article</a> for details). For some real heavy lifting I would probably go for the client, but decided to stay plain for the sake of simplicity.</p><h4>Web endpoint</h4><p>Locust makes use of Flask to deliver it’s web user interface (see: <a href="https://github.com/locustio/locust/blob/master/locust/web.py">https://github.com/locustio/locust/blob/master/locust/web.py</a>).</p><p>Adding your own endpoint is just a matter of proper `route` annotation. <br>In this example, I created two endpoints — one responsible for displaying rescale form and one for performing the action.</p><p>See the following pseudo-code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd0fbe8a7b172bf8a5ba02e33b2a4067/href">https://medium.com/media/fd0fbe8a7b172bf8a5ba02e33b2a4067/href</a></iframe><p>The actual code resides in the enclosed <a href="https://github.com/karol-brejna-i/locust-experiments/blob/extend-web-ui/extend-web-ui/locust-scripts/locustfile.py">locustfile</a>.</p><h4>Web form</h4><p>Being a flask app, the UI can utilize <a href="http://jinja.pocoo.org/">Jinja2</a> to generate the pages. For this experiment I trimmed down <a href="https://github.com/locustio/locust/blob/master/locust/templates/index.html">the template used by Locust itself</a> so my web form looks at least a bit similar to the rest of the UI.</p><p>The key part of [rescale-form.html](./locust-scripts/rescale-form.html) is the following form:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/67b77f38ae4d72bd3895378e52c1eaf4/href">https://medium.com/media/67b77f38ae4d72bd3895378e52c1eaf4/href</a></iframe><p>And these are all the required parts.</p><h3>Deployment</h3><p>This work depends on <a href="https://medium.com/locust-io-experiments/locust-io-experiments-running-in-kubernetes-95447571a550">previous experiment</a> which point was to create Kubernetes descriptors for Locust. See the <a href="https://github.com/karol-brejna-i/locust-experiments/tree/extend-web-ui/kubernetes">https://github.com/karol-brejna-i/locust-experiments/tree/extend-web-ui/kubernetes</a> for the scripts and info on how to use them.</p><p>Basic K8s mechanics here are intact, the difference being <a href="https://github.com/karol-brejna-i/locust-experiments/blob/extend-web-ui/extend-web-ui/kubernetes/scripts-cm.yaml">the config map that holds update Locust scripts</a>.</p><p>In short, creating Locust cluster in K8s can be done these commands (if Kubernetes is up and running):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/740378759256b4aff480ec1e9ce0e79f/href">https://medium.com/media/740378759256b4aff480ec1e9ce0e79f/href</a></iframe><p>I am assuming you have done it already (while running the previous experiments) and locust nodes are running.</p><p>So, now let’s go to <em>extend-web-ui</em> folder (here are the sources for the current exercise) and update the configmap:</p><pre>$ cd ../extend-web-ui<br>$ kubectl replace -f kubernetes/scripts-cm.yaml<br>configmap &quot;scripts-cm&quot; replaced<br>$ kubectl delete pods -l &#39;role in (locust-master, locust-slave)&#39;</pre><p>The last line (restart of pods) is here to make sure new configuration is used by Locust.</p><h3>In action</h3><p>So, let’s open rescale form in the browser — <a href="http://192.168.1.123/locust/rescale-form">http://192.168.1.123/locust/rescale-form</a> (my K8s cluster’s IP is 192.168.1.123):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*JiP3V7YY9rBRmD_7pBC0RA.png" /><figcaption>Rescale form</figcaption></figure><p>Remember when I claimed I trimmed down the original template? Well, actually: I crippled it greatly which you can see by yourself. I didn’t want to focus on the UI part, but just to prove the point — so I made a minimal effort there.</p><p>Still, this simple form let’s us submit a new number of workers. My cluster has two workers (these are the defaults provided by my K8s scripts). Let’s go for 4 slaves now. After submitting, proper command is sent to Kubernetes and the user gets redirected to the main page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/1*kNTtjz2ElTHqqf6vSspMAQ.png" /><figcaption>First upscale</figcaption></figure><p>In the upper right corner you can see that after a while the number of slaves has been increased.</p><p>We are doing so well, so we could try something braver: let’s do a downscale (to 1 worker) and then another upscale (to 3) straight away.</p><p>And here is a little surprise. Instead of <strong>expected 3 workers </strong>at the end, I see 6 of them on the UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*7UN-dDyLW6JNAFG-FEIp-g.png" /></figure><p>Checking how many workers are really there (each Locust worker should correspond to one K8s pod):</p><pre>$ kubectl get pods<br>NAME                            READY     STATUS    RESTARTS   AGE<br>locust-master-5d8fbbc6-46k6n    1/1       Running   0          5m<br>locust-slave-54c84df478-nh7wn   1/1       Running   0          31s<br>locust-slave-54c84df478-rt8fn   1/1       Running   0          31s<br>locust-slave-54c84df478-v65ng   1/1       Running   0          5m</pre><p>There are three of them, as there should be.</p><p>Inspecting the master’s logs (kubectl logs -l=’role=locust-master’ | grep “^\[“) confirms, that communication with K8s was effective (Kubernetes did what we asked it to). The following lines seems to bring us closer to the diagnosis:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e42304109560a055cfc0e39aca09840/href">https://medium.com/media/7e42304109560a055cfc0e39aca09840/href</a></iframe><p>These are the only traces of worker-client communication. My reasoning is that Locust workers are able to report for duty, but they are unable to inform the master when they cannot server the load anymore. Nor is there any keep-alive or heartbeat mechanism in place. I’ll leave inspection of the sources for later (Locust’s internal mechanics is not the core concern of this article, scaling is just an exemplary new action).</p><p>In the meantime…</p><h3>Conclusions</h3><p>I think I managed to prove that<strong> extending the UI for Locust is quite easy</strong>: both introducing new endpoints and creating new web pages.</p><p>Aside from the problem mentioned earlier, scaling Locust with Kubernetes gives great flexibility for handling different/changing loads. It could be even cooler to use it in conjunction with <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/"><strong>K8s’ autoscaling</strong> </a>features. Maybe this should be a subject for some future experiments…</p><p>As usual, the sources mentioned in this article are stored in <a href="https://github.com/karol-brejna-i/locust-experiments">https://github.com/karol-brejna-i/locust-experiments</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ee809ac91e1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/locust-io-experiments/locust-io-experiments-extending-the-ui-6ee809ac91e1">Locust.io experiments — extending the UI</a> was originally published in <a href="https://medium.com/locust-io-experiments">Locust.io experiments</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Locust.io experiments — running in Kubernetes]]></title>
            <link>https://medium.com/locust-io-experiments/locust-io-experiments-running-in-kubernetes-95447571a550?source=rss----84c32fc5c3de---4</link>
            <guid isPermaLink="false">https://medium.com/p/95447571a550</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[experiment]]></category>
            <category><![CDATA[locust]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Karol Brejna]]></dc:creator>
            <pubDate>Wed, 10 Jan 2018 10:24:38 GMT</pubDate>
            <atom:updated>2018-02-24T08:10:56.056Z</atom:updated>
            <content:encoded><![CDATA[<p>Locust supports running load tests on multiple machines. It’s a perfect fit for Kubernetes which makes distributed deployments, container orchestration and scaling easy.<br>In this installment of my Locust experiments, I’ll prepare Kubernetes (alias K8s) scripts for deploying and managing Locust cluster and see if there are any surprises along the way.</p><p>This article is not about explaining K8s. There are already great sources of knowledge on the topic, including the product’s homepage — kubernetes.io. I’ll focus on describing how to set up and how to use Locust in K8s.</p><h3>Locust distributed mode</h3><p>Running Locust in distributed mode is described in <a href="https://docs.locust.io/en/latest/running-locust-distributed.html">its documentation</a>. In short:</p><ul><li>you can have one master and multiple slave nodes (workers)</li><li>you need to tell the workers where the master is (give them the IP address and ports)</li><li>you need to supply each worker with the tests code (locustfiles)</li><li>the workers need to be able to connect to the master</li></ul><p>That’s pretty much it.</p><h3>Kubernetes components</h3><p>Let’s dive in and go through the building blocks of the Locust cluster to be deployed.</p><p>Kubernetes’ <strong><em>configmap</em></strong>s will help configure Locust. One <a href="https://github.com/karol-brejna-i/locust-experiments/blob/master/kubernetes/scripts-cm.yaml">configmap </a>will hold a proper locustfile and will be mounted as a volume to the master and workers, <a href="https://github.com/karol-brejna-i/locust-experiments/blob/master/kubernetes/locust-cm.yaml">the second</a> will contain other configuration settings (URL for tested host/service, for example) that will be injected to running Locust nodes as environment variables.</p><p>I’ll use <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/"><strong><em>deployment</em></strong></a><em> </em>in order to “ask” K8s for making sure our <a href="https://github.com/karol-brejna-i/locust-experiments/blob/master/kubernetes/master-deployment.yaml">master</a> and <a href="https://github.com/karol-brejna-i/locust-experiments/blob/master/kubernetes/slave-deployment.yaml">slaves</a> are up and running. The <a href="https://github.com/karol-brejna-i/locust-experiments/blob/kubernetes/kubernetes/service.yaml"><strong><em>service</em></strong></a><strong><em> </em></strong>will make the master addressable within the cluster.</p><h3>Cluster deployment</h3><p>All the source files used below are stored in <em>/kubernetes </em>directory of the experiments repo. Let’s set up the cluster. (I assume that kubernetes is up and running and kubectl is able to connect to the cluster.)</p><pre>$ git clone <a href="https://github.com/karol-brejna-i/locust-experiments.git">https://github.com/karol-brejna-i/locust-experiments.git</a><br>$ cd locust-experiments<br>$ cd kubernetes</pre><pre>$ kubectl create -f locust-cm.yaml -f scripts-cm.yaml -f master-deployment.yaml -f service.yaml -f slave-deployment.yaml</pre><p>kubectl command connects to my minikube cluster and crate the components mentioned above. If ran for the first time, it may take a while to complete (if there is no locust docker image on the cluster, it needs to be downloaded first).</p><p>To see, if locust nodes are running we can use terminal to inspect if the pods are up.</p><pre>&gt; kubectl get -w pods<br>NAME                             READY     STATUS    RESTARTS   AGE<br>locust-master-754dc88dd8-zgs7m   1/1       Running   0          3m<br>locust-slave-7c89bfc5b7-4cms8    1/1       Running   0          3m<br>locust-slave-7c89bfc5b7-f7p4l    1/1       Running   0          3m</pre><p>Let’s also confirm this using K8s dashboard (brought up to the default browser with minikube dashboard):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z-Ik6WbMXNibC-vxmC2bCg.png" /></figure><p>From the output we can pick up the master name can also take a look at the logs. kubectl logs locust-master-754dc88dd8-zgs7m should include the following information:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/38fbea0272b2e07de8c046356c954ba3/href">https://medium.com/media/38fbea0272b2e07de8c046356c954ba3/href</a></iframe><p>We can see that the master has started (line 1 and 2) and the slaves “volunteer” to do some work (lines 3–4).</p><h3>First test run</h3><p>Nothing more is happening there, because we still need to tell the master to start the tests. Normally, you’d probably use neat Locust UI for this. You can also take advantage of HTTP API, which is particularly useful for test automation etc. (for example, when you want to start a stress test as a part of your CI/CD pipeline).</p><p>I’ll use two master’s endpoints here: <em>/swarm</em> to start the tests and /stop to finish them. We have a K8s service defined for the master, so the URLs gonna be:</p><ul><li><a href="http://locust-master:8089/swarm">http://locust-master:8089/swarm</a></li><li><a href="http://locust-master:8089/stop">http://locust-master:8089/stop</a></li></ul><blockquote>Quick note: Kubernetes job is to make sure that containers in the cluster can communicate with each other. Please, mind that by default the containers are not accessible externally. So, in order to make the calls, we’ll create a short live container which sole role will be to issue a single curl command. Alternatively, you could use port forwarding to be able to access a pod from your local machine (for example, use kubectl port-forward locust-master-754dc88dd8-zg7m 8089 and open the browser — explorer http://localhost:8089).</blockquote><p>In the deployment scripts we configured locust master to periodically dump activity logs to the console (with <em>print-stats </em>switch) so we can listen to the logs in one terminal (to check the results):</p><pre>kubectl logs -f locust-master-754dc88dd8-zgs7m</pre><p>and start the test in the second one:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bb448809612af45806102bb52bf030e8/href">https://medium.com/media/bb448809612af45806102bb52bf030e8/href</a></iframe><p>After some time, stop the test:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/62abb18460b7e4ec6c856ea43449c6b7/href">https://medium.com/media/62abb18460b7e4ec6c856ea43449c6b7/href</a></iframe><p>For me, all went well, which the following log proves:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/36774f6a514f9ac01f0af31ace07d987/href">https://medium.com/media/36774f6a514f9ac01f0af31ace07d987/href</a></iframe><p>What you can see here is:</p><ul><li>some inactivity period, stats from before the test was started (lines 1–4)</li><li>the moment that the master delegated the work to the slaves (line 6)</li><li>growing number of request (starting from line 7)</li></ul><h3>External access — initial attempt</h3><p>As mentioned before, typically, services and pods have addresses only routable by the cluster network. You need to put some effort in order to expose them to the outside world.</p><p>Let’s examine, how making Locust web console accessible from the internet would work.</p><p>The final piece of the puzzle here is an <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/"><strong><em>ingress</em></strong></a><strong><em> </em></strong>which is K8s’ mechanism that allows inbound connections to reach the cluster services.</p><blockquote>Note: If you are using minikube (as I am), mind that ingress controller is delivered as an add-on, that is not enabled by default. You need to do it yourself. For me, the following command did the trick: `minikube addons enable ingress`</blockquote><p>According to this simple rule:</p><pre><strong>apiVersion: </strong>extensions/v1beta1<br><strong>kind: </strong>Ingress<br><strong>metadata:<br>  annotations:<br>    ingress.kubernetes.io/rewrite-target: </strong>/<br>    <strong>ingress.kubernetes.io/ssl-redirect: </strong>&quot;false&quot;<br>  <strong>name: </strong>locust<br><strong>spec:<br>  rules:<br>  </strong>- <strong>http:<br>      paths:<br>      </strong>- <strong>backend:<br>          serviceName: </strong>locust-master<br>          <strong>servicePort: </strong>8089<br>        <strong>path: </strong>/locust</pre><p><em>locust-master </em>service is exposed so the users can reach Locust web UI under <em>/locust </em>URL on the host.</p><p>Let’s create the ingress with kubectl create -f ingress.yaml (check the file location in “Cluster deployment” section), get the cluster IP with minikube ip (192.168.1.123 in my case) and put expected locust URL in the browser (explorer <a href="http://192.168.1.123/locust).">http://192.168.1.123/locust</a>/<a href="http://192.168.1.123/locust).">).</a></p><p>Unfortunately, although Locust web page is accessible, it’s all mangled and nonfunctional.</p><p>This screenshot shows both the symptoms and the source of the problem:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YFenyqq3c6UE0df8MHDXLQ.png" /></figure><p>It turns out that the URLs for page resources (CSS, js) and the HTTP API are absolute (actually: root-relative). By using the ingress we “moved” the page address from the root (/ ) to /locust and the page got confused. Programmed this way, Locust cannot be properly accessed via ingress (or put behind a reverse proxy, load balancer for that matter).</p><h3>External access — the fix</h3><p>Well, Locust.io is an open source project, for good and for bad. Let’s make use of it.</p><p>After inspecting the sources, it turns out that the fix (that would allow serving Locust UI under URL different than /) is pretty simple. Take a look at this pull request: <a href="https://github.com/locustio/locust/pull/692/files">https://github.com/locustio/locust/pull/692/files</a></p><p>When the PR is merged or you build Locust docker image with these changes (see: <em>/docker-url-fix </em>folder in <a href="https://github.com/karol-brejna-i/locust-experiments">the sources GitHub repository</a>), the UI starts working just fine.</p><p>Here’s the evidence (explorer <a href="http://192.168.1.123/locust/">http://192.168.1.123/locust/</a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*mkpBZU6yD7KqIeHpo5ZZqg.png" /></figure><p>and (after starting the test):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*o8n-i9Ybipjzos7KtXIBKw.png" /></figure><h3>Conclusions</h3><p>As usual, the sources mentioned in this article are stored in <a href="https://github.com/karol-brejna-i/locust-experiments">https://github.com/karol-brejna-i/locust-experiments</a>.</p><p>Lessons learned from this experiment:</p><ul><li>Deploying Locust to Kubernetes seems “natural” and is pretty straightforward</li><li>Locust.io codebase is not without its flaws, but it’s legible enough even for an inexperienced developer to introduce changes</li><li>Change management process is not so clear. There are many living git branches (going as far as 7 years ago) and many open pull requests (44 ATTOW, dating back to 2014) without clear acceptance criteria. Submitting a PR there may be interesting experience</li></ul><p>Anyhow, for now it seems that using Locust.io with Docker and Kubernetes is a way to go. Having this covered, I can focus at some more specific problems.</p><p>So, stick around!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95447571a550" width="1" height="1" alt=""><hr><p><a href="https://medium.com/locust-io-experiments/locust-io-experiments-running-in-kubernetes-95447571a550">Locust.io experiments — running in Kubernetes</a> was originally published in <a href="https://medium.com/locust-io-experiments">Locust.io experiments</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Locust.io experiments — running in Docker]]></title>
            <link>https://medium.com/locust-io-experiments/locust-io-experiments-running-in-docker-cae3c7f9386e?source=rss----84c32fc5c3de---4</link>
            <guid isPermaLink="false">https://medium.com/p/cae3c7f9386e</guid>
            <category><![CDATA[locust]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Karol Brejna]]></dc:creator>
            <pubDate>Wed, 20 Dec 2017 17:00:28 GMT</pubDate>
            <atom:updated>2019-04-25T07:48:48.094Z</atom:updated>
            <content:encoded><![CDATA[<p>Containerizing Locust makes the development of load test easier and — which is more important to me — is a precondition to run Locust on Kubernetes cluster. So, let’s deal with this right at the beginning.</p><h4>Word of caution</h4><p>Before we start for good, let me give you a word of warning that applies to a whole series of the following posts in this series.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FWX2s7GqZYQtDq%2Ftwitter%2Fiframe&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Fufc-monday-rowdy-WX2s7GqZYQtDq&amp;image=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FWX2s7GqZYQtDq%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="252" frameborder="0" scrolling="no"><a href="https://medium.com/media/7df710bf4716ca207223573bd884db7c/href">https://medium.com/media/7df710bf4716ca207223573bd884db7c/href</a></iframe><p>You may encounter things that you may find disturbing or even horrifying. OK, let’s spit it out: for coding and running most of the experiments I am using <strong>Windows</strong>. I don’t remember how it exactly started, but now it became kind of exercise in patience (my colleagues who observe my struggle call it a penance), mortification or self-flagellation. Never mind the reason, you will find some Windows-specific commands, terminal screenshots and similar <strong>eyesore images.</strong></p><p>You’ve been warned, so let’s continue.</p><h4>Preparing docker image</h4><p>For me, one of the biggest advantages of using Docker is that Your application and all its dependencies can be bundled into a single container. This container can be moved to some other machine and executed regardless underlying host OS distribution or version (assuming it can run Docker).</p><p>Also, I am planning to have a scalable deployment of Locust. For that purpose, I will use Kubernetes, which in turn utilizes Docker containers. The basic prerequisite for achieving that goal, then, will be having a docker image definition for Locust.</p><p>So, here is what my requirements for the image are. I want it to:</p><ul><li>use Python 3</li><li>use the latest version of Locust</li><li>be as small as possible</li><li>be simple to use</li><li>take Locust scripts by means of mounting a volume</li></ul><p>I did a quick research and most of the images found on docker hub are old (1–2 yo) so I decided to give it a try myself.</p><p>After some investigation, it turned out that Locust has some run-time dependencies (libraries that need to be present in order for it to run properly), and dependencies required only for building/installing Locust.</p><p>The simple plan for creating the image is to use small python image as a baseline, add the dependencies and — after installation completes — get rid of anything that is not required at run-time. Then execute a script that starts Locust.</p><p>Here is the initial attempt on the Dockerfile:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31feb4d7c3f9689d58fd52f9594565f3/href">https://medium.com/media/31feb4d7c3f9689d58fd52f9594565f3/href</a></iframe><p>And the start script:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/21e147c01031c5ea9939c415c1074607/href">https://medium.com/media/21e147c01031c5ea9939c415c1074607/href</a></iframe><p>In order to build the image the following command could be used:</p><pre>docker build -t grubykarol/locust:latest .</pre><h4>Running the image</h4><p>Before we continue, there are three things to mention.</p><p>First, you can run Locust in the following modes:</p><ul><li><strong>standalone </strong>— single node “cluster”, where the instance plays both the role of the master and the worker (single Locust instance is responsible for running load tests)</li><li><strong>distributed </strong>— where <em>master</em> controls the behavior of the <em>workers</em>, collects the results and workers (or, in other words, <em>slaves</em> ) do the actual job of generating the load. In this mode, you’ll need one master and one or more slaves</li></ul><p>Second, a “prescription” of what to test and how to do this needs to be supplied to the Locust cluster (via so-called <em>locustfile</em>; see <a href="https://docs.locust.io/en/latest/writing-a-locustfile.html">https://docs.locust.io/en/latest/writing-a-locustfile.html</a>).</p><p>Third, you need to indicate which website/host you are going to test against.</p><p>So, let’s go on and see how these are addressed with the dockerfile I am proposing.</p><p>You influence the behavior of Docker containers (defined by this docker image) with environment variables.</p><p>You can set desired run mode (<em>standalone</em>, <em>master</em>, <em>slave</em>) using L<em>OCUST_MODE</em> variable. The host that is being tested is set through <em>ATTACKED_HOST</em> variable.</p><p>As far as the “prescription” is concerned, the image assumes that the test scripts will be supplied on run-time. In other words, when you run the image, you mount a volume (to <em>/locust </em>path) that holds the files describing the test.</p><p>The full list of environment variables to adjust Locust configuration is presented below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ecc3e5a899898c7a47a1c2d0c6216899/href">https://medium.com/media/ecc3e5a899898c7a47a1c2d0c6216899/href</a></iframe><p>Example run-time configurations are presented below.</p><h4>Example run-time configurations</h4><p>Let’s run some actual docker containers.</p><p>I’ll use the following locustfile. It is supposed to be simple, so it just makes requests against Locust web interface itself.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8f0f2e5482b5e6a7d49b7d8c544af90d/href">https://medium.com/media/8f0f2e5482b5e6a7d49b7d8c544af90d/href</a></iframe><p>Let’s assume it is stored on the host machine in a known location (<em>c:\locust-scripts</em> for the following examples).</p><p>Issuing the following command (in PowerShell):</p><pre>docker run --rm --name standalone --hostname standalone `<br> -e ATTACKED_HOST=http://standalone:8089 `<br> -v c:\locust-scripts:/locust `<br> -p 8089:8089 -d `<br> grubykarol/locust</pre><p>will start <strong>Standalone</strong> Locust node:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tEEwqN9NZ--uxTwa_eRj9A.png" /></figure><p>For <strong>distributed mode</strong>, you can run the master:</p><pre>docker run --name master --hostname master `<br> -p 8089:8089 -p 5557:5557 -p 5558:5558 `<br> -v c:\locust-scripts:/locust `<br> -e ATTACKED_HOST=&#39;http://master:8089&#39; `<br> -e LOCUST_MODE=master `<br> --rm -d grubykarol/locust</pre><p>and then a couple of slaves:</p><pre>docker run --name slave0 `<br> --link master --env NO_PROXY=master `<br> -v c:\locust-scripts:/locust `<br> -e ATTACKED_HOST=http://master:8089 `<br> -e LOCUST_MODE=slave `<br> -e LOCUST_MASTER=master `<br> --rm -d grubykarol/locust</pre><pre>docker run --name slave1 `<br> --link master --env NO_PROXY=master `<br> -v c:\locust-scripts:/locust `<br> -e ATTACKED_HOST=http://master:8089 `<br> -e LOCUST_MODE=slave `<br> -e LOCUST_MASTER=master `<br> --rm -d grubykarol/locust</pre><p>If everything goes well (which it should) you’ll see you Locust cluster up and running:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ty-rY9toN9EN-mF7B2H5pQ.png" /></figure><h4>In conclusion</h4><p>It looks like we got ourselves a simple, light (~124MB) docker image for Locust.</p><p>The source files for this experiment are <a href="https://github.com/karol-brejna-i/locust-experiments/">stored on GitHub</a> (tagged: <a href="https://github.com/karol-brejna-i/locust-experiments/releases/tag/dockerfile">https://github.com/karol-brejna-i/locust-experiments/releases/tag/dockerfile</a>).</p><p>The image is also <a href="https://hub.docker.com/r/grubykarol/locust/">published on Docker Hub</a>. ATTOW the latest version of the image is 0.8.1-py3.6.</p><h4>Edit</h4><p>As it turned out, this article has quite a few views/reads.</p><p>I decided to update the docker image for the newer versions of Locust, Python, and Alpine. I moved the sources to a dedicated GitHub repo: <a href="https://github.com/karol-brejna-i/docker-locust">https://github.com/karol-brejna-i/docker-locust</a></p><p>You’ll find some information here: <a href="https://medium.com/locust-io-experiments/locust-experiments-docker-image-updates-c7576e410efa">https://medium.com/locust-io-experiments/locust-experiments-docker-image-updates-c7576e410efa</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cae3c7f9386e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/locust-io-experiments/locust-io-experiments-running-in-docker-cae3c7f9386e">Locust.io experiments — running in Docker</a> was originally published in <a href="https://medium.com/locust-io-experiments">Locust.io experiments</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Locust.io experiments — Introduction]]></title>
            <link>https://medium.com/locust-io-experiments/locust-io-experiments-introduction-eb0d06bbbbb2?source=rss----84c32fc5c3de---4</link>
            <guid isPermaLink="false">https://medium.com/p/eb0d06bbbbb2</guid>
            <category><![CDATA[experiment]]></category>
            <category><![CDATA[locust]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Karol Brejna]]></dc:creator>
            <pubDate>Sat, 16 Dec 2017 11:29:58 GMT</pubDate>
            <atom:updated>2017-12-16T11:29:58.206Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently we were supposed to conduct a load/stress testing session.</p><p>There were already some python scripts in place and the team that produced them suggested using locust.io for this purpose.</p><p>The tool was new to me. At the first glance, as far as feature list is concerned, Locust seems inferior to other software we used before, like Gatling or even JMeter (the latter is turning 20 soon, but it doesn&#39;t make it an invalid testing tool!).</p><p>It looks like Locust generates load and that&#39;s pretty much it (note: ATTOW, the current version of locust is 0.8.1). With the exception of the single feature that I think is its strongest selling point: the ability to distribute the load (single master, multiple slaves) out of the box.</p><p>The authors also claim that it&#39;s very easy to use and to extend Locust. So, instead of discarding it (using Locust), I decided to give it a quick try and attempt to address potential needs/problems with series of short experiments.</p><p>Some of the things I would like to confirm are:</p><ul><li>Running on Kubernetes (but first in the docker)</li><li>Testing non-REST apps</li><li>Dealing with long-running requests</li><li>Capturing atomic results (now Locust saves pre-aggregated results)</li><li>Enriching test result data (tags, etc.)</li><li>Emitting the result to external data store</li><li>Alternating locust behavior</li><li>and some more…</li></ul><p>I plan to conclude particular experiments with a short write-up and keep the source code in the following GitHub repository: <a href="https://github.com/karol-brejna-i/locust-experiments">https://github.com/karol-brejna-i/locust-experiments</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb0d06bbbbb2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/locust-io-experiments/locust-io-experiments-introduction-eb0d06bbbbb2">Locust.io experiments — Introduction</a> was originally published in <a href="https://medium.com/locust-io-experiments">Locust.io experiments</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>