<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Stav Alfi on Medium]]></title>
        <description><![CDATA[Stories by Stav Alfi on Medium]]></description>
        <link>https://medium.com/@stavalfi?source=rss-6ba9df8f6bb7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*0jbyXKYFlPsaL3qSa88CMA.jpeg</url>
            <title>Stories by Stav Alfi on Medium</title>
            <link>https://medium.com/@stavalfi?source=rss-6ba9df8f6bb7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 18:04:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@stavalfi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Solidity — How Was I Able To Hack OpenZepplin Challange Level 19 — Alien Codex]]></title>
            <link>https://stavalfi.medium.com/solidity-how-was-i-able-to-hack-openzepplin-challange-level-19-alien-codex-70048ac848d1?source=rss-6ba9df8f6bb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/70048ac848d1</guid>
            <category><![CDATA[evm]]></category>
            <category><![CDATA[openzeppelin]]></category>
            <category><![CDATA[solidity]]></category>
            <category><![CDATA[solidity-contract-audit]]></category>
            <dc:creator><![CDATA[Stav Alfi]]></dc:creator>
            <pubDate>Tue, 11 Apr 2023 17:56:47 GMT</pubDate>
            <atom:updated>2023-04-12T10:17:20.517Z</atom:updated>
            <content:encoded><![CDATA[<h3>Solidity — How Was I Able To Hack OpenZepplin Challange Level 19 — <strong>Alien Codex</strong></h3><p>Well that was the hardest so far.</p><p>Let’s get into it.</p><p>Level: <a href="https://ethernaut.openzeppelin.com/level/19">https://ethernaut.openzeppelin.com/level/19</a></p><p>The easy part:</p><ol><li>understanding the task — make your self the owner of the contract</li><li>be able to call all the functions — just call `make_contact` and you are good to go.</li></ol><p>The hard part:</p><p>How to acomplish the task — this contract looks solid.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AdI2IuXXKYemHqHOUJ_sBQ.png" /></figure><h3>Storage Layout</h3><p>Recap: all the solidity class fields are saved in a sequental order (except for dynamic arrays and maps).</p><p>So our contract has 3 properties:</p><ol><li>address ownable (I’m guessing it doesn’t have anything else in the Ownable-05.sol file)</li><li>bool public contact</li><li>bytes32[] public codex</li></ol><p>I won’t explain here about memory slots (go and read about it if you don’t know what that are).</p><p>The first 2 fields share the first slot: 0.</p><p>The third property is located in the second slot: 1. But it’s a dynamic array so in the second slot, the value is just the length of the array. After we made an overflow (calling `retract`), the second slot should contrain the value: 0-1 = 0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff.</p><p>Important to note that array length is always uint256 so it will always take a full slot.</p><p>Reading the slots value:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*35FwNsA1L_RHuZC1uxbVAQ.png" /></figure><pre>curl -X POST -H &quot;Content-Type: application/json&quot; --data &#39;{&quot;jsonrpc&quot;:&quot;2.0&quot;, &quot;method&quot;: &quot;eth_getStorageAt&quot;, &quot;params&quot;: [&quot;0x2ACDe8bc8567D49CF2Fe54999d4d4A1cd1a9fFEA&quot;, &quot;0x0&quot;, &quot;latest&quot;], &quot;id&quot;: 1}&#39; http://localhost:8547</pre><p>the 0x0 is the slot number you want to read.</p><p>In solidity, all the fields (static or dynamic) are located from address 0x000…0 (0) to 0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff (2²⁵⁶-1).</p><blockquote>Slot’s are just addresses in memory!</blockquote><p>So the ownable field is located at address 0x000.</p><p>So if we could change the value of slot 0, we could modify who the owner is.</p><p>But how?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5qNxp_4fxK92CiZNaI2evA.png" /><figcaption>mmmm… need more clues?</figcaption></figure><h3>More Clues</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*IA7dnFIwSphuy8oH3efipw.png" /><figcaption>Everytime you see that, think about over/undeflows: uint256(0–1) === ²²⁵⁶-1</figcaption></figure><p>How is that going to help us? Search for places in the code that has a potential over/underflows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*Ri-2SzGx4tI0S7I1PRY_xg.png" /></figure><p>By calling it, the length of the array will be equal to ²²⁵⁶-1.</p><p>So it means that now the array is covering all the memory of our solidity program!</p><p>We can use the following function to override any memory we want:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*1wNqLPo_PfNSOvN3u_c1dw.png" /></figure><blockquote>Now the question is, what should be i so I can reach any memory we want.</blockquote><p>Well, arrays cells are located sequentally so we need to understand the address of the first cell of the array:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S1z-wsjhNfzvfrmy8Z4ICg.png" /><figcaption>Source: <a href="https://programtheblockchain.com/posts/2018/03/09/understanding-ethereum-smart-contract-storage/">https://programtheblockchain.com/posts/2018/03/09/understanding-ethereum-smart-contract-storage/</a></figcaption></figure><p>In simple words, the address of the first cell of an array is: keccak256(slot-of-the-array)</p><p>Let’s use `chisel` (solidity CLI tool provided by <a href="https://github.com/zenstruck/foundry">https://github.com/zenstruck/foundry</a>) to run some commands:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sjjvwWZ-NYziOnFz6OMvRw.png" /></figure><p>Now we know that the array located in 0xb10e2d527612073b26eecdfd717e6a320cf44b4afac2b0732d9fcbe2b7fa0cf6.</p><p>So how much cells we need to move until we reach cell zero (0x000..0)?</p><p>Step by step:</p><ol><li>We are at: 0xb10e2d527612073b26eecdfd717e6a320cf44b4afac2b0732d9fcbe2b7fa0cf6</li></ol><p>2. We need to react 0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff</p><p>3. Then we need to add 1 so there will be an overflow and we will be at address 0x000…0</p><p>How do we solve 2? 0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff — 0xb10e2d527612073b26eecdfd717e6a320cf44b4afac2b0732d9fcbe2b7fa0cf6=0x4ef1d2ad89edf8c4d91132028e8195cdf30bb4b5053d4f8cd260341d4805f309</p><p>How do we solve 3? 0x4ef1d2ad89edf8c4d91132028e8195cdf30bb4b5053d4f8cd260341d4805f309+1 = 0x4ef1d2ad89edf8c4d91132028e8195cdf30bb4b5053d4f8cd260341d4805f30a</p><p>All in all, if we reach cell number 0x4ef1d2ad89edf8c4d91132028e8195cdf30bb4b5053d4f8cd260341d4805f30a, we can modify the first solidity slot where the ownable field is.</p><p>But before doing the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q2_4t4lxc2U90D43rFrgjg.png" /></figure><p>We need to remember that the first slot contain 2 fields:</p><ol><li>address ownable (I’m guessing it doesn’t have anything else in the Ownable-05.sol file)</li><li>bool public contact</li></ol><p>So we should run:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ip3lTz-i-1peNV7tZvrymA.png" /><figcaption>My player address is 0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266</figcaption></figure><p>Why the `contact` field is located before the `ownable`? I will leave that to you for additional reading.</p><p>— —</p><p>To see the differnce between the value of the first slot before and after the change:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2n9S0zbStQKqNaiAouBUIQ.png" /><figcaption>My player address is 0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266</figcaption></figure><pre>curl -X POST -H &quot;Content-Type: application/json&quot; --data &#39;{&quot;jsonrpc&quot;:&quot;2.0&quot;, &quot;method&quot;: &quot;eth_getStorageAt&quot;, &quot;params&quot;: [&quot;0x2ACDe8bc8567D49CF2Fe54999d4d4A1cd1a9fFEA&quot;, &quot;0x0&quot;, &quot;latest&quot;], &quot;id&quot;: 1}&#39; http://localhost:8547</pre><p>The 0x0 is the slot number you want to read.</p><p>Enjoy!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70048ac848d1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NodeJS Devtools Performance (CPU) Optimizations — Example 1]]></title>
            <link>https://medium.com/octopol-engineering/nodejs-devtools-performance-cpu-optimizations-example-1-2a6b73e0eab2?source=rss-6ba9df8f6bb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a6b73e0eab2</guid>
            <category><![CDATA[devtools]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[processing]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[realtime]]></category>
            <dc:creator><![CDATA[Stav Alfi]]></dc:creator>
            <pubDate>Sun, 10 Oct 2021 19:54:01 GMT</pubDate>
            <atom:updated>2021-10-11T07:06:30.869Z</atom:updated>
            <content:encoded><![CDATA[<h3>NodeJS Devtools Performance (CPU) Optimizations — Example 1</h3><p>Today me and my cool colleague (@AmiBenSon <a href="https://github.com/amibenson">https://github.com/amibenson</a>) waited too much time to our app to load it’s first data and it made me thinking what the hell takes so much time to process events from pub/sub (Kafka) internally inside NodeJs process. We consume messages in batches. In the first 2–3 minutes, our procuder sends us alot of messages so each batch may contain 10k-30k messages. After that, batches may increase up to 100 messages at most. Some of the big batchs takes may take from 2s to process (too much :S) up to 30s processing.</p><p>That’s alot. And most of the processing is synchronous. that’s a huge problem because:</p><ol><li>NodeJs can’t react to readiness/liveness checks of monitoring systems (k8s).</li><li>Prometheus can’t reach out to the pod to collect metrics :S</li><li>other internal IO operations can’t occur concurrenctly.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a0l1pnCb9B5E5Rc_lPJi2w.png" /><figcaption>Shit, we are in trouble</figcaption></figure><p>I’m considering my self a begginer with Devtools so I bet other folks here will do much great job explaining what I’m going to show and what other conclusions we can take from my small investigation.</p><h3><strong>Let’s Start</strong></h3><p>The first step is to identify the problems; What takes up most of the CPU.</p><p>Let’s run our nodeJs program (better to compile it first):</p><pre>node --cpu-prof packages/csm-im/dist/src/index.js</pre><p>Now we wait until something interesting happen (see the horrible screenshot above).</p><p>And we hit ctrl+c to exit our NodeJs process and stop the cpu-profiling (NodeJs will save the cpu-profiling in the same dir we run our command).</p><p>Go to Devtools:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q1g-4A6mCMm0pRsYX_vkqQ.png" /><figcaption>Go to the performance tab</figcaption></figure><p>Click on:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/92/1*VMQsEJoyKOh_Wwgp70BSTg.png" /><figcaption>Load profile…</figcaption></figure><p>And pick your generated cpu-profilling result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V4zockCLKUGvm3_BHlq6tQ.png" /><figcaption>ignore the details</figcaption></figure><p>Well, that’s the fun part. understanding what the hell do we even see here.</p><p>from the logs of my app, I saw that there were exactly 4 batches which my app processed during the cpu-profilling. I can only guess that the following blocks represent something related to them:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*azvC0umtiyZNVNDU3KGZRQ.png" /></figure><p>Each execution of a function is printed in here. Rule: The more long the function line is, it means more time to run. Maybe it did only I/O stuff and the process did not do much CPU stuff, we don’t know yet.</p><p>In my case, each yellow line:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*z6MJK_0F3co__Oy0SA4iXQ.png" /></figure><p>represents a call of the nodejs-kafka-client to our processing function. every time the kafka-client sent 10k–30k events so that function call took time, as you can see.</p><p>if we look a little bit deeper, each line may have more lines underneath it. If a line has 4 lines underneath it, it means that line represent a function call that called 4 functions (maybe some/all of them are the same). We can measure how much each function call took in ms (milliseconds). Remember that a function may do something by it self without calling other functions and it may take time as well: for example running for-loop from 0 to 1,000,000.</p><p>In my screenshot, I scrolled down and deeper to see what are the most inner functions represent and how much time they took:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RftUlEK_ShhvmOVUe-gkIA.png" /></figure><p>I saw that the following line:</p><pre>/Users/stavalfi/projects/singer/packages/csm-im/dist/src/persist-im/persist-molly.service.js:27</pre><p>occured alot and it took 11ms to run!</p><p>I went to the source code to see what that was:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dT8MA-eHGN2cFek-HJ5Mvw.png" /><figcaption>I saw some other lines as well that pointed out something really bad is going wrong in this screenshot</figcaption></figure><p>It made me think:</p><p>con1 and con2 are strings which represents Dates (UTC): 2021–10–10T19:38:47.490Z.</p><p>To sort the array (for some reason we wanted to sort it — I don’t care why at this point), we are converting each date to an epoch so we can have a comparation in our sort.</p><p>All in all, we are doing tons of parsings and tons of sortings in every batch.</p><h3>First Solution</h3><p>I went over the code and our db and converted every representation of a date from string to number. Now I don’t need any parsings. We do have tons of sortings but at least no parsings.</p><p>You probably have alot of questions about what is the code doing, why I have dates in strings or what is the damage of sorting them as numbers. Let’s leave all these questions aside because they are irrelevant to this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WGwoNHvSW0jMFi53q56iHg.png" /></figure><p>Now let’s run the profiller again:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HUUx5ROgr70aM4QnPuZM4Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*43sDgfcE7Gb-pHIkXTvcGw.png" /></figure><p>Looks much much better: 1500% improvement!! (20–30s → 1–2s)</p><p>The processing durations still very bad (1.5s+-) but at least we made a huge improvements.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a6b73e0eab2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/octopol-engineering/nodejs-devtools-performance-cpu-optimizations-example-1-2a6b73e0eab2">NodeJS Devtools Performance (CPU) Optimizations — Example 1</a> was originally published in <a href="https://medium.com/octopol-engineering">octopol-engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Redis Server— Debugging Performance]]></title>
            <link>https://medium.com/octopol-engineering/redis-server-debuging-performance-b1ab0594733b?source=rss-6ba9df8f6bb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1ab0594733b</guid>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[redis]]></category>
            <dc:creator><![CDATA[Stav Alfi]]></dc:creator>
            <pubDate>Sat, 07 Aug 2021 22:33:40 GMT</pubDate>
            <atom:updated>2025-10-11T10:04:00.271Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post I will share some techniques on how to debug performance problems.</p><p>Topics:</p><ol><li>Setup Redis on your bare metal local machine</li><li>Run your service with Production configurations</li><li>redis-cli monitor</li><li>Batching commands</li><li>Redis slowlog</li><li>Redis Server CPU, RAM, SSD IOPS (IO Per Second)</li></ol><p>What’s Not On The Menu:</p><ol><li>Networking Issues</li><li>Overview about Redis Multi, Pipelines, Lua Scripts Pros/Cons/Best Practices — You need all of them.</li><li>Monitoring your service: Prometheus/Grafana/K8S-Metrics</li><li>Redis-Clusters</li></ol><h3>Setup Redis On Your Bare Metal Local Machine</h3><p>I created a small NodeJS script to setup a single instance or cluster of redis-masters (3+): <a href="https://gist.github.com/stavalfi/f5c8c5373c1e619dfde53853b178935c">https://gist.github.com/stavalfi/f5c8c5373c1e619dfde53853b178935c</a></p><p>Check if you are using Redis-Cluster in production or not. To run single instance: ts-node &lt;file-name&gt;.ts 1 or ts-node &lt;file-name&gt;.ts 3 for redis-cluster of master-instances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DeYy6oQirDQA7VpZFzE-oQ.png" /><figcaption>Run Single Redis Instance (Not a Redis-Cluster)</figcaption></figure><h4>Why Setting Up Redis For This Test</h4><p>Well, we are going to test performance so we want to *avoid* any network latencies, any virtualizations performance degregations: k8s, docker, aws EC2, VMs, ….</p><p>— — — — — — — — —</p><h3>Run your service with Production configurations</h3><p>Generate some requests: fake or not. Just make sure your service will call Redis at some point. More calls are better.</p><p>— — — — — — — — —</p><h3>redis-cli monitor</h3><p>In my service, I use multi, pipeline and lua-scripts. My goal is to see which redis-calls were made over time. At this point, I care less which operation made the redis-call (multi/pipeline/lua/…).</p><p>Open a new terminal and run `redis-cli -p &lt;port&gt; monitor`. It will print every command that has been sent to redis.</p><p>Note: In this section, it is best to use single redis instance (not cluster) because if you are using cluster, then you will need to monitor every instance seperatly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LFqZ6ksOPPCtSQsa_ZVrkg.png" /><figcaption>make sure something is printed during your test</figcaption></figure><p>The monitor command will print every call to redis in this format:</p><p>&lt;unix timestamp in microseconds&gt; [@ &lt;client-ip&gt;:&lt;client-port&gt;] “&lt;redis-command&gt; “…redis-command params…”</p><ul><li>Note: microsecond = second/1,000,000</li></ul><p>We can make use of the monitor output to investigate how much calls were made using each command by some bash commands: `cat log | awk ‘{if(NR&gt;1) print $4}’ | sort | uniq -c | sort -r`</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nI62ISSpUZgia_giPBKSJA.png" /><figcaption>format: &lt;count-of-repeats&gt; &lt;redis-command&gt;</figcaption></figure><p>Well, what did I just ran and what are we seeing?</p><ol><li>cat — read everything.log and send it to stdin of the next command using “|” (linux pipe).</li><li>awk — filter-out and string manipulation of each line in everything.log.</li><li>{if(NR&gt;1) ....} — filter out the first line in everything.log because it is “OK” and is not part of the monitor output format.</li><li>awk &#39;{... print $4}&#39; — each line in the monitor output format is: &lt;unix timestamp in microseconds&gt;<strong>SPACE</strong>[@<strong>SPACE</strong>&lt;client-ip&gt;:&lt;client-port&gt;]<strong>SPACE</strong>“&lt;redis-command&gt;<strong>SPACE</strong>“…redis-command params…”. print $4 will print the 4&#39;th word which is the &lt;redis-command&gt;</li><li>sort — sort in asc (small to big). this is needed for the next command uniq because we want to group by each command and print how much times it was called. uniq will group by only same lines that apear one after the other so we need to sort before that.</li><li>uniq — count the re-appearance of same-lines</li><li>sort -r — sort the ouput of uniq in desc order (big to little). (Note:sort will evaluate each line as a hole and not only the first word in the line).</li></ol><p><strong>What Can We Do With That</strong></p><p>See each line and try to understand if it makes any sense that some command has x calls during your test. In my case, I called 80k times the “evalsha” redis-command which is a call to a custom LUA script.</p><p>At This point you can identify some bugs or too much calls and decrease the amount of calls to redis from your production enviroment and from custom LUA scripts.</p><h3><strong>Batching Commands</strong></h3><p>We can do even more with awk . We can group by something stronger — &lt;count&gt; &lt;redis-command&gt; &lt;redis-key&gt; : awk ‘{if(NR&gt;1) print $4 $5}’</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0WZpCWuM1sTa-MuDuaGPwg.png" /><figcaption>In some cases we might see we called the same redis-key too much using the same command.</figcaption></figure><h4>Example</h4><p>Today I noticed that I called HINCRBY (<a href="https://redis.io/commands/HINCRBY">https://redis.io/commands/HINCRBY</a>) too much times on the same key.</p><p>HINCRBY — incr by x a value inside a hashmap.</p><p>Problem — HINCRBY does not allow to incr multiple keys in the same hashmap Why it’s a problem — In my usecase I need to call HINCRBY at most 30 times everytime. This is alot.</p><p>Will MULTI will solve it — With MULTI I’m able to send multiple redis-commands at once to redis to avoid sending multiple redis-calls seperatly and it will be a network panelty. Even with MULTI it was pretty slow command when you are doing it alot.</p><p>Solution? After some reading online, it seems that Redis created BITFIELD to solve this problem (and other problems as well) — it is a datastructure over string to manipulate bits. It is pretty convient command and it’s easy to setup with that an array of counters. It does supports batching so I was able to decrease 1–30 calls to HINCRBY to one call to BITFIELD . I gained 10% performance optimizations :)</p><h3>Redis SlowLog</h3><p>Next, we want to see how much time it took Redid-server to run every redis-command in microseconds (microsecond = second/1,000,000).</p><p>Redis implemented a feature called slowlog . it will print each command if it’s execution (inside the redis-server!!) took more than x microseconds.</p><p>To set x :</p><h4>redis.conf</h4><ol><li>Go to <a href="https://redis.io/topics/config">https://redis.io/topics/config</a></li><li>Check which config file you need to read to check it’s options. Do that after you check your redis version. For Redis 6: <a href="https://raw.githubusercontent.com/redis/redis/6.0/redis.conf">https://raw.githubusercontent.com/redis/redis/6.0/redis.conf</a></li><li>Search for slowlogs:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MAa8v29ubNlkBQvqK-QhAw.png" /><figcaption>for this test — set 0 to track on all commands</figcaption></figure><p>4. restart your redis service.</p><h4>Set In Runtime</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UV0CzqPhLTOQIFF_396tnw.png" /><figcaption>for this test — set 0 to track on all commands</figcaption></figure><p>Notes:</p><ol><li>To delete all the slowlog until now, run: slowlog reset</li><li>Slowlog in Redis is a append-only to in-memory data-structure so it won’t descrease your performance. it may end up your free RAM so be carefull with that.</li></ol><p>Now, run your service and then call:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*b7BiqAwOBenM5m-nnPthXQ.png" /><figcaption>-1 will print all the logs. 10 will print only the 10 last commands. choose -1 for this test.</figcaption></figure><p>Format: Each command is a different “slowlog”:</p><ol><li>Command-id (increasing as your can see: you see the end of the slowlog so the top is the newest log).</li><li>UNIX timestamp and seconds when the log was made (or the command was executed — I’M NOT SURE!)</li><li>How much time in micro seconds it took Redis to run your command. Lower is better. microsecond = second/1,000,000. Remember that there is also a network panelty outside of Redis and there is your CPU, RAM, Virtualization panelty. Redis may not be the core reason for the performance degradation.</li><li>Redis-Command: &lt;command-name&gt; &lt;…params…&gt;</li></ol><p>Note: the output is not sorted. It’s just an append-only data-strucutre for logs in-memory</p><p>I built a small script in NodeJS script to evaluate the output of `slowlog get -1`:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bIOZabtzo2trjQ2cUU_Whg.png" /><figcaption><a href="https://gist.github.com/stavalfi/285eebfc93872d7eb83e31286ae845a5">https://gist.github.com/stavalfi/285eebfc93872d7eb83e31286ae845a5</a></figcaption></figure><p>The above code will show me how much time it took Redis to execute each command in seconds:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gjyggyljTMuwDjAR6zKjaQ.png" /></figure><p>You can manipulate the script to group by commands and evaluate AVG execution time of every command.</p><h3>Redis Server CPU, RAM, SSD IOPS (IO Per Second)</h3><p>The above techniques where used to improve performance by changing the code. Now, we will evaluate the CPU, RAM and your SSD inside your Redis-Server.</p><p>If you have prometheus or any other monitoring tool over your redis-server, use that to evaluate the CPU, RAM and SSD IOPS (SSD in case you are using persistency in Redis).</p><p>Locally or inside your redis-production server, you can use htop to evaluate CPU and RAM usage:</p><p>Run htop inside your redis server when you run your test. check if each CPU Core (1–4 in my case) is increasing too much.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cqc4af3thuQ0IKQ4Y1IYBg.png" /></figure><p>Redis is mosly single threaded but still, you need to make sure redis does not blow up your CPU or RAM.</p><p>Also, check if you can disable Swp by evaluating how much RAM you will need for your app in Redis and if it’s less than the amount of RAM in your Redis server, you can disable Swp. because as Redis-labs recommend, there is no point of transfering some of the RAM to the SSD and back if you have more RAM than you actually use at any given point in time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1ab0594733b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/octopol-engineering/redis-server-debuging-performance-b1ab0594733b">Redis Server— Debugging Performance</a> was originally published in <a href="https://medium.com/octopol-engineering">octopol-engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Producing & Consuming (Much) More With Kafka]]></title>
            <link>https://medium.com/octopol-engineering/producing-consuming-much-more-with-kafka-eb0e499daec5?source=rss-6ba9df8f6bb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb0e499daec5</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[kafkajs]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Stav Alfi]]></dc:creator>
            <pubDate>Tue, 27 Apr 2021 21:19:33 GMT</pubDate>
            <atom:updated>2021-08-07T22:34:09.602Z</atom:updated>
            <content:encoded><![CDATA[<p>On the last couple of days, Me and @AmiBenson are working on a new task which involves consuming large amount of data from multiple kafka topics into a NodeJS service.</p><p><strong>Requirements:</strong></p><ol><li>We need to consume in a rate 1–4k messages per second on 2 Kafka topics.</li></ol><p>2. Proccessing/consuming Delay/Lag is unacceptable.</p><p>3. Single consumer* — The Data must be consumed in the same order which the producer is producing. Let’s leave that aside for now.</p><p><em>Tech &amp; Env:</em> Prometheus for monitoring, NodeJS, KafkaJS, this article is based on experiments on my macbook pro 2020(16g RAM and 8 CPU), Postgres (using docker postgres:13.2-alpine image and kafka using bitnami/kafka:2.7.0-debian-10-r114 image (with zookeeper)).</p><p>To improve the development velocity, we created a small mock-service which produces the same data we expect.</p><p><strong>To sum it all up, we have 2 challenges:</strong></p><ol><li>Be able to produce the same amount of data.</li><li>Be able to consume this data without lag.</li></ol><p>— — — — — — — — — — — — — — — — — — — — — — — — — —</p><h3>Mock-Producer</h3><p>What it does is generating array of messages forever and send each array to our Kafka broker. Each generation of a message involves some json → xml parsing so it may take some time as well.</p><h4>First Attempt</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E9epPK83qrxtKMLGy650Zw.png" /><figcaption>How bad can it be?!</figcaption></figure><p>Well, pretty bad:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j5lLQa4EQz_MUjJBOmn6nQ.png" /></figure><p>We were able to produce 700 messages per second.</p><p>Let me explain, internally, some kafka clients has an internal queue that collectes all the messages should be sent and actually sends it every `X`ms. This feature is <em>linger. KafkaJS currently doesn’t support it so what it does is to really send every single message. This is extemrly slow.</em></p><h4><strong>Second Attempt — Produce In Batch</strong></h4><p>Let’s try to send messages in batch to our kafka broker. It will help us reach out to our kafka much less times and each time we will send much more data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BuisrQojqZDjCxtnQRlZ6A.png" /></figure><p>For start, let’s try to send batches of 700+- messages on every batch:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*06fXufTYO-6stFqo48Izkw.png" /></figure><p>Now we are talking. We were able to send 4k messages per second.</p><p>But can we do better?</p><p>When we try increase the batch size from 700 to 1000+, KafkaJS throws an error:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OCg12TOdLhbYKd2E_WAetA.png" /></figure><h4>Third Attempt — Compress The Batch</h4><p>KafkaJS allow us to compress the data before we sent it. Let’s try it out and send 3000 messages pet batch:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*lW_S_HKR4e64nR2klmje0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YcMjob2J6o1fuHMVsVWrqQ.png" /></figure><p>Interstingly, there is no difference at all. Let’s try to increase the batch size just to be sure to 6,000 messages (Yes, GZIP let us increase the batch size from 700 to 6,000!)</p><blockquote>We can even send more than 20,000 but the service performance was too slow.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SMMwfX1e6zxWzVrJQiUYtg.png" /></figure><p>We improved from 3900+- to 4100+-. I also tested it with batch = 10,000 and 20,000 and there was no addtional improvement.</p><p>To sum it all up, this attemp did not do any major performance improvement.</p><p>Addtional ways of improvement should come from the actual message-generation logic. for this article, this is out of scope.</p><p><strong>We achieved our goal.</strong></p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — —</p><h3>Consumer</h3><p>Consume from kafka → save in PostgreSQL. In the middle, we parse XML messages to JSON. I bet that’s gonna heart.</p><blockquote>We are using Typeorm.</blockquote><h4>First Attempt</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S-efoPbmAcF4pPKebIlOPQ.png" /><figcaption>How bad can it be?!</figcaption></figure><p>And again, we have the same problem: we are sending too many requests to our postgres, so we waste time on waiting until postgres will receive the request and send us a response:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QT9aSuO1QDQxYGR4WAUJHQ.png" /></figure><p>All we were able to process is 500+- messages per second.</p><h4>Second Attempt — Batch Inserts To PostrgeSQL</h4><p>In our case, all the data we received is a representation of a state in a given time. There are no db-records updatesm, just inserts.</p><p>In this case, we can send batch INSERT queries to our DB. To do that, first we need to consume in batch from our Kafka. That’s easy with KafkaJS:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vJeibGl1s_QNxULHTpkXlg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bg5PJM1aSFwMM1tgmPD1gw.png" /><figcaption>What could go wrong here?!</figcaption></figure><p>It should work, but there was a problem, each kafka-consumer-batch can contain between 1 to 60K (or even more) messages. depends on the amount of unread messages in the topic. It means that the DB query may contain 60k records. Postgres throw us errors about that. To over come them, we did our own batching mechanism using a little bit of lodash magic:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ao1kBAA5FqvsfbFqflUysw.png" /><figcaption>lodash-chunk: split up an array to sub-arrays</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mY-BRVsLnTp1G35ToUH5lQ.png" /></figure><p>In the first section (23:36 → 23:40), we are processing 4K messages per second. But can we do more? Yes. We stopped my service and let the mock fill up our local kafka between 23:40 → 23:43. In 23:43 we run our service to see how it behaves when he needs to catch up.</p><p>We can see that the rate is now 6K + messages per second.</p><p>— — — — — — — — — — — — — — — — — —</p><h3>Proccessing/Consuming Delay/Lag Is Unacceptable</h3><p>Ny now, we are able to produce and consume in a rate of 1–4K messages per second. As requested.</p><p>Let’s combine both of the consumer and the producer to a single metric — lag-calculation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ll59I4motT3F2ljnOmOHw.png" /></figure><p>Well, this is unacceptable! Luckily we are not in production yet :P</p><p>See you on the next chapter :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb0e499daec5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/octopol-engineering/producing-consuming-much-more-with-kafka-eb0e499daec5">Producing &amp; Consuming (Much) More With Kafka</a> was originally published in <a href="https://medium.com/octopol-engineering">octopol-engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Improving WebSocket callback Processing Time]]></title>
            <link>https://medium.com/octopol-engineering/improving-websocket-callback-processing-time-fb5f1ef2ce32?source=rss-6ba9df8f6bb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb5f1ef2ce32</guid>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[websocket]]></category>
            <dc:creator><![CDATA[Stav Alfi]]></dc:creator>
            <pubDate>Sat, 20 Mar 2021 21:53:24 GMT</pubDate>
            <atom:updated>2021-04-01T13:41:50.047Z</atom:updated>
            <content:encoded><![CDATA[<h3>Improving WebSocket Callback Processing Time</h3><p>As part of the creation of one of our NodeJS services, me and part of my team (@NoyEliyahu and @OrFins) investigated a critical performance problem. The service wasn’t ready yet anyways but it was cool enough to share this story.</p><p>The flow was: at the initialization of the service, we connect to a web-socket (<a href="https://github.com/websockets/ws">https://github.com/websockets/ws</a> ) to other service (let’s call it ws-producer), receiving 100k of events (snapshot of what the other side has from the begining of time until now) at the start of the connection and then recieve real-time events at a much lower rate:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zz1FnJ-XvVhNC8yWdkG-sg.png" /><figcaption>1k events per second</figcaption></figure><p>Each incomming ws-message, should be sent to our kafka.</p><p>The messages has a relation between them, and the behavior of the ws-producer has a wierd API:</p><p>It sends us 10 types of events. 2 of them is: “add_something” and “remove_something”. The problem was, sometimes, instead of sending me “add” → “remove” → “add” on the same entity, the ws-producer sends me: “add” → “add”. So to make our life easier on the rest of our services, this service will identify this situation by receiving ws events of: “add” → “add” and translate them to kafka events of: “add” → “remove” → “add”.</p><p>To do that, we use in-memory (there is a reason for that, it’s out of scope of this article) hash-map to remember all the “add” and the “remove” events.</p><p>Our ws processing function was async so as how ws callbacks work, ws will call the callback without waiting until the last call to the callback resolved. To avoid race-conditions between calls of our callback, we used an in-memory queue (<a href="https://caolan.github.io/async/v3/docs.html#queue">https://caolan.github.io/async/v3/docs.html#queue</a>) with concurrency-level=1 so we can safely process each incoming ws-message in the original order in a async way.</p><p>It was a big failure :S<br>We received 100k of events at the first 40s of the service and most of them waited in the queue for up to a minute.</p><p>There were 2 problems:</p><ol><li>We sent each message to the kafka one by one (not in batch)</li><li>The queue was too slow.</li></ol><p>To fix (1), we had to find some kind of a queue that we can consume from it in a batch and use this to send kafka messages in a batch. We used rxjs for that:</p><pre>private readonly producerSubject = new Subject&lt;{ topic: string; message: Kafka.Message }&gt;()</pre><pre>this.producerSubject<br>      .pipe(<br>        // every 100ms send the internal queue to `concatMap` and reset the queue<br>        bufferTime(100),<br>        concatMap(async array =&gt; {<br>          if (array.length &gt; 0) {<br>            const batch = // internal processing of the array<br>            await this.kafkaProducer.sendBatch({ topicMessages: batch })<br>          }<br>        }),<br>      )<br>      <br>// send events to rxjs:<br>// NOTE: `next` returns void!<br>this.producerSubject.next({topic: &#39;topic1&#39;, messages: [message1]})<br>this.producerSubject.next({topic: &#39;topic2&#39;, messages: [message2]})<br>this.producerSubject.next({topic: &#39;topic1&#39;, messages: [message3]})<br>this.producerSubject.next({topic: &#39;topic3&#39;, messages: [message4]})</pre><p>cool!</p><p>Now, the processing of each ws-message does not send messages to kafka so it’s synchronous (We also removed some other async stuff from the ws-processing callback).</p><p>How does it help us?</p><p>Alot!</p><p>As of now, the ws-callback is synchronus, so we don’t need any queue (<a href="https://caolan.github.io/async/v3/docs.html#queue">https://caolan.github.io/async/v3/docs.html#queue</a>) to help us process all the incoming messages one after the other (because of how node.js works — if a callback is not async, the ws will wait until the end of the processing and only then call the callback with the next message).</p><p>Here are the results (no queue, send batch messages to kafka): (90% of the messages has processing time of 0.01s — we decreased it from 60s!).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qGLDZB878GnEitGQswkqrg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N2xATVeiyxjJYEVw2NBf-Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bY5eonEKI3Nr_mR0986p3w.png" /></figure><p>This is a huge improvement.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb5f1ef2ce32" width="1" height="1" alt=""><hr><p><a href="https://medium.com/octopol-engineering/improving-websocket-callback-processing-time-fb5f1ef2ce32">Improving WebSocket callback Processing Time</a> was originally published in <a href="https://medium.com/octopol-engineering">octopol-engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>