<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Szymon Mentel on Medium]]></title>
        <description><![CDATA[Stories by Szymon Mentel on Medium]]></description>
        <link>https://medium.com/@mentels?source=rss-76c3137d6ba7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*PkMCL_cCIqkdK47eL3KWPw.png</url>
            <title>Stories by Szymon Mentel on Medium</title>
            <link>https://medium.com/@mentels?source=rss-76c3137d6ba7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 03:52:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mentels/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Today I learned about the BEAM (a.k.a. ErlangVM)]]></title>
            <link>https://medium.com/series/today-i-learned-about-the-beam-a-k-a-erlangvm-d46187d1dd37?source=rss-76c3137d6ba7------2</link>
            <guid isPermaLink="false">https://medium.com/p/d46187d1dd37</guid>
            <dc:creator><![CDATA[Szymon Mentel]]></dc:creator>
            <pubDate>Sat, 05 Oct 2019 17:00:33 GMT</pubDate>
            <atom:updated>2019-10-09T11:01:47.680Z</atom:updated>
            <content:encoded><![CDATA[<h4>Where is my process?</h4><p>While working on a <a href="http://rrug.pl/meetup/2019/09/27/spotkanie23.html">presentation discussing key features of the Elixir platform</a> I was testing some code illustrating processes’ <em>location transparency</em>. And out of the sudden, I discovered that</p><blockquote><em>Process.alive?/1</em></blockquote><p>works only for the processes at the local node. Yeah, that’s stated clearly in the <a href="https://hexdocs.pm/elixir/Process.html#alive?/1">docs</a> but for some reason I didn’t expected that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*BvNNH4IgV7qMoFwIi2SeOQ.png" /><figcaption>Checking if a remote process is alive</figcaption></figure><h4>Pinning an undefined variable in Elixir</h4><p>In Elixir the “=” operator is not an assignment operator — it’s a pattern matching one.</p><p>Coming from Erlang it looked strange to me at first, that one can bind a variable and then rebind it like:</p><blockquote>iex(1)&gt; x = 11<br>iex(2)&gt; x = 12</blockquote><p>In Erlang that would cause an exception. Anyway, if there is a variable on the left hand side of the “=” operator it will be rebound unless it is “pinned”:</p><blockquote>iex(3)&gt; ^x = 13 # error</blockquote><p>Then we say: <em>Do not try to rebind, pattern match against the existing value.</em></p><p>However, it the pinned variable is undefined we get an error:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*MPW9n03AdkThM-QU6o-V-A.png" /><figcaption>Pattern matching against undefined variable</figcaption></figure><p>The<a href="https://elixir-lang.org/getting-started/pattern-matching.html"> official doc</a> as well as the <a href="https://medium.com/@Jules_Corb/understand-the-pin-operator-in-elixir-a6f534d865a6">linked article</a> explain these concepts well.</p><h4>Running an <em>iex</em> shell for a project without starting the app</h4><p>Usually to start an interactive shell for an Elixir project one uses:</p><blockquote>iex -S mix</blockquote><p>It compiles the sources, boots the Erlang VM, loads the current app and its dependencies into the VM and starts them. Actually it is a shortcut of</p><blockquote>iex -S mix run</blockquote><p>as <em>run</em> is the default mix task. You can verify that your application is running with:</p><blockquote>Application.started_applications/0</blockquote><p>If you want mix not to start you app, you can pass a <em>— no-start</em> option to the mix <em>run</em> task. Then the two last steps of the above process are skipped, i.e. that app and its dependencies are neither loaded nor started. You need to start it manually with:</p><blockquote>Application.ensure_all_started/1</blockquote><p>When is this useful? E.g. if you want to debug the start process of the application as it crashes upon starting.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d46187d1dd37" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A short guide to refc binaries]]></title>
            <link>https://medium.com/@mentels/a-short-guide-to-refc-binaries-f13f9029f6e2?source=rss-76c3137d6ba7------2</link>
            <guid isPermaLink="false">https://medium.com/p/f13f9029f6e2</guid>
            <category><![CDATA[elixir]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[erlang]]></category>
            <category><![CDATA[functional-programming]]></category>
            <dc:creator><![CDATA[Szymon Mentel]]></dc:creator>
            <pubDate>Tue, 12 Dec 2017 21:29:34 GMT</pubDate>
            <atom:updated>2017-12-12T21:29:34.298Z</atom:updated>
            <content:encoded><![CDATA[<p>According to the <a href="http://erlang.org/doc/efficiency_guide/binaryhandling.html">Erlang Efficiency Guide</a> there are four different types of <a href="https://hexdocs.pm/elixir/Kernel.SpecialForms.html#%3C%3C%3E%3E/1">binaries</a> implementation-wise. This blog post focuses on reference counted binaries (<em>refc binaries</em>). It also visits the <em>sub-binaries </em>to the extent they are related to the refc ones.</p><p>But why is it worth knowing the nitty-gritty of the binaries implementation?The reason is simple: in some circumstances, like performance tuning or tracing down bottlenecks, this knowledge helps.</p><h4>Basics</h4><p>In short, a refc binary is a container for the binary data which consists of<br>two elements:</p><ol><li>a reference to the actual binary stored on a process heap — <em>ProcBin</em></li><li>the referenced binary object stored in a dedicated shared memory space<br>in the VM (<em>binary space</em>).</li></ol><p>A binary object keeps track of the number of processes referencing it so that it can be garbage collected when no one is using it. In contrast, <em>heap binaries</em> are stored directly on a process’s heap and are garbage collected as any other process data.</p><p>The binary space is only for <em>large binaries</em>, i.e. those larger than 64 bytes. Within the binary space there is a notion of a <em>process’ binary virtual heap </em>which spans over all the refc binaries referenced by a process. Because these binaries are shared between processes, the virtual heaps overlap. But keep in mind — they are <em>virtual</em> which means that there’s no special data structure in the VM to represent them. In other words, each binary in the binary space belongs to at least one process’ virtual heap (or an ETS table — but we’re not talking about them here).</p><p>The following sketch illustrates a process with two binaries: a heap one and a refc one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*-83ZtQGZsjDIKy4YI-UZzA.png" /><figcaption>A process with an <strong><em>on-heap</em></strong><em> and </em><strong><em>off-heap</em></strong><em> binaries</em></figcaption></figure><p>The reason behind such implementation is performance. Because sending a message between two processes in Elixir involves copying data, there would be a performance penalty if the large binaries were copied between them. Instead, only the ProcBin (a pointer) is copied and the binary is shared between the processes. This is illustrated in the picture below where two processes share the same <em>off-heap binary.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*mEWFPG9lYmMhaqdtaBaF1A.png" /><figcaption>Two processes referencing <strong>the same off-heap binary</strong></figcaption></figure><p>To put the theory into a context, below is <strong>Bins</strong> module which demonstrates the issues discussed. The entry point is the <strong>run/0</strong> function that performs four<br>steps:</p><ol><li>spawn two processes which receive binaries and keep them in a loop;</li><li>send 1 MB large binary to the processes (the same to both);</li><li>force garbage collection of the current process;</li><li>output the information about binaries currently referred to by the processes.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4753ece9ff1cdebe9b8d39348b029d60/href">https://medium.com/media/4753ece9ff1cdebe9b8d39348b029d60/href</a></iframe><pre>iex(1)&gt; Bins.run<br>[<br>  {{:binary, [{390074408, 1048576, 2}]}, :p1},<br>  {{:binary, [{390074408, 1048576, 2}]}, :p2}<br>]</pre><p>The snippet contains some interesting bits that deserve additional comments.</p><p>First, we call the <a href="http://erlang.org/doc/man/erlang.html#garbage_collect-0"><strong>:erlag.garbage_collect/0</strong></a><strong> </strong>to get rid of the reference to the binary generated with <strong>one_mb_bin/1</strong> in the context of the current process (if you run the snippet from the shell it will be the shell process). It will become clear once we get to the last paragraph which touches garbage collection. Next, in the <strong>bin_info/0</strong> we call the <strong>Process.info/2</strong> with <strong>:binary </strong>which returns a list of 3-element tuples describing the ProcBins the process has references to. As per the <a href="http://erlang.org/doc/man/erlang.html#process_info-2"><strong>:erlang.process_info/2</strong></a><strong> </strong>documentation the three integers indicate the binary id, the binary size and the number of references to the binary respectively.</p><p>Having that said, we can finally focus on the output of the <strong>run/0</strong> function. As you can see, the two processes we spawned reference the same binary. This is <em>our</em> binary because it is 1 MB large (1048576 = 1024 x 1024) and is referenced by 2 processes.</p><p>Equipped with some basics regarding refc binaries, we can go on to the <br><em>sub-binaries</em>.</p><h4>Sub-binaries</h4><p>A sub-binary is a reference into a part of another binary: either a refc or heap one. We’ll focus on the first one.</p><p>A sub-binary can be created when a process is only interested in a part of a binary and that part is matched out. Again, there is a performance gain associated with this approach as the data is not copied. The following snippet introduces <strong>SubBins</strong> module which together with <strong>Bins</strong> illustrates a scenario involving sub-binaries.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9bce8903258437c962fa9c074d4e33f0/href">https://medium.com/media/9bce8903258437c962fa9c074d4e33f0/href</a></iframe><pre>iex(1)&gt; SubBins.run<br>[<br>  {{:binary, [{498941992, 1048576, 2}]}, :p1},<br>  {{:binary, [{498941992, 1048576, 2}]}, :p2}<br>]</pre><p>As in the previous example, two processes are spawned. This time, however, the second one pattern matches received binaries and keeps just the matched out sub-binary.</p><p>The output leads to the same conclusion: both processes reference the same<br>binary. And most importantly, both processes reference <em>the whole binary</em>. No additional binaries were created but the process holding the sub-binary keeps a reference to the original one. The diagram below illustrates the discussed scenario.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*LyUxBj1-G0d9_u6u3y_C9w.png" /><figcaption>Two processes referencing the same off-heap binary: the 1st<strong> uses the whole</strong> while the 2nd <strong>only a part of it</strong></figcaption></figure><p>If for some reason we don’t want to keep a reference to a whole binary when matching out only a part of it, we can resort to a <a href="http://erlang.org/doc/man/binary.html#copy-1"><strong>:binary.copy/1</strong></a> function. It returns a brand new copy of a binary passed as an argument. It means that if we pass it a sub-binary that refers to a larger refc binary, only the referenced part will be copied. Then the process will no longer need to reference the original binary. There’re good reasons to do so but I consider them out of scope for this blog post.</p><p>As we got to know some aspects of sub-binaries, let’s go forward to the last topic: refc binaries and garbage collection.</p><h4>Garbage Collection (<em>GC</em>)</h4><p>Refc binaries are garbage collected differently from all the other terms managed by a process. As it has already been mentioned they are <em>reference counted</em>.</p><p>If a process no longer needs a refc binary the attached reference counter is decreased <em>on a garbage collection</em>. Thus, if <em>all the processes that have ever touched the binary</em> are garbage collected after dropping it, the counter eventually reaches zero. Then the VM gets rid of the binary as it is no longer needed by any process. In other words, a binary stays alive as long as there’s at least one process that still has a reference to it (which may be the case even if the process no longer needs the binary but waits to be garbage collected).</p><p>The following snippet, which contains extended <strong>Bins</strong> module, illustrates the <em>GC</em> of ProcBins. Compared to the previous version of <strong>Bins</strong> module, the <strong>keep_binary/1</strong> function now handles a <strong>reset</strong> message that empties the binaries’ lists the process holds. What’s more, after sending the binaries to the processes, a <strong>reset_one/0</strong> function is called. It sends the <strong>reset</strong> message to the first process to make it forget its binary list. In the end, the <strong>spam_until_gc/0</strong> function is invoked which is the most tricky one in the presented example. It does the following:</p><ul><li>spawns the process that keeps “spamming” the first of the “binary processes” with 2 MB binaries,</li><li>set ups tracing on the spammed processes to monitor its garbage collection so that spamming stops once the garbage collection on a process starts,</li><li>prints <em>GC info</em> and <em>binary info</em> after the spammed process gets garbage collected.</li></ul><p>Below is what <strong>Bins</strong> looks like after modifications. Don’t worry if you don’t<br>understand it in the first place — it’s followed by an explanation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ad8c49d939b155f35880dced5539bc92/href">https://medium.com/media/ad8c49d939b155f35880dced5539bc92/href</a></iframe><pre>iex(1) Bins.run<br>[<br>  {{:binary, [{524288040, 1048576, 2}]}, :p1},<br>  {{:binary, [{524288040, 1048576, 2}]}, :p2}<br>]<br>iex(2)&gt; Bins.reset_one<br>[<br>  {{:binary, [{524288040, 1048576, 2}]}, :p1},<br>  {{:binary, [{524288040, 1048576, 2}]}, :p2}<br>]<br>iex(3)&gt; Bins.spam_until_gc<br>Trace for :gc_minor_start:<br>[<br>  ..., # keys skipped for readability<br>  bin_vheap_size: 393_216,<br>  bin_vheap_block_size: 196_630<br>]</pre><pre>Trace for :gc_minor_end:<br>[<br>  ..., # keys skipped for readability<br>  bin_vheap_size: 262_144,<br>  bin_vheap_block_size: 514_761<br>]<br>Spammed with 1 bins<br>[<br>  {{:binary, [{525340712, 2097152, 1}]}, :p1},<br>  {{:binary, [{524288040, 1048576, 1}]}, :p2}<br>]</pre><p>As this example is fairly complicated, let’s break it down command by command.</p><p>Command <strong>(1)</strong> does nothing special: the binary is sent to our two processes<br>and both reference it.</p><p>In <strong>(2)</strong>, the <strong>reset</strong> message is sent to the 1st process which makes it drop the binary. However, we can see that the binary created in the first place is still referenced by two processes. This is expected because even though the 1st process dropped the binary it has done too little work to get garbage collected. It is due to how the garbage collection works in the Erlang VM. It runs in the context of a process (there’s no <a href="https://en.wikipedia.org/wiki/Tracing_garbage_collection#Stop-the-world_vs._incremental_vs._concurrent">“stop the world”</a> GC) and is initiated when the process doesn’t have enough space on the heap to allocate a new object (more details on the garbage collector can be found in this <a href="https://www.erlang-solutions.com/blog/erlang-19-0-garbage-collector.html">blog post</a>).</p><p>That’s why we spam the <strong>:p1</strong> in command <strong>(3)</strong> until garbage collection occurs. After the process is garbage collected, the binary info proves that the original<br>binary (524288040) is now referenced only by the 2nd process (<strong>:p2</strong>), and the 1st one (<strong>:p1) </strong>references a new binary received during the spamming.</p><p>The <strong>spam_until_gc/0</strong> function does a couple of interesting things that are worth explaining. First, it set ups tracing on the <strong>:p1</strong> with the <a href="http://erlang.org/doc/man/erlang.html#trace-3"><strong>garbage_collection</strong></a> flag which generates trace messages when garbage collection for the process starts and ends. These messages are then received in the <strong>do_spam_until_gc/3</strong> function and once the <strong>:gc_minor_start</strong> event is received the spamming stops (if you’re new to tracing in the end of this article I post a few resources on that topic).</p><p>The keyword lists just after the command <strong>(3)</strong> come from these trace events. The first one describes some process’ characteristics when the GC is about to be started while the latter one refers to the situation afterwards. Their format is described in detail in the documentation of <a href="http://erlang.org/doc/man/erlang.html#gc_minor_start"><strong>:erlang.trace/3</strong></a> but let’s have a look at the bits that are interesting in the context of binaries. The <strong>bin_vheap_size </strong>tells us that<strong> </strong>before the GC, the process referenced 393216 words of binary data in total which amounts to 3 MB (see <a href="http://erlang.org/doc/man/erlang.html#system_info-1"><strong>:erlang.system_info(:wordsize)</strong></a>) This is correct according to the following calculation:</p><pre>iex(4)&gt; (393216 * :erlang.system_info(:wordsize))  / (1024*1024)<br>3.0</pre><p>Just before the GC started, the process referenced the original 1 MB binary and the new one (spam) weighting 2 MB. What can be also interesting for us is the <strong>:bin_vheap_block_size</strong>. Its value determines how much binary data can be put in the binary space before the process gets garbage collected. As you can see, before the GC the process “was allowed” to have ~ 1.5 MB of binaries while after the spamming the space grew to ~ 4 MB. The increase was caused by the garbage collector which noticed that the current size was insufficient to host the new 2 MB binary.</p><p>The following picture illustrates this example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tRSVQxerVPqZeBtKZEgOYg.png" /><figcaption>Off-heap binaries and <strong>garbage collection</strong></figcaption></figure><h4>Summary</h4><p>Before we wrap up: note that you have just completed a <em>very selective</em> tour over Elixir binaries. There is much more to say about them. However, I hope this blog post shed some light on the problem.</p><p>After carefully reading this post you should know:</p><ul><li>when your binary data is internally represented as a <em>refc binary</em>,</li><li>what are the consequences of matching out a <em>sub-binary</em> out of a refc one,</li><li>how the refc binaries are <em>garbage collected</em>.</li></ul><p>All the presented examples were run on Elixir 1.6.0-dev and Erlang 20.0.</p><p>Resources on taring in Erlang/Elixir:</p><ol><li><a href="https://www.safaribooksonline.com/library/view/erlang-programming/9780596803940/ch17.html"><em>Trace BIFs, the dbg Tracer, and Match Specifications</em></a><em> </em>chapter of Erlang Programming by Cesarini &amp; Thompson</li><li><a href="http://erlang.org/doc/man/erlang.html#trace-3"><strong>:erlang.trace/3</strong></a><strong> </strong>documentation</li><li><a href="http://erlang.org/doc/man/dbg.html"><strong>dbg</strong></a><strong> </strong>documentation</li><li><a href="http://ferd.github.io/recon/recon_trace.html"><strong>recon_trace</strong></a><strong> </strong>documentation</li><li><a href="http://www.erlang-factory.com/upload/presentations/316/dbg[1].pdf"><strong>dbg/ttb</strong></a> presentation</li><li>My <a href="https://youtu.be/EYmQ3YkJRYc">talk</a> and <a href="https://github.com/mentels/beam-tracing/blob/7782751281062bd1372e3ac88cda5e3066830e4e/elixir/docs/Crystalize%20your%20mixture.pdf">presentation</a> on Elixir tracing techniques overview</li></ol><p>Thank you <a href="https://medium.com/u/2f421853ecad">Kacper Mentel</a> <a href="https://medium.com/u/adf825c95e97">Radek Szymczyszyn</a> and <a href="https://github.com/studzien">Rafał Studnicki</a> for the reviews.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f13f9029f6e2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>