<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jonathas Conceição on Medium]]></title>
        <description><![CDATA[Stories by Jonathas Conceição on Medium]]></description>
        <link>https://medium.com/@Jonathas_Conceicao?source=rss-f3e705b63d7c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*ObOPTgdq7DPE0RxvDf2wJQ.jpeg</url>
            <title>Stories by Jonathas Conceição on Medium</title>
            <link>https://medium.com/@Jonathas_Conceicao?source=rss-f3e705b63d7c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 00:04:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@Jonathas_Conceicao/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[We just released UpdateHub Agent v2.0]]></title>
            <link>https://medium.com/os-systems/we-just-released-updatehub-agent-v2-0-eaba11c21458?source=rss-f3e705b63d7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/eaba11c21458</guid>
            <category><![CDATA[ota]]></category>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[updatehub]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[rust]]></category>
            <dc:creator><![CDATA[Jonathas Conceição]]></dc:creator>
            <pubDate>Tue, 09 Mar 2021 11:50:42 GMT</pubDate>
            <atom:updated>2021-03-09T11:50:42.139Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IHXLrwFopIpIof0vzwgwLA.png" /></figure><p>We are happy to announce the release of UpdateHub Agent v2.0, a fully rewritten version of our IoT Application for OTA updates with some overall improvements. In this article, we will be talking about some of the main changes to the UpdateHub Agent that come with this new version.</p><p>The UpdateHub has been fully rewritten in Rust, but be aware, <strong>this is not a language comparison</strong> as the internal implementations have also changed considerably from the original Go implementation we had for the UpdateHub Agent v1.1.</p><p>We want to start by sharing some of the major problems we’ve faced with the past version of the UpdateHub Agent. The main one we faced was about “memory usage predictability”, given the tight restrictions that some embedded devices enforce us, Go’s model of occasionally running a Garbage Collector, took away some of the control we needed to achieve the required performance. Memory usage would reach a peak only when installing a new update due to the need to decompress data and heavy write to disk, which was mostly handled by libarchive, but even when the agent was idle (e.g. waiting for a time to probe the UpdateHub Cloud for a new update) there would still be a considerable memory footprint.</p><p>Another point that definitely needed an improvement was the overall logging of the application. Finding both software and hardware malfunctions is a common task we get from our clients here at O.S. Systems, so having a complete and clean log for our update process is very important to help to look for potential issues. The application would also send its log to the UpdateHub Cloud when an update had failed, but the overall logging lacked some important points that would have helped us find the source of the error.</p><p>In order to fix the major issues we had with the UpdateHub Agent, we decided to rewrite the software from scratch. Mainly due to how memory management works in Rust, we decided to give the language a try for this version, not to mention that we are always excited about new technologies. Rust model’s of ownership and the guarantee of having a segfault-free software was satisfactory since the early development stages.</p><p>We also starting adding detailed logging of operations in different levels from the beginning. But the main improvement to log was the new feature we added to access the log of the last update cycle by the agent; this makes it especially useful for investigating bugs when an update process doesn’t go as expected. This last operation log is also sent to the UpdateHub Cloud if something goes wrong, making identifying faulty updates easier even without having direct access to the device.</p><p>We also took advantage of the full rewrite to simplify some aspects of the agent internal code structure to be more extensible and easy to maintain. The State Change Callback Scripts, which can be used to add custom hooks to specific states (e.g. turning Wi-Fi on before probing the UpdateHub Cloud for updates), were simplified to avoid some redundancy while maintaining the same usability. The internal State machine has also been updated to better represent the handle of use requested actions like the installation of a local package, or the command to cancel the download of a new update.</p><p>We have collected some data to compare how the memory usage has changed from the UpdateHub Agent v1.1 to this new v2.0 release. We used a RaspberryPi 3B+ to fetch and install the very same system update package, we used <a href="https://pypi.org/project/memory-profiler/">mprof</a> to measure the residual memory the application consumed from the probe time till the agent was ready to reboot the device into the new installation. Here’s the memory usage plot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*6PLb3iulSbwKQa5V63YLSg.png" /><figcaption>Memory usage from fetch to install of an update package.</figcaption></figure><p>Both versions have a very similar increase in memory usage since the process to install a package has not changed much, and the new version also uses libarchive for handling compressed archives. But the residual memory both before and after the heavier workload had some significant decrease. The overall installation time has also been smaller for the UpdateHub Agent v2.0 which is always a good thing.</p><p>We also captured the residual memory usage of the Agents without having an update available and probing the server every minute. In this case, we can see that the new version not only has a lower memory footprint but also keeps a steady memory usage over time. Agent v1.1 shows a small (but constant) increase in memory usage due to how to Go’s runtime doesn’t fully release resilient memory unless the OS actually needs more memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*TnJLTqnPqEV193ByFmYGrQ.png" /><figcaption>Idle application’s memory usage.</figcaption></figure><p>Another important parameter when it comes to IoT devices is binary size. On this front, however, we actually had a little fallout. Rust’s micro-dependency model has not been kind to us, and even after some tweaking of our major dependencies we still ended up with a slightly bigger binary.</p><p>What we did to have some improvement in binary size was to combine the Agent application with its command-line app, which was previously separated from the agent in v1.1. Since not everyone that uses the UpdateHub Agent actually needs the CLI controller, this is not exactly a win for the v2.0 release but is still an important parameter to share.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/964c0a214ed8fbd03a97a0d548dbba58/href">https://medium.com/media/964c0a214ed8fbd03a97a0d548dbba58/href</a></iframe><p>We have been working on this v2.0 release on and off for a long time due to some other project for our clients. But through the whole process over these past two years of development, we have always been excited about the new changes. The biggest advantage is that the overall code structure of the newly rewritten agent has been improved, and now that it’s finished we can start to look at the new features and improvements we can make.</p><p>The next feature we are eager to release is the support for Differential Updates, a way to only download and install the chunks of data that has actually changed from one version to another. Right now we are exploring the use of <a href="https://github.com/oll3/bita">bita</a> for it, a well-performing tool also written in Rust that supports Differential Updates for file synchronization.</p><p>If you wanna keep up to date with the lasted changes to the UpdateHub Agent be sure to follow us on GitHub at <a href="https://github.com/UpdateHub/updatehub">https://github.com/UpdateHub/updatehub</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eaba11c21458" width="1" height="1" alt=""><hr><p><a href="https://medium.com/os-systems/we-just-released-updatehub-agent-v2-0-eaba11c21458">We just released UpdateHub Agent v2.0</a> was originally published in <a href="https://medium.com/os-systems">O.S. Systems</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Benchmarking HTTP Client-Server Binary Size in Rust]]></title>
            <link>https://medium.com/os-systems/benchmarking-http-client-server-binary-size-in-rust-3f4398f2aa07?source=rss-f3e705b63d7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f4398f2aa07</guid>
            <category><![CDATA[binary-size]]></category>
            <category><![CDATA[benchmark]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Jonathas Conceição]]></dc:creator>
            <pubDate>Mon, 06 Jul 2020 17:39:10 GMT</pubDate>
            <atom:updated>2020-07-08T06:34:43.284Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RXA3NocNLu3MrP6f9eP9sA.png" /></figure><p>Choosing the right dependencies for your can been a complicated task. From performance to code style there are so many aspects to take into account for making this decision, and choosing the right dependencies early can save programmers hours of rework and refactoring in the long run. One of the important aspects when dealing with IoT devices is the binary size of the application, and that’s what we will be focusing today in this report.</p><p>Our usual choices for for server and client frameworks in Rust used to be actix-web and reqwest, they are both among the most popular in each segment and provide a lot of features. At some point, given that they have a lot of different dependencies, we decided to swap the client framework, reqwest, for awc, hoping to see a decresce in our total binary-size. The result we got, however, was a incresse of about 3% of the total binary-size after the full migration.</p><p>As HTTP Client and Server is a very recurrent component of our IoT applications and that lower binary size is always a goal to achieve, we have decided to benchmark how different pairs these frameworks would vary in regards to binary size. Doing so we can choose the most suitable dependencies to our current and future projects.</p><p>Firstly we have defined some basic requirements for each of the benchmark contestants based on our common use case. We then selected a few popular pairs of applications to test. After that we separated what aspects of the applications would be interesting to analyse, compare, and we finally went straight to hacking.</p><p>We have also built a interactive web app to showcase the data we collected to help us, and anyone interested, to analyse our results. We’ve analysed not only the total binary size, but also the estimated influence of each of it’s dependencies. With this we have been able the new pair of Server and Client framework we will be using from now on.</p><p>The rules we decided to specify for our benchmark were based on the needs of HTTP Client and Server that we have on our Updatehub Agent. There, we have a periodic task to run — checking and possibly installing an update — and it’s also important to keep the Agent fully responsive at all times, as other applications or the user himself can query the agent to know what it’s doing. The agent offers via a small HTTP API a set of endpoints that local applications can access to either request information about the agent’s execution, or to trigger some actions, e.g., canceling a download.</p><p>With all these in mind, these are the requirements that we have settled for our:</p><ul><li>Asynchronous API: One of the main reasons we love Rust is the current state of the async/await support of the ecosystem. It offers a really nice way to implement concurrency, offering a simple way of make the agent responsive at all times.</li><li>Internal State Sharing: During the update process, there are several variables that will be kept in memory and might be queried at anytime on the Agent. That why we require that our benchmark supports a sharing internal state between the client used to query the cloud and the server used to respond the local API.</li><li>One route to server and one to query: We decided to serve and access only one route on the benchmark. We expect it to be enough to link the relevant functions of the libraries to the final binary so we can compare them.</li><li>Use OpenSSL if available: On Updatehub Agent we already use OpenSSL for validating digital signatures, and since in our use case the remote link would usually use HTTPS protocol, we have defaulted to OpenSSL when ever available.</li></ul><p>With the rules setup about what would be required for the apps, these are the contestants we decided to test:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/68a50960b3669fe574ea30e034406a3e/href">https://medium.com/media/68a50960b3669fe574ea30e034406a3e/href</a></iframe><p>For our data analysis we have used <a href="https://github.com/RazrFalcon/cargo-bloat">cargo-bloat</a> to estimate the weight of each dependencies towards the final binary. Although it’s not 100% accurate, it does leave clear the “unknown” bits at the binary — marked as a Crate named “[Unknown]” — and on our use cases it was always a small amount.</p><p>For all projects we have also tested a feel different compiling flags in order to check for binary-size optimization. The following table shows all the parameters used.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2b63d2b9f4fde3ddd56e8a5bb02bd07/href">https://medium.com/media/e2b63d2b9f4fde3ddd56e8a5bb02bd07/href</a></iframe><p>We have crated a interactive dashboard to display our interactive graphs available at <a href="https://jonathas-conceicao.shinyapps.io/web-client-server-binary-size-benchmark-rs/">this shinyapp</a>. The app allows for a more detailed view of all the data we collected, here we will be focusing on some of the main points.</p><p>We start the results showdown by showing the comparison between actix-reqwest and actix-full, the initial contestas the motivated this benchmark. Even with optimizations tuned, the context we’ve setup to test binary sizes also shows a advantage in we comparing awc to reqwest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oKwjhC7gDT1g38qjIXLU3g.png" /><figcaption>awc vs reqwest (<a href="https://jonathas-conceicao.shinyapps.io/web-client-server-binary-size-benchmark-rs/">interactive graph</a>).</figcaption></figure><p>Another important point we like to highlight is that the micro-dependency model that Rust tends to follow doesn’t seam to affect very much the binary size. In this next plot we show the size distribution of all crates usage in all contestants. The box plot shows that most crates contribute very little to the total binary size, and that a few larger crates are actually responsible for raising the binary size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P9K8yX4GBa9ZEkn8Vo7oEg.png" /><figcaption>Crates size distribution across all projects (<a href="https://jonathas-conceicao.shinyapps.io/web-client-server-binary-size-benchmark-rs/">interactive plot</a>).</figcaption></figure><p>As far as optimization flags goes we had no surprises. Setting optimization level to z, lto to fat, and number of codegen units to 1, has show better results in all of the contestants. The following plot shows optimization on the tide-surf contestant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XAZD6qhWoFb1y_5b6XxFyQ.png" /><figcaption>Optimization parameters on <strong>tide-surf </strong>(<a href="https://jonathas-conceicao.shinyapps.io/web-client-server-binary-size-benchmark-rs/">interactive plot</a>).</figcaption></figure><p>The <a href="https://github.com/async-rs/async-std">async-std</a>, tide-surf, is our new choice of HTTP client-server pair for embedded applications. Despite having a high level API, it still had better results that more lower level APIs such as hyper_full.</p><p>If we were to choose a pair that’s built fully on top of <a href="https://github.com/tokio-rs/tokio">tokio</a>, another of the main runtimes for rust’s async ecosystem, we will definitely be considering <a href="https://github.com/seanmonstar/warp">warp</a>. The crate has presented a small Binary Size overhead over <a href="https://github.com/hyperium/hyper">hyper</a> for a much more simpler API for building HTTP servers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B1h1s6-59pam_N2tJpC63g.png" /><figcaption>All projects size (<a href="https://jonathas-conceicao.shinyapps.io/web-client-server-binary-size-benchmark-rs/">interactive plot</a>).</figcaption></figure><p>If you wanna keep updated with future updates on the benchmark be sure to follow the <a href="https://github.com/OSSystems/web-client-server-binary-size-benchmark-rs/">repository on GitHub</a>. Release tags will be used to track the results change through time. If you wanna more details about the data we collected checkout our <a href="https://jonathas-conceicao.shinyapps.io/web-client-server-binary-size-benchmark-rs/">interactive shinny app</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f4398f2aa07" width="1" height="1" alt=""><hr><p><a href="https://medium.com/os-systems/benchmarking-http-client-server-binary-size-in-rust-3f4398f2aa07">Benchmarking HTTP Client-Server Binary Size in Rust</a> was originally published in <a href="https://medium.com/os-systems">O.S. Systems</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[compress-tools: A Swiss Army Knife for handling compressed data in Rust]]></title>
            <link>https://medium.com/os-systems/compress-tools-a-swiss-army-knife-for-handling-compressed-data-in-rust-9a3c3a75b6b3?source=rss-f3e705b63d7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a3c3a75b6b3</guid>
            <category><![CDATA[embedded-linux]]></category>
            <category><![CDATA[compression]]></category>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[libarchive]]></category>
            <category><![CDATA[rust]]></category>
            <dc:creator><![CDATA[Jonathas Conceição]]></dc:creator>
            <pubDate>Wed, 15 Apr 2020 11:53:45 GMT</pubDate>
            <atom:updated>2020-06-30T16:35:34.731Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Someone working on computer and laptop" src="https://cdn-images-1.medium.com/max/1024/1*GKS0DQW_Mg6fJz0VpUJqtg.png" /></figure><p>Here at <a href="https://www.ossystems.com.br/"><strong>O.S. Systems</strong></a> we have been working with Rust for a over a year now.<br>From type safety and a well designed syntax, to high performance and nice memory usage, Rust has been a great tool for the new products we have been working and even for older ones we are migrating.</p><p>Working with Embedded Linux often brings into discussion some common issues to the table when specifying a new product; we can mention binary size, disk usage, memory scalability just to name a few. The possibility to use dynamically linked libraries is mostly a welcome solution to tackle this problems.</p><p>We present here our motivation into developing <a href="https://crates.io/crates/compress-tools"><strong><em>compress-tools</em></strong></a><em>,</em> a Rust library we have created to offer some of <em>libarchive</em>’s functionalities with handy integration to Rust’s type system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JKZU5EDVghF3Gp6T2R_hqg.png" /></figure><p>Rust itself has a really nice and robust compression and decompression library called <a href="https://crates.io/crates/flate2"><em>flate2</em></a>, with support for <em>DEFLATE</em> based compression formats. Its has a native implementation and support for a <em>libz</em> and <em>miniz</em> backends for the user to choose from, it has served us really well in past projects. However, for the new version of <strong>Updatehub Agent</strong>, which is been fully rewritten in Rust, we are working on we need to support for a larger pool of compression and archive formats, leading us to search for alternatives.</p><p><em>libarchive </em>was already used in the original version of the <strong>Updatehub Agent</strong> for handling this compressed objects, and being able to handle so many compression and archives formats made it our main choice once again. <a href="https://crates.io/">Rust’s package registry</a>, unfortunately, didn’t seem to have any library for <em>libarchive</em> usage being actively maintained to the best that we could find. This crate published as <a href="https://crates.io/crates/libarchive"><em>libarchive</em></a> would probably serve well but it has not seen a release in over 4 years, and there was basically no activity on the issues and pull requests open on their Github repository.</p><p>Maintaining a full port of <em>libarchive </em>with the Rust type safe binds to make it more usable is not a simple task, but what we need for this project at this point is just a small set of it’s functionalities, which shouldn’t be hard to use nor to maintain. That was when we decided to create <strong>compress-tools </strong>offering some simple API to handle compressed data that we can ingrate our embedded software projects in Rust.</p><p><a href="https://updatehub.io/"><strong>Updatehub</strong></a><strong> </strong>is an enterprise-grade solution for Software and Firmware Update Over The Air. The update packages are always compressed to improve network usage and multiple different formats of compression are supported. The <strong>Updatehub Agent</strong> is the application that runs on the embedded device to install the updates, hence having to deal frequently with uncompressing packages.</p><p>Initial version of <strong>compress-tools</strong> was aimed at better defined the API we would need to provided in order to use it on <strong>Updatehub</strong> as well as to test a hole execution cycle of the <strong>Updatehub Agent</strong> implementation we had at the time. To avoid complexity we started a initial implementation that instead of using the <em>libarchive</em> library, just did the objects uncompressing using shell commands under the hood. After testing with disk space and memory usage we had defined three main functions that would need to be implemented by the <strong>compress-tools</strong> in order to have it fully meet our needs:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/88548b9aff935a15d139f5aa97cc923c/href">https://medium.com/media/88548b9aff935a15d139f5aa97cc923c/href</a></iframe><p>The functions should all be able to handle a generic reference to a std::io::Read, so the compressed data can come straight from memory, from a buffered OS file read, or even a special read object that can support timeouts to handle possible disk fails that are fairly common with flash memory devices as provided by the <a href="https://crates.io/crates/timeout-readwrite">timeout-readwrite</a> crate.</p><p>The output would be a generic std::io::Write for most cases so it can also be manipulated more freely, but when uncompressing an archive file, a single Write object won’t do the trick, and there are always proprieties of the files we need to preserve, such as ownership and permissions, so for this case the function actually accepts a Path as it’s target and write it’s output straight to disk.</p><p>Using this <strong>compress-tools </strong>handy functions on <strong>Updatehub Agent</strong> has allowed us to have more control when handling compressed data. This are some of the changes we were now able to made:</p><ul><li>We can now read the Metadata structure stored inside a <strong>uhupkg </strong>(<strong>Updatehub</strong>’s update package format) as well as validate it’s Digital Signature without the need to extracting the hole package, greatly improving disk usage.</li><li>We were able to implement our <strong>Raw Install </strong>mode, that installs a update from a compressed source directly into one of the devices’s disk, much like a gzip -d foo.gz | dd of=/dev/bar would do in plain shell.</li><li>The <strong>Tarball Install Mode </strong>can install files while keeping the expected permissions, ownership and timestamps without the need of the <em>tar</em> to be installed in the Embedded device.</li></ul><p>Another advantage of using <em>libarchive</em> for handling compression on this IoT context is that the library itself can be built with different configurations depending on what the device will have to support, which works really well to adapt it to the embedded device constrains.</p><p>Any new requirement that we might have for using <em>libarchive</em> with Rust now will be handled though our new <strong>compress-tools</strong> crate. Other handy functions might become necessary and we consider at some point adding support for compressing and archiving objects there too, so feel free to follow the <a href="https://github.com/OSSystems/compress-tools-rs/">repository over on Github</a> to keep track of new releases.</p><p>Be sure to check the <a href="https://docs.rs/compress-tools/">compress-tools documentation</a> if you want more details and examples on how to use this tools.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a3c3a75b6b3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/os-systems/compress-tools-a-swiss-army-knife-for-handling-compressed-data-in-rust-9a3c3a75b6b3">compress-tools: A Swiss Army Knife for handling compressed data in Rust</a> was originally published in <a href="https://medium.com/os-systems">O.S. Systems</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>