<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Johannes Foufas on Medium]]></title>
        <description><![CDATA[Stories by Johannes Foufas on Medium]]></description>
        <link>https://medium.com/@johannes.foufas?source=rss-8ca435c3cd7e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*yb6xDOnfQvP9TJA2IPY8tA.jpeg</url>
            <title>Stories by Johannes Foufas on Medium</title>
            <link>https://medium.com/@johannes.foufas?source=rss-8ca435c3cd7e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 01:02:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@johannes.foufas/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Agnostic Build System: Powertrain build]]></title>
            <link>https://medium.com/volvo-cars-engineering/agnostic-build-system-powertrain-build-53cb007dfa7f?source=rss-8ca435c3cd7e------2</link>
            <guid isPermaLink="false">https://medium.com/p/53cb007dfa7f</guid>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[c-coding]]></category>
            <category><![CDATA[automotive-software]]></category>
            <category><![CDATA[autosar]]></category>
            <category><![CDATA[build-system]]></category>
            <dc:creator><![CDATA[Johannes Foufas]]></dc:creator>
            <pubDate>Tue, 18 Jun 2024 09:48:37 GMT</pubDate>
            <atom:updated>2024-06-18T09:48:37.394Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Powering Volvo Cars’ Software Builds</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eeLb_qyQzaA5GEZgJxQ-gw.jpeg" /></figure><p>We are thrilled to announce the release of the Volvo Cars Powertrain Build project, now available on <a href="https://opendev.org/volvocars/powertrain-build">OpenDev</a>!</p><p>As a build system tailored for embedded software development using DSpace Target Link or Mathworks Embedded coder, this project is set to improve the way we as automotive embedded engineers handle our code bases. The idea is to make it easy to move between different targets, enable Software in the Loop (SIL) testing, and streamline our continuous integration (CI) processes.</p><p>Powertrain build offers to efficiently move embedded source code between different embedded targets, both Posix based and old school embedded ECU’s. This build system does what AUTOSAR never did, and is a tool that prepares us for the future of the car OS. Check out the article below to learn more about AUTOSAR:</p><p><a href="https://medium.com/volvo-cars-engineering/the-reality-of-autosar-and-the-way-forward-36af39ec4099">The reality of AUTOSAR and the way forward</a></p><p>Our Powertrain Build project is designed to work harmoniously with the Zuul CI system. This ensures that all code changes are automatically tested and validated, providing continuous feedback to developers and maintaining the integrity of our codebase.</p><h3><strong>What is it?</strong></h3><p>The build system is written in Python for ease of use and flexibility, <a href="https://opendev.org/volvocars/powertrain-build/src/branch/master/docs/build_cmd.md">and these are the build options</a>.</p><pre>py -3.10 pybuild.wrapper --build ABC_123</pre><p>Projects are defined in .json using <a href="https://opendev.org/volvocars/powertrain-build/src/branch/master/docs/project_config.md#project-config">Project</a>, <a href="https://opendev.org/volvocars/powertrain-build/src/branch/master/docs/project_config.md#base-config">base</a> and <a href="https://opendev.org/volvocars/powertrain-build/src/branch/master/docs/project_config.md#units-config">Unit</a> configs.</p><p>The project config file contains project specific configuration settings and references to other configuration files. The base config file shares the same structure as the project config file and can be used for non project-specific configuration settings and default values, these could for instance be diagnostic core dummy file names. The units config file contains information about included software units and scheduling rasters. The software units are executed in the order they are defined within each time raster definition.</p><pre>{<br>    &quot;SampleTimes&quot;: {<br>        &quot;VcExt10ms&quot;: 0.01<br>    },<br>    &quot;VcExt10ms&quot;: [<br>        &quot;VcDebug&quot;,<br>        &quot;VcCarCfg&quot;,<br>    ]<br>}</pre><p>Schedulers are not included in the powertrain build, but they could easily be added.</p><p>Usually, an interface configuration is used, where individual c-functions variables are connected to com-variables that are sent on different busses.</p><p>The system utilises C Preprocessor directives, instead of code switches, in TargetLink models, allowing for a superset C code representation of Simulink models.</p><p>Variables use different <a href="https://opendev.org/volvocars/powertrain-build/src/branch/master/pybuild/memory_section.py#L13">variable classes</a>, to direct them to appropriate memory addresses.</p><p>The system includes unit tests for build scripts and HTML documentation for transparency and maintainability. One feature is the consistency checks that are done by creating JSON files of the code. <a href="https://opendev.org/volvocars/powertrain-build/src/branch/master/docs/signal_consistency_in_check.md#checks">Signal consistency is a wonderful tool</a>, where we have the classic signals traveling in CAN frames. The tool reports if we delete signals that are used, or if we produce signals that are not consumed.</p><pre>py -3.10 pybuild.wrapper --build ABC_123 --interface</pre><p>A developer would usually follow this workflow, which is also detailed in the picture below:</p><p>1. Design the Simulink model and generate code using powertrain-build command.<br>2. Build the relevant powertrain-build project.<br>3. Inspect the generated artifacts, and commit those if desired.<br>4. Build the SIL binary, and write the unit and smoke tests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*e-NAO5pY9g9MTo4w9bnDWQ.jpeg" /><figcaption><a href="https://en.wikipedia.org/wiki/Object_Process_Methodology">Object Process Methodology</a> diagram of a typical flow.</figcaption></figure><p>A typical file structure looks like this:</p><pre>gitrepo/<br>├── ConfigDocuments/<br>│   ├── .<br>│   ├── .<br>│   ├── .<br>│   ├── SPM_Codeswitch_Setup.csv<br>│   ├── SPM_Codeswitch_Setup_ICE.csv<br>│   ├── SPM_Codeswitch_Setup_PVC.csv<br>│   ├── .<br>│   ├── .<br>│   ├── .<br>├── Models/<br>│   └── PVCTM/<br>│       └── VcPvcDemo/<br>│           ├── pybuild_cfg/<br>│           │   ├── config_VcPvcDemo.json<br>│           │   └── VcPvcDemo.yaml<br>│           ├── pybuild_src/<br>│           │   ├── tl_defines_PvcDemo.h<br>│           │   ├── VcPvcDemo.a2l<br>│           │   ├── VcPvcDemo.c<br>│           │   ├── VcPvcDemo.h<br>│           │   └── VcPvcDemo_OPortMvd_LocalDefs.h<br>│           ├── tests/<br>│           │   ├── _cumulated_code_coverage_<br>│           │   │   ├── ctcpost_merge_options.txt<br>│           │   │   └── experiment.spec<br>│           │   ├── VcPvcDemo_UnitTests<br>│           │   │   ├── 00_test_setup<br>│           │   │   │   ├── dataset.DCM<br>│           │   │   │   ├── sut_config.txt<br>│           │   │   │   ├── sut_interface.py<br>│           │   │   │   └── twTest.sil<br>│           │   │   ├── 01_stimulus<br>│           │   │   │   └── U_VcPvcDemo_ExplorativeStimulus.py<br>│           │   │   └── 02_watchers<br>│           │   │       └── U_VcPvcDemo_watcher.py<br>│           │   ├── ctc_env.bat<br>│           │   └── project.testweaver<br>│           ├── VcPvcDemo.mdl<br>│           ├── VcPvcDemo_par</pre><h3><strong>Why open source?</strong></h3><p>As we’re transitioning to become a fully electric carmaker, we no longer have powertrain software development for combustion engines in house. However, there was a need to be able to develop and share the build system, since it is still used for electric cars.</p><p>The easiest option was of course to put the code as open source.</p><p>Since we like Zuul to run tests, and we do that internally, we asked <a href="https://opendev.org/">Opendev</a>. They kindly offered us our own tenant in their Zuul <a href="https://zuul.opendev.org/t/volvocars/buildsets">that you’ll find here</a>.</p><p>We also want to fill Industry Gaps, as the build system addresses gaps that AUTOSAR couldn’t fill, providing a tool that is well-prepared for the upcoming advancements in car OS. Sharing the code behind these alternatives can foster positive feedback and collaboration with other OEMs.</p><p>It is also much more fun for us to work with our code in the open!</p><p>Now, we look forward to new collaborations, in the same open and friendly way that OpenDev has described <a href="https://opendev.org/volvocars/powertrain-build/src/branch/master/CONTRIBUTING.md#">in their contribution guidelines</a>.</p><p>Go ahead and check out the code and let’s collaborate!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53cb007dfa7f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/volvo-cars-engineering/agnostic-build-system-powertrain-build-53cb007dfa7f">Agnostic Build System: Powertrain build</a> was originally published in <a href="https://medium.com/volvo-cars-engineering">Volvo Cars Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The reality of AUTOSAR and the way forward]]></title>
            <link>https://medium.com/volvo-cars-engineering/the-reality-of-autosar-and-the-way-forward-36af39ec4099?source=rss-8ca435c3cd7e------2</link>
            <guid isPermaLink="false">https://medium.com/p/36af39ec4099</guid>
            <category><![CDATA[autosar]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[automotive-software]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Johannes Foufas]]></dc:creator>
            <pubDate>Fri, 16 Jun 2023 12:20:40 GMT</pubDate>
            <atom:updated>2023-06-16T12:20:40.366Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CR4KseKZxZsBWBdEqjdedQ.jpeg" /></figure><p>The evolution of car software has reached a point where there needs to be a radical change. Fully electric, autonomous, and connected vehicles require many million lines of code, and the updates need to be made over the air.<br>Many car suppliers talk about their OS, and there is a need for an ecosystem, that makes it easier to access off-board and on-board systems, with alluring features such as dynamic discovery of services and network agnostic APIs.</p><p>Another trend is the arrival of centralized computing platforms, the need for a discrete microcontroller for each vehicle’s functions is not needed anymore.</p><p>Many, but not all of these microcontrollers ran Classic <strong>AUT</strong>omotive <strong>O</strong>pen <strong>S</strong>ystem <strong>AR</strong>chitecture (AUTOSAR) based operation systems. AUTOSAR is a development partnership of automotive interested parties founded in 2003.</p><p>AUTOSAR was preceded by the OEM’s desire to use <a href="https://en.wikipedia.org/wiki/OSEK">OSEK</a> -certified (<em>Offene Systeme und deren Schnittstellen für die Elektronik in Kraftfahrzeugen</em>; English: <em>“Open Systems and their Interfaces for the Electronics in Motor Vehicles</em>”) operating systems (only the scheduling part to begin with). Many Tier-1s had to get rid of their mature, target optimized and efficient proprietary schedulers. Instead, they had to use bulky, sub-optimized operating systems which may carry many features not needed by their application.</p><p>The objective with AUTOSAR was to create and establish an open and standardized software architecture for automotive electronic control units (ECUs). This standardization was to provide scalability to different vehicle and platform variants, and to enable transfer of software components between different suppliers and target ECUs.</p><p>The main concept of the standardized ECU software architecture was the separation of hardware-independent application software and hardware-oriented basic software (BSW) by means of the software abstraction layer RTE (runtime environment). RTE is a middleware which abstracts from the network topology for the inter and intra-ECU information exchange between the application software components and between the Basic Software and the applications. One of the biggest problems with AUTOSAR is just the RTE. It is very complex and like a mini-OS with its own execution rates. The top abstraction layer above the RTE enables the development of OEM-specific software applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_mSjFuvSlQ032tqE8-yr4A.png" /><figcaption>Simplified AUTOSAR stack</figcaption></figure><p>Now, this sounds like an appealing idea, but there are many drawbacks.<br>Since AUTOSAR is developed by a committee (consisting of OEMS, Tier-1s, various component vendors, etc.), most of their nice-to-have features are present in the standard. That means that instead of a lean standard in some cases the standard is bloated. The amount of configuration parameters is endless, and complexity-by-design often breeds problems and high CPU load.</p><p>A software developer, who experiences an AUTOSAR OS for the first time, can have great difficulties. These quotes pretty much sums it all up:</p><blockquote>there are now AUTOSAR experts and AUTOSAR tools and AUTOSAR Tool Experts and within that tool there is an ARXML generator which is generated with another tool … (source: Reddit)</blockquote><blockquote>I was considered an “AUTOSAR expert” for a time, and that essentially meant having enough programming and systems knowledge to work on the entire stack; but never writing a single line of code, only clicking buttons in these tools and watching them crash continuously, loosing hours upon hours of work. (source: Hacker News)</blockquote><p>First, AUTOSAR is just a standard, and you need an implementation of it. Often, different vendors have different interpretations of the standard, so if we mix vendors in the same car project, we might have some surprises. Then the standard would most likely break. On top of this, each vendor makes tools, always GUI based, for which you need to configure your OS. This in turn causes problems, because these tools are difficult to automate, if at all possible, and put in a modern Continuous Integration (CI) system. Also, it’s hard to know what code all the configurations and their combinations will produce.</p><p>You might have a promising idea that you want to test under limited time (e.g. in a spike, during a hackathon, etc.), but the whole DevOps experience drowns and delays this innovation in a myriad of clicks lost through thousands of different options while reading the manuals in a not so intuitive or stable GUI. The end result also might be unexpected due to an obscure option on an unrelated menu. There might even be cases where you click one option, and then the tool changes old settings silently, that you did not think it would do.</p><p>The typical AUTOSAR workflow is by modelling a GUI application resulting in a project in some database. Then, the GUI produces a descriptions recipe file in ARXML, which defines the formal exchange format (AUTOSAR Schema) and the semantic constraints which go along with the exchange format.</p><p>Often then, this ARXML file goes into another GUI tool and then finally is used to generate source code; .c and .h files.<br> A huge problem is that the ARXML files are enormous in size, and they are extremely difficult to parse and generate.</p><p>Overall, it is a manual process, which is both error-prone and results in long lead times.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/1*X7jut3MMi2Gw7AgbAvVKuw.png" /><figcaption>A generic CI flow for GUI based AUTOSAR systems, || are manual steps * are automatic</figcaption></figure><blockquote><em>Is there a way around this?</em></blockquote><p>Well, you don’t need to use the RTE and all the manual steps producing ARXML. You could work with source files and machine-readable recipe files like .json or .YAML. Especially, if you implement a <a href="https://www.autosar.org/fileadmin/standards/R22-11/CP/AUTOSAR_EXP_CDDDesignAndIntegrationGuideline.pdf">Complex Device Driver (CDD)</a>, then you could avoid a lot of manual ARXML productions.</p><p>A CDD is a software entity, not standardized by AUTOSAR, which can access or be accessed via AUTOSAR Interfaces and Basic Software Modules APIs. The main goal of the CDD is to implement complex sensor evaluation and actuator control with direct access to the microcontroller, using specific interrupts and complex microcontroller peripherals, that is, external devices like communication transceivers, ASIC, etc., to fulfill the special functional and timing requirements.</p><p>In addition, it might be used to implement enhanced services and protocols or encapsulate legacy functionality of a non-AUTOSAR system. So, this is a way out in some cases.<br> <br>Of course, such an approach may be seen risky, since one needs to write code and not just trust the code generation of some tool. And it might indeed be problematic, if done in a less professional setting (e.g., lack of experience, lack of testing, low coverage, etc.). But, if a company owns the stack, they also need to own the code, so there might not be a way out of this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*p4y8yApipOCCTEZzL-bCaA.png" /><figcaption>A generic CI flow without AUTOSAR GUI tools</figcaption></figure><h3>CPU load</h3><p>During the years of developing embedded software in vehicles, I had the chance to talk to one of the largest suppliers of ECUs and of embedded OS platforms. Their point of view was that sometimes there is a hard requirement to use AUTOSAR, but still, a cheap CPU had to be used. Due to this, they claimed that it is often necessary to have a team of experienced engineers (10 engineers or more) to work for significant amounts of time (years+) modifying their AUTOSAR implementation to optimize and trim the AUTOSAR performance.</p><p><em>Many engineers have told me, and I have observed firsthand, that different AUTOSAR OS’es add cpu load, how much cpu load that is added is dependent on the implementation</em>.</p><p>On cars with combustion engines, the engine ECU was many times the one handling the most complex algorithms. When the requirement to convert these to AUTOSAR came, it was observed that the CPU load increased by more than 30%, in the same codebase just by running the COM stack. This led to solutions where one core ran just the OSEK OS and sometimes in combination with CDDs in some instances. Signaling between the different layers is handled by .c and .h files, configuration was done in .json and .csv and developers were still happy. These developers did not see how the developers working with ARXML files and databases suffered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*jqOu4l-6EUQz-OkRd36LMQ.png" /><figcaption>A mix of AUTOSAR and non-AUTOSAR</figcaption></figure><p>Still, there were some teams who had their algorithms distributed in pure manual ARXML populated development kits. During this time, we got a lot of support in offering a CI chain that could produce downloadable and tested software in less than 45 minutes, compared to 12–48h.</p><blockquote><em>So, what could be the future? What are the alternatives?</em></blockquote><p><a href="https://medium.com/volvo-cars-engineering/why-volvo-thinks-you-should-have-rust-in-your-car-4320bd639e09">As already written about, we at Volvo Cars Corporation implement a full ECU in RUST</a>.</p><p>There is an interesting upcomer <a href="https://oxidos.io/">https://oxidos.io/</a> that offers a safe OS, written in RUST and based on <a href="https://www.tockos.org/documentation/design">Open-source</a>.</p><p>Another interesting project is <a href="https://github.com/zephyrproject-rtos/zephyr">https://github.com/zephyrproject-rtos/zephyr</a>. It might not be Safety Certified for some years, but it looks promising, and it is Open-source.</p><p>Then there is of course FreeRTOS. The FreeRTOS kernel was originally developed by Richard Barry around 2003 and was later developed and maintained by Barry’s company, Real Time Engineers Ltd. In 2017, the firm passed stewardship of the FreeRTOS project to A<a href="https://aws.amazon.com/freertos/">mazon Web Services (AWS)</a> . Barry continues to work on FreeRTOS as part of an AWS team.</p><p>Then we have SAFERTOS®, that is based on the functional model of the FreeRTOS kernel. This is available and pre-certified to ISO 26262 ASIL D by TÜV SÜD, but not Open-source.<br> <br>So, what is needed is a developer-friendly environment that uses source code, and machine-readable recipe files. It would be needed to be CPU efficient, and many solutions that are Opensource utilize shared memory solutions (example: <a href="https://iceoryx.io/latest/getting-started/what-is-iceoryx/#fast">https://iceoryx.io/latest/getting-started/what-is-iceoryx/#fast</a> ).</p><p>When it comes to middleware and API standards, the <a href="https://www.dds-foundation.org/what-is-dds-3/">Data Distribution Service (DDS™)</a> is very much appealing. Configured correctly, it might even be used in embedded systems like car ECUs, and it could very well provide a scalable architecture of a car OS of the future. DDS conceptually sees a local store of data called the “global data space.” To the application, the global data space looks like local memory, accessed via an API; You write what looks like your local storage. In reality, DDS sends messages to update the appropriate stores on remote nodes. You read from what looks like a local store. Altogether, the local stores give applications the illusion of having access to the entire global data space.</p><blockquote>The global data space shares data between embedded, mobile, and cloud applications across any transport, regardless of language or system,</blockquote><p><em>sounds lovely right?</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36af39ec4099" width="1" height="1" alt=""><hr><p><a href="https://medium.com/volvo-cars-engineering/the-reality-of-autosar-and-the-way-forward-36af39ec4099">The reality of AUTOSAR and the way forward</a> was originally published in <a href="https://medium.com/volvo-cars-engineering">Volvo Cars Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Darth Unix]]></title>
            <link>https://medium.com/volvo-cars-engineering/darth-unix-7ba7538d75e7?source=rss-8ca435c3cd7e------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ba7538d75e7</guid>
            <category><![CDATA[software]]></category>
            <category><![CDATA[unix]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[simplicity]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Johannes Foufas]]></dc:creator>
            <pubDate>Wed, 14 Dec 2022 15:14:20 GMT</pubDate>
            <atom:updated>2022-12-14T15:14:20.852Z</atom:updated>
            <content:encoded><![CDATA[<p>When we formed our teams, we looked for principles that resonated with our ideas of simplicity, modularity and continual testing of ideas by prototyping. We found that simplicity is crucial because complexity breeds chaos and disaster, especially when things go wrong, as they often do.</p><p>We found inspiration in the east, where we read about inventors that made products with incredible robustness while under the constraints of lacking financial resources. First, they got the basics working. Next, they dragged their inventions through dirt and mud and dropped them from heights.</p><p>Now the team stumbled upon similar principles, the Unix philosophy, which came from Bell labs in the late seventies.</p><p>From the Bell system Technical Journal, July-August 1978, one can read about the <strong>Guiding Principles,</strong> and several maxims have gained currency among the builders of the Unix system to explain and promote its characteristic style.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*nr2-PhDsI7qrHcvzop8ldw.png" /><figcaption><a href="https://emulator.pdp-11.org.ru/misc/1978.07_-_Bell_System_Technical_Journal.pdf">https://emulator.pdp-11.org.ru/misc/1978.07_-_Bell_System_Technical_Journal.pdf</a></figcaption></figure><pre>1. Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new &quot;features&quot;.</pre><pre>2. Expect the output of every program to become the input to another, as yet unknown, program. Don&#39;t clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don&#39;t insist on interactive input.</pre><pre>3. Design and build software, even operating systems, to be tried early, ideally within weeks. Don&#39;t hesitate to throw away clumsy parts and rebuild them.</pre><pre>4. Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you&#39;ve finished using them.</pre><p>Of course, we fail to follow these principles, but we try, and they act as our inspiration and aim.</p><p>Then of course we stumbled upon the legendary book ‘<a href="http://www.catb.org/~esr/writings/taoup/html/index.html">The Art of Unix Programming’ by Eric Steven Raymond</a>.</p><p>Although Eric condensed the Unix philosophy as <a href="https://en.wikipedia.org/wiki/KISS_Principle">KISS Principle</a> of</p><blockquote>“Keep it Simple Stupid”</blockquote><p>from his book <a href="http://www.catb.org/~esr/writings/taoup/html/ch01s06.html">we extracted 17 rules</a>,</p><pre>Rule of Modularity: Write simple parts connected by clean interfaces.<br>Rule of Clarity: Clarity is better than cleverness.<br>Rule of Composition: Design programs to be connected to other programs.<br>Rule of Separation: Separate policy from mechanism; separate interfaces from engines.<br>Rule of Simplicity: Design for simplicity; add complexity only where you must.<br>Rule of Parsimony: Write a big program only when it is clear by demonstration that nothing else will do.<br>Rule of Transparency: Design for visibility to make inspection and debugging easier.<br>Rule of Robustness: Robustness is the child of transparency and simplicity.<br>Rule of Representation: Fold knowledge into data so program logic can be stupid and robust.<br>Rule of Least Surprise: In interface design, always do the least surprising thing.<br>Rule of Silence: When a program has nothing surprising to say, it should say nothing.<br>Rule of Repair: When you must fail, fail noisily and as soon as possible.<br>Rule of Economy: Programmer time is expensive; conserve it in preference to machine time.<br>Rule of Generation: Avoid hand-hacking; write programs to write programs when you can.<br>Rule of Optimization: Prototype before polishing. Get it working before you optimize it.<br>Rule of Diversity: Distrust all claims for “one true way”.<br>Rule of Extensibility: Design for the future because it will be here sooner than you think.</pre><p>In our teams, there is a lot of python code written, and from scanning through our code base, I can see that there are traces of both rule of modularity, composition, separation, simplicity and extensibility. What comes in the way are deadlines and, sometimes, difficult missions. Still I see that small modules are reused and built in new combinations to match the evolution of our build systems. Probably a bit later than the rest of the industry, we found that many of the things our teams wrote were to be found in build systems like <a href="https://bazel.build/">Bazel</a>, so now we are working to use less python and more Bazel. Naturally, python and Bazel blends well.</p><h3>Code Generation with Matlab Simulink</h3><p>We also applied many of the Unix principles to the guidelines of our function developers, the engineers who use Matlab and Simulink to draw control diagrams that in turn generate C code. We often hear from programmers who come from other industries, and especially those who love C++, that this method is inferior, but having used both, I can clearly see the benefits of <a href="https://en.wikipedia.org/wiki/Simulink">Simulink code generation</a>. For large control systems, which are used in vehicles and which can prompt vehicles to exhibit unwanted behavior, software also needs to be well-documented, and in such a task this method is hard to beat. <br> One reason is that the code generator limits possible constructs. Another is that we only allow certain safe libraries. And finally, we have hundreds of check scripts of the design patterns in Simulink, as well as checks of the generated code. A typical Simulink <a href="https://zuul-ci.org/">Zuul</a> check pipeline:</p><pre>- project:<br>   name: repo<br>   check:<br>    jobs:<br>     - buildavoidance-nodeless<br>     - common-gcc_check<br>     - common-pybuild_diff<br>     - common-signal_consistency<br>     - common-cppcheck<br>     - common-checkscript<br>     - common-unittests_shared<br>     - ProjectA-unittests<br>     - common-simdiff<br>     - common-mxray<br>     - common-mxam<br>     - common-mxam_safety<br>     - common-ci_of_ci<br>     - Polyspace code prover</pre><p>We also use the tool <a href="https://model-engineers.com/en/quality-tools/mxray/">MXRAY from Model Engineering</a> on Simulink models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VPhbpbFzjJvsGeGjExCh5g.png" /><figcaption>The MXRAY complexity report.</figcaption></figure><p>This tool uses three primary metrics, McCabe Cyclomatic Complexity, Halstead Complexity, and Incoherence. The primary number is the weighted Halstead volume that fits very well with the design of Simulink. We made a subjective judgment of 500 Simulink models, where we graded their complexity from 1–10. After we scanned the same models with the tool, we got a result very close to our own judgment.</p><p>For some tasks, it is better to write the code by using a keyboard, but this may easily be integrated with the Simulink models. In our Software Development kits, which we have put together, we stress that developers who generate code are also responsible for the code. That’s one of the reasons we let these developers push the code to Gerrit and also write unit tests that verifies the compiled code, and not pressing play in a Simulink simulation.</p><p>So, the aspects from the Unix philosophy that we stress for Simulink design are:</p><pre><strong>Rule of Modularity: </strong>‘Write simple parts connected by clean interfaces’<br><strong>Rule of Simplicity: Design for simplicity;</strong> add complexity only where you must.<br><strong>Rule of Transparency: Design for visibility to make inspection and debugging easier<br>Rule of Robustness: Robustness is the child of transparency and simplicity.<br>Rule of Least Surprise: Pay attention to your expected audience.<br>Rule of Optimization: Prototype before polishing.</strong> Get it working before you optimize it.<br><strong>Rule of Extensibility: Design for the future, because it will be here sooner than you think.</strong></pre><p>We also drew parallels to <a href="http://www.bonsaiempire.com/basics">the cultivation of bonsais</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*d3x29s0HeZ2y5q336VhowA.png" /><figcaption>Keeping it all tidy</figcaption></figure><p><strong>Grow and cultivate trees</strong>. First step is to acquire a tree, which can be done by buying a prebonsai (rough material to be pruned and wired) or by using one of several possible cultivation techniques. Very important, however, is to select a tree species that fits your circumstances.</p><p><strong>Train and style techniques.</strong> Let’s begin with the single most important technique to Bonsai; pruning. Pruning is crucial in keeping trees miniaturized as well as to shape them. The goal is to create a Bonsai that resembles nature as close as possible. Remove branches with unnatural twists and turns. Remove disproportionately thick branches from the top of the tree</p><p><strong>Care and maintenance.</strong> A crucial part of information about how to grow a Bonsai tree is its maintenance and care.</p><p>Converted to Simulink models…</p><p><strong>Get a tree.</strong> Think about the best flow and use of subsystems before you implement!</p><p><strong>Pruning.</strong> As the function grows, use subsystems to keep a reasonable size. Create a flow with as few branches and signal lines as possible. Place subsystems so that they feed each other. Small decisions could be kept in subsystems. Try to contain signaling to avoid spaghetti.</p><p><strong>Care and maintenance.</strong> If the system grows, change of subsystem order might be needed. It could also be of benefit to structure different kind of logic in different subsystems. A good practice is to let each subsystem have a single responsibility; it should do one thing. This in turns leads to good cohesion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*MSMklWPJwt8biGVHe00lvw.png" /><figcaption>Control flow</figcaption></figure><h3>Code written by a keyboard</h3><p>When it comes to code written by a keyboard, we do not have any good methods for analyzing and gating code complexity in our CI system Zuul. The only measurement we have now is Cyclomatic complexity, and that more or less just tells us how hard it will be to test. Although, we do have something in the pipeline, and that is from my colleague and comrade <a href="https://www.researchgate.net/profile/Vard-Antinyan/publication/339377604_Evaluating_Essential_and_Accidental_Code_Complexity_Triggers_by_Practitioners%27_Perception/links/5f983739458515b7cfa1bca5/Evaluating-Essential-and-Accidental-Code-Complexity-Triggers-by-Practitioners-Perception.pdf?origin=publication_detail">Vard Antinyan</a>, who has the researched the subject in great detail.</p><p>As Eric Steven Raymond states,</p><blockquote><strong>you have to be loyal to and pursuit excellence.</strong></blockquote><p>You have to come to the conclusion that software design is a craft worth all the intelligence, creativity, and passion you can muster. Otherwise, you will be tricked into the easy path, stereotyped ways of approaching design and implementation; you’ll rush into coding when you should be thinking. Especially if you work in an Agile setting, you might just run in your sprint wheel, and you might carelessly complicate when you should be</p><blockquote><strong>relentlessly simplifying</strong></blockquote><p>and then you’ll wonder why your code bloats and debugging is so hard.</p><blockquote><strong>If someone has already solved a problem once, don’t let pride or politics suck you into solving it a second time rather than re-using.</strong></blockquote><p>If any of my colleagues have something better, I will steal it with pride. And of course, we want to automate everything we can, it will save time in the long run.</p><p>Programming should be a joyous art, something we appreciate and are passionate about. If we lack this, then maybe we should do something else?</p><blockquote><strong>You need to care. You need to play. You need to be willing to explore.</strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ba7538d75e7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/volvo-cars-engineering/darth-unix-7ba7538d75e7">Darth Unix</a> was originally published in <a href="https://medium.com/volvo-cars-engineering">Volvo Cars Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Volvo thinks you should have Rust in your car]]></title>
            <link>https://medium.com/volvo-cars-engineering/why-volvo-thinks-you-should-have-rust-in-your-car-4320bd639e09?source=rss-8ca435c3cd7e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4320bd639e09</guid>
            <category><![CDATA[software]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[rustlang]]></category>
            <dc:creator><![CDATA[Johannes Foufas]]></dc:creator>
            <pubDate>Fri, 23 Sep 2022 08:01:53 GMT</pubDate>
            <atom:updated>2022-10-27T11:57:35.180Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why Rust is actually good for your car.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lf61QbASBkew69V-jBRLeg.jpeg" /></figure><p>This post is an interview made with one of our embedded Rust Pioneers, Julius Gustavsson who is a Technical Expert and System Architect at Volvo Cars Corporation.</p><p>JF = Johannes Foufas (interviewer), JG = Julius Gustavsson (interviewee).</p><p>JF: How did you discover Rust?</p><p>JG: I think this was back in 2014 and by then I had been doing C and C++ for 15 years. I was working in a new team at a large Swedish tech company. They had quite an advanced code style, and they were proud of their code base. It looked really nice. You know, I was excited to start working there but then, lo and behold, I was debugging the same kind of obscure memory issues as before.</p><blockquote>A thought just struck me, is this as good as it gets?</blockquote><blockquote>Is this how my career is going to be?</blockquote><blockquote>Am I going to spend the rest of my life doing this?</blockquote><p>And interestingly enough, I think it was the same week or the week after that, I saw this Reddit post where they were closing in on the final Rust 1.0 release. And apparently, it still was living up to its original claims, which I found when I first discovered Rust somewhere around 2009.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*Z0g4j3QYapjaMIxni6mgKg.jpeg" /><figcaption>Julius to the left, Johannes to the right</figcaption></figure><p>From then on, I followed the Rust project from the sidelines. When I came to Volvo Cars a few years later, I was already sold on it, and I thought this would be useful for Volvo Cars because it embodies the same type of Ideology that you want when you’re developing safety-critical software. You really want to have quality up front.</p><p>JF: That’s what we struggled with the most, the obscure and difficult-to-find errors. They don’t happen that often, but when they happen it’s a pain.</p><p>JG: Yeah, definitely.</p><p>JF: Did you code Rust on the side then, privately or?</p><p>JG: Nothing major. But when I started at Volvo, my first project was to make an Android integration with our Signal Broker, what is now the RemotiveBroker, on our prototype version of the <a href="https://www.youtube.com/watch?v=8WyV487QG9Q">Core Computer</a>. It was a kind of Hardware Abstraction Layer (HAL) towards the broker, itself written in Elixir. I did that using Rust and async Futures. That was also a big confirmation that this was indeed something useful. Everything just worked from the start. I mean, once you get it to build it usually always works when you run it. Of course, it does not solve any logic errors in your implementation. But, if your logic checks out, it just magically works as soon as it compiles. But then you might of course struggle sometimes to get it to compile, especially if you are trying to do things that the compiler sees as erroneous.</p><p>JF: From there you went more serious?</p><p>JG: Then I pitched to the managers, “If we want to do a serious Rust effort within the company, then I want to take part in that and become an employee”, since I was a consultant at that point.</p><p>So that’s how my employment started at Volvo and that’s when I met Niko (Nikolaos Korkakakis) who had the same ambition. And so, we teamed up and started to work on the Low Power node of the core computer. That was basically a fluke that we had this node that no one was really paying attention to. Everyone was busy with the other nodes. And it happened to run on the architecture that was best supported in the embedded bare metal space of Rust at the time. It was also not safety critical component, so we did not have to worry about safety certifications.<br> <br> JF: Yeah, that’s always a hassle.</p><p>JG: We didn’t have to worry about the safety-critical stuff.</p><blockquote>But at the same time, it must be extremely reliable, because if it doesn’t work, the car will not start.</blockquote><p>Also, since the functionality scope was limited, we could be a small team for the first project.</p><p>JF: Was anything lacking for you?<br>Did you need anything from the Open Source community or did you develop it all by yourselves?</p><p>JG: No, we did not have everything. Some parts were missing. We run on an <a href="https://www.microchip.com/en-us/product/ATSAME51N20A">ATMEL/Microchip ATSAME chip</a>, and there is an Open Source project for it.<br>In Rust, you have this hardware abstraction layer, similar to an MCAL in Autosar.</p><p>So that project had been recently started, but it only had a few of the peripherals, and many of the peripherals that we needed were not yet in place. This was the case for drivers like <a href="https://en.wikipedia.org/wiki/CAN_bus">CAN</a> that we use a lot in the car industry. Because of this, we have been active in that project.</p><p>JF: Did you team up with anyone then?</p><p>JG: Yes, we meet <a href="https://www.grepit.se/">Grep</a>it at a Rust conference. Grepit was founded in 2014 as a spin-off from Luleå University of Technology. They are the authors of: <a href="https://crates.io/crates/cortex-m-rtic">https://crates.io/crates/cortex-m-rtic</a>, and it’s a concurrency framework for building real-time systems.</p><p>With it, you can achieve real-time behaviour in a system, but it&#39;s only that. It does not provide you with any higher-level abstractions or things like services that many other RTOS would typically give you. But you get that instead from many of the open source components that are available. So yes, we teamed up with Grepit to get us up and running. To develop drivers and push them upstream to the projects. There have also been some implementations lacking that we had to develop from scratch, but that has not hindered us much. There are quite a lot of neat tools out there that you can use.<br> <br> JF: Now you have experience from the C++ world. What benefits do you see straight off? I mean now, when you have something working in Rust?</p><p>JG: I would say the benefits I saw from the start were that you didn’t have to think about race conditions and memory corruption, and memory safety in general. You know, just writing correct and robust code from the start. So that was basically my first impression, but now I have also come to realize that there are many other aspects. You get just as big benefits from the side effects of that first aspect. First let’s back up and describe briefly how memory safety is achieved in Rust, which is quite unique. It is based on static analysis of the lifetime of data within the program and making sure that any references to that data never outlives the lifetime of the data itself. You also never allow for more than one mutable instance at any given time or you can have multiple read instances to the data, but you may never mix those two. By enforcing this statically at compile time you get the memory safety for free, because then you know when the lifetime ends, and the compiler will then inject clean up code at that location.</p><p>JF: Unless you use the unsafe keyword?</p><p>JG: Well, the unsafe parts give you some extra flexibility to shoot you in the foot. But still it’s not like all bets are off. There are still a lot of invariants that are upheld, even in unsafe. For example, the lifetime stuff is still being upheld. The difference is that you are allowed to use raw pointers. They basically erase all lifetime information or you may bypass things that the compiler would not allow you to do. This also includes things that the compiler can’t reason about on its own but instead must leave to the human and that’s always done in a clearly marked unsafe block, so that you can audit it specifically.</p><p>But it’s not like, as I think many assume, just because it’s unsafe you’re back to C. That’s not the case. You have a lot of safety measures, even though they are more relaxed compared to the safe subset. But the nice thing about having this memory model, lifetime and ownership model enforced by the compiler means that everyone is on the same page, which in turn means that you can import and use third party components in a much easier and straightforward way. Since Rust comes with a built-in tool chain that takes care of building and that fetches and resolves dependencies, it is also much easier and safer to add new dependencies.</p><p>You no longer have to check:</p><blockquote>Does it build?</blockquote><blockquote>Is it my build system that makes it fail?</blockquote><p>You do not have to make changes to your build system to make it build and link. And if I get it that far,</p><blockquote>does the library have the same assumptions about memory and ownership as I do?<br> Who’s going to free this memory?<br> If I must create a buffer, who’s going to delete it, all that stuff!</blockquote><p>By not having to worry about that, you can feel much more at ease when you use third party components.</p><p>JF: So there is a rule for how that is done?</p><p>JG: Yes.</p><p>JF: How about packages? Do you just take anything, or do you have reasoning for what you use?</p><p>JG: We try to keep our dependencies to a minimum, because we are creating products that will live for a long time. So, all the dependencies that we pull in is something that we deem maintainable by ourselves in case we need to make bug fixes or whatever. There’s a built-in plugin for Cargo that does auditing of the code and checks it towards a database and reports any vulnerabilities or any other issues that may turn up during the lifetime of the product.</p><p>One other benefit that I did not realize in the beginning is onboarding new people is much easier because the new person is free to play with the with the code base, to try to improve it, change or refactor it, and the compiler is not going to compile until all the invariants are upheld again. This means that you can refactor without any fear and that new people can start writing code without being minutely reviewed, because you know there are so many unwritten invariants that only a few people know about. And I’m quite confident that this will also result in fewer warranty issues over time because you get higher quality up front. <br> <br>JF: Yes, you do not need to run so many tools on the product to know if it’s safe or not, we run a whole array of tools on C and C++ code to try to find these hard to find errors.</p><p>JG: Yes, you do not need that to the same extent. There are some runtime characteristics and things like that that, that you may need to check. It also depends on how formally verified you want the code to be. Do you want to statically ensure that these things will never happen?</p><p>There are some runtime behaviors that you always will need to have some other tool to help you with.</p><p>JS: Are there tools like this available?</p><p>JG: They are becoming more and more available. I’m not sure if there are any like out of the box that can do everything we need, but we are experimenting with a few. For example, <a href="https://github.com/rust-lang/miri">Miri</a>, which is basically running the code in a virtual machine during compile time to figure out any unsoundness in your code base.</p><p>JF: Is there not also another tool made by some university people?</p><p>JG: Yes, there is for example <a href="https://klee.github.io/">KLEE</a>, a dynamic symbolic execution engine built on top of the LLVM compiler infrastructure. It has been adapted towards Rust. This tool can give you definitive answers on whether something can panic in a certain situation or not. We haven’t started using it ourselves, but it’s something we want to explore. There is also the <a href="https://model-checking.github.io/kani/">Kani verifier</a> that looks promising.</p><p>JF: How about profilers, does that come with the language or?</p><p>JG: Unfortunately, not on embedded targets, or at least not yet. So in a recent release of Rust, there’s an instrumented code coverage flag that you can give to the compiler, so that you get extra instructions. This will afterwards show you what code paths you have actually executed. We need to do some work there to apply this on an embedded target. I know it works for desktop because there it can generate those files on the fly. But of course, on an embedded target you don’t have file systems or files, so then you need to take care of that yourself writing it to some internal buffer. For profiling you can use the standard desktop tools, but I am not sure how well that translates to the target. When we are testing we try to isolate all logic that is hardware independent to their own crates. Because then we can run them and use the built in tests support that the language provides. We can run the regular test infrastructure on those, but then when we do integration testing on actual target hardware, then we use integration test tools.</p><p>JF: So you can rely a lot on X86?</p><p>JG: Yes, we build the code for host and test it there. And that is part of strength of Rust. It’s fairly seamless across platform support. It’s just a part of the tool chain, which is a completely different experience compared to C or C++.</p><p>JF: How is testing them?<br> I mean, are there any test frameworks supplied with Rust?</p><p>JG: There is a build in unit test framework. Any function in the code can be run as a test just by annotating it with a specific tag. You can intersperse tests within regular code, and when you build for tests, those are run. The threshold to do unit testing is almost nonexistent. It’s built-in, which is quite amazing. You can easily write benchmark tests to see how your function improves. The same test framework can be used also to write integration tests, which are easy to run on your desktop, but when you’re doing it on target, it is not as seamless. But there’s this new Rust project called <a href="https://probe.rs/">Probe-rs</a> that allows us to communicate with the target hardware in in a fairly seamless way. It offers a <a href="https://en.wikipedia.org/wiki/GNU_Debugger">GDB </a>type of interface as a library, so you can write your test application for target. For your host it just loads probe-rs as a library and then you can interact and send GDB like commands or doing low level hardware manipulation over the wire, as part of the test.<br> <br> JF: What do you foresee for the future?</p><p>JG: We have quite ambitious plans going forward.</p><blockquote>We want to expand Rust here at Volvo Cars</blockquote><p>to enable it on more nodes and to do that, we need to get compiler support for certain hardware targets and OS support for other targets. There is no point in replacing already developed and well tested code, but code developed from scratch should definitely be developed in Rust, if at all feasible. That is not to say that Rust is a panacea. Rust has some rough edges still and it requires you to make certain trade-offs that may not always be the best course of action. But overall, I think Rust has huge potential to allow us to produce higher quality code up front at a lower cost which in turn would reduce our warranty costs, so it’s a win-win for the bottom line.</p><p>JF: But Rust could co-exist with a code base based on C?</p><p>JG: It can co-exist on an almost arbitrary granularity, on a module level or on a function level, depending on what you’re doing. You could for instance rewrite parts that need cyber security, parts that are vulnerable. There is zero overhead between C and Rust, C can call into Rust and vice versa. Even for C++ there are ways, although you have to go through the C interface, there are some nice crates that can help you generate extra boilerplate so that C++ and Rust can seamlessly communicate.</p><blockquote>There is also a lot of uptake in the safety critical industry, and we try to support these efforts as much as possible.</blockquote><p>For example, there’s this<a href="https://ferrous-systems.com/ferrocene/"> Ferrocene project</a> that is aiming to ASIL D certify the compiler. Ferrocene will be an ISO26262 qualified version of the existing open-source compiler, rustc.</p><p>AUTOSAR and SAE have as well started a Rust working group.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4320bd639e09" width="1" height="1" alt=""><hr><p><a href="https://medium.com/volvo-cars-engineering/why-volvo-thinks-you-should-have-rust-in-your-car-4320bd639e09">Why Volvo thinks you should have Rust in your car</a> was originally published in <a href="https://medium.com/volvo-cars-engineering">Volvo Cars Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simple but not easy software development]]></title>
            <link>https://medium.com/volvo-cars-engineering/simple-but-not-easy-software-development-f90937b50b0?source=rss-8ca435c3cd7e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f90937b50b0</guid>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Johannes Foufas]]></dc:creator>
            <pubDate>Fri, 17 Jun 2022 10:02:00 GMT</pubDate>
            <atom:updated>2022-10-28T13:49:28.020Z</atom:updated>
            <content:encoded><![CDATA[<p>This post is a tribute to, in my view, one of the best YouTube clips about software development, Mike Acton’s talk: “CppCon 2014: Mike Acton “Data-Oriented Design and C++”. There are so many aspects of his talk that me and my colleagues resonate with.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FrX0ItVEVjHc%3Fstart%3D194%26feature%3Doembed%26start%3D194&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DrX0ItVEVjHc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FrX0ItVEVjHc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/49b9a6a907bdad3b577415a0868e8c4d/href">https://medium.com/media/49b9a6a907bdad3b577415a0868e8c4d/href</a></iframe><blockquote><em>In order to solve new problems, we have to make space for them…performance is king. [06:33].</em></blockquote><p>Well, the same goes for an embedded system, and especially for an <a href="https://www.youtube.com/watch?v=8WyV487QG9Q">autonomous core computer</a>. Here we have a lot of high-resolution sensors like radars and cameras, and developers need to optimize the code for throughput, latency, correctness and determinism. So, we need to write fast, memory-efficient low-level code to capture high-frequency, high-volume data from our sensors, and to share it with multiple consumer processes — without impacting central memory access latency or starving critical functional code from CPU cycles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*Wk4u3iLxpcI9idHoPtj76A.png" /><figcaption>the future sensor setup</figcaption></figure><p>From the lowest levels of the stack, the software needs to be tightly integrated with the hardware. Mike points out that this is one of the three lies of the Software industry:</p><blockquote>1. Software is a platform</blockquote><blockquote>2. Code designed around a model of the world</blockquote><blockquote>3. Code is more important than data</blockquote><p>As Mike points out about the first lie, obviously hardware is the platform, and with different hardware we will have different solutions to our problem. We can never forget this, abstractions too far away from the hardware will not work for performance critical features.<br>Mike also states:</p><blockquote><em>If you don’t understand the cost of solving the problem, you don’t understand the problem</em> and i<em>f you don’t understand the hardware, you can’t reason about the cost of solving the problem</em>.</blockquote><p>One day in the shower, I tried to figure out a key performance index, that would tell the cost of communicating in the stack versus performing, what I call, the end customer related functions. I called this middleware overhead. Of course, this metric works best when one compares the same or a similar code base, and it will also look lower and better if one implements some inappropriate CPU load consuming code in the module. Automotive engineers might react on the <a href="https://en.wikipedia.org/wiki/AUTOSAR">Autosar </a>case, and that is in my view a good example of a standard that conforms heavily to the three lies. Adding software layers and abstracting away the hardware has a cost. Just to mention, the optimized hack had just three layers, the code, a hardware abstraction, and an Ethernet sniffer and dumper.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*JkjyLl6DAWqTcnp1Pkymww.png" /><figcaption>Middleware overhead, either the time (or the CPU) load it takes to communicate with the world compared to execution of the code that does the magic</figcaption></figure><p>One of the most striking sentences in the video clip is:</p><blockquote><em>Solving problems you probably don’t have creates more problems you definitely do.</em></blockquote><p>This, I would argue, is one of the most dangerous problems. Sometimes software architects get carried away by a beautiful vision, that does not solve any real engineering problem. The real engineering problem comes from analyzing data and maybe with a task such as avoiding a collision, and if the problem is imaginary, it goes out of the window. Mike makes the rather fun remark:</p><blockquote><em>Software does not run in a fairy aether powered by the fevered dreams of CS PhDs</em>.</blockquote><p>I think one aspect that I have seen, that relates to this, is the amounts of levels of abstraction. Ok, I am not a complete fundamentalist, we could have use of one or two layers on certain platforms, but maybe not twelve. That will cost CPU cycles, and we need to reason about what the cost of all these are. When it comes to large scale sensor data, there are not many options, we need to make use of shared memory mechanisms, and then we might get away with some layers.</p><p>Anyway, sometimes when we analyze the data and discuss the solution, the simplest solution will just not solve the problem well enough. Consider a control theory problem, where we want to control one of the most critical aspects of our product. In the past, a simple <a href="https://en.wikipedia.org/wiki/PID_controller">PID control algorithm</a> might be the only thing what we could fit in our micro controller, but today, there is a wide range of control systems, that would fit an embedded micro controller or a larger core computer. So maybe we actually need a physical model of parts of our product and maybe we need to invest in something like a <a href="https://en.wikipedia.org/wiki/Model_predictive_control">Model Based Prediction control</a> algorithm where we control not the sensor data but customer related cost functions, if the performance improvement may justify the CPU load and memory consumption. To make this call, we need to understand our data and our problem, and we need to weigh in aspects like robustness. If we are over engineering a problem, like controlling a simple aspect of our product with a large-scale neural network, that consumes a major part of our modules CPU load, then we might be doing it wrong.</p><p>In the end of Mike Acton’s video, he shows some nice simple examples of how to reason about data and its performance. In one of the examples (49:38) he looks at the information density over time of a is_spawn value by printing it out and then zipping it. Quick, dirty and efficient, but in the simple calculation there is still a quite accurate assumption of the percentage of waste.</p><p>On a bit similar tangent some of my friends use the free tool <a href="https://godbolt.org/">https://godbolt.org/</a> where one can choose compiler and target, input C/C++, Go, <a href="https://www.rust-lang.org/">Rust</a> and D code and check the assembler output. This way, one can analyze snippets of code and compare a rough estimate of instructions, and assess what is a good and what is a bad idea.</p><p>On a higher level, I thought of mentioning other ways we look at our code. The most basic one, is to instrument a type of time consumption for all the major calls, that is lightweight in a sense that it does not add a lot of overhead calculation time. Looking at these and comparing them to the actual value of adding code execution time will give us a rough hint of where we are heading.</p><p>Then of course, for more brutal instrumentation, I do like the open source tool <a href="http://kcachegrind.sourceforge.net/html/Home.html">http://kcachegrind.sourceforge.net/html/Home.html</a>. Although some people argue that it’s not as accurate as pure timing measurements, I think it is accurate enough for its purpose, to let us know of most of the task performances of our module. It does add a lot of overhead, so it’s maybe not for evaluating real production performance.</p><p>The user interface is quite nice too, it is intuitive to look at the Callee map view and the call graph. At times, we have used this tool both on target hardware and on servers running x86 images in the cloud. The cloud jobs run on different hardware, but still give a quick and dirty relative assumption of what that might have gone wrong. Comparing x86 runs with appropriate target runs is interesting, because although the processes to some extents are different, the major problems are as far as I have seen visible in both.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*4WkKTpNyVqnPziMOszIdYQ.png" /><figcaption>Calle map</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f90937b50b0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/volvo-cars-engineering/simple-but-not-easy-software-development-f90937b50b0">Simple but not easy software development</a> was originally published in <a href="https://medium.com/volvo-cars-engineering">Volvo Cars Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OpenInfra Summit Berlin 2022]]></title>
            <link>https://medium.com/volvo-cars-engineering/openinfra-summit-berlin-2022-3c2e52dbbf22?source=rss-8ca435c3cd7e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c2e52dbbf22</guid>
            <category><![CDATA[openinfra]]></category>
            <category><![CDATA[ci]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[zuul]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Johannes Foufas]]></dc:creator>
            <pubDate>Mon, 13 Jun 2022 04:32:34 GMT</pubDate>
            <atom:updated>2022-10-25T06:43:03.405Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_tQZyjFKNHs1o9Cr53XHqw.jpeg" /><figcaption>Johannes Foufas and James Blair (the founder of Zuul) keynote at the <a href="https://www.linkedin.com/feed/hashtag/?keywords=openinfrasummit&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A6940384351725469696">#OpenInfraSummit</a>, Berlin</figcaption></figure><p><a href="https://zuul-ci.org/">Zuul</a> is an Open Source CI system originally developed in the Open Stack community, now called <a href="https://openinfra.dev/">Open Infra</a>. Here at Volvo Car Corporation (VCC) Zuul is our default CI/CD chain at Research and Development, and we use it because of this mantra:</p><blockquote><strong>Stop merging broken code!</strong></blockquote><p>Our system is built in gating, and mainly uses the check, gate and release pipelines when the gate merges to master. The good thing is that this is not just for a single change in one repository, but for any set of dependent changes. This is especially great for us having many nodes in the car that depend on each other — each developed by a different team.</p><p>Because of this, Zuul tests the upcoming future state of the product, but only let changes in that pass tests. To my knowledge, Zuul is the only CI system that can have dependencies between changes in Gerrit, Github and Gitlab, and carry these changes through all the way to git merges in the respective repositories.</p><h3>Speculative Merge</h3><p>To be able to test each change in isolation before a merge, Zuul uses <a href="https://zuul-ci.org/media/simulation.webm">speculative merge</a>. Speculative merge saves a lot of time by testing in parallel. The only limit is the build capacity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l5o1Bi8WWCXuS0FaDrHfJw.jpeg" /><figcaption>testing the future state</figcaption></figure><h3>Input Scalability</h3><p><a href="https://zuul.opendev.org/t/openstack/status">Zuul is still the CI</a> for one of the largest Open Source projects on the planet. Although our Zuul usage is not as big, it’s still quite a lot with around 4000 builds each 24 hours.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*q8CEnfLrLQllF4_YtjcdRA.jpeg" /><figcaption>Random comparison of build nodes and executor ques between OpenDev and Volvo Cars Corporation</figcaption></figure><h3>Community</h3><p>And last but not least, Zuul is an active open source developed tool with a fantastic community, it gives developer the chance to participate, if they so like.</p><p>For the second year, I have the honor to speak at the Open Infra Summit, and here is some of my experiences from the conference.</p><p>The last summit was virtual. But this time around traveling to Berlin was fantastic. I have always appreciated east Berlin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w0jV29MpN3N-qVVdpQtHqA.jpeg" /><figcaption>building just next to the conference center</figcaption></figure><p>This year Zuul celebrates 10 years, so this was a special occasion. The keynotes of day 1 were quite interesting, especially <a href="https://youtu.be/ZlPLGmBfaVc?t=1335">Mark Collier´s presentation.</a> Mark is the CEO of the OpenInfra Foundation and he really showed what is the potential of the cloud market and Open Source Software. Some of the greatest software stacks in the world have come into existence through different companies collaborating. Open stack is a good example with over 8700 developers contributing from 450 companies. Mark predicts that over 50 billion devices will be connected by 2030 and that there are interesting areas of development, like mobile tele medicine, where specialists can perform surgery from a distance and of course, and self driving cars. These problems can not be solved by isolated companies alone, so collaboration is necessary.</p><p>A talk that really stood out, was <em>Building Environmental Dashboards for OpenStack at BBC R&amp;D. </em>They build Grafana dashboards that let users of their private OpenStack cloud track the carbon dioxide emissions resulting from each of their VMs. Combining real-time data from electricity producers with metrics from the compute hosts, they gave users a tool to monitor, and eventually to reduce, the environmental impact of their various workloads.</p><p><a href="https://youtu.be/aWA8HFxFpos">There were many interesting talks about Zuul</a>, from its <a href="https://youtu.be/0-AVWf1BMwU">advanced features</a> to a discussion among some of the developers in how to develop the security aspects to the system. For me, the talk from a large German car brand, <a href="https://youtu.be/4FCNagEYS9s?list=PLKqaoAnDyfgqUDjoZhZnDKF5aJrLMnm9E"><em>No gain without pain</em></a>, was very interesting. As the title indicates, they struggled too with optimizing all the components of the system to match scale. They actually run up to 6000 jobs in one hour!</p><p>They made a company direction decision early on to use Zuul, where we went from shadow IT to be default.</p><p>Then of course, J<a href="https://youtu.be/Jpu7SPLvjjE?t=7730">ames Blair and I gave a keynote about our growth and usage of Zuul here at Volvo Cars</a>.</p><p>James Blair, the creator of Zuul and we collaborate with his company <a href="https://acmegating.com/">ACME Gating</a>. James built an enterprise version of Zuul for us, where we sometimes are ahead of the upstream development, and sometimes after. Of course, all work is pushed up streams to the Open Source project.</p><p>In our keynote we spoke about how we went from shadow IT to become the default CI chain for the major part of the code that enter the core computer.</p><p>The cooperation with James has been a game changer, and we got many features developed for us: improved Azure drivers, semi-static nodes and enterprise wide semaphores. Our CI software stack contains of these technologies:</p><blockquote>Zuul, Ansible, Gerrit, Artifactory, Grafana, Splunk, AWS, Azure &amp; Kubernetes.</blockquote><p>Since 2021 up until today we grew a lot, we went from around 200 Gerrit repositories up to over 600, and this proved to be a challenge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*tiws3-Ai66E_nReAUhU8YA.jpeg" /><figcaption>Added Gerrit projects that Zuul listens to, during 2021 to the present</figcaption></figure><p>Of course all the new teams needed support, but our back end of Zuul also needed some love.</p><p>Basically, teams that went from small scale Jenkins setups complained about the speed or to be more precise: time it takes from when a developer pushes a change to Gerrit until the first job starts in the pipeline. When it was at its worst, this could for short periods take up to 2o minutes.</p><p>Now, the average is around 1.5 minutes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HKp3IBYarG6VDYExjMFChQ.jpeg" /><figcaption>The time between a Gerrit event to the start of the first job, a normal morning</figcaption></figure><p>Basically, the solutions was: more detailed metrics, better use of Zuul tenants and optimization of the Zuul reconfiguration.</p><p>We also mentioned our current setup of the back end, we use <a href="https://zuul-ci.org/docs/zuul/latest/components.html?highlight=scheduler">6 schedulers</a>, <a href="https://zuul-ci.org/docs/zuul/latest/components.html?highlight=executors">10 executors</a> and 6 web pods all run in an <a href="https://aws.amazon.com/eks/">EKS K8</a> cluster.</p><p>All in all, it was a fantastic experience to talk to all Open Source developers from around the globe, #weareopeninfra.</p><h3>Watch the talk</h3><p>You can watch the whole talk <a href="https://www.youtube.com/watch?v=Jpu7SPLvjjE">here</a>, or just skip to the part where I talk about our setup at VCC:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FJpu7SPLvjjE%3Fstart%3D7730%26feature%3Doembed%26start%3D7730&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DJpu7SPLvjjE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FJpu7SPLvjjE%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee4a7ce4cf7672044b0664e41382c47a/href">https://medium.com/media/ee4a7ce4cf7672044b0664e41382c47a/href</a></iframe><p>The second and more detailed talk:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fzi2A8uyhRlU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dzi2A8uyhRlU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fzi2A8uyhRlU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/28426b6899a661b03a5e3c753c5f4870/href">https://medium.com/media/28426b6899a661b03a5e3c753c5f4870/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c2e52dbbf22" width="1" height="1" alt=""><hr><p><a href="https://medium.com/volvo-cars-engineering/openinfra-summit-berlin-2022-3c2e52dbbf22">OpenInfra Summit Berlin 2022</a> was originally published in <a href="https://medium.com/volvo-cars-engineering">Volvo Cars Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>