<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kapernikov on Medium]]></title>
        <description><![CDATA[Stories by Kapernikov on Medium]]></description>
        <link>https://medium.com/@kapernikov?source=rss-5a452e0b0e84------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*rUgh_8zgmKjGsLPJujgVDQ.png</url>
            <title>Stories by Kapernikov on Medium</title>
            <link>https://medium.com/@kapernikov?source=rss-5a452e0b0e84------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 21:24:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kapernikov/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Reduce embedded systems development time using ROS — Kapernikov]]></title>
            <link>https://medium.com/kapernikov/reduce-embedded-systems-development-time-using-ros-kapernikov-f69ad0254518?source=rss-5a452e0b0e84------2</link>
            <guid isPermaLink="false">https://medium.com/p/f69ad0254518</guid>
            <category><![CDATA[machine-vision]]></category>
            <category><![CDATA[ros2]]></category>
            <category><![CDATA[robot-operating-system]]></category>
            <dc:creator><![CDATA[Kapernikov]]></dc:creator>
            <pubDate>Thu, 08 Dec 2022 09:25:59 GMT</pubDate>
            <atom:updated>2023-01-06T15:40:54.665Z</atom:updated>
            <content:encoded><![CDATA[<h3>Reduce embedded systems development time using ROS — Kapernikov</h3><p>Large systems nowadays are generally divided into many small components, “microservices”, that are easy to isolate, replace or integrate into other solutions. Those components often need to run on different hardware architectures and may be written in different programming languages: C++, C and Python are among the most popular options in the embedded world.</p><p>The question is then, how do you connect those components together? More importantly, how can you avoid losing sleep over the often tedious debugging &amp; optimization of custom messaging infrastructures?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*0Iwbw5K_6CYNzRabXpV27A.png" /></figure><p>In this article we’ll look at one possible solution called ROS (short for Robot Operating System). We believe that ROS provides not only a fast and reliable messaging system but also access to a huge ecosystem of building blocks that will easily interface with your own components.</p><p>ROS has been adopted in the products of different companies in the industry. For instance, <a href="https://roscon.ros.org/2018/presentations/ROSCon2018_Aibo.pdf">Sony embraced the use of ROS in the Aibo project</a>. Another example is the Spanish company <a href="http://wiki.ros.org/RobotnikAutomation">Robotnik</a>, which designs and creates mobile industrial ROS-based robots. <a href="https://fetchrobotics.com/">Fetch Robotics</a>, which offers a range of warehouse robots for automation in manufacturing and warehousing operations, also employs ROS in its products.</p><h3>Principles</h3><p>Despite its name, ROS is not an operating system. At its core, it’s rather a communication backbone along with a lot of higher level tools that are suited for robotics.</p><h3>Communication infrastructure</h3><p>Communication endpoints in ROS are called nodes. Nodes exchange structured data over two types of channels:</p><p><strong>Topics </strong>provide a publish/subscribe pattern where some nodes emit data over a named bus and other nodes listen over the same bus to receive the data. It’s conceptually similar to a radio broadcast where subscribers “tune in” to the right frequency according to their interest, with the difference that multiple nodes can publish over the same topic without signal degradation. Thanks to the use of multicast, adding subscribers to a topic doesn’t generate any additional traffic over the network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/0*QzZ-nTG0hMVHn5zq" /></figure><p>Topics are great for data flow, where for example some component may regularly publish updates on a ship’s position or the progression of some background task to be consumed by other components, like a user interface or a path finding algorithm.</p><p><strong>Services</strong>, on the other hand, are a request/response mechanism where a single node handles requests from one or more clients.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/0*aV11QKjqNZL5mqkC" /></figure><p>This is better suited for control flow: operations that change the state of the system. For example, a user interface may request a system reboot or load a different configuration using the appropriate service.</p><p>In both cases, messages exchanged between nodes are structured according to message definition files which are user defined and domain specific. Both topics &amp; services are created and removed at runtime and can be dynamically discovered using standard ROS requests.</p><h3>Ecosystem &amp; tools</h3><p>On top of this communication infrastructure, ROS provides tools to help build robot applications efficiently. These tools accelerate the progress of software development and include debugging, visualization, and logging functionalities.</p><p>For instance, the software tool <a href="https://github.com/ros2/rviz">Rviz</a> provides support for 3D visualization of data from your robot application. It allows the combination of data from different sensors and the robot’s current state into a combined view. <a href="https://github.com/ros2/rosbag2">Rosbag2</a> also provides an interesting functionality allowing the recording of data from the system. This recorded information includes data from sensors and the robot’s trajectory and can be further used in debugging and data analysis. For an overview of the system, illustrating the ROS nodes and topics, the <a href="https://github.com/ros-visualization/rqt_graph/">Rqt_graph</a> can be used. This tool provides a GUI plugin for visualizing the ROS graph and can be especially useful for large systems with several nodes interacting with each other.</p><h3>Benefits</h3><h3>Robustness</h3><p>Since 2007, ROS has been used everywhere, from robotic arms to underwater vehicles. By default, ROS uses eProsima Fast DDS as a communication backend. That open source implementation has been used in safety critical &amp; high performance scenarios such as self driving cars, planes and battleships. At Kapernikov, we’ve used ROS <a href="https://kapernikov.com/cases/where-exactly-is-my-ship-accurate-dynamic-positioning-of-vessels/">aboard a robotic ship</a> and <a href="https://kapernikov.com/cases/machine-learning-for-optical-sorting-machines/">inside food sorting machines</a>.</p><p>Why reinvent the wheel? Even using ROS as a simple communication backbone considerably reduces development time, allowing you to focus on your core business and making your product stand out from the competition.</p><h3>Large ecosystem</h3><p>One of the benefits of using a standard communication interface is to be able to communicate with 3rd party components. Some hardware manufacturers even support ROS out of the box, making it easy for both experimental or production code to interface with their products.</p><p>For example, Basler provides cameras alongside a ROS driver that directly exposes the raw image captured by the sensor on a ROS topic, allowing any software with a ROS interface to access the image pixels.</p><p>The ROS community has also developed hundreds of algorithms that are presented as ROS components and solve usual problems seen in robotics, such as path finding and geometry transformation.</p><h3>Programming language independence</h3><p>ROS nodes communicate with each other using clearly defined interfaces that are language independent. As a result, it is very common to see systems based on ROS use a combination of Python and C++, and this brings additional benefits at every step of the development process:</p><ul><li>Python can be used to develop integration tests (with a framework such as pytest-bdd) that test your components (no matter in which language they are written) using the exact same interfaces that runtime clients will use.</li><li>Your deployed solution may consist of a heterogeneous set of nodes written in C++ or Python, depending on the type of problem they solve or on which framework they are built.</li><li>It’s future proof! No matter what programming languages will be used in a few years, chances are there will be a binding for ROS that enables interfacing with today’s components. While C++, Python and even Lisp are officially supported, a quick Google search brings client libraries for the usual suspects such as Java, Rust &amp; C#.</li></ul><h3>Architecture flexibility</h3><p>Using ROS means that your components don’t need to know on which computers their communication peers are running. While some early deployment of your product may run two nodes together on the same machine, a need for more processing power or a different hardware architecture later in development can most of the time be addressed by simply moving one node to another system. The code of both components should stay exactly the same. And it will always be easy to replace a node (or multiple nodes) by a new implementation, e.g. to quickly test a new reasoning engine.</p><h3>Conclusion</h3><p>ROS provides a robust, flexible framework for applications, robotics and other. It supports a wide range of programming languages, comes with a ton of modules to connect to existing hardware or provide common functions and allows a flexible application architecture.</p><p><em>Originally published at </em><a href="https://kapernikov.com/reduce-embedded-systems-development-time-using-ros/"><em>https://kapernikov.com</em></a><em> on December 8, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f69ad0254518" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kapernikov/reduce-embedded-systems-development-time-using-ros-kapernikov-f69ad0254518">Reduce embedded systems development time using ROS — Kapernikov</a> was originally published in <a href="https://medium.com/kapernikov">Kapernikov</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Best practices for model training and serving in machine learning — Part 2 — Kapernikov]]></title>
            <link>https://medium.com/kapernikov/best-practices-for-model-training-and-serving-in-machine-learning-part-2-kapernikov-1ac4cf926c65?source=rss-5a452e0b0e84------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ac4cf926c65</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-vision]]></category>
            <dc:creator><![CDATA[Kapernikov]]></dc:creator>
            <pubDate>Thu, 10 Nov 2022 10:43:28 GMT</pubDate>
            <atom:updated>2022-11-10T12:46:43.712Z</atom:updated>
            <content:encoded><![CDATA[<h3>Best practices for model training and serving in machine learning — Part 2 — Kapernikov</h3><p>In our <a href="https://kapernikov.com/best-practices-for-model-training-and-serving-in-machine-learning-part-1/">previous article</a>, we discussed what are the parameters controlling the training process of machine learning algorithms. We explained how to run and log experiments to find the optimal set of these parameters and achieve good performance results for your model. Now we will look at what to do when your network isn’t working as expected and how to convert the final model into a format suitable for deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*NOYFFmSPrNxkSW-xN9OVIw.jpeg" /></figure><h3>Debugging</h3><p>When training a neural network, you may encounter situations where the network does not work as expected and you need to find the source of the problems. This task can be quite challenging given the fact that deep learning models operate like black boxes. In other words, their behavior cannot be fully interpreted by humans. However, there are still some general principles that might help you debug it:</p><h4>Data inspection</h4><p>The first thing to do is inspect your input data. Make sure your data is uploaded and has correctly passed all pre-processing steps, including augmentation, before entering your network. The easiest way to do this is to visually check the images.</p><h4>Patterns</h4><p>Often the network makes wrong predictions in specific cases. You can try to identify patterns in mispredictions. These patterns point to missing examples in the training dataset.</p><h4>Outliers</h4><p>Outliers are data that differ significantly from other data in your data set. Significant number of outliers reduce the performance of your algorithm. Since your network is unlikely to be able to learn from them, they can be safely eliminated from the dataset. The outliers could be found by analyzing image metadata (aspect ratio, resolution, brightness <em>etc</em>), embeddings or confidence score of your classifier.</p><h4>Reduced dataset</h4><p>Running your first experiments on a full dataset is slow and inefficient. Instead, you can start with a small subset that represents your problem. Make sure you can run the entire learning pipeline and get meaningful predictions. Keep in mind that machine learning algorithms are likely to overfit on small datasets.</p><h4>Model capacity</h4><p>Model performance on training images can tell you a lot about possible problems. Poor performance might mean that your network does not have enough capacity to deal with this particular problem. In this case you can try a deeper network with more layers and parameters. Near-ideal performance is also suspect, probably as a result of overfitting. To fix this, the network can be made smaller or the current network can use more training data.</p><h4>Vanishing/exploding gradients</h4><p>Model training uses a backpropagation algorithm to update network parameters at each training step. The algorithm propagates backward from the output layer to the input layer calculating the error gradients (that is, derivatives) of the parameters on the way. There might be situations when the gradients are getting vanishingly small or extremely large as the backpropagation algorithm progresses towards the input layer.</p><p>In the former case, the optimization algorithm never reaches the optimum solution (i.e. the training process gets stuck). In the latter case, the network weights are updated too heavily and the algorithm diverges. If possible, look at the gradients and activations inside the network.</p><p>To cure this error, one can try different weights initialization strategies, change non-linear activation functions or apply batch normalization between the convolution layers. Another solution to the exploding gradient problem could be to set a threshold value for the gradient that they can never exceed (“gradient clipping”).</p><h3>Model format</h3><p>Model training is only half the way to a real application, since the model is likely to be executed in a different environment compared to the one in which it was trained. For example, a model can be trained on a laptop, but executed on a mobile phone or an embedded device. Thus, the original model might need to be converted into the format suitable for specific hardware it will be run on.</p><p>Today, each deep learning framework has its own model format: <strong>.pt</strong> for PyTorch; <strong>.pb</strong> for TensorFlow, <strong>.caffemodel</strong> for Caffe or Intermediate Representation for Intel’s <a href="https://docs.openvino.ai/latest/index.html">OpenVino</a>. This causes many difficulties, since not all machine learning operations are implemented in all libraries. <a href="https://onnx.ai/">ONNX</a> is a mediator used for conversion between different machine learning frameworks. <a href="https://kapernikov.com/future-proof-your-ai-investment/">Converting your model into <strong>.onnx</strong> format is a good idea</a>, as it helps to simplify the portability of your model and increase cross-functionality.</p><p>In this article, we have discussed possible ways to debug your deep learning algorithm when it is not working properly as well as how to prepare your final model for deployment. The outlined steps will help you achieve your goals in creating and serving a reliable AI solution for the problem you are trying to solve.</p><p><em>Originally published at </em><a href="https://kapernikov.com/best-practices-for-model-training-and-serving-in-machine-learning-part-2/"><em>https://kapernikov.com</em></a><em> on November 10, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ac4cf926c65" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kapernikov/best-practices-for-model-training-and-serving-in-machine-learning-part-2-kapernikov-1ac4cf926c65">Best practices for model training and serving in machine learning — Part 2 — Kapernikov</a> was originally published in <a href="https://medium.com/kapernikov">Kapernikov</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cloud independent solutions — Kapernikov]]></title>
            <link>https://medium.com/kapernikov/cloud-independent-solutions-kapernikov-a7baa874347?source=rss-5a452e0b0e84------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7baa874347</guid>
            <category><![CDATA[cloud-data-platform]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[cloud-architecture]]></category>
            <dc:creator><![CDATA[Kapernikov]]></dc:creator>
            <pubDate>Fri, 28 Oct 2022 15:07:21 GMT</pubDate>
            <atom:updated>2022-11-10T12:44:14.937Z</atom:updated>
            <content:encoded><![CDATA[<h3>Cloud independent solutions — Kapernikov</h3><p>At Kapernikov, we build robust, scalable and cost-efficient cloud-native data platforms and applications. When we start discussing architecture, vendor lock-in is a topic that keeps popping up. This article describes our vision on cloud vendor lock-in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r-6fUzHdUYxoYPDwwnEPzA.png" /></figure><p>The big cloud vendors now offer a huge number of building blocks for building your data platform (both Amazon AWS and Azure cloud claims to offer over 200 products/services, with google cloud listing over 100 products on their <a href="https://cloud.google.com/terms/services">summary page</a>). Some of them are basic infrastructure (a virtual machine or data storage), and some of them are value-added services that offer a lot of functionality (an IoT gateway).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X_MMyZGPTb8vdIQS" /></figure><p>In addition to this, independent software vendors also have their offering on the public cloud: you can provision software from Confluent, Databricks, Ververica on all major cloud providers.</p><p>For a lot of these value added services (but not all), a self-managed (sometimes open source) alternative exists, that offers functionality comparable to the SaaS versions. However, this shifts the burden of maintenance, monitoring and upgrades on your team.</p><p>This makes for a huge solution space: for every component of the software architecture you are designing, multiple options are available on all public clouds. The optimal choice depends on a lot of factors. Avoiding cloud vendor lock-in is one of them.</p><p>In order to answer the question “what about cloud vendor lock in”, we need to:</p><ul><li>Understand <strong>what is cloud vendor lock in and if it is harmful in your situation</strong>;</li><li>Understand <strong>the other motivations for selecting components </strong>in an architecture;</li><li>Understand <strong>what trade-off </strong>we are making when selecting components;</li><li>Understand <strong>how we can avoid </strong>vendor lock in.</li></ul><p>Let’s dive in.</p><h3>About cloud vendor lock-in</h3><p><a href="https://en.wikipedia.org/wiki/Vendor_lock-in">Vendor lock-in</a>, as defined on Wikipedia, makes a customer dependent on a vendor for products, unable to use another vendor without substantial switching costs. With <em>cloud vendor lock-in </em>we mean the inability to switch to another cloud or technology vendor.</p><p>There are several reasons why vendor lock-in can be problematic:</p><ul><li>Your customers might impose a specific cloud provider. Large corporations and government institutions often do. Some customers might even have good reasons to stick to an on-premise deployment. Even when you can choose your cloud vendor, the customer might not want to use certain services.</li><li>When negotiating with your vendor, lock-in puts you in a weak position.</li><li>When you are stuck to a certain technology, you could be stuck to a certain pricing model, that might become prohibitively expensive as you grow. Note that some SaaS solutions have very flexible pricing models.</li><li>The technology or platform you choose might become obsolete, and it might become very hard to find skilled manpower in certain cases.</li><li>A vendor lock-in can hamper the speed at which you innovate: the vendor you chose might have missed a new disruptive trend.</li></ul><p>Apart from these reasons, cost savings (due to pricing differences) are often cited as a reason to avoid cloud vendor lock-in. In our experience, costs savings alone has rarely been the motivation to switch from one cloud vendor to another one.</p><p>Note that cloud vendor lock-in isn’t all that new: vendor lock-in exists for much longer than cloud vendor lock in. Cloud vendor lock-in isn’t <em>worse </em>than “traditional” vendor lock-in, but with the advent of the cloud, we have much more tools at our disposition to fight lock in than we had before.</p><h3>Choosing components: the trade-off</h3><p>The question of avoiding cloud vendor lock-in is hence a question of limiting the cost of switching to a different cloud or technology vendor.</p><p>In order to understand what is the risk of a vendor lock-in, we must first ask ourselves how desirable the ability to switch (for the whole cloud vendor or for a specific component) might become in the future. This can be hard to predict, but here are some questions that can help you:</p><ul><li>Do you want to innovate and stay ahead the competition ? Is the solution you are developing critical to reach that goal ? If so, will you select components that you will need to “own” (e.g. modify or differentiate). If you sell an IoT platform, you might want to internalize the IoT communication layer. If you have a project that involves a couple of IoT devices, you will be much better of by sticking to some SaaS.</li><li>Or … is this just commodity software that should not be particularly innovative, and instead be dependable. Your accounting software is probably not where you are going to innovate, but you want it to always work.</li><li>Or is it a side-project or something an experiment ?</li><li>Are you building something that should for 10 years ? Or is this a first version which you expect to either throw away or rewrite 2 to 3 years from now ?</li></ul><p>Answering these questions might help to judge the importance of avoiding vendor-lock in. The decisions are mostly made case by case. For instance, suppose that you are developing what will become the core innovative product of your company. In that case, avoiding lock-in can be very important (for reasons listed above), but maybe there is one cloud SaaS service from one of the vendors that is unmatched in relevant functionality, and not using that service would hamper the product or be a huge cost to put in place yourself.</p><h3>Tricks for avoiding cloud vendor lock in</h3><ul><li>Try to <strong>estimate the lifetime cost of your decisions</strong>, not just the set-up or development cost.</li><li>Don’t put all your eggs in one basket: separate applications &amp; deployments where they make sense and <em>design </em>a proper facade (API) between them, even when run on the same platform. In other words: make sure your components are <strong><em>loosely coupled</em></strong>. This is good design practise and will allow you to switch out one component for another more easily.</li><li>Use <strong>SaaS with a standardized API rather than SaaS with a custom API</strong>. For instance, when looking for stream processing at Amazon, you could choose between <strong>Amazon MSK </strong>and <strong>Amazon Kinesis</strong>. Both are Amazon-supported SaaS services, but the former uses the Kafka API’s (so you could move to confluent easily if you ever wanted), while the latter uses a proprietary API.</li><li>Beware of <strong>stateful components </strong>(like databases, …), because moving them does not only require some rework but also data migration. An open storage format might make such a migration much more doable. As an example, apache parquet is now used as storage format by multiple SAAS solutions in the analytics world.</li><li><strong>Use Kubernetes instead of VMs or lone containers</strong>: all major cloud vendors offer it, and compatibility between the cloud vendors is actually quite good. Kubernetes covers much more than just running containers in a completely standardized API.</li><li>Some SaaS offerings now have a very good <strong>Kubernetes-native equivalent</strong>, and choosing the Kubernetes-native equivalent will immediately make you cloud vendor independent. However, do factor in both the maintenance burden and the quality of the Kubernetes operator: some operators are quite mature and will be very reliable, while others are beta quality at most.</li></ul><h3>Conclusion</h3><p>Dealing with cloud vendor lock-in or independence is not something you do or don’t, but it is a trade-off you make, balancing the risk of being unable to switch with the extra costs a cloud-vendor-independent solution might impose. Doing so needs a good understanding of what risks and what costs we are exactly balancing, as well as a clear view on the positioning of the software you are trying to build.</p><p>Cloud technology selection can make the difference between a failure and a successful platform. However, it is difficult, because you need to take into account the risks and costs and also the team skill-set, the need of the organization to standardize on a portfolio rather than a single product, and the existing relations with software vendors and integrators.</p><p>Kapernikov has helped various small and big organisations navigating this trade-off. Are you also building a cloud platform, and do you need help ?</p><p><em>Originally published at </em><a href="https://kapernikov.com/cloud-independent-solutions/"><em>https://kapernikov.com</em></a><em> on October 28, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7baa874347" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kapernikov/cloud-independent-solutions-kapernikov-a7baa874347">Cloud independent solutions — Kapernikov</a> was originally published in <a href="https://medium.com/kapernikov">Kapernikov</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Book review: what we learned from The Data-Centric Revolution — Kapernikov]]></title>
            <link>https://medium.com/kapernikov/book-review-what-we-learned-from-the-data-centric-revolution-kapernikov-508076c2a2f?source=rss-5a452e0b0e84------2</link>
            <guid isPermaLink="false">https://medium.com/p/508076c2a2f</guid>
            <category><![CDATA[book-review]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-management]]></category>
            <dc:creator><![CDATA[Kapernikov]]></dc:creator>
            <pubDate>Thu, 13 Oct 2022 08:44:13 GMT</pubDate>
            <atom:updated>2022-11-10T12:37:53.278Z</atom:updated>
            <content:encoded><![CDATA[<h3>Book review: what we learned from The Data-Centric Revolution — Kapernikov</h3><p>Implementing enterprise software systems for utility or manufacturing companies can be a gargantuan task. The cost and effort of setting up such systems is often underestimated, not to mention that derailing budgets are not so exceptional. The industry’s application-centric approach is to blame for this, says Dave McComb, author of <a href="https://www.amazon.com/Data-Centric-Revolution-Restoring-Enterprise-Information/dp/1634625404">The Data-centric Revolution</a>. In his book, McComb argues that enterprise system implementation needn’t be that complex. Even more, a lot of the money spent on IT today is wasted. Instead, companies should apply a data-centric approach. Here’s our take on this eye-opening book.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rpursFB53X4JZRMiGE5Uig.png" /></figure><p>Every once in a while, a book comes along that changes the way you think profoundly. McComb’s The Data-Centric Revolution might well be such a book. Recently, one of our customers who became interested in the data-centric approach recommended us reading it. And we are glad we did.</p><h3>Enterprise software systems</h3><p>The book talks about enterprise systems, typically deployed in large corporations. Belgian rail infrastructure company Infrabel is a good example. This organization has <a href="https://kapernikov.com/establishing-the-digital-railway/">a lot of operational assets</a> to manage (railway tracks, overhead lines, electrical cabinets, trackside poles, etc.). In addition, the company also manages projects, customers, suppliers, subcontractors, etc.</p><p>To manage all these concepts, we need software applications. Historically — let’s say from the late sixties onwards — enterprise systems were introduced for every application area or problem that needed to be solved, silo per silo. Typically, companies would start with a payroll or bookkeeping system, and then add new applications along the way. For every new problem, a new application project was launched.</p><p>The problem with this way of working is that implementing new applications got increasingly difficult, because of all the previous applications that were introduced. Trying to make all different applications talk to each other has become more complex, and this complexity has been building up with every new application that is launched.</p><blockquote><em>Implementing new applications has become increasingly difficult, and complexity has been building up with every new application that is launched.</em></blockquote><h3>Runaway complexity</h3><p>The big drawback of the application-centric approach is that software applications have a much shorter lifetime than the actual data and concepts they describe. In the case of Infrabel, a railway asset will be operational for many decades on end. A software application will never last that long. So, what happens when an application no longer meets the needs of the organization? In a first stage, updates are added onto the original application, which will partially solve the problem. However, during the upgrade, developers will start tampering with the original data structure and user interface, which will require new software integration work.</p><p>Over time, the ability to efficiently upgrade the system will decrease and the need for a new software application, requiring a new data structure, will arise. Because of the application’s new data structure, the original data will need to be forced into the system (data conversion) and new integrations will need to be made. This is what the author calls ‘runaway complexity’, and it’s costing organizations a fortune in integration work.</p><h3>The data-centric revolution</h3><p>But it doesn’t have to be that way, McComb argues. In the data-centric approach, organizations see data as a permanent shared asset, while applications that are built around it come and go. In this approach, the data model from which you start needs to be as simple and as close to the organization’s reality as possible (although it cannot be oversimplified). Changing applications then adapt to the data model, instead of the other way around. As a result, no data conversion is needed, and migrating to a new application is seamless.</p><blockquote><em>In the data-centric approach, organizations see data as a permanent shared asset, while applications that are built around it come and go.</em></blockquote><h3>Standardization or customization?</h3><p>Obviously, this all sounds very nice. But the data-centric approach has an important implication, namely that standard software solutions are no longer interesting. Instead, the application software needs to be customized from the ground up to reflect the organization’s reality.</p><p>Is this wise? Isn’t there a good case for standardization? Don’t we want standard software applications because of their cost-effectiveness and extensive features? Let’s nuance this a bit.</p><ul><li><strong>More cost-effective</strong>: Yes, standard enterprise systems have a lower acquisition cost, compared to customized solutions. However, if you have to count the data conversion cost and integration cost with every new application, then a customized application is often much more cost-effective.</li><li><strong>More features</strong>: An enterprise software application that can do it all and that checks all the boxes of your requirement checklist may sound great, but in practice it isn’t. More often than not, more features — some of which you might never need — only add to the complexity of the system. Apart from that, adding more whistles and bells and more configurability almost always comes at the expense of user-friendliness.</li></ul><h3>Is data-centric the new way to go?</h3><p>After reading this book, a data-centric approach, with the data model at the center of your enterprise, and with customized applications tailored to your organization, makes perfect sense. In this approach, adding new applications is becoming much less painful, and the cost of migration is negligible.</p><p>More organizations are starting to see the benefits of the data-centric approach. And the good thing is, you can implement your data-centric enterprise in steps, with one project at a time. However, if you really want to make an impact with the data-centric enterprise, then you need to make the upfront investment. There’s no denying that.</p><p>But in contrast to the early days of software development, today’s technology and linked data structures make a data-centric approach perfectly possible. The only thing that might restrain developers and data architects from choosing a data-centric approach, is that they always have done things differently in the past.</p><p><em>Originally published at </em><a href="https://kapernikov.com/what-we-learned-from-the-data-centric-revolution/"><em>https://kapernikov.com</em></a><em> on October 13, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=508076c2a2f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kapernikov/book-review-what-we-learned-from-the-data-centric-revolution-kapernikov-508076c2a2f">Book review: what we learned from The Data-Centric Revolution — Kapernikov</a> was originally published in <a href="https://medium.com/kapernikov">Kapernikov</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From experimentation to production: how MLOps helps to deliver high-quality machine learning…]]></title>
            <link>https://medium.com/kapernikov/from-experimentation-to-production-how-mlops-helps-to-deliver-high-quality-machine-learning-7e87493bde7f?source=rss-5a452e0b0e84------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e87493bde7f</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[machine-vision]]></category>
            <dc:creator><![CDATA[Kapernikov]]></dc:creator>
            <pubDate>Thu, 29 Sep 2022 11:57:46 GMT</pubDate>
            <atom:updated>2022-11-10T12:24:55.031Z</atom:updated>
            <content:encoded><![CDATA[<h3>From experimentation to production: how MLOps helps to deliver high-quality machine learning applications — Kapernikov</h3><p>Deploying machine learning models into a production environment can be a long and winding road. A lot can go wrong between the experimentation phase and the production floor, and a successful deployment depends on a complex interaction between data, machine learning model and code. The MLOps methodology is a good way to streamline this interaction. MLOps borrows heavily from DevOps, but there are important differences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D6vkppHlx3qTcrJ-2AoqOA.png" /></figure><p>According to a<a href="https://venturebeat.com/2019/07/19/why-do-87-of-data-science-projects-never-make-it-into-production/"> VentureBeat article of 2019</a>, 87% of data science projects never make it to production. Why is it so hard to transform machine learning experimentation to applications deployed in production floors? There may be several reasons.</p><h4>Experiment</h4><p>One of the inherent challenges of machine learning is that a lot of experimentation is involved. Features, parameters, and models are constantly tweaked in the process. Problems occur when data scientists and machine learning engineers forget to track versions, record parameters or keep details of the environments in which they run their models. Then it tends to become very messy, and the result of the data experiments will be hard to reproduce.</p><h4>Collaborate</h4><p>Another challenge is collaboration. Machine learning projects are typically executed by a mixed group of talents, including data engineers, data scientists, machine learning engineers. For these projects to generate any usable results, it’s essential that everyone is able to collaborate with one another. Unfortunately, working in siloes is often the rule. Machine learning experiments are often done on computational notebooks that reside in the cloud or on a data scientist’s personal workstation. More often than not, the evolution of the code is not (sufficiently) exposed to the team, which makes it challenging to reproduce experiments in other computing environments.</p><h3>Why not use DevOps?</h3><p>In addition to experimentation and a high need for collaboration, machine learning projects are characterized by fast change. Machine learning applications evolve rapidly. State-of-the-art models that are used as a starting point are often refined and extended in the process to better suit their purpose, making the original model rapidly inefficient compared to the new model. This makes deployment fairly complex.</p><p>To be able to cope with fast change and to avoid rework and inefficiency, machine learning projects need a stable, but agile process for bringing applications into production, which allows teams to adapt to changing conditions fast. But, wait a minute, isn’t that what DevOps is for?</p><p>Indeed, <a href="https://kapernikov.com/why-kapernikov-loves-devops-and-why-you-should-too/">DevOps</a> is a methodology in traditional software development that ensures a stable deployment process and a clear lineage of the code. DevOps has helped companies to break down the silos of development and operational teams, and has stimulated collaboration between both teams to come to better results. DevOps works with a never-ending, iterative loop through which engineering teams can continuously improve. This way, teams go through different stages, from planning to monitoring, and then go back to the beginning to start a new loop.</p><p>The agile principles of DevOps could work for machine learning, were it not for the fact that machine learning is about more than just code. In machine learning there is a constant interaction between the code, the data and the model. That’s why a traditional software workflow cannot work. But isn’t there a way to apply the DevOps principles to machine learning? Yes, there is. It’s called MLOps.</p><h3>What is MLOps?</h3><p>MLOps is a methodology, based on DevOps, which improves the collaboration between data scientists and operations professionals. Applying this methodology helps teams to deploy machine learning models in large-scale production environments much faster and with much better results. Let’s look at some of the building blocks of the MLOps methodology.</p><h4>Version control</h4><p>Code versioning is an essential part of DevOps. In MLOps, there is also a need to<a href="https://kapernikov.com/how-we-keep-track-of-our-data-experiments/"> version the data</a> and the models that are produced. In addition, all these versioning processes need to be correlated. For every model, you need the corresponding datasets and the corresponding versions of the code.</p><h4>Pipelines</h4><p>A pipeline connects the development environment to the production environment. This enables us to transfer the model we used for inference into production. Computational notebooks need to be converted into executable code, so that it can be versioned and integrated into the deployment pipeline.</p><h4>Cloud environment</h4><p>By training the model in the cloud, you have a centralized, or at least a shared, environment, which facilitates collaboration among data scientists. A centralized environment and automated workflows also offer uniformity and reproducibility, which are prerequisites for the successful delivery of a machine learning project. At Kapernikov, we typically rely on a DevOps platform for storing the code in the cloud. The data can be stored in some cloud storage and be downloaded/mounted at the beginning of the training. For the model, we prefer to host the trained model in a dedicated model registry.</p><h3>The MLOps life cycle</h3><p>The MLOps lifecycle offers a complete workflow to connect experimentation with the production environment. As part of this pipeline, short cycles of development can bring new features into production.</p><p>Work packages for data engineers, data scientist and machine learning engineers are clearly defined:</p><ul><li><strong>Data engineers</strong> focus on <strong>data storage and data preparation</strong>.</li><li><strong>Data scientists </strong>use that data to take care of <strong>model development</strong>. The result is a trained model.</li><li><strong>Machine learning engineers deploy </strong>the trained model in production and <strong>monitor </strong>it.</li></ul><h3>What are the benefits of MLOps?</h3><h4>Easy to reproduce</h4><p>Now that your versioned data and models are saved in the cloud, along with your code, it’s much easier to reproduce models. Whenever data scientists need to rework a model, they can reproduce it safely on their machine or in the cloud.</p><h4>Stimulating collaboration</h4><p>Model training can be automated and scheduled in the cloud, so that the best performing model is always available for production. The cloud also makes collaboration much easier between members of a team working on the same project. No more lost models, no more lost code.</p><h4>Faster deployment</h4><p>Applying MLOps principles leads to faster and more robust deployment of applications whenever there is a code or data change. Granted, training models in the cloud may be resource-intensive and time-consuming. However, it’s still perfectly possible with ML pipelines to run training locally. For example, you could train on a local machine in the early stage of your project to decrease the latency and to speed up development. Then, following your ML pipeline, you could regularly share your code, data and models in the cloud when you reach key milestones.</p><h3>What does an MLOps task look like?</h3><p>MLOps is an ensemble of processes, supported by a wide range of tools and platforms. Some tools like <a href="https://dvc.org/">DVC</a> and <a href="https://mlflow.org/">MLFlow</a>, will manage part of the process. To do MLOps, you will need a selection of tools, also called a framework. Some MLOps platforms will offer most of these processes as one solution.</p><p>To perform the training in a remote environment, you will need to clearly define tasks that will lead to the building of the model. This includes preprocessing (data transformation, data augmentation), the training process itself, and the validation process. These tasks, depending on the MLOps framework/platform, are defined at a different level of abstraction:</p><ul><li>code: tasks are defined using a piece of code or by using specific classes</li><li>notebook: cells that are considered as tasks</li><li>file: file scripts that define a task</li><li>docker container: the execution of a file script inside a docker image</li></ul><p>With codes and notebooks, the frameworks/platforms are language dependent. Also, dependencies need to be provided and supported by the solution, which will be more difficult for recent libraries. Even if they are supported, there are risks of failures every time you update these dependencies.</p><p>On the contrary, docker containers as tasks offer a solution where you keep control over the dependencies. That’s a dedicated artifact that offers a clean isolation and that you can functionally test. Moreover, docker containers are interoperable meaning that you are not stick to one framework/platform.</p><h3>Improving your MLOps workflow</h3><p>Versioning of code, data and models, a pipeline that connects the development environment to production and training in the cloud: these best practices form a solid foundation for MLOps. Once you have these components in place, you can start extending your workflow with additional or improved components.</p><h4>ML pipeline</h4><p>The ML pipeline includes all tasks required to build a model:</p><ul><li>Input data preprocessing</li><li>Model training and testing</li><li>Reporting on the quality of the models (metrics, plots)</li><li>Registering and deployment of the model</li><li><a href="https://kapernikov.com/the-devops-8/">CI/CD pipeline</a> of the code</li></ul><p>The ML pipeline can be triggered by code changes, or by adding new data, or it can be scheduled. Complex non-linear workflows can be designed.</p><h4>Model serving (inference)</h4><p>The model can be put into production by using automated tools ( <a href="https://www.tensorflow.org/tfx/guide/serving">TFServing</a>, <a href="https://pytorch.org/serve/">TorchServe</a>, <a href="https://readthedocs.org/projects/mlserver/">MLServer</a>), but it can also be deployed through a homemade API. This offers end users more flexibility and customization possibilities. In both cases, you get an API that offers an interface between the model and the outside world.</p><h4>Reporting &amp; monitoring</h4><p>When a model is in production, you can collect statistics to get insights. You can also set up alerts that warn you of possible problems. By closely monitoring your model, you can detect model drift in time. This means that a model will become less accurate, because it is receiving different data than the data it was trained on. An often-cited tool for performing drift detection is Seldon.</p><h4>Experiment tracking</h4><p>You can track and share experiments using tools like DVC and MLFlow. Tracking includes the execution of the ML pipeline along with all the produced artifacts (logs, metrics).</p><h3>Ready to start with MLOps?</h3><p>MLOps helps teams to deploy machine learning applications into a production environment much faster and with higher quality results. By connecting the experimentation environment to the production environment using a well-defined deployment process, it is much easier for teams to adapt to changes. And by carefully versioning all data, models and code in the cloud, there is no risk of losing valuable work.</p><p>MLOps offers a controlled development process, which may require a learning curve for your team, but at Kapernikov, we couldn’t be more convinced of the benefits of this methodology. Need to get your team up to speed with MLOps? Let us know, maybe we can help.</p><h3>Author</h3><p>You might expect that studies in Chemistry may primarily lead to working in a lab, but Ludovic had already moved far beyond that as he was doing his PhD in Quantum Chemistry…</p><p><a href="https://kapernikov.com/authors/ludovic-santos/">https://kapernikov.com/authors/ludovic-santos/</a></p><p><em>Originally published at </em><a href="https://kapernikov.com/from-experimentation-to-production-how-mlops-helps-to-deliver-high-quality-machine-learning-applications/"><em>https://kapernikov.com</em></a><em> on September 29, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e87493bde7f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kapernikov/from-experimentation-to-production-how-mlops-helps-to-deliver-high-quality-machine-learning-7e87493bde7f">From experimentation to production: how MLOps helps to deliver high-quality machine learning…</a> was originally published in <a href="https://medium.com/kapernikov">Kapernikov</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Escape from the land of confusion: quality data for the energy market — Kapernikov]]></title>
            <link>https://medium.com/kapernikov/escape-from-the-land-of-confusion-quality-data-for-the-energy-market-kapernikov-f37428b3db80?source=rss-5a452e0b0e84------2</link>
            <guid isPermaLink="false">https://medium.com/p/f37428b3db80</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[databases-big-data]]></category>
            <category><![CDATA[data-architecture]]></category>
            <dc:creator><![CDATA[Kapernikov]]></dc:creator>
            <pubDate>Thu, 22 Sep 2022 08:08:14 GMT</pubDate>
            <atom:updated>2022-09-22T08:08:14.534Z</atom:updated>
            <content:encoded><![CDATA[<h3>Escape from the land of confusion: quality data for the energy market — Kapernikov</h3><p><a href="https://kapernikov.com/data-science">Data science</a></p><p>When you’re dealing with the energy market, you’re dealing with big data.<br> In Belgium, customers can easily switch between energy suppliers-and they often do. The companies then have to speak to clearing house Atrias for updated info, drawing from a pure jungle of data.<br> Enter Kapernikov, to bring order into chaos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zcyAVHIM2fVsneSDiEfdMw.jpeg" /></figure><p>The Belgian energy market was liberalised in 2007. Since then, gas and electricity suppliers have sprung up like mushrooms and they fiercely compete for every customer. People switch back and forth easily, especially now that <strong>sustainability</strong> has become an additional motivator. Meanwhile, distribution grid operators also see the climate goals ahead and they are working hard to build a greener and <strong>smarter power grid</strong>. To tackle the data challenges, they jointly founded the clearing house Atrias.</p><p>There are many clients to cater for,<br>and they all need tailored information.</p><p>Atrias runs a Central Market System ( <strong>CMS</strong>) to simplify data exchange between all the parties: the grid operators as well as the energy transport companies and the actual suppliers. That’s a lot of clients to cater to, and they all need their own tailored info. On top of the CMS, Atrias manages an Operational Data Store ( <strong>ODS</strong>), which turns out daily aggregates. For the much-needed data quality control of this ODS, Atrias called in Kapernikov.</p><h3>Challenge</h3><p>To fully understand the challenge, try not to be dazzled by all the factors involved.</p><ul><li>The <strong>complexity </strong>of the CMS content. Consumers send in readings of their energy metres once every year, which are then verified by grid operators every two years. Customers can switch suppliers at any moment, meaning their data needs to be corrected. This is why it is so hard to have exact figures at all times on who consumed or produced how much, at what time. In fact, it requires looking at data going back as far as five years.</li><li>We’re dealing with multiple time dimensions. It’s a bit like ‘Back to the Future’: every error or change of contract requires a retroactive update. This is what we call <strong>versioning</strong>: every version of the information needs to be kept in the system. It requires very careful abstract conceptualisation if you don’t want to get lost in data (or time).</li></ul><p>Working with multiple time dimensions<br>is a bit like ‘Back to the Future’.</p><ul><li>Over the course of five years, the data stored in the back-end as well as the reporting system easily reaches up to 40–50 TB. It is rare to come across a project with such <strong>gigantic volumes of data</strong>. And we haven’t seen the end of it. Instead of one metre reading per year, the new smart meters will provide information every 15 minutes of every hour of every day…</li><li>The energy consumption of the entire market passes through the central application, but that doesn’t mean every user should be able to see all of it. Private customer data needs to be protected, and information cannot fall into competitors’ hands. Proper <strong>data segregation</strong> is crucial.</li><li>Technically speaking, a clearing house application can be tough to work with, especially for business users who just need to turn out reports. The data needs to be self-explanatory, easy to locate and available in their own terms. <strong>Modelling the data</strong> from the huge back end to such a tight fit-that calls for quite a tailor.</li><li>Most of the data in the CMS is not created by Atrias but by the market parties. Considering that <strong>data quality</strong> is a challenge for most companies, get ready for inconsistencies when you pile everything up. The big challenge is knowing when your data is bad. Sometimes it is easier to see it at the end of the chain, in your reporting application, than in your source system. But it is in the source system that the bug needs to be fixed…</li></ul><h3>Role</h3><p>Kapernikov built a framework that guarantees the quality of the ODS by comparing its actual output with the required one. If that’s a match, the application can be trusted to deliver quality data. Usually, such a comparison would involve lots of manual checks, which is an awful lot of work and only yields superficial results. Remember, we are dealing with billions of records. <strong>Automatisation</strong> is key. But how? Sit tight, because this is where it gets technical.</p><p>Our developers first screened the data from both the CMS and ODS databases, according to Atrias’ <strong>logical data model</strong> (LDM). The LDM describes how the business is run: the required data (for example, every contract has to be linked to a person or company), the relations between data, etc. The logical model applied to both databases, but there was a big difference between the technical implementation of the two. Which data went into which table in which format? That was just one of many questions.</p><p>To solve the riddle, Kapernikov brought its own <strong>new data store</strong> into play. Basically, it converts data from both databases to a canonical data model, so they speak the same language. Rather than writing endless amounts of individual SQL queries implementing the same logic, it defines functionalities once and sets up dynamic code generation from there. Easy does it. Any correction only has to be done in one place, rather than in hundreds or thousands. Besides, this approach runs a much lower risk of flaws in the code.</p><p>Developers took a shortcut by organising the work in DAGs (Directed Acyclic Graphs — tasks that run in a specific order, building on previous tasks). They applied modern data pipeline practices and used <a href="https://kapernikov.com/using-luigi-to-power-a-reporting-pipeline/"><strong>Luigi</strong></a> as the dataflow orchestration tool. To make the flow smoother, the data warehouse itself takes care of all data transformations, expressed in SQL queries. Long story short: it was all about parallelisation of tasks, huge data throughput and logical transparency.</p><h3>Results</h3><p>Our approach allowed us to verify <strong>billions of records</strong> and spot even the tiniest error or inconsistency. From there, all that was required was analysing the root causes and weeding them out with Atrias’ team. The quality of the data in the ODS improved, and even long-lived glitches in the CMS could now be fixed. Both Atrias and the grid operators can rest assured about their reporting system. In the end, it all looks simple.</p><h3>Applied Solutions &amp; Technologies</h3><p><em>Dynamically generated SQL</em></p><p><em>Originally published at </em><a href="https://kapernikov.com/cases/escape-from-the-land-of-confusion-quality-data-for-the-energy-market/"><em>https://kapernikov.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f37428b3db80" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kapernikov/escape-from-the-land-of-confusion-quality-data-for-the-energy-market-kapernikov-f37428b3db80">Escape from the land of confusion: quality data for the energy market — Kapernikov</a> was originally published in <a href="https://medium.com/kapernikov">Kapernikov</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Best practices for model training and serving in machine learning — Part 1 — Kapernikov]]></title>
            <link>https://medium.com/kapernikov/best-practices-for-model-training-and-serving-in-machine-learning-part-1-kapernikov-dc2b0b216b74?source=rss-5a452e0b0e84------2</link>
            <guid isPermaLink="false">https://medium.com/p/dc2b0b216b74</guid>
            <category><![CDATA[machine-vision]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Kapernikov]]></dc:creator>
            <pubDate>Fri, 19 Aug 2022 11:43:06 GMT</pubDate>
            <atom:updated>2022-09-22T07:56:07.887Z</atom:updated>
            <content:encoded><![CDATA[<h3>Best practices for model training and serving in machine learning — Part 1 — Kapernikov</h3><p>Now that we have chosen <a href="https://kapernikov.com/which-machine-learning-algorithm-does-your-computer-vision-project-need/">the right algorithm</a> for our machine learning problem, and we have <a href="https://kapernikov.com/want-to-use-ai-on-your-problem-but-struggling-to-get-a-good-dataset-part-1/">collected</a> and <a href="https://kapernikov.com/want-to-use-ai-on-your-problem-but-struggling-to-get-a-good-dataset-part-2/">prepared</a> a dataset, we can finally start training our network to obtain a model that will solve our problem. But the training depends on many parameters that are hard to choose. Let’s take a look at the most important ones and discuss strategies that can be used to intelligently manage your experiments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gcS-gpijC70x8XAGEDs-oQ.png" /></figure><h3>Hyperparameters</h3><p>In deep learning, we look for the set of network parameters ( <em>weights</em>) resulting in the best score for our task. The network architecture determines how many parameters we have and how they depend on each other (model depth, width and topology). But we also need to define how we update these parameters during training as well as to provide a quantitative score function. These control parameters are often called <em>hyperparameters. </em>They affect the speed and quality of the learning process.Let’s discuss a few of the most important hyperparameters.</p><p>The network weights are found using an iterative <em>optimization</em> procedure that minimizes an objective function called <em>the loss function</em>. <em>The loss function</em> must well represent the problem that the algorithm is trying to solve, otherwise the training will never achieve its goal. Examples of good loss function choice are <em>cross-entropy function</em> for classification, <em>mean squared error</em> for regression or <em>dice coefficient</em> for segmentations. One can consult the literature on whether there are variants of loss functions for specific tasks: <a href="https://link.springer.com/article/10.1007/s11263-019-01247-4">object detection</a>, <a href="https://arxiv.org/pdf/2006.14822.pdf">semantic segmentation</a>.</p><h4>Learning rate</h4><p>The choice of possible optimization algorithms includes <a href="https://pytorch.org/docs/stable/generated/torch.optim.SGD.html#torch.optim.SGD">SDG</a>, <a href="https://pytorch.org/docs/stable/generated/torch.optim.Adam.html#torch.optim.Adam">Adam</a> or <a href="https://pytorch.org/docs/stable/generated/torch.optim.Rprop.html#torch.optim.Rprop">RProp</a>. The algorithm updates the model weights according to <em>the learning rate</em>.A good learning rate is a trade-off between slow convergence and overshooting. If the learning rate is too small, training will take a very long time. If it is too large, the network will be driven far away from the minimum and might even diverge. Practically, a good learning rate range can be found using the following pre-training procedure.</p><p>Split your training set into mini batches and load them into your neural network one by one, varying the learning rate over a wide range of values (10–7–100). You can then plot the loss function as a function of the learning rate, which should decrease as we increase the learning rate and then reach a minimum (see the illustration). The learning rate interval corresponding to the area where the loss function decreases is a good range for your learning rate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/0*g5XGo6wzktwAOCYq" /></figure><p>It is recommended to use a variable learning rate in training instead of a fixed value. A good learning rate scheduler is usually a function that grows in the beginning and then drops down. Using a cyclic scheduler with the learning rate continuously oscillating in between the minimum and maximum bounds allows to avoid the problem of saddle points in the optimization.</p><h4>Other sets</h4><p>There are many more hyperparameters than the choice of optimization algorithm, loss function, and learning rate scheduler. Often these hyperparameters are algorithm or network specific. But how to choose the optimal set of these parameters?</p><p>As a starting point, it is recommended to first experiment with the hyperparameters that can be found in the literature. The authors have already experimented with the settings, and the best values are published. In general, the optimal set of hyperparameters can be found using a certain search strategy.</p><p>In grid search strategy, we divide the domain into a discrete grid and then try every combination of values from the grid. In a random search strategy, we pick up random hyperparameter combinations instead of exploring all possibilities from the grid. Eventually, more advanced Bayesian methods of hyperparameter search can be employed.</p><h3>Logging</h3><p>When training your network or looking for the optimal set of hyperparameters, It is very important to keep track of all your experiments to make sure they are repeatable. Logging your experiments in a structured way enables you to discover patterns in parameter effectiveness and make smart choices for the next experiment. They can also be used to transfer knowledge to other people about what has already been tried or to report progress.</p><p><a href="https://www.tensorflow.org/tensorboard">TensorBoard</a> is the visualizer of the TensorFlow framework that contains functionalities to save parameters and compare trained networks. For example, one can track the loss function and metrics evolution, quantify the computational performance of the model and display the images with predictions. The open source <a href="https://mlflow.org/">MLflow</a> platform is agnostic of the underlying environment and contains some extra functionalities.</p><p>In this article, we discussed which control parameters govern the training of neural networks, as well as possible strategies for finding the optimal set of these parameters and tracking of experiments. In our next article, we will discuss what to do if something goes wrong while training the network, how to debug your network and prepare the final model for deployment.</p><h3>Author</h3><p>As a consultant, Maksim is now enjoying his work days with a wide range of machine vision and machine learning projects. The variety of work appeals to him: “Up to now, I ha …</p><p><em>Originally published at </em><a href="https://kapernikov.com/best-practices-for-model-training-and-serving-in-machine-learning-part-1/"><em>https://kapernikov.com</em></a><em> on August 19, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc2b0b216b74" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kapernikov/best-practices-for-model-training-and-serving-in-machine-learning-part-1-kapernikov-dc2b0b216b74">Best practices for model training and serving in machine learning — Part 1 — Kapernikov</a> was originally published in <a href="https://medium.com/kapernikov">Kapernikov</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scalable cloud architecture gears MYND up for the commercial market — Kapernikov]]></title>
            <link>https://medium.com/kapernikov/scalable-cloud-architecture-gears-mynd-up-for-the-commercial-market-kapernikov-399ce84b9d87?source=rss-5a452e0b0e84------2</link>
            <guid isPermaLink="false">https://medium.com/p/399ce84b9d87</guid>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[cloud-architecture]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[co-development]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Kapernikov]]></dc:creator>
            <pubDate>Tue, 12 Jul 2022 15:27:18 GMT</pubDate>
            <atom:updated>2022-07-12T15:27:18.026Z</atom:updated>
            <content:encoded><![CDATA[<h3>Scalable cloud architecture gears MYND up for the commercial market — Kapernikov</h3><p>MYND is a topic modeling software, based on machine learning. It allows users to automatically analyze and cluster vast amounts of text data. It’s an ideal tool for innovative companies that want fast access to external intelligence and information sources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Zfrjw2-vW9yEVFBRxs-rQ.png" /></figure><p>“Companies that want to bring their innovation to the market fast are often confronted with large amounts of information available today in the form of technical documentation, research papers, and patents,” says <strong>Jelle Demeulemeester</strong>, Managing Partner — COO at MYND. “It’s just not humanly possible to read it all. With MYND, our customers can quickly acquire vast amounts of information, and use that as a basis for their innovation efforts.”</p><p>MYND started off as an internal tool for Belgian innovation agency <a href="https://creax.com/">Creax</a>. But soon, customers of Creax started to see the value of it too. That’s why Creax decided to launch MYND as a spin-off commercial SaaS product.</p><h3>Challenge: flexible software architecture</h3><p>“Our MYND team includes some very talented data scientists, but commercializing a tool as a user-friendly and reliable SaaS product is a whole different ball game,” says <strong>Jelle</strong>. “Since we did not have the knowledge in house, we called upon Kapernikov for support.”</p><p>The technical challenge consisted of finding an architecture that was able to cope with the highly variable, on-demand use of the tool. To calculate a topic model, the MYND software requires a lot of computational power, but not continuously. It’s only needed for a short amount of time when the calculation is running. Choosing the wrong cloud service to make this happen could quickly amount to huge operating costs. That’s why it was important to set up a flexible, dynamic architecture that could cope with both periods of inactivity and peak demand.</p><blockquote><em>By carefully designing a </em><strong><em>flexible cloud architecture</em></strong><em>, MYND has lots of computational power when needed, whilst </em><strong><em>limiting the operating cost</em></strong><em> to a minimum when not in use.</em></blockquote><h3>Our role: commercial deployment of MYND</h3><p>Kapernikov helped MYND to set up a scalable, <a href="https://kapernikov.com/local-kubernetes-development/https://kubernetes.io/">Kubernetes</a>-based software architecture. We helped the MYND team to make the right technical decisions for the further commercial deployment, provided conceptual advice, but also supported them with the practical implementation.</p><p>“Kapernikov offered a wide range of services, and also <a href="https://kapernikov.com/co-development-co-creation-kapernikov/">helped us to grow and mature as a software development team</a>,” says Jelle. “They helped us to become more competent with the <a href="https://kapernikov.com/why-kapernikov-loves-devops-and-why-you-should-too/">DevOps</a> methodology and with continuous integration and development. This knowledge transfer from Kapernikov to MYND is invaluable for the further growth of our team.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xVbRPXaPiETGqnZZ" /></figure><blockquote><em>This </em><strong><em>knowledge transfer</em></strong><em> from Kapernikov to MYND is invaluable for the further growth of our team.</em></blockquote><h3>Strategic partner</h3><p>“Kapernikov offers a unique combination of expertise: they know AI and data science, but they can also help you develop a solid, cloud-native commercial product. I haven’t seen this combination in any other company.”</p><p>MYND operates in a competitive market, so the team was looking for a partner to help them bring a commercial product to the market very fast. Kapernikov helped MYND to build up that speed. At the same time, Kapernikov’s lean approach helped the team to keep costs under control, which is essential for a start-up like MYND.</p><blockquote><em>Kapernikov offers </em><strong><em>a unique combination of expertise</em></strong><em>: they know AI and data science, but they can also help you develop a solid, cloud-native commercial product.</em></blockquote><p>“Of course, you can always try to develop an entire product by yourself, but when you don’t have the knowledge or resources in house, that would take a lot of time and effort,” says <strong>Jelle</strong>. “That’s why it was very comforting for us to have a partner like Kapernikov by our side, who has the experience and who helped us make the right strategic decisions to bring MYND to the market fast.”</p><p>Discover MYND at <a href="http://www.mynd-ware.com/">www.mynd-ware.com</a>.</p><p><em>Originally published at </em><a href="https://kapernikov.com/cases/scalable-cloud-architecture-mynd/"><em>https://kapernikov.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=399ce84b9d87" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kapernikov/scalable-cloud-architecture-gears-mynd-up-for-the-commercial-market-kapernikov-399ce84b9d87">Scalable cloud architecture gears MYND up for the commercial market — Kapernikov</a> was originally published in <a href="https://medium.com/kapernikov">Kapernikov</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Want to use AI on your problem, but struggling to get a good dataset? Part 2. — Kapernikov]]></title>
            <link>https://medium.com/kapernikov/want-to-use-ai-on-your-problem-but-struggling-to-get-a-good-dataset-part-2-kapernikov-3cd6d167344c?source=rss-5a452e0b0e84------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cd6d167344c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-vision]]></category>
            <dc:creator><![CDATA[Kapernikov]]></dc:creator>
            <pubDate>Wed, 06 Jul 2022 09:53:25 GMT</pubDate>
            <atom:updated>2022-07-12T15:23:00.843Z</atom:updated>
            <content:encoded><![CDATA[<h3>Want to use AI on your problem, but struggling to get a good dataset? Part 2. — Kapernikov</h3><p>Data is a key component of machine learning. Without a good dataset, it is impossible to build a good model, no matter how complex your algorithm may be. In this blog post series, we will look at some of the best practices to keep in mind when working with data for your machine learning project. In <a href="https://kapernikov.com/want-to-use-ai-on-your-problem-but-struggling-to-get-a-good-dataset-part-1/">Part 1</a>, we discussed <a href="https://kapernikov.com/want-to-use-ai-on-your-problem-but-struggling-to-get-a-good-dataset-part-1/">how to collect, annotate, and analyze a dataset</a>. We now move on to a discussion of best practices that lead to better and more robust model training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z5h4jwOROImjfM9FIsEZ-g.png" /></figure><h3>Data Partitioning</h3><p>Machine learning algorithms should never be trained on all available data. Overwise, your algorithm will simply “remember” it, giving artificially high results on the data it has already seen, but failing on the new unfamiliar data in production. This situation is called <em>overfitting.</em> To avoid it, we must be able to estimate the performance of our model independently and reliably.</p><p>Having validation and test sets for evaluation helps us perform an unbiased hyperparameter search. The reasonable separation ratio is 60–20–20%, respectively. The data is usually randomly shuffled before splitting. However, it’s a good idea to keep the distribution of the entire set’s data in each subset so that each image category is present in equal proportions throughout.</p><p>Another way to overcome biased or uneven data splitting is to use K-fold cross-validation splitting. In the cross-validation approach, we split our dataset K times (K = 3, 5, 10) into K different training and validation subsets. Further, our algorithm is trained on the data for each split, resulting in K different models tested on different validation sets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/0*PrCh3uEQIIXwXrXA" /></figure><p>By comparing the model performances, we can be more confident in our algorithm performance. For example, large variations in results between the folds is a sign of bad data distribution. For production, we should use the best model among the K folds.</p><h3>Data Augmentation</h3><p>If you want to increase the amount of data in your dataset, you don’t always have to collect new data. The first strategy you should think about is data augmentation. Data augmentation is the process of creating <em>variants </em>of input data from existing data samples and using them in the training phase. Some common forms of image augmentation are geometric transformations (mirroring, rotations, translations), cropping/zooming and changes in the contrast/brightness of images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WO0Yc1MXqhidNH1Q" /></figure><p>The key is to apply transformations that result in images that we could see in the original dataset. This will cause the training set to better represent the distribution of the data we want to learn. By performing data augmentation, we get more annotated data without the costly step of collecting and annotating additional data. The popular libraries for data augmentation are <a href="https://github.com/albumentations-team/albumentations">albumentations</a>, <a href="https://github.com/aleju/imgaug">imgaug</a> and <a href="https://opencv.org/">OpenCV</a> packages.</p><h3>Data Versioning</h3><p>During model development, our dataset is constantly changing. Importantly, a different dataset results in a different model. Thus, it is important to always be able to reconstruct the exact dataset that was used for an experiment, to ensure repeatability and ease of debugging. A data version control system (DVC) is a tool that allows you to keep track of the version of your dataset. Check out our article on DVC to know more about data version control.</p><p>In this small series of articles about data we have covered different stages of the data lifecycle from data collection and preparation to data processing and versioning. Following these guidelines will help you efficiently use data for your project, whether it’s a small personal project or a complex project for your business.</p><p><em>Originally published at </em><a href="https://kapernikov.com/want-to-use-ai-on-your-problem-but-struggling-to-get-a-good-dataset-part-2/"><em>https://kapernikov.com</em></a><em> on July 6, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cd6d167344c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kapernikov/want-to-use-ai-on-your-problem-but-struggling-to-get-a-good-dataset-part-2-kapernikov-3cd6d167344c">Want to use AI on your problem, but struggling to get a good dataset? Part 2. — Kapernikov</a> was originally published in <a href="https://medium.com/kapernikov">Kapernikov</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Three reasons why you shouldn’t rush blindly into deep learning — Kapernikov]]></title>
            <link>https://medium.com/kapernikov/three-reasons-why-you-shouldnt-rush-blindly-into-deep-learning-kapernikov-26aa8b38b0d7?source=rss-5a452e0b0e84------2</link>
            <guid isPermaLink="false">https://medium.com/p/26aa8b38b0d7</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-vision]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Kapernikov]]></dc:creator>
            <pubDate>Wed, 22 Jun 2022 07:35:24 GMT</pubDate>
            <atom:updated>2022-07-12T15:18:44.407Z</atom:updated>
            <content:encoded><![CDATA[<h3>Three reasons why you shouldn’t rush blindly into deep learning — Kapernikov</h3><p>Got a data-related problem? No worries, let deep learning sort it out. This may be a natural reaction, but it’s not always the best one. There are a few reasons why this advanced form of AI is not always the best option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1wWM5f-Ld8KUjj_bq_MqMw.png" /></figure><p><em>If you only have a hammer, then everything starts to look like a nail.</em></p><p>Something similar is going on with AI, and more specifically with deep learning. This is a form of AI that uses neural networks to enable machines to make complex decisions.</p><p>Now that we have deep learning at our disposal, it seems that every problem needs to be solved with it. Nothing could be further from the truth. Don’t get us wrong, we love deep learning and it’s still part of our core business. Deep learning offers a powerful way to solve complex problems, such as image classification, object detection and semantic segmentation. But before you start thinking about using it, you need to ask yourself whether it’s the right technique for the job. In a number of cases, <a href="https://kapernikov.com/traditional-machine-learning-algorithms-for-machine-vision/">traditional machine learning techniques</a> may help you to reach your goals faster and cheaper.</p><p>There are <strong>at least three reasons</strong> why you shouldn’t rush blindly into deep learning.</p><h3>1. You don’t have the data.</h3><p>As we mentioned before,<a href="https://kapernikov.com/which-machine-learning-algorithm-does-your-computer-vision-project-need/"> deep learning is great at solving complex problems</a>. But to do that, it needs high-quality data, lots of it. And although techniques such as data augmentation and synthesizing can help you to make better use of existing data, you will still need more data for deep learning than for traditional methods. High-quality data means that your data should be accurate, labeled correctly and diverse enough to cover all possible environment variables. If that is not the case, then the accuracy of your results will be poor. Everybody knows that <em>garbage in </em>means<em> garbage out.</em></p><p>Sometimes, there simply is not enough data to work with. This is often the case with very specific, complex problems. For example, a food sorting machine that needs to remove bad carrots from a conveyor belt will need to see many bad carrot examples first, because these may come in all types of shapes and colors. To make this work, you may need to develop your data set from scratch, which is costly and time-consuming.</p><p>The rarity of an event may also be the reason why it’s hard to obtain good data. Just think of a manufacturing defect that only occurs once a month. With an event that is so hard to reproduce, you may need to look out for techniques that do not require so much data.</p><h3>2. You don’t have the computing power</h3><p>Deep learning can solve complex problems. This is thanks to a significant increase in computing power. In recent years, computers have become faster and more powerful, and they can execute algorithms much faster. But for some projects, this may exactly be the Achilles heel.</p><p>Sometimes, machine learning developers don’t want to achieve the highest level of performance and technical sophistication. Instead, they need to be creative and make use of the technology that is available within a limited budget. This was the case for the<a href="https://kapernikov.com/counting-power-to-the-people/"> Telraam project</a>, which included the development of a traffic counting sensor for cities.</p><p>In the Telraam case, Kapernikov was able to <a href="https://kapernikov.com/high-framerate-deep-learning-low-cost/">reduce the required computing power</a> by splitting up the traffic detector’s counting function into a detection and a tracking function. The traffic sensor now uses deep learning to detect people in the camera image. Here, deep learning makes sense, because people detection is a fairly complex problem, but one that is supported by a quality data set. Counting people also implies that you can track them in the image, so that you only register them once. The Telraam traffic sensor uses traditional algorithms to do that, because they are better suited and more cost-effective.</p><h3>3. You need insight into the decision-making process</h3><p>One of the most important disadvantages of deep learning is that it’s a black box. This means that you exactly know the input (data) and output, but every operation in between, from input to output, remains a mystery. This is fine, until something unexpected or problematic happens. When the output is not what you expected, you cannot debug or reverse-engineer your deep learning network and find out what went wrong.</p><p>Deep learning uses neural networks, which are inspired by the human brain, mimicking the way biological neurons signal to each other. Depending on the problem you want to solve, these networks contain many hidden layers in which the computing is done, and where input is transformed into output. Neural networks can learn their own features during the training phase, which makes them so suitable for complex problems. This contrasts with traditional machine learning algorithms which have a human element of feature engineering and a transparent, traceable path for debugging.</p><p>In the insurance industry, for example, deep learning allows providers to assess risks and optimize pricing strategies for their customers. Insurance companies can now determine their policies based on a wealth of data about the previous behavior and activities of their customers, but also increasingly based on the use of social media, reviews and other online sources. Deep learning may have allowed insurance companies to be more efficient. On the other hand, the obscurity of the technology has also raised many questions about fairness, discrimination and transparency in insurance programs.</p><p>If you are only interested in the output of your machine learning process, then deep learning may be a good option. However, if you want to have control over your process and insight into the inner mechanics of the decision-making process, then traditional machine learning may be a better idea.</p><h3>The most efficient algorithm for the job</h3><p>Deep learning enables us to perform tasks with unseen accuracy and speed. Unfortunately, this does not make deep learning the ideal solution for every project. Just know that, when data quality is a problem, when computing power is restricted, or when you need clarity about the intermediate stages of your development process, there are other options.</p><p>That’s why, at Kapernikov, we don’t head blindly into deep learning. We like to assess first what is the most efficient technology for the job. That could well be deep learning, but we have other tools in our toolkit as well.</p><p><em>Originally published at </em><a href="https://kapernikov.com/three-reasons-why-you-shouldnt-rush-blindly-into-deep-learning/"><em>https://kapernikov.com</em></a><em> on June 22, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26aa8b38b0d7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kapernikov/three-reasons-why-you-shouldnt-rush-blindly-into-deep-learning-kapernikov-26aa8b38b0d7">Three reasons why you shouldn’t rush blindly into deep learning — Kapernikov</a> was originally published in <a href="https://medium.com/kapernikov">Kapernikov</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>