<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Fabian R on Medium]]></title>
        <description><![CDATA[Stories by Fabian R on Medium]]></description>
        <link>https://medium.com/@fa.re.wal.fr?source=rss-1c2105b597b5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*qH-17iAHfMxDx-jv3M0SqA.jpeg</url>
            <title>Stories by Fabian R on Medium</title>
            <link>https://medium.com/@fa.re.wal.fr?source=rss-1c2105b597b5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 02:40:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@fa.re.wal.fr/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Understanding and Managing 1:1 vs. 1:*]]></title>
            <link>https://medium.com/mendix/understanding-and-managing-1-1-vs-1-relation-pitfalls-in-mendix-a7c333b7aa4c?source=rss-1c2105b597b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7c333b7aa4c</guid>
            <category><![CDATA[domain-modeling]]></category>
            <category><![CDATA[low-code]]></category>
            <category><![CDATA[how-to]]></category>
            <category><![CDATA[mendix]]></category>
            <category><![CDATA[tips]]></category>
            <dc:creator><![CDATA[Fabian R]]></dc:creator>
            <pubDate>Wed, 26 Feb 2025 11:57:22 GMT</pubDate>
            <atom:updated>2025-06-13T11:44:23.057Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aaWHjdM07EzJP5Bu7wX4Nw.png" /></figure><h4>Relation Pitfalls in Mendix</h4><h4>Mendix simplifies application development with its visual domain modeling capabilities. However, even seasoned developers can stumble when choosing between a 1:1 (one-to-one) and 1:* (one-to-many) association, leading to unexpected behavior that’s tricky to debug. Let’s explore a common issue, its implications, and how to handle it effectively, along with ideas for improving the experience in Mendix.</h4><h3>The Problem: Overwriting in 1:1 Relations</h3><p>Imagine a scenario with two entities: <strong>Process </strong>and <strong>Contact</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*F__aHaPo4Kukb1oZR_UD-A.png" /></figure><p>You set a 1:1 association between them, linking the <strong>Process</strong> P to a <strong>Contact </strong>C, defining this as the process owner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*_LwiA9Ery1tqqCeucKpsdg.png" /></figure><p>Later, you associate <strong>Process </strong>Q with <strong>Contact </strong>C. In a 1:1 relationship, Mendix enforces exclusivity — each instance of entity <strong>Process </strong>can only link to one instance of entity <strong>Contact</strong>, and vice versa. As a result, the original association between <strong>Process </strong>P and <strong>Contact </strong>C is silently overwritten, leaving P disconnected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*S8j042fixxCxbYEWILg81w.png" /></figure><p>This is the expected behavior of a 1:1 relation, but if your use case actually requires multiple <strong>Process </strong>entities to connect to a single <strong>Contact </strong>entity this can lead to data loss and unpredictable application behavior.</p><p>The errors are insidious because Mendix doesn’t throw an obvious warning when this happens — it’s just how the association works. Pages might suddenly show missing data, microflows might fail to retrieve expected objects, and tracking down the root cause becomes a headache, especially in complex applications.</p><h4>Why It Happens</h4><p>This issue often stems from a misunderstanding of the domain model. A 1:1 relation implies a strict, exclusive pairing — like a <strong>User </strong>and their unique <strong>Personal </strong>information. In contrast, a 1:* relation allows one entity to associate with multiple instances of another — a <strong>Contact </strong>can be the owner of many <strong>Process </strong>instances. If the real-world relationship is 1:* but modeled as 1:1, the system will overwrite associations as new links are made.</p><h4>Handling the Issue</h4><ol><li><strong>Audit Your Domain Model</strong>: Start by revisiting your entities and their real-world relationships. Ask: <br>- Can one instance of <strong>Entity Alpha</strong> (e.g. Process) be associated with <em>only one</em> instance of <strong>Entity Beta</strong> (e.g., Contact),<br>- And can one instance of <strong>Entity Beta</strong> (Contact) be associated with <em>only one</em> instance of <strong>Entity Alpha</strong> (Process)?<br>Only if both questions can be answered with yes, switch to a 1:1 association.</li><li><strong>Data Migration</strong>: If you’ve already built data with the wrong association, you’ll need to migrate it (create a microflow to reassign existing associations to the new 1:* structure). In some cases the original data may be lost.</li><li><strong>Validation Logic</strong>: If a 1:1 relation was introduced on purpose you can add validation in your application to prevent unintended changes.</li></ol><h4>Is there a better approach?</h4><ul><li><strong>General Best Practices</strong>: Always map out entity relationships before modeling in Mendix. Use real-world examples (e.g., “Can two users share a profile?”) to clarify cardinality. In most cases a 1:* relation is the right choice. Document your assumptions in the domain model using annotations to guide future developers.</li><li><strong>Platform Enhancement</strong>: Mendix could improve this experience by adding optional runtime warnings or consistency checks. For instance, when a 1:1 association is about to be overwritten, a warning could be logged. Alternatively, Studio Pro could offer a “relationship advisor” tool that analyzes your domain model and suggests 1:* over 1:1 when it detects potential cardinality mismatches based on usage patterns.</li></ul><h4>Conclusion</h4><p>The 1:1 vs. 1:* mix-up in Mendix is a classic gotcha that highlights the importance of aligning your domain model with reality. By auditing your design, migrating data, and adding safeguards, you can resolve and prevent these quirks.</p><p>Looking forward, a little platform polish — like smarter warnings — could make this even smoother. For now, a keen eye and proactive testing are your best tools to keep unexpected behavior at bay.</p><h4>Read more</h4><ul><li><a href="https://docs.mendix.com/refguide/domain-model/">Data in the Domain Model</a></li><li><a href="https://docs.mendix.com/refguide/associations/">Associations</a></li><li><a href="https://docs.mendix.com/refguide/association-member-properties/">Association Tab Properties</a></li></ul><p><strong><em>From the Publisher -</em></strong></p><p><em>Inspired by this article to bring your ideas to life with Mendix? Sign up for a </em><a href="https://signup.mendix.com/?utm_medium=content_syndication&amp;utm_source=medium.com&amp;utm_id=11172331"><em>free account</em></a><em>! You’ll get instant access to the </em><a href="https://academy.mendix.com/link/home"><em>Mendix Academy,</em></a><em> where you can start building your skills.</em></p><p><em>For more articles like this one, visit our </em><a href="https://medium.com/mendix"><em>Medium page</em></a><em>. And you can find a wealth of instructional videos on our </em><a href="https://www.youtube.com/c/MendixCommunity/community"><em>community YouTube page</em></a><em>.</em></p><p><em>Speaking of our community, join us in our </em><a href="https://join.slack.com/t/mendixcommunity/shared_invite/zt-hwhwkcxu-~59ywyjqHlUHXmrw5heqpQ"><em>Slack community channel</em></a><em>. We’d love to hear your ideas and insights!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7c333b7aa4c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mendix/understanding-and-managing-1-1-vs-1-relation-pitfalls-in-mendix-a7c333b7aa4c">Understanding and Managing 1:1 vs. 1:*</a> was originally published in <a href="https://medium.com/mendix">Mendix Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Nothing but Mendix Lists]]></title>
            <link>https://medium.com/mendix/nothing-but-mendix-lists-ca3b44395c43?source=rss-1c2105b597b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca3b44395c43</guid>
            <category><![CDATA[tips]]></category>
            <category><![CDATA[empty-check]]></category>
            <category><![CDATA[lists]]></category>
            <category><![CDATA[mendix]]></category>
            <category><![CDATA[low-code]]></category>
            <dc:creator><![CDATA[Fabian R]]></dc:creator>
            <pubDate>Wed, 18 Sep 2024 07:41:02 GMT</pubDate>
            <atom:updated>2025-02-24T12:37:29.826Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Nothing but Mendix Lists (Banner Image)" src="https://cdn-images-1.medium.com/max/1024/1*rfkauUq5eCz4w550PjXsHA.png" /><figcaption>Nothing but Mendix Lists</figcaption></figure><h4>In a developer&#39;s life you come to think about the emptiness of things.</h4><p>This article is not a philosophical essay — it’s about the <em>empty </em>expression within Mendix and what that implies to the usage of lists.</p><p>The <em>empty </em>expression in Mendix relates to the <em>null </em>value in Java. If something is <em>empty </em>in Mendix, no actual value or data object is present. For example an integer variable which is <em>empty </em>is not the same as a variable that is <em>0</em>.</p><h4>Creating empty lists</h4><p>The same principle that applies to numbers and strings also applies to lists. Let’s consider a simple example where a submicroflow returns a list of roles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*u-qPz_5H-7FvoL4rDDIBvg.png" /></figure><p>Based on a condition, the flow either returns a retrieved list, or it directly returns <em>empty</em>.</p><p>While the $UserRoleList is an instance of a list which may have 0 or more entries, the second expression is not an actual list.</p><h4>Working with empty lists</h4><p>The return value of this subflow can be used in list operations in microflows. E.g. you can add something to the list:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/174/1*agkRGF8BTy-PRXSh07Ymsg.png" /></figure><p>Or you can add the list itself to another list. All these operations are passed by the runtime without any error.<br>However, we already stated that <em>empty </em>is not really a list. Since it isn’t a list you <strong>cannot add something</strong> to it (the operation is just skipped).</p><p>This behaviour can lead to unexpected behaviour if you modify a list returned from a subflow for later operations, since all modifications will not have any effect on the end result.</p><h4>Impact on other processes and actions</h4><p>While the Mendix runtime usually handles <em>empty </em>the same way as a list with no entries there may be exceptions to this.</p><p>Java actions operating on lists might not be doing an empty check before calling a list method, resulting in an exception.</p><p>The workflow assignment microflow in the Mendix workflow engine will cause an error, if the returned value is <em>empty</em>.</p><h4>Conclusion</h4><p>When working with lists in Mendix you should be aware, that empty is not a list and how this may impact other application logic. It is recommended to not use the <em>empty </em>expression as a return value for a list type.</p><h4>Read more</h4><ul><li><a href="https://docs.mendix.com/refguide/special-checks/">Special Checks</a></li><li><a href="https://docs.mendix.com/refguide/working-with-lists-in-a-microflow/">Working with Lists in a Microflow</a></li></ul><p><strong><em>From the Publisher -</em></strong></p><p><em>Inspired by this article to bring your ideas to life with Mendix? Sign up for a </em><a href="https://signup.mendix.com/?utm_medium=content_syndication&amp;utm_source=medium.com&amp;utm_id=11172331"><em>free account</em></a><em>! You’ll get instant access to the </em><a href="https://academy.mendix.com/link/home"><em>Mendix Academy,</em></a><em> where you can start building your skills.</em></p><p><em>For more articles like this one, visit our </em><a href="https://medium.com/mendix"><em>Medium page</em></a><em>. And you can find a wealth of instructional videos on our </em><a href="https://www.youtube.com/c/MendixCommunity/community"><em>community YouTube page</em></a><em>.</em></p><p><em>Speaking of our community, join us in our </em><a href="https://join.slack.com/t/mendixcommunity/shared_invite/zt-hwhwkcxu-~59ywyjqHlUHXmrw5heqpQ"><em>Slack community channel</em></a><em>. We’d love to hear your ideas and insights!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca3b44395c43" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mendix/nothing-but-mendix-lists-ca3b44395c43">Nothing but Mendix Lists</a> was originally published in <a href="https://medium.com/mendix">Mendix Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Face Detection with Mendix ML Kit]]></title>
            <link>https://medium.com/mendix/face-detection-with-mendix-ml-kit-3bba19d944f7?source=rss-1c2105b597b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/3bba19d944f7</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[how-to]]></category>
            <category><![CDATA[mendix]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[low-code]]></category>
            <dc:creator><![CDATA[Fabian R]]></dc:creator>
            <pubDate>Thu, 23 Nov 2023 08:28:50 GMT</pubDate>
            <atom:updated>2024-02-20T11:45:58.914Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Face Detection with Mendix ML Kit (Banner Image)" src="https://cdn-images-1.medium.com/max/1024/1*q5rSZCsXQO4d3sUN6FlWcQ.png" /><figcaption>Face Detection with Mendix ML Kit</figcaption></figure><h4>The <a href="https://docs.mendix.com/refguide/machine-learning-kit/using-ml-kit/">Mendix ML Kit</a> allows importing pre-trained machine learning models (ML models) using the <a href="https://onnx.ai/onnx/intro/">ONNX format</a>.</h4><p>A pre-trained model is a machine learning model that is already trained with a training data set and includes all the parameters for executing the model. A common use for ML models is computer vision, after importing such an ONNX model in Studio Pro it can be executed in a microflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9g23QBf18sQ06NJTnNnnNA.gif" /><figcaption>People in a meeting from <a href="http://shuoyang1213.me/WIDERFACE/">http://shuoyang1213.me/WIDERFACE/</a></figcaption></figure><p>This article explains how to integrate a specific face detection model in a Mendix application. A significant part of this step is to add the required pre- and postprocessing steps as Java code. Detailed Java code examples are presented here:</p><p><a href="https://github.com/mendix/mlkit-example-app">GitHub - mendix/mlkit-example-app: Demo for Mendix MLKit</a></p><h4>Face Detection model</h4><p>The ML model we want to use is a lightweight face detection model from the ONNX model zoo. There are two models available which differ in the input resolution. We choose the RFB-640 model which expects an RGB image with a resolution of 640x480 pixels. The output of this model are two lists containing boxes and corresponding scores. A box is defined by its left, top, right, and bottom border position.</p><p><a href="https://github.com/onnx/models/tree/main/vision/body_analysis/ultraface">models/vision/body_analysis/ultraface at main · onnx/models</a></p><p><strong>Model mapping in Mendix</strong></p><p>Importing and invoking an ONNX ML model is done in a few simple steps which are well described in the Mendix documentation. The face recognition model generates the following input and output mapping for calling the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*iM_Mb_20kUmod1hi2pcpbQ.png" /><figcaption>Input mapping for the RFB-640 model</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*t2LtWKY8hanx477tEOoPhQ.png" /><figcaption>Output maping for the RFB-640 model</figcaption></figure><p>For calling the model in Mendix we create an instance of the input entity and pass it to the <em>Call ML model</em> action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/1*12ziD62zeIvXSABNyXQ1Wg.png" /></figure><p>As we can see the input and output data require a special format here. To convert that data into a format we can work some pre- and postprocessing is required.</p><h4>Pre- and Postprocessing</h4><p>There are no standard processing methods available in Mendix for this specific problem. The processing required here is thus done in two Java actions that handle the heavy lifting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*WihQ8T-u9N5Pri6cxqslLw.png" /></figure><h4>Mapping input data</h4><p>The image must be passed as binary data containing a tensor with the shape [1,3,480,640]. You may read more about tensors and shapes here:</p><p><a href="https://www.tensorflow.org/guide/tensor">Introduction to Tensors | TensorFlow Core</a></p><p>For running the model the following steps are required:</p><ul><li>Scale the input image to 640 x 480 pixels in RGB format.</li><li>Convert the image to a matrix with matching shape and normalized values.</li><li>Encode the matrix to a format that can be passed to the ML model.</li></ul><h4>Image reading and scaling</h4><p>For scaling the image we will use the OpenCV library with its Java wrapper. Since OpenCV requires native libraries we follow the same approach as the Mendix ML learning kit and use a library that is packaged with the native libraries.</p><p>Before we can use any OpenCV functionality we need to load the shared libraries. This can be done in a static code section.</p><pre>// BEGIN EXTRA CODE<br>static {<br>  nu.pattern.OpenCV.loadShared(); //OpenCV initialization<br>}</pre><p>We can read an input image and convert it using the OpenCV resize method.</p><pre>// read the image into a matrix<br>InputStream istream = com.mendix.core.Core.getFileDocumentContent(getContext(), sourceImage.getMendixObject());<br>MatOfByte matOfByte = new MatOfByte(istream.readAllBytes());<br>Mat sourceImage = Imgcodecs.imdecode(matOfByte, Imgcodecs.IMREAD_COLOR);<br><br>// resize the image to the target size<br>Mat rim = new Mat();<br>Imgproc.resize(sourceImage, rim, new Size(640,480));</pre><h4>Convert the image to a Matrix</h4><p>The next step is to convert the image to a matrix and normalize the values. The face detection model expects the input data to be scaled to the range [-1;1]. This conversion is done with the following code.</p><pre>float[][][][] inputArray = new float[1][3][480][640];<br>for(int i = 0; i &lt; 480; i++) {<br> for(int j = 0; j &lt; 640; j++) {<br>  for(int k = 0; k &lt;= 2; k++) {<br>   double[] rawValue = rim.get(i, j);<br>   float normalizedValue = ((float) (rawValue[Math.abs(k - 2)] - 127) / 128);<br>   inputArray[0][k][i][j] = normalizedValue;<br>  }<br> }<br>}</pre><h4>Encode the Matrix for processing</h4><p>The resulting inputArray matrix now must be passed to the model. The Mendix ML Kit already provides some helper functions for base64 encoding this data. The result from this Java action is written to the input object of the model.</p><pre>InputStream is = MLKit.toInputStream(inputArray);<br>String base64data = MLKit.toBase64(is);</pre><h4>Mapping output data (postprocessing)</h4><p>The model generates two multidimensional matrices (scores and boxes) which are binary encoded in base64. For handling that output we use a postprocessing java action. For decoding the raw data we can again use a helper function from the MLKit.</p><pre>float[][][] _box = new float[1][17640][4];<br>InputStream isb = MLKit.fromBase64(rfb640Output.getBoxes());<br>MLKit.toArray(isb, _box);</pre><p>The same code is used for decoding the calculated scores. The shape of the output data is [1,17640,4]. That means the model always generates 17640 candidates for potential faces. To identify a final list of detected faces some additional filtering is required which is not done by the model itself.</p><h4>Filtering model output</h4><p>You can find the example Python code for this filtering in the <a href="https://github.com/onnx/models/blob/main/vision/body_analysis/ultraface/demo.py">ONNX model repository</a>.</p><p>The postprocessing consists of two steps. Since we cannot easily execute this Python code from the Java runtime it must be rewritten to Java action. The first step is to filter results by their score and limit the candidates to a smaller set. The result <em>sortedCandidateIdx</em> is a set of indices referring to candidates.</p><pre>List&lt;Integer&gt; sortedCandidateIdx = IntStream.range(0, 17640)<br>.filter(idx -&gt; score[idx][1] &gt; detectionThreshold)<br>.boxed().sorted(new Comparator&lt;Integer&gt;() {<br>    @Override<br>    public int compare(Integer idx1, Integer idx2) {<br>    return Float.compare(score[idx2][1] ,score[idx1][1]);<br>  }<br>}).limit(candidates).collect(Collectors.toList());</pre><p>The score is an indicator of how likely a specific box contains a face. These candidates are now valid detections of a face in the input image. However, these boxes will have significant overlap.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f4DOS8kESUgv3NYbDpFqvg.png" /></figure><p>If you use this data unfiltered there will be multiple boxes indicating a detected face in the same area of the image as shown above. As a final output, we want to have a single box per face.</p><h4>Non-maximum suppression</h4><p>The method used for filtering these boxes is called <a href="https://learnopencv.com/non-maximum-suppression-theory-and-implementation-in-pytorch/">non-maximum suppression (NMS)</a> and uses the IoU (intersection over union) to determine boxes that overlap. A Java equivalent for this process is given here:</p><pre>for (Integer idx1 : sortedCandidateIdx) {<br>  if (!finalCandidates.isEmpty()) {<br>    boolean hasIoUAboveThreshold = false;<br>    for (Integer idx2 : finalCandidates) {<br>      float l = Math.min(box[idx1][0], box[idx2][0]);<br>      float t = Math.min(box[idx1][1], box[idx2][1]);<br>      float r = Math.max(box[idx1][2], box[idx2][2]);<br>      float b = Math.max(box[idx1][3], box[idx2][3]);<br>      if (l&gt;=r || t&gt;=b) continue;<br>      <br>      float intersection = (r-l)*(b-t);<br>      float area1 = (box[idx1][2]-box[idx1][0])*(box[idx1][3]-box[idx1][1]);<br>      float area2 = (box[idx2][2]-box[idx2][0])*(box[idx2][3]-box[idx2][1]);<br>      <br>      float iou = intersection / (area1 + area2 - intersection + 0.0000001f);<br>      if (iou&lt;iouThreshold) continue;<br>      hasIoUAboveThreshold=true;<br>      break;<br>    }<br>    if (hasIoUAboveThreshold) continue;<br>  }<br>  finalCandidates.add(idx1);<br>}</pre><p>With this method in place, the resulting candidates are now clearly separated into 4 distinct sections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7o-NO5sf81TbNeko3qX7Lg.png" /></figure><h4>Using result data</h4><p>The actual result data at this point is still a set of coordinates for boxes in a normalized space [0,1]x[0,1]. We can use methods from OpenCV to draw the results onto the source image as here or we can pass the information to any consecutive process.</p><pre>for (Integer idx : finalCandidates) {<br>  Rect rec = new Rect(new Point(box[idx][0] * width, box[idx][1] * height), new Point(box[idx][2] * width, box[idx][3] * height));<br>  Imgproc.rectangle(sourceImage, rec, color, linethickness);   <br>};</pre><h4>See it in action</h4><p>You can <a href="https://facedetection111-sandbox.mxapps.io/">test the face detection model in the sandbox app</a> (please do not upload pictures without permission).</p><h4>Conclusion</h4><p>Importing and running a pre-trained ONNX model has become so much easier through the Mendix ML kit. Mendix provides the kind of interfaces we are already used to for mapping input and output data to the actual model.</p><p>You will still need the expertise for the pre-and postprocessing of data. Some of these steps are common across similar problems and there’s a good chance that there will be more methods available that can be used out of the box.</p><p>Some of the operations required were a challenge the integration of the model into the platform made it easy to quickly progress and get a working solution within a few days.</p><h3>Further Reference</h3><ul><li><a href="https://docs.mendix.com/refguide/machine-learning-kit/using-ml-kit/">Using ML Kit</a></li><li><a href="https://onnx.ai/onnx/intro/">Introduction to ONNX - ONNX 1.16.0 documentation</a></li><li><a href="https://github.com/mendix/mlkit-example-app">GitHub - mendix/mlkit-example-app: Demo for Mendix MLKit</a></li><li><a href="https://docs.opencv.org/4.7.0/">OpenCV: OpenCV modules</a></li><li><a href="https://github.com/openpnp/opencv/tree/develop">GitHub - openpnp/opencv: OpenCV Java bindings packaged with native libraries, seamlessly delivered as a turn-key Maven dependency.</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3bba19d944f7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mendix/face-detection-with-mendix-ml-kit-3bba19d944f7">Face Detection with Mendix ML Kit</a> was originally published in <a href="https://medium.com/mendix">Mendix Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Efficiently stream large data files on Mendix]]></title>
            <link>https://medium.com/mendix/efficient-stream-processing-of-large-data-files-on-mendix-310537866189?source=rss-1c2105b597b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/310537866189</guid>
            <category><![CDATA[low-code]]></category>
            <category><![CDATA[mendix]]></category>
            <category><![CDATA[how-to]]></category>
            <category><![CDATA[data-import-and-export]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Fabian R]]></dc:creator>
            <pubDate>Thu, 05 Jan 2023 15:32:38 GMT</pubDate>
            <atom:updated>2023-04-20T11:36:52.490Z</atom:updated>
            <content:encoded><![CDATA[<h4>This article shows how to stream process large files in <a href="https://mendix.com/">Mendix</a>.</h4><figure><img alt="Efficiently stream large data files on Mendix — A hard drive repeated infinitely as if moving away from the viewer" src="https://cdn-images-1.medium.com/max/1024/1*H5fXZIXf85F_YyvXi2dteg.png" /><figcaption>Efficiently stream large data files on Mendix — A hard drive repeated infinitely as if moving away from the viewer</figcaption></figure><h4>Let’s begin with the story first. It started some years ago when Mendix&#39;s offline capabilities were first released, fresh on the market, with the vision of building an offline app for a third-party system.</h4><p>To make data available offline it needs to exist in the Mendix database. Therefore, data in the remote system must be synchronized to our Mendix app. After several iterations, we reached a solution using a message queue to send any change directly to the Mendix app.</p><p>As part of a setup process, the app must be bootstrapped with the full data from the source system — about <strong>30 tables</strong> with up to <strong>one million records</strong>. This is done for initial configuration and at regular intervals for restoring data on development systems. This is where the problem started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g8IslYRemF32AqhG1Stq2Q.png" /></figure><h3>Transporting data from the source system</h3><p>We investigated some options for exporting the data from the source system into the Mendix app. The easiest and most efficient solution we found consisted of exporting the data to JSON files. Compressing these files with a total size of 1.5GB to an 80MB ZIP file allowed us to easily transfer them into our app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*01iIRrT908oG8_C9JLc8AQ.png" /></figure><p>The problem we had to solve was how to process all of the compressed data without rewriting the core processing logic that was already put in place for the message queue. The process would need to consist of these steps:</p><ul><li>Upload zipped file to App</li><li>Unzip JSON files</li><li>Read all entries from each JSON file</li><li>Process individual entries</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*5CP2viM6_n16Cbi4DOhBxw.png" /></figure><p>Due to the amount of data, this process would not only <strong>run for some time</strong>, but it might also <strong>use up a considerable amount of memory and storage</strong> resources. We also had to consider that this data amount only applied to our first app instance. Other instances were expected to have <strong>3–5 times that</strong> <strong>amount of data.</strong></p><h3>Memory and storage efficient processing</h3><p>Sequentially executing each step after another turned out to be not feasible and we decided to process the data <strong>as a stream directly from the ZIP file</strong>. With this processing approach, <strong>neither the source JSON data nor any intermediate data are ever fully kept in memory, storage, or the database</strong>. This keeps the process scalable and stable.</p><h3>Technical Implementation — from code to low-code</h3><p>This process could not be implemented with Mendix alone without using intermediate data storage. When importing JSON, the data would either be written to a persistent table or kept in memory. With the help of a Java action, it is possible to integrate the file reading process smoothly into a Mendix flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*7lpuXsqPSOAMApbJmkIVrA.png" /></figure><p>A simplified representation of the process is shown above. The zipped file is passed to our Java action. This will call the selected processing microflow for each single JSON entry.</p><h3>Reading the zipped file</h3><p>The zip file is opened as a stream with standard Java. To avoid problems with connectivity to the storage service, we copy the file to the server’s temporary file folder.</p><pre>Files.copy(istream, tempPath,StandardCopyOption.REPLACE_EXISTING);<br>InputStream tmpstream = Files.newInputStream(tempPath);<br>ZipInputStream zipstream = new ZipInputStream(tmpstream);<br>ZipEntry entry = zipstream.getNextEntry();</pre><p>By reading the ZipEntry <strong>we can move the stream to a specific JSON file</strong> for further processing. You can also inspect each entry to <strong>look up a specific item in the zipped file.</strong></p><h3>Reading JSON files to individual JSON objects</h3><p><strong>Processing of the JSON content</strong> is now done using the <strong>Jakarta JSON Library</strong>. The InputStream from the zipped file is again processed in a streaming approach by <strong>using a JsonParser</strong>.</p><pre>JsonParser parser = Json.createParser(zipstream);<br>JsonParser.Event event = parser.next();<br>...<br>JsonObject obj = parser.getObject();</pre><p>The JsonParser allows us to traverse the data one element at a time. In our case, the JSON data consisted of a nested structure containing a list of objects. Surrounding data is skipped until the relevant “body” identifier is found. Then we read each object from the list directly to a JsonObject.</p><pre>{<br>  &quot;body&quot;: [ {...}, {...} ]<br>}</pre><p>The extracted JsonObject is then passed to the processing microflow.</p><h3>Transaction Handling</h3><p>Now we have managed to process all the data in the stream, <strong>without ever</strong> <strong>loading</strong> <strong>a single object into memory</strong>. Implementing this process that way has still one drawback. The full processing would be running in <strong>one single database transaction</strong>. This would be putting <strong>an intense load on the database</strong> and also come with the <strong>risk of failure</strong> if the process is interrupted before fully finishing.</p><p>To solve that problem, <strong>we introduce separate transactions</strong> for every batch of N objects. All database actions are executed in that individual transaction.</p><pre>// create a separate transaction context for batches<br>ISession session = getContext().getSession();<br>IContext localContext = session.createContext();<br>localContext.startTransaction(); <br>// do something... <br>localContext.endTransaction();</pre><h3>Conclusion</h3><p>This shows the essential components for setting up a memory and storage efficient process for importing large data files in a Mendix application. Additional mechanisms like <strong>error handling</strong> or <strong>resume on failure are left out</strong> to better outline the main concept. A stable implementation <strong>still needs to be complemented with such features.</strong></p><p>This approach is shown in the context of processing JSON data but <strong>can be adapted to any other format that supports stream processing of source data</strong> (e.g. CSV or XLS).</p><h4>Read More</h4><ul><li><a href="https://docs.mendix.com/refguide/java-actions/">Java Actions</a></li><li><a href="https://jakarta.ee/specifications/jsonp/2.1/">Jakarta JSON Processing 2.1 | The Eclipse Foundation</a></li></ul><p><em>From the Publisher -</em></p><p><em>If you enjoyed this article you can find more like it on our </em><a href="https://medium.com/mendix"><em>Medium page</em></a><em>. For great videos and live sessions, you can go to </em><a href="https://www.mendix.com/live/"><em>MxLive</em></a><em> or our community </em><a href="https://www.youtube.com/c/MendixCommunity/community"><em>Youtube pag</em></a><em>e.</em></p><p><em>For the makers looking to get started, you can sign up for a </em><a href="https://signup.mendix.com/link/signup/?source=direct"><em>free account</em></a><em>, and get instant access to learning with our </em><a href="https://academy.mendix.com/link/home"><em>Academy</em></a><em>.</em></p><p><em>Interested in getting more involved with our community? Join us in our </em><a href="https://join.slack.com/t/mendixcommunity/shared_invite/zt-hwhwkcxu-~59ywyjqHlUHXmrw5heqpQ"><em>Slack community channel</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=310537866189" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mendix/efficient-stream-processing-of-large-data-files-on-mendix-310537866189">Efficiently stream large data files on Mendix</a> was originally published in <a href="https://medium.com/mendix">Mendix Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>