<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Tech at Trax - Medium]]></title>
        <description><![CDATA[The Trax Engineering Blog - Medium]]></description>
        <link>https://medium.com/tech-at-trax?source=rss----ce231112e8a0---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Tech at Trax - Medium</title>
            <link>https://medium.com/tech-at-trax?source=rss----ce231112e8a0---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 18:34:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/tech-at-trax" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Improving class representation in training dataset — from greedy algorithm to linear programming]]></title>
            <link>https://medium.com/tech-at-trax/improving-class-representation-in-training-dataset-from-greedy-algorithm-to-linear-programming-34aa7b79b944?source=rss----ce231112e8a0---4</link>
            <guid isPermaLink="false">https://medium.com/p/34aa7b79b944</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[linear-programming]]></category>
            <category><![CDATA[algorithms]]></category>
            <dc:creator><![CDATA[Tamar David]]></dc:creator>
            <pubDate>Tue, 31 Jan 2023 13:33:38 GMT</pubDate>
            <atom:updated>2023-01-31T13:33:38.731Z</atom:updated>
            <content:encoded><![CDATA[<h3>Improving class representation in training dataset — from greedy algorithm to linear programming</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lzUPdsGjpT3BIX6KFAJ6Rw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@luckaslh?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Lucas Hobbs</a> on <a href="https://unsplash.com/photos/gEHSvDMHnEA?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>The problem</h3><p>Occasionally when we create a dataset for training, we don’t have enough samples for all the classes. Sometimes we have many classes with a low amount of samples (long tail) so the model won’t be able to classify those classes well. Even if we train the model several times or try different models we still get the same poor results - because the problem here is that we don’t have enough examples from those classes. How can we collect more data from those specific classes?</p><p>Let&#39;s take this example: we want to classify the colors of cars in a parking lot. We have images of the parking lot and our model already classified those images. Still, it doesn’t recognize well some of the colors (for example the results for pink and red are poor), so how can we improve our model performance on those colors and as a result improve the model?</p><p>The first choice will be to collect more data on those classes, but in this case, we want to be very accurate with the data we collect. Meaning, we want to collect exactly the data from the classes we miss, and don’t collect data from the classes we already have enough data on them. So how we can collect this kind of data?</p><h3>First solution — the Greedy algorithm</h3><p>Let’s assume we have some images that we can revalidate part of the data manually, and we have some suspects on where to find each color of the car.</p><p>One option to improve the number of samples from each class is to use a greedy algorithm with these steps:</p><ol><li>Calculate for each color how many samples are missing for the low limit we assume we need for good recognition of this color</li><li>For each image, we calculate how many samples we suspect there are in this image from each class</li><li>Then we sort these images from the maximum number of tags per image to the lowest</li><li>We will take the first image and calculate again how many we need from each color (after reducing the number of tags per color in the image we select)</li><li>We will return to steps 2–4 until we have enough tags per each class or we don&#39;t have more images</li></ol><p>The problem with the algorithm in the production environment is that:</p><ol><li>There is a limited number of images we can collect (~5K) because it is difficult to sort the images above a specific limit</li><li>Each step takes a lot of time so when we try to run it on a reasonable amount of classes it takes more than 12 hours which is too long time to run it</li></ol><p>For these reasons, we need to think of another solution…</p><h3>Second solution — the linear programming algorithm</h3><p>In order to solve this problem in another way, we tried to define this problem as a linear programming problem, with the following steps:</p><ol><li>Calculate the colors needed and their missing appearances</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*pRRw3D2wCexEbtaTAqQ60g.png" /></figure><p>2. Find the images in which we suspect those colors appear in them (based on the current model)</p><p>3. Create this matrix (for example):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/1*8g3ePv3clEfy7oRLo7vh6A.png" /></figure><p>4. Calculate for each color — the minimum between how many appearances missing to the actual amount of this color we have</p><p>* For example in the table above:</p><p>For the color blue - we need 10 appearances for the training, and we can have an additional 20 from those images then the minimum is 10. For the color orange we will need 35 appearances, but the actual number we can get is 33 so the minimum is 33.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*w7VpJBcAwenJr89Pab5isQ.png" /></figure><p>5. Define and solve the linear problem</p><p>The output:</p><p>colors list, chosen images to validate again</p><h3>Linear problem definition</h3><p><em>Linear Programming (LP)</em> is a type of mathematical optimization, in a selection of variables that make a function takes maximum or minimum value, which to be an optimized function, and a linear relationship represents all requirements. (<a href="https://medium.com/@mengsaylms/a-brief-introduction-to-linear-programming-2107e769a1fe">A brief introduction to Linear Programming</a>, <a href="https://medium.com/codex/optimization-and-linear-programming-f0b1faab1e5a">Optimization, and Linear Programming</a>)</p><p>Lets us express this problem in a mathematical way:</p><p>We will define the chosen images:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/1*Bps4OTEa97rNSFYH6bFk0Q.png" /></figure><p>If Mi=1 we will select this image and if Mi=0 we won’t</p><p>We also define this variable:</p><p>Cji- the appearance of a car with the color j (j=1,2,…m) in image i</p><p>So we can define this minimum target function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/133/1*3xx9n6LAKcWiPfr5BnuqBA.png" /></figure><p>And these are the constraints:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*iKBh8A7lDpokhr6urxMaXg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*x8aeIBIj4_isM0HDBBtnOw.png" /></figure><p>….</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*v93kdKhDZ0C2VGFeM4jedQ.png" /></figure><p>In the above example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*_avz50kICFxDKXDZHp-Nmw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/229/1*22vaZye95QkZldib2pMipA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/240/1*DXXT2X52taojbjHbphGfyw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/179/1*Lh_oQ_jxDhLZKG-6pET9vw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/189/1*ake1m_q3bBP-RKHmaVMH0w.png" /></figure><p>When the problem is solved, we will get a list of images with which colors we need to recheck and validate. After it, we can retrain the engine again and examine if the change in the training dataset influences the engine&#39;s ability to recognize those cars.</p><h3>Results</h3><p>In the trials we did, we saw a significant improvement in the accuracy of the engine, its ability to recognize weak colors got bigger and the total accuracy get improved by 5%</p><h3>Future Work</h3><p>In order to improve this algorithm, we can consider the following options:</p><p>Instead of finding an images list and a colors list- that are separate lists (images list, colors list), we can find a combined list of the image and color ((image 1- colors),(image 2- colors),….,(image n- colors))</p><p>In this way, we can get more accurate color appearances per image. We won’t validate more colors than what we want- because, in the separate lists situations, we have at the end more tags than we need, for example, a situation when we send an image that was selected because of the red color. Still, it includes also blue color and this color is also on the list of colors. Therefore in order to solve this, we will need to define a variable for color j in image i.</p><h3>Summary</h3><p>In this post, I show how we initially solved a problem with a naive, greedy algorithm. Since it did not perform well enough we replaced it with a linear programming algorithm. The linear algorithm has the ability to run in a reasonable time, has a big positive influence on the training dataset, and improves the engine results.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34aa7b79b944" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-at-trax/improving-class-representation-in-training-dataset-from-greedy-algorithm-to-linear-programming-34aa7b79b944">Improving class representation in training dataset — from greedy algorithm to linear programming</a> was originally published in <a href="https://medium.com/tech-at-trax">Tech at Trax</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to save Heroku dyno formation with the CLI]]></title>
            <link>https://medium.com/tech-at-trax/how-to-save-heroku-dyno-formation-with-the-cli-6f860fd63db?source=rss----ce231112e8a0---4</link>
            <guid isPermaLink="false">https://medium.com/p/6f860fd63db</guid>
            <category><![CDATA[command-line]]></category>
            <category><![CDATA[terminal]]></category>
            <category><![CDATA[cli]]></category>
            <category><![CDATA[heroku]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <dc:creator><![CDATA[Loïc CHOLLIER]]></dc:creator>
            <pubDate>Wed, 23 Nov 2022 10:42:23 GMT</pubDate>
            <atom:updated>2022-11-23T10:42:23.931Z</atom:updated>
            <content:encoded><![CDATA[<p>At ̶Q̶u̶r̶i̶ <a href="http://traxretail.com/">Trax Retail</a> (Quri was acquired by Trax Retail in January 2018) we use Heroku to power most of our back-end systems. <em>(tl;dr </em><a href="https://medium.com/quri-engineering/how-to-save-heroku-dyno-formation-with-the-cli-6f860fd63db#1403"><em>down there</em></a><em>)</em></p><p>One of the powerful features with Heroku is that you can easily scale up and down your different services independently. Here is what the Heroku Resources Dashboard looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t5Mqcxvfw-3mT_IbTJDjbw.png" /><figcaption>Screenshot of the Dyno Formation UI</figcaption></figure><p>Now let’s imagine you want to do some maintenance on your app, or simply shutdown your staging or sandbox servers during the night.</p><p>For every single row in the above the screenshot, you would need to turn down the dyno count to 0, then click confirm:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u_dDA8FDrhlhxWogWpNvDg.png" /></figure><p>This gets cumbersome, especially when you are scaling the dynos back up and would need to look at a screenshot like the first one to remember what was the <em>dyno formation.</em></p><h4>Here comes the <a href="https://devcenter.heroku.com/categories/command-line">Heroku CLI</a>:</h4><p>Most — if not all — the actions you can perform in the UI can be performed with the command line tool:</p><pre>$ heroku help<br>Usage: heroku COMMAND [--app APP] [command-specific-options]</pre><pre>Primary help topics, type &quot;heroku help TOPIC&quot; for more details:</pre><pre>  addons    #  manage add-on resources<br>  apps      #  manage apps (create, destroy)<br>  auth      #  authentication (login, logout)<br>  config    #  manage app config vars<br>  domains   #  manage domains<br>  logs      #  display logs for an app<br>  ps        #  manage dynos (dynos, workers)<br>  releases  #  manage app releases<br>  run       #  run one-off commands (console, rake)</pre><p>We will focus here on the heroku ps command that lets us manage the dynos and workers. With heroku ps:scale, we’re able to get the same screenshot that the first one I shared describing the <em>dyno formation</em> but as text:</p><pre>$ heroku ps:scale</pre><pre>clock=1:Standard-1X console=0:Standard-1X mq_workers_high_priority=1:Performance-L mq_workers_low_priority=2:Performance-M mq_workers_photo=10:Standard-2X rake=0:Standard-1X renderer=1:Performance-L scheduler=1:Standard-1X web=3:Performance-M</pre><p>Now heroku ps:scale can also take as input the exact same format it outputs. This opens an interesting perspective: instead of having to manually scale every single dynos or worker, we would be able to automate that.</p><p>Saving your dyno formation to a text file can be achieved with:</p><pre>$ heroku ps:scale &gt; mydynoformation</pre><p>Scaling all dynos to zero with some regular expression magic becomes as <em>simple</em> as:</p><pre>$ heroku ps:scale $(heroku ps:scale | sed -E &#39;s/=([0-9]+):/=0:/g&#39;)</pre><p>Scaling back your app to it’s initial state becomes:</p><pre>$ heroku ps:scale `cat mydynoformation`</pre><h4>TLDR;</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f29418d3f1e76c59f2df146e296df93d/href">https://medium.com/media/f29418d3f1e76c59f2df146e296df93d/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f860fd63db" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-at-trax/how-to-save-heroku-dyno-formation-with-the-cli-6f860fd63db">How to save Heroku dyno formation with the CLI</a> was originally published in <a href="https://medium.com/tech-at-trax">Tech at Trax</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mirror Mirror on the Wall: BigTable to BigQuery Mirroring]]></title>
            <link>https://medium.com/tech-at-trax/mirror-mirror-on-the-wall-bigtable-to-bigquery-mirroring-fcf5bd5bb0f9?source=rss----ce231112e8a0---4</link>
            <guid isPermaLink="false">https://medium.com/p/fcf5bd5bb0f9</guid>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[configuration]]></category>
            <category><![CDATA[mirroring]]></category>
            <category><![CDATA[big-table]]></category>
            <dc:creator><![CDATA[Yaniv Bordeynik]]></dc:creator>
            <pubDate>Wed, 23 Nov 2022 10:42:03 GMT</pubDate>
            <atom:updated>2022-11-23T10:42:03.192Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9qfUGgzstJJDVhpOY4al-Q.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@gabcsika?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Gabor Szucs</a> on <a href="https://unsplash.com/s/photos/mirroring?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In the world of Big Data, there are cases when data needs to be saved in more than one database. It may sound wasteful, there may different reasons to do this:</p><ul><li>Different DB usage (direct access vs scanning).</li><li>Costs considerations regarding saving the data and querying it.</li><li>One may be more suitable for BI and reporting than the other.</li></ul><p>These are some of the considerations for writing the same data into more than one database. However, there are implications to this practice. When a micro service handles millions of events per day, you must need consider the message processing performance time, the memory usage, the connection to the DB’s and of course - lines of code.</p><p>Lately, I have been working on a project where I wrote data into BigQuery for BI needs. Later on there was a demand to write the same data to BigTable in order that another service will be able to read from it with direct access. First thing I did was to add another function that all it did was to write the same data into BigTable. After a while we noticed that more teams across the company was needed to write the data not only for BI and reporting use but also for real time purposes.</p><p>We thought on how we can make it better and decided on the concept of data mirroring. The idea was to create a new service and it will be responsible on the writing part.</p><p>This helped to solve our problem by:</p><ul><li>Reducing the message processing time in the queues.</li><li>Reduce memory usage.</li><li>Remove lines of codes.</li><li>Maintain code to write to two databases.</li></ul><p><strong>TLDR</strong> — Data Mirroring from BigTable to Bigquery by a serivce.</p><p>In our case the databases in use are BigQuery and BigTable. BigQuery and BigTable are both cloud native databases created by Google. While BigQuery is a data warehouse for relational structured data, commonly used for storing and analysing data, BigTable is a key-value store, NoSQL database, ideal for heavy reads &amp; writes and direct access.</p><p>The proposed mirroring method takes advantage of the fact<strong> that each time data is written to BigTable, an event is likewise published to a PubSub queue</strong>. The mirroring service will get the data from the pubsub and will determine (with the help of a configuration file) whether the event is configured to be mirrored into BigQuery.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tuARsHa-D-Ql9DO2oQ6NLA.png" /><figcaption>Service architecture (image by author)</figcaption></figure><p>When you have mutliple services and ETL processes which are all producing huge amounts of data, you want to save the data in a relational database to be able to query it later on for reports, dashboards and analysis. You may want to save the data as is and duplicate it (or just part of it).</p><p>When writing such a service, we want to make it as generic as possible so it can handle different format of events and to build it as a simple task for the user as much as possible without refactoring the service and deploying it every time new data needs to be written.</p><p>When the mirroring serivce gets a new event from the queue, it will check whether this event need to be handled or not.</p><p><strong>Configuration</strong></p><p>In order to manage the services and destination tables orchestration, there is a configuration file which is stored in Git repository.</p><p>Why do we even need a configuration file?!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PoU6CMaYh6uNWgYn0AX7bg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@flowforfrank?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Ferenc Almasi</a> on <a href="https://unsplash.com/s/photos/configuration?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>A configuration file will allow the end user to set the properties that he needs. The user will be able to do it simply and without the need of code writing and deployments.</p><p>In the configuration file we will define the table that the data needs to be written to with some other properties: the project name and the dataset on BigQuery. The advantage is that the configuration file can also define the scehma. If the data meets the schema requirements the table is expecting, the service will write it to BigQuery table. If not, the event will simply be ignored.</p><p>Using a configuration file, also helped us to add an option to validate the data if needed (if the user wish to), or choose to ignore it if not.</p><p><strong>Final Words</strong></p><p>As you saw in this article, our goal was to reduce performance time issues caused by services and ETLs having to write data into two databases.. The solution was to do data mirroring using a configuration file. The largest gain from that was that the configuration file is generic enough to handle different use cases. Working on the process, we took advantage the fact that event is already published to the Pub/Sub, simply read it with the new mirroring service, and used it as we wish in order to write the data as we need.</p><p>There is a little prior work that is required of by the end-user before the uese:<br>to create the table the user wishes to write to in BigQuery and update the configuration file with the new table. A quite simple task without the need of new deployments and code changes.</p><p>Please feel free to comment and ask anything, and clap if it make sense :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fcf5bd5bb0f9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-at-trax/mirror-mirror-on-the-wall-bigtable-to-bigquery-mirroring-fcf5bd5bb0f9">Mirror Mirror on the Wall: BigTable to BigQuery Mirroring</a> was originally published in <a href="https://medium.com/tech-at-trax">Tech at Trax</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[BigQuery — Update? Delete & Insert? Simply Insert!]]></title>
            <link>https://medium.com/tech-at-trax/bigquery-update-delete-insert-simply-insert-6d002f58319d?source=rss----ce231112e8a0---4</link>
            <guid isPermaLink="false">https://medium.com/p/6d002f58319d</guid>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Michal Nitzani]]></dc:creator>
            <pubDate>Wed, 23 Nov 2022 10:28:59 GMT</pubDate>
            <atom:updated>2022-11-23T10:28:59.251Z</atom:updated>
            <content:encoded><![CDATA[<h3>BigQuery — Update? Delete &amp; Insert? Simply Insert!</h3><p>This post presents a recommended approach for dealing with data updates in Google Cloud BigQuery managed storage. I will describe the learning process that has led us to choosing this approach, as it emphasizes some of the main concepts BigQuery is designed for. If you prefer to skip the detailed description, you can jump directly to the <a href="#47d9">summary section</a>.</p><h3>Background</h3><p>My team has recently developed a service that receives image recognition results of supermarket shelves, and writes them into BigQuery in real time. The results are queried using an API to provide up-to-date information about the history of the shelves with respect to product locations. The volume of data this service writes every day is large (~200MB), and the volume of data that is queried in each API call is much larger (~2GB). Due to the large volumes our service and API deal with, we have chosen BigQuery as our data storage tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*Mm_PCpJR-IDPTePPxMyTOQ.png" /><figcaption>Photo by <a href="https://unsplash.com/@franki?utm_source=medium&amp;utm_medium=referral">Franki Chamaki</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The below table represents a simplified version of the information our application tracks. Each row refers to an image. It captures the locations of all the products that were found in the image. The column ‘recognition_data’ is a repeated struct of the product’s id and its location coordinates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*FIxv5gdzOPTJ0qOVuGNLqA.png" /><figcaption>Image recognition data — image by author</figcaption></figure><p>It happens that an image is processed twice, sometimes within a very short timeframe, therefore, there is a need to update its information. In the next section I will describe the approaches we have tested for handling these kind of real time updates, and in particular, the approach we have found as best appropriate for our use case.</p><h3>Real Time Data Updates</h3><p>Let’s assume that you develop an application that needs to load real time data into BigQuery’s managed storage. The <a href="https://cloud.google.com/bigquery/streaming-data-into-bigquery">streaming api</a> allows you to stream records (represented in JSON) in an efficient near real time manner. This was the recommended approach in use cases of frequent single row inserts (<a href="https://cloud.google.com/blog/products/bigquery/life-of-a-bigquery-streaming-insert">a thorough overview</a>) before the <a href="https://cloud.google.com/bigquery/docs/write-api#advantages">storage write api</a> was introduced. The data loaded to BigQuery’s managed storage needs to be updated, sometimes, within minutes since the time it was first loaded.</p><h4><strong>1. UPDATE/DELETE statements</strong></h4><p>The first approach I have tested to deal with data updates was the most obvious one - using standard SQL UPDATE statements. It didn’t take long to understand that this approach is not well fitted to BigQuery’s design, and particularly when working with streaming inserts.</p><p>Let’s get back to our recognition example. Suppose that there was an issue with the recognition engine, and that image #101 was reanalyzed five minutes after the previous rows were inserted to the table (using the streaming api). Let’s assume that this is the data we wish to see for image #101:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*0TdSp-FPdV7j-lsIQXxH0w.png" /><figcaption>Image #101 most updated data - image by author</figcaption></figure><p>If we’ll try to update the previous row (using SQL UPDATE statements), or to delete the previous row and insert the new row (using SQL DELETE and INSERT statement), we might get the following error:</p><p><strong>“BigQuery Error : UPDATE or DELETE statement over table would affect rows in the streaming buffer, which is not supported”</strong></p><p><strong>Data availability limitation: </strong>This error indicates that there is an attempt to modify rows which are still in the streaming buffer. BigQuery does not allow that in order to avoid consistency issues. In a worst case scenario, rows can stay in the streaming buffer for up to 90 minutes. Therefore, if updates can happen in a shorter duration, like it is in the case of our application, this approach is not appropriate.</p><p>Even if data updates occur within a longer duration, you should reconsider the UPDATE/DELETE approach.</p><p><strong>High cost of single row manipulations:</strong> BigQuery is designed for Online Analytical Processing (OLAP) by using table scans, rather than single row lookups that are used by Online Transaction Processing (OLTP) systems (this is why BigQuery doesn’t use indexes). For this reason, UPDATE/DELETE statements that affect a single row are not cost efficient.</p><p>At this point I understood that the use of UPDATE statements is not suitable for my application, nor does the use of DELETE statements followed by new streaming inserts. What is the solution then?</p><h4><strong>2. Don’t throw away history - simply insert</strong></h4><p>When you update rows you lose historic information. If you just insert new rows that have a timestamp or some sequential id field, you can access the latest information, and you also have the previous information available for different analysis purposes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*PW18be2TcwEasmmLp_ytVg.png" /><figcaption>Insert only —both historic information and updated information are kept — image by author</figcaption></figure><p>Now, all you need is to make sure that you access the latest copy of the rows in your production environment.</p><p><strong>Deduplication:</strong></p><p>To access the most updated information, you can run a query that removes duplicate rows. A typical deduplication query includes a numbering function like ROW_NUMBER() or RANK(), and a PARTITION BY clause.</p><p>The query below returns the most updated image recognitions. In the subquery, sequential row numbers are given to each group of rows that share the same partition (the same image_id, and the same image_date). The row numbers are given in a descending order so that the rows with the latest timestamps (in each partition) are given the number 1. The outer query selects the rows that were given the number 1, and removes the row_number column (with the Except function).</p><pre>SELECT * EXCEPT(row_number)<br>FROM (<br>      SELECT *, ROW_NUMBER()<br>                OVER (PARTITION BY image_id, image_date<br>                      ORDER BY record_creation_timestamp DESC) row_number<br>      FROM image_recognition_table)<br>WHERE row_number = 1</pre><p>The result of running the above query:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*QsM9_N5nHY7EWOWFAZ8zjg.png" /><figcaption>Deduplication results — image by author</figcaption></figure><p>Creating a view* over the above query allows one to run queries on a deduplicated table.</p><pre>CREATE OR REPLACE VIEW image_recognition_view AS<br>(SELECT * EXCEPT(row_number)<br>FROM (<br>      SELECT *, ROW_NUMBER()<br>                OVER (PARTITION BY image_id, image_date<br>                      ORDER BY record_creation_timestamp DESC) row_number<br>      FROM image_recognition_table)<br>WHERE row_number = 1)</pre><p>Note that any query that references the above view, triggers the deduplication query, thus, all the rows/columns accessed in the deduplication query are scanned. This could have a significant impact on the query costs.</p><p>* We define a simple view and not a materialized view, as a materialized view wouldn’t add any performance advantage in this case (<a href="https://cloud.google.com/bigquery/docs/materialized-views-intro">more on BigQuery’s materialized views</a>).</p><h3>To Conclude</h3><ul><li>A best practice for dealing with frequent data updates:<br>1. Do not use UPDATE/DELETE statements.<br>2. Insert a new row with up-to-date information, include a timestamp column or a sequential id column that will indicate the order of insertion.<br>3. To access the most up-to-date rows run a deduplication query.<br>4. To run queries over the most up-to-date rows, create a view over the deduplication query and query the view.</li><li>Note that at the time I worked on this project, legacy streaming was the recommended approach for inserting real time data into BigQuery’s managed storage. As of today, the recommended approach is streaming using <a href="https://cloud.google.com/bigquery/docs/write-api#advantages">storage write api</a>.</li></ul><p>For more information on BigQuery’s<a href="https://cloud.google.com/bigquery/docs/best-practices-performance-patterns"> best practices</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d002f58319d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-at-trax/bigquery-update-delete-insert-simply-insert-6d002f58319d">BigQuery — Update? Delete &amp; Insert? Simply Insert!</a> was originally published in <a href="https://medium.com/tech-at-trax">Tech at Trax</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing Your IoT Device Energy Consumption]]></title>
            <link>https://medium.com/tech-at-trax/optimizing-your-iot-device-energy-consumption-3e8e90d4a02d?source=rss----ce231112e8a0---4</link>
            <guid isPermaLink="false">https://medium.com/p/3e8e90d4a02d</guid>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[energy-efficiency]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[battery]]></category>
            <category><![CDATA[internet-of-things]]></category>
            <dc:creator><![CDATA[Tal Klinger]]></dc:creator>
            <pubDate>Mon, 08 Feb 2021 12:14:10 GMT</pubDate>
            <atom:updated>2021-02-08T12:14:09.991Z</atom:updated>
            <content:encoded><![CDATA[<p>Here is a word, and many engineers are not going to like this word: <strong>Batteries.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/0*6Al93O_VxMzFyo1k.jpg" /><figcaption><a href="https://www.flickr.com/photos/30713600@N00/6997580445">“low battery”</a> by <a href="https://www.flickr.com/photos/30713600@N00">twicepix</a> is licensed under <a href="https://creativecommons.org/licenses/by-sa/2.0/?ref=ccsearch&amp;atype=rich">CC BY-SA 2.0</a></figcaption></figure><p>Yes, these things that make our lives so simple by not connecting every device to a wall outlet as consumers makes our lives hard as engineers.</p><p>Imagine how the world would look if we had to plug pacemakers into an outlet. Contrarily, how would it feel if we had to change batteries every week?</p><p>This is where trade-off engineering turns into an art — how do we design a device that consumes less energy without compromising on functionality.</p><p>The pacemaker example is an extreme yet clear representation of this trade-off — this device that must operate without any quality compromises over a period of 5–10 years.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/254/0*pRqmU62kdKo73haF.png" /><figcaption>Creative Commons <a href="https://xkcd.com/license.html">XKCD</a></figcaption></figure><p>So how is it related to our work? Well, IoT devices are all around us— monitoring your beer freshness, counting people within a store or even helping your local grocery store never run out of milk — all of this magic, on batteries.</p><p>So how can one reduce a device’s battery consumption? In the following article, we’ll review several techniques to reduce battery consumption in “on” states — all of this was done with the <a href="https://www.qoitech.com/">Otii Arc</a> — an engineer’s multitool that serves as a power supply and analyzer.</p><h3>Frequency Matters</h3><p>CPU/MCU tend to consume more energy when they run at higher frequencies, but higher frequencies do not guarantee better or faster operation for your IoT device.</p><p>Lets take a concrete example: here is my base profile and the improved one with a lower frequency:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uZqzbYg2RVwjkGcSNAafBg.png" /><figcaption>160Mhz compared to 80 Mhz</figcaption></figure><p>And if we dive deeper, we can see that the average current in similar tasks is 4~7mA lower while maintaining the original operation time, meaning, no additional compute was needed to conduct the operation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JzNsJVc-zyQ8DeHkokCNKw.png" /><figcaption>Same region is highlighted and compared on the upper right part</figcaption></figure><p>Just a side note — in my specific example, the application does not conduct heavy compute, so there is no effect on timing at all when frequency is raised or reduced — I would recommend to test both options (raising and lowering) just to see what parts of your states machine benefit from this change.</p><h3>Cut Connections</h3><p>Or at least reduce your packet transmission. If you have a WiFi enabled device and you need to keep a socket open, try to reduce the keep alive packet transmission to minimum.</p><p>For BLE — depending on your chosen topology, you can strategize how you send your data in bigger intervals and still maintain full functionality.</p><p>And the key takeout — if you don’t need to keep a connection open or send data — <strong>turn of your emitter immediately or put it to standby</strong> if you’re going to retransmit within a period of time shorter than your cold start.</p><h3>Turn Things Off — Immediately</h3><p>This may be very obvious, but turn off your peripherals as soon as you don’t need them anymore, don’t wait the extra milliseconds. Sometimes we forget to turn off things as soon as possible (boiler, oven, AC) just because we turn them on a few moments later, but due to the fact that we need to save each and every joul, it matters.</p><h3>Idle - The Silent Killer</h3><p>Let&#39;s say you have an I2C operation, and you’re sleeping 10ms in between operations, even though I2C operations should take <a href="https://en.wikipedia.org/wiki/I%C2%B2C#Design">micro-seconds</a> . You are adding up (10-x)*NumberOfOperations ms to your idleness, this could sum up to quite a lot of energy in case your base profile is quite high.</p><p>Another common mistake that is related to idleness is using pre-defined waits for operations that send a notification when they are done. By waiting extra time for the confirmation, we waste extra Joules on nothing. This can be improved by using interrupts instead of polling for signal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OYg4j0ysCfxyQrzH60Gm9w.png" /><figcaption>Original VS optimized</figcaption></figure><h3>Your Engineers’ Artifacts</h3><p>So your engineers made it and released this amazing product, but they didn’t clean up their debugging components, such as blinking LEDs, printing messages, enabling debug tools (UART) — these all sum up to quite a lot of energy.</p><p>Assuming you are using a LED at full brightness, it could <a href="https://learn.adafruit.com/battery-power-for-led-pixels-and-strips/estimating-running-time">potentially</a> drain to 10–20mA per LED.</p><p>You’d be amazed how much you can save just by cleaning before deploying to production.</p><h3>Restructuring The State Machine</h3><p>A similar example planning a multi-stop commute plan — you can organize your trip by multiple methods:</p><ul><li>Priority -opening/closing time for destinations, I must arrive when something is still open.</li><li>Distance between points —<a href="https://en.wikipedia.org/wiki/Travelling_salesman_problem">the travelling salesperson problem.</a></li><li>Logical order — I can drop off the pizza only after I pick it up from the restaurant.</li></ul><p>The same thing applies to your device — if we are focused on <strong>functionality</strong> and <strong>energy efficiency</strong>, we can do some non-trivial ordering to reduce cold start penalties.</p><h3><strong>Summary</strong></h3><p>By following these simple guideline, I was able to reduce the overall energy consumption in <strong>on</strong> state by 66% (7.5mWh to 2.5mWh per iteration), which is the major part of the energy consumption throughout the day.</p><p>To conclude, these guidelines could significantly help you reduce your overall energy consumption when developing a battery based IoT product.</p><p>Of course we can explore the <strong>deep-sleep</strong> realm, but this is something that would be more board specific or related more to the selected architecture.</p><p>If you are interested in deep sleep techniques, I highly recommend that you check out <a href="https://www.kevindarrah.com/">Kevin Darrah</a> and his <a href="https://www.youtube.com/channel/UC42d7zFnWU0dYVk_M0JED6w">Youtube channel</a>.</p><h3>TL;DR</h3><ul><li>Reduce your MCU/CPU <strong>core frequency to minimum</strong>.</li><li><strong>Reduce the WiFi/BLE transmit interval</strong> for socket keepalive.</li><li>Turn off any unused peripherals<strong> as soon as possible</strong>.</li><li><strong>Avoid predefined idle time frames</strong>, use interrupts instead.</li><li><strong>Remove</strong> <strong>any</strong> “debug in release” code — <strong>debug LEDs, prints, UART access.</strong></li><li><strong>Reconsider your state machine design</strong>, sometimes you’d be surprised how much you can save with a different sequence — cold start vs warm standby.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e8e90d4a02d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-at-trax/optimizing-your-iot-device-energy-consumption-3e8e90d4a02d">Optimizing Your IoT Device Energy Consumption</a> was originally published in <a href="https://medium.com/tech-at-trax">Tech at Trax</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How (Not) to Screw up an Analytic Presentation?]]></title>
            <link>https://medium.com/tech-at-trax/how-not-to-screw-up-an-analytic-presentation-11f81d6c025e?source=rss----ce231112e8a0---4</link>
            <guid isPermaLink="false">https://medium.com/p/11f81d6c025e</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[presentations]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Danit Berger Zalmanson]]></dc:creator>
            <pubDate>Thu, 14 May 2020 07:10:01 GMT</pubDate>
            <atom:updated>2020-05-14T07:10:01.153Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oy4nO_txiFuqbnccbliWHg.jpeg" /></figure><p>In the data world, communication is crucial.</p><p>Have you ever sat in a forum where you presented results of an analysis to stakeholders/clients, and everyone doubted you? Have you presented a genius data model that you had been working on for several months, and didn’t understand why you were getting hollow stares at the end?</p><p>Because we have all experienced this feeling at least once, I would like to argue that you should invest energy and thought in preparing a presentation — at least the enthusiasm you put into the analysis itself.</p><p>Let me show you several ways how not to screw up an analytic presentation — so you’ll be able to pass this phase in flying colors and be recognized for the great value that your work is bringing.</p><h3><strong>Giving it all (or: structure as the first step)</strong></h3><p>During the analysis phase, you have a long process of cleaning the data, arranging it, characterizing it, building a model based on it, and creating several tests. So why wouldn’t you share this long process with the client?</p><p><strong>Well, you can… but you shouldn’t.</strong></p><p>The client is not looking to hear about your journey (or that great chart you created in plotly) — they are there for the bottom line. So in every analysis you present, you should spend time thinking about what the appropriate structure is for the story coming out of the data.</p><ol><li><strong>Don’t start with more questions:</strong> It is important to remember that our internal/external customers are coming to us to get answers — not additional question marks. That’s why you should start your flow with clear conclusions and recommendations.</li><li><strong>Have a repetitive template:</strong> In most cases, start with the business question and the upshots (two slides max), then review your methodology and continue to the analysis itself. Only then, present the results that lead you to the conclusions you presented earlier.</li><li><strong>Use your appendix wisely:</strong></li></ol><ul><li>You’ve created a slide and are having trouble finding what the stakeholder should take from it? Hide it in the appendix.</li><li>You’d like to talk about a complex technical level, or think that the client might ask you about something specific — prepare an easy to understand section in the appendix. Try to avoid data-analyst-scientist buzzwords without being ready to explain them (yes, I’m talking about all the “Pearson’s chi-squared test” or “Random forest algorithm”).</li></ul><figure><img alt="Show complicated ideas by simple visualization" src="https://cdn-images-1.medium.com/max/1024/0*a7v2G-htqz2p5Ycg" /><figcaption>Show complicated ideas by simple visualization</figcaption></figure><h3><strong>The simpler the better (or: let’s talk about your content)</strong></h3><p>The quickest way to confuse the people sitting in front of you is to copy/paste outputs from your data (tables/model/code) to the presentation itself. We already discussed the importance of structure, but how do we not bore people to death? And how do we help them distill the relevant conclusions from the analysis?</p><ol><li><strong>And… Action:</strong> I make sure my output is actionable and shows what are the next steps that my customers can take, or what is the value of my analysis. Is there a new way to order the shelf? Who are the best and worst auditors in each store? Can you spot frauds?</li><li><strong>Data paradox: </strong>Geeks like us loooove data, and we tend to forget that very few people feel this way about numbers too. Here are some tips to communicate your results:</li></ol><ul><li>Not everyone can look at a spreadsheet or table and draw quick and clear conclusions about what the data indicates. Anyone can follow a trend line or the size of bars on a bar chart.</li><li>You can start simple, by describing the trend — “For each unit decrease in [add variable], we expect to see a lift of X% on average”. In this way, you avoid talking about the technical aspects of the analysis and focus on the business value it brings.</li><li>Pick one data point and demonstrate how your story applies to it. At Trax, we collect data from the shelf itself. I tend to use the actual images from the store itself to back up my conclusions from the analysis. A real-life example that brings the discussion down from a highly technical one to an easily digestible and simple to relate to.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*A_4AlTS54p3JUpMr" /><figcaption>Present your argument in a way that will make it actionable (in this case, we showed two different ways of data collection)</figcaption></figure><h3><strong>Design is where science and art break even (or: how to be clear?)</strong></h3><p>Once you have submitted the presentation, it is no longer in your possession — and may well get a lot of viewers. How do you ensure that the messages you conveyed orally will also pass to those who only read it? Here are some tips…</p><ol><li><strong>Be bold:</strong> Each headline should represent the slide and give insight.</li><li><strong>Creative mode:</strong> In the presentation — the idea is to convey the message clearly, easily, and quickly. Did it take more than 5 seconds to understand what the slide is about? If so, you’re in the wrong direction. Whenever I have a coherent slide idea but no clue on how to display it, I search for its template in Google. You’ll be amazed by what you’ll find.</li><li><strong>Don’t force the chart to speak for itself</strong>: Make sure that you have labels on your charts and a few lines to describe them. If you’re displaying data in a chart, only show one graph at a time. Be sure to explain what it shows and what it means in the broader context of the problem you’re addressing. Lastly, don’t be creative and overuse colors — green is good, red is bad, and blue helps to emphasize things.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RTUyylofwOAN-aRV" /><figcaption>Example of templates for a presentation you can find on the internet</figcaption></figure><p><strong>To conclude</strong>, to (not) screw up your analytics presentation, there are three parts that you should focus on: structure, content, and design. We have reviewed some notable tips that have helped me in each of those areas, but obviously, we only reviewed the tip of the iceberg.</p><p>I’m sure you have quite a few extra in mind…is it a mid-font change? Or maybe a spelling mistake?</p><p>Share what disturbed you with others, and what you learned from your previous presentations, so we’ll be able to ‘make our presentations great again!’</p><p>— — -</p><p>Did you like it? Don’t forget to clap!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11f81d6c025e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-at-trax/how-not-to-screw-up-an-analytic-presentation-11f81d6c025e">How (Not) to Screw up an Analytic Presentation?</a> was originally published in <a href="https://medium.com/tech-at-trax">Tech at Trax</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rendering Objects in Augmented Reality with Metal in IOS]]></title>
            <link>https://medium.com/tech-at-trax/rendering-objects-in-augmented-reality-with-metal-in-ios-96602cc0db40?source=rss----ce231112e8a0---4</link>
            <guid isPermaLink="false">https://medium.com/p/96602cc0db40</guid>
            <category><![CDATA[ar]]></category>
            <category><![CDATA[rendering]]></category>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[metal]]></category>
            <dc:creator><![CDATA[Muhammad Abed]]></dc:creator>
            <pubDate>Sun, 03 May 2020 10:54:34 GMT</pubDate>
            <atom:updated>2020-05-03T10:54:34.588Z</atom:updated>
            <content:encoded><![CDATA[<p><em>NOTE: before you dive into this, you need to be familiar with: Swift, IOS development, ARKit, basic linear algebra, basic 2D metal rendering.</em></p><p><em>If you are, then go grab yourself a cup of coffee and come back, I’ll wait for you!</em></p><p>My story begins with the <a href="https://traxretail.com/">Trax Retail App</a>. In short we use our crowdsourcing app to gather data about retail stores by taking pictures of the store shelves. We use these pictures to extract (using complicated machine learning algorithms) valuable data that retailers can use in order to know and monitor what’s happening in their stores.</p><p>We had one major problem - users were uploading pictures with bad angles, in order to enhance the user’s experience, we decided to integrate AR in our app, and we saw that users that use AR had better quality pictures!</p><h3>The Trax App</h3><p>You open the AR camera, let it detect the shelf</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/0*-pyUi8v8Ss-6gE3u" /></figure><p>After the AR session detects the shelf, the user can start taking pictures:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/0*yXAJpVzQvhnXrEnc" /></figure><p>With each picture the user takes, we draw a rectangle (plain) on the shelf to mark the area that the user covered:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/0*j1ObUMLfJa4kaUfw" /></figure><p>In our app we use SceneKit to render these turquoise walls, but because I love challenges, I found myself looking at how to render objects in AR using Apple’s <a href="https://en.wikipedia.org/wiki/Metal_(API)">Metal</a>. I had a lot of challenges doing it, due to the lack of proper and detailed documentation from Apple (as always), and the surprisingly poor and old material on the internet, yet I loved every single moment of it.</p><p>Moving to Trax and the joy of working on this part of our app made me very passionate about it.</p><p>So if you are a challenge lover, and you like to go deep and understand how these libraries work, then you’ve come to the right place!</p><h3><strong>Math Background</strong></h3><h4>Handedness</h4><p>Handedness refers to the orientation of the z-axis in a given 3D space. If the z-axis conforms to the so-called right-hand rule, then the space is said to be <em>right-handed</em>. Alternatively, the space is <em>left-handed</em>.</p><p>We can use both, as long as we understand this and know how to move between one and the other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/0*6N9DLZMF3BEjbMft" /></figure><h4>Linear Transformations</h4><p>Geometric transformation is a function that maps one point to another point.</p><p>Linear transformation (also called linear mapping, linear function) is a <a href="https://en.wikipedia.org/wiki/Map_(mathematics)">mapping function.</a></p><p><em>V</em> → <em>W</em> between two <a href="https://en.wikipedia.org/wiki/Module_(mathematics)">modules</a> (for example, two <a href="https://en.wikipedia.org/wiki/Vector_space">vector spaces</a>) that preserve (in the sense defined below) the operations of addition and <a href="https://en.wikipedia.org/wiki/Scalar_(mathematics)">scalar</a> multiplication — wikipedia</p><p>f(u + v) = f(u) + f(v) //addition</p><p>f(c*u) =c*f(u), c is scalar // scalar multiplication</p><p>Each linear transformation can be written as a matrix.</p><p>The most common transformations in computer graphics are Scale, Rotation and Translation.</p><p>Scale and Rotation are linear transformations (we won’t be covering proofs here, or it’d be a course in linear algebra 😅). After playing with Translation a little, you’ll find out that it’s not a linear transformation, so we need a trick to make it work.</p><h4>Scale</h4><p>Can be represented by the following 3X3 matrix</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/1*BJGXMCGBYp7DPzUcXhSPwA.png" /></figure><h4><strong>Rotation</strong></h4><p>If <em>u</em> is the direction vector and the angle (in radians), then the rotation matrix should look like this…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xNgb47IpNvJqwCuF" /></figure><p>We won’t be covering the source of this matrix here, but you are free to check out this wikipedia <a href="https://en.wikipedia.org/wiki/Rotation_matrix">post</a> which explains it very well.</p><h4><strong>Translation</strong></h4><p>In order to translate the point <em>p</em> by the normalized translation vector <em>t,</em> we need this simple equation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*61rs29KRqjfKV1wa" /></figure><p>We can’t make a 3X3 matrix that we multiply with the point, to get this result.</p><p>To do this we will use a handy and widely known trick, we’ll add a new dimension <em>w</em>, and make it equal 1, so we’ll have this 4X4 matrix.</p><p>So when you multiply it with a point (with addition of <em>w</em> dimension), you get a 4D point, and by removing the fourth dimension we get our desired point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/0*sHu6pcm3geBtysC3" /></figure><p>The above linear transformation (in 4D) is called a <a href="https://en.wikipedia.org/wiki/Shear_mapping">Shear transformation</a>.</p><h4><strong>Projection</strong></h4><p>One of our goals in 3D rendering is to “squash” 3D objects into 2D, and create the perspective illusion. We achieve this by creating a perspective projection matrix.</p><p>To understand how a perspective projection matrix works, you need this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/0*7ChLrOJhso6jtNad" /></figure><p>This is called the view frustum. By choosing an aspect ratio (ratio between width and height of viewport) and a field of view, we implicitly determine the <em>clipping planes</em> that make up the sides of the view frustum.</p><p>The pyramidal shape of the frustum is a natural outcome of our demand for perspective. The perspective projection scales point relative to their distance from the virtual camera, which in turn causes the sloped sides of the view frustum to become parallel as it undergoes this transformation. Points far away are squeezed closer to the axis of view, which causes the phenomenon of <em>foreshortening</em>, an important depth cue.</p><h3>Clip Space</h3><p>The device has it’s coordinate system, and it expects the given points to render, to respect this coordinate system: x[-1,1], y[-1,1], z[0,1]. This is called the clip space of the system. It’s where the system determines which triangles should be shown, which are clipped, which are hidden.</p><p>Now that we know what the clip space is, we can construct the projection matrix that scales points from one clip space to another :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/0*qMKx9W5KsP0EAhBf" /></figure><p>All the symbols refer to the view frustum illustration.</p><p>The construction of this matrix is beyond the scope of this post, but you’re free to check <a href="https://en.wikipedia.org/wiki/Camera_matrix">this</a> and <a href="https://en.wikipedia.org/wiki/3D_projection">this</a> wikipedia post about it.</p><h3><strong>Code</strong></h3><p>To put this into practice, we’ll make a little app that puts a 3D cube on the location you clicked every time you click on the screen.</p><p>We won’t cover the Metal setup and the Assets setup here, since I am assuming you know all these things.</p><p>We’ll start with the Vertex Data function (functions we set of necessary data for drawing) before we pass the data to the shaders.</p><p>The first interesting function is the one where we set the necessary matrices, this is how we define our frustum:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8648cf0cdf044cb627a80bb4b958627c/href">https://medium.com/media/8648cf0cdf044cb627a80bb4b958627c/href</a></iframe><p>You can see that at the start of the function, we set the view matrix and the projection matrix — by setting view port size, near and far properties we talked about.</p><p>The other lines are for setting the light, in order to give our shapes 3D lighting.</p><p>We won’t be covering this here, but you’re welcome to <a href="https://en.wikipedia.org/wiki/Computer_graphics_lighting">read about it</a>.</p><p>Next, we’ll update the anchors:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77f27bad5224ee9e19af3f18b501e21d/href">https://medium.com/media/77f27bad5224ee9e19af3f18b501e21d/href</a></iframe><p>We’re going through the anchors here, and from these anchors we create a model matrix — the matrix that defines the object we want to draw as a result of your click on the screen.</p><p>That’s the two main functions we use to set the data for drawing (Vertex Data stage).</p><p>Wasn’t really as tough and scary as you thought it would be! Right!? Now that you understand the whole purpose of this “complicated” mathematics, you can now connect the dots to figure out why we need all these matrices.</p><p>Next we’re moving to the shader functions, this is where we apply all the mathematics we’ve learned.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b1c085c9c6ef0f33b36e0ec7989e8be/href">https://medium.com/media/2b1c085c9c6ef0f33b36e0ec7989e8be/href</a></iframe><p>Here we simply apply each vertex position with projection, model and any other transformations (Translation, Rotation, Scale). We also set the Eye Position (frustum top point).</p><p>Don’t worry about the lighting properties, they’re just there to make things look better, we won’t be covering the lighting here.</p><p>The surprise is that you can find the whole source code if you open a new project in your Xcode and select the Augmented Reality App, and then choose Metal for Content Technology in the next screen.</p><p>But now instead of starting from zero, you can focus on the important parts — Yes, the parts we covered here — and leave the “complicated” technical setup for later.</p><p>From this point you can start playing with AR using Metal, trying to create a more interactive app, handling clicks and swipes on these cubes you create, and trying to Rotate or Scale them.</p><p>In short… HAVE FUN!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96602cc0db40" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-at-trax/rendering-objects-in-augmented-reality-with-metal-in-ios-96602cc0db40">Rendering Objects in Augmented Reality with Metal in IOS</a> was originally published in <a href="https://medium.com/tech-at-trax">Tech at Trax</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Architecture Made Easy: 4 Simple Ways you can Perfect your Code Design]]></title>
            <link>https://medium.com/tech-at-trax/architecture-made-easy-4-simple-ways-you-can-perfect-your-code-design-74ee86ed636?source=rss----ce231112e8a0---4</link>
            <guid isPermaLink="false">https://medium.com/p/74ee86ed636</guid>
            <category><![CDATA[software-design]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[clean-code]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Bar Zach]]></dc:creator>
            <pubDate>Thu, 23 Apr 2020 20:46:02 GMT</pubDate>
            <atom:updated>2020-04-26T10:30:58.170Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MRCuk12J0-KF7i-7qgH6XQ.jpeg" /><figcaption>Image by <a href="https://pixabay.com/users/StockSnap-894430/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=2569802">StockSnap</a> from <a href="https://pixabay.com/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=2569802">Pixabay</a></figcaption></figure><p>For many years I’ve been working as a software engineer in various roles and companies. Whether I was working on new projects or editing existing ones, a few questions have always come back to mind.<br>Am I designing the right project architecture? How much time will it take someone to read/refactor my code in the future?<br>Most senior developers, hopefully, consider the way they architect a code and what they’re going to write about. But because there is no “correct” architecture for a program, many developers aren’t sure how their design will handle future changes.</p><p>In this article I’m going to talk about a few simple steps I’ve learnt to create a design that scales over time.</p><h3>Ask the Right Questions</h3><p>Taking the time to organise a project before starting, significantly helps us get things done easier. These are my favourite “4-Whats” questions when starting to form a solution:</p><p>● What entities do you have?</p><p>● What part of the code is breakable over time?</p><p>● What are your project’s external dependencies and I/O?</p><p>● Last but not least — What could change?</p><p>To show the power of these questions, I’ll take a simple coding task of creating an ordering system for restaurants. Meaning, managing the order data from end-to-end.</p><h3>1. Planning your Entities</h3><p>Have you seen an entire code written inside “main”? Probably not often. Although this is an obvious ‘code smell’, it doesn’t mean that all other code is considered perfect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*681KMVDpBY_4EWoHN5I0Zw.png" /><figcaption>First thought of dividing our implementation into different entities</figcaption></figure><p>Using the “high cohesion, low in coupling” concept, we would like to make loose connections between entities. Clients place orders with restaurants. Restaurants have menus, with their own menu items and chefs making them. Each order is packed and sent as a delivery, which the client receives.</p><h3>2. Finding Breakable Code</h3><p>Every developer knows that testing is a must, because without it the code may react differently than we expect. But what if we could try and use the test approach while designing? That way we could make a cleaner design by “playing” with our entities before we code.</p><p>How do you test a design to find breakable code? I usually try and imagine possible flows of deployment and see what problems I get. One might call it “debugging on paper”.</p><p>There are a few issues with this flow, but let’s focus on only one for this example.</p><p><em>Business Flow:</em> “Clients choose menu items and make orders. These orders are placed in a restaurant and passed to the chefs. The chefs make the orders, and when they’re ready they are delivered to the client.”</p><ul><li><strong><em>Chefs</em> </strong>— Which kind of chefs should receive the order? Could chefs work in different restaurants on the same day?</li></ul><p>For that I might introduce two new entities — a “Shift” and an “Employee”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*FixrD8pf3qK5wD-tpABXRQ.png" /><figcaption>New connections between entities that might otherwise break the code in future</figcaption></figure><ul><li><strong><em>Employee (base class)</em></strong> — Having a Chef inherit Employee would help us expand different roles (Pastry Chef, Waiter, etc) as needed. That way we can add chef-specific functionalities.</li><li><strong><em>Shift</em></strong> — Chefs might only work part time. This object would match a list of chefs with specific times and restaurants, providing a dynamic way to control the number of chefs needed in each restaurant at a specific time.</li></ul><h3>3. Packaging External Dependencies and Making APIs</h3><p>Even though our newer design seems cleaner, we can make tremendous progress by diving into its external parts. I usually ask myself — do I have any particular code that can benefit other projects of mine? Something using any external I/O? Using the example from above, we can try and create packages out of it. There are many examples of packaging and “API-ing” our code, but I’ll focus on these two:</p><ul><li><strong><em>Orders</em></strong> — Most of the order information is quite generic to any type of orders. For example — clothes, art, etc. You might create an Order package and use an inherited class to implement the unique data — menu items.</li><li><strong><em>Menu items and prices</em></strong> — The data these classes use is placed in a DB, so you can obviously “guard” it from direct queries using some kind of API. That way future changes to the DB are easy, because the data in the DB isn’t coupled with the code itself. Thus, you only need to update the API.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*wNmhIn8XWLJPyiWSF_IdXQ.png" /><figcaption>Phase three, creating packages and making APIs</figcaption></figure><h3>4. What could change?</h3><p>This last question might seem like a small one at first, but actually it is of the utmost importance to ask. So how can you adopt a futuristic mindset? My favourite strategy is to look down at my program’s UML diagram, and start exploring. In the example I presented above, I’d take each entity in turn and try to rethink it. For example –</p><p><strong><em>Shifts</em></strong> — I implemented it as a list of chefs and a time range, such as “chefs #1, #2 at 14:00 to 16:00”. But what would you do if you needed to split shifts? For example, “chef #1 from 13:00 to 16:00, chef #2 from 14:00 to 15:30 and chef #3 from 15:00 to 17:00”? The employer might not care when the chefs come and go as long as there are X chefs at every time of the day.</p><p>For that, we might change the Shift entity to be chef-specific, and add a new class called “Capacity” to handle the number of chefs per hour.</p><p><strong><em>More changes may include:</em></strong></p><ul><li>Different prices on menu-items (in different restaurants)</li><li>Deliveries as external service with API</li><li>Restaurant service by client location</li><li>and more</li></ul><h3>Future Thoughts</h3><p>The topics we discussed in this post can be considered highly relevant in creating a new service. It seems that similar skills are required in refactoring an existing code that has already been deployed. As services get bigger without the appropriate design, it tends to be harder to maintain over time and make changes. It’s often possible to refactor such code appropriately (a topic that should get its own post), but smartly adapting a futuristic mindset and techniques would save much time and energy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=74ee86ed636" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-at-trax/architecture-made-easy-4-simple-ways-you-can-perfect-your-code-design-74ee86ed636">Architecture Made Easy: 4 Simple Ways you can Perfect your Code Design</a> was originally published in <a href="https://medium.com/tech-at-trax">Tech at Trax</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why should you build an organizational Slack bot?]]></title>
            <link>https://medium.com/tech-at-trax/why-should-you-build-an-organizational-slack-bot-c7cec814a0de?source=rss----ce231112e8a0---4</link>
            <guid isPermaLink="false">https://medium.com/p/c7cec814a0de</guid>
            <category><![CDATA[slack]]></category>
            <category><![CDATA[bots]]></category>
            <category><![CDATA[slackbot]]></category>
            <category><![CDATA[organizational-culture]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Yanai Ron]]></dc:creator>
            <pubDate>Tue, 14 Apr 2020 12:21:27 GMT</pubDate>
            <atom:updated>2023-01-05T11:28:08.749Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why you should build an organizational Slack bot?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fx1T_9H33xTTDag_8Niabg.png" /></figure><p>If you’re working in a tech organization, most chances are that you are using Slack to interact with coworkers. Did you know that you can build your own organizational Slack bot to enhance interaction between co-workers and to share knowledge?</p><p>In this post I will show you the simplicity and value of making time for you and the team to build one. I’ll also throw in some ideas to get you started.</p><h3>What are bots? Here’s the explanation from <a href="https://api.slack.com/bot-users">Slack</a>:</h3><blockquote><em>“A bot is a type of Slack App designed to interact with users via conversation…the same as a regular app: it can access the same range of APIs and do all of the magical things that a Slack App can do.</em></blockquote><blockquote><em>But when you build a bot for your Slack App, you’re giving that app a face, a name, and a personality, and encouraging users to talk to it. Your bot can send DMs, it can be mentioned by users, it can post messages or upload files, and it can be invited to channels — or kicked out. Bots are not cybernetic infiltration units, and it is unlikely that they dream of electric sheep, though we can’t rule it out. 🤖”</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*EjrAyDNpYA3QBPVd25P3cQ.png" /></figure><h3>Why do I need an organizational Slack bot?</h3><p>To be honest, you don’t really need it, but it can solve many problems your organization may have (and it’s easy and fun to write), such as:</p><h4>Knowledge Sharing</h4><p>Your Slack bot can be used to retrieve internal information, for example:</p><ul><li>Customer statistics</li><li>Employee roles, departments, birthdays</li><li>KPIs, open support tickets, system health</li><li>Product adoption, Sales data, top customers / customers at risk</li><li>Data for Salesforce: release notes, new versions, how-tos</li></ul><p>At Trax, we have many internal data sources (SQL DBs, No-SQL, internal APIs, etc.). The Slack bot can consolidate the interesting stuff and present it in an easily digestible way, the options are endless…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*m7WnVGhKw7ONmYlU1J_whg.png" /><figcaption>Supply useful information about our customers</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*DEytG089n7NNwZcBxX07Pw.png" /><figcaption>Supply useful information about your mobile versions</figcaption></figure><h4>Provide easy access to common actions</h4><p>Your Slack bot can be a great tool to initiate actions, such as:</p><ul><li>Environment setup</li><li>Running internal scripts</li><li>Accessing APIs</li></ul><p>At Trax, we can copy a production environment to a testing environment in order to perform tests on real data, it’s a killer feature used daily by multiple departments:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*bICFKEQj98zGMHbRvq2Z9Q.png" /></figure><p>This is a valuable feature for multiple stakeholders in the organization, but in order to get wide usage we needed to make it accessible, and a Slackbot is the simple UI-free way to accomplish that.</p><p>We can also run internal jobs (we call it “Simon”):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*2jHndz2YbBqcmUgMf9zyJg.png" /></figure><h3>Alright, I get it, I need all that, but why choose Slack?</h3><p>Great that you asked, there are many reasons to choose Slack for that:</p><ul><li>It’s already installed for everybody in your organization: HR, Finance, Sales, Technology, upper management, etc.</li><li>It’s a “UI without UI”, you don’t need to develop a fancy dashboard and employ designers to get the information to the organization</li><li>It’s so easy to implement, a few hours to set up everything for any passionate developer</li></ul><h3><strong>So it’s good for the company, but what’s in it for me?</strong></h3><p>Well, for starters, it’s an opportunity to work on something new, whether you’re a Frontend developer who wants to experience some Backend, or a Backend developer who wants to try out a new technology.</p><p>It’s small and well defined, and you get to do it on work hours because it’s useful at work! Perfect for a side-project, <strong>win win!</strong></p><h3>Okay, I’ll build it, but where should I start?</h3><p>Alright, firstly you’ll need a server with a public address/DNS, then…</p><ul><li>Pick the programming language of your choice, Node.js, Python or Golang are good starters for web servers</li><li>Write a simple REST API using <a href="https://slack.com/intl/en-il/help/articles/115005265703-Create-a-bot-for-your-workspace">this guide</a></li><li>Pick a cool <a href="https://www.userlike.com/en/blog/bot-names">name and avatar</a></li><li>Bonus: Track your bot usage using your favorite analytics platform (I use <a href="http://mixpanel.com/">Mixpanel</a>)</li></ul><p>You can start with <a href="https://github.com/yanairon/slack-bot-template">this</a> skeleton I’ve made using Node.js:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*et0opkcP4xsqn-BoQBfZGg.png" /></figure><p>Once the skeleton is set up, you can add more and more to it, the sky’s the limit! Here are some ideas:</p><ul><li>Provide visibility on customer’s statistics about system usage, features adoption, amount of data, etc.</li><li>Bonus: add some buttons that will lead to open issues, customers’ dashboards, etc.</li><li>Send a “Did you know” message from your bot once a week, with a trivia detail about the company, events that are happening, “kudos” to those special employees, etc.</li><li>Send notifications about employees’ birthdays, and let the other employees share a personalized message</li><li>Think about manual recurring operations your organization is doing, think about automating it with Slack</li></ul><p>Later on when it catches like wildfire in your organization, you can make it an internal open-source project for the rest of the company developers to contribute to.</p><h3>Wrapping up</h3><p>Now you should have a general idea of how an organizational slack bot can benefit you and your organization.</p><p>I’d love to hear about it, please share your ideas and success stories in the comments section, maybe I’ll implement them in Trax as well :)</p><p>Hope you enjoyed it, don’t forget to clap!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7cec814a0de" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-at-trax/why-should-you-build-an-organizational-slack-bot-c7cec814a0de">Why should you build an organizational Slack bot?</a> was originally published in <a href="https://medium.com/tech-at-trax">Tech at Trax</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GraphQL beyond the buzzword; When to use GraphQL]]></title>
            <link>https://medium.com/tech-at-trax/graphql-beyond-the-buzzword-when-to-use-graphql-d85283150f91?source=rss----ce231112e8a0---4</link>
            <guid isPermaLink="false">https://medium.com/p/d85283150f91</guid>
            <category><![CDATA[rest-api]]></category>
            <category><![CDATA[graphql]]></category>
            <category><![CDATA[frontend]]></category>
            <dc:creator><![CDATA[Shani Dar]]></dc:creator>
            <pubDate>Thu, 20 Feb 2020 10:02:35 GMT</pubDate>
            <atom:updated>2020-02-20T11:16:21.512Z</atom:updated>
            <content:encoded><![CDATA[<p>GraphQL, a query language for APIs, became very popular since it was first publicly released by Facebook in 2015. Many companies increasingly adopted GraphQL that provides a simple declarative way to retrieve data.</p><p>At Trax we started to use GraphQL over the last year. As part of the adoption process, we discussed the cost vs. value of using GraphQL in our services. While GraphQL has many advantages, it is also introduces a new set of challenges which need to be evaluated.</p><p>In this post you can find guidelines for deciding when it makes sense to use GraphQL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4KCQfhMT6YSZXC-S71xggQ.jpeg" /><figcaption>credit: <a href="https://www.piqsels.com/en/public-domain-photo-zknni">https://www.piqsels.com/en/public-domain-photo-zknni</a></figcaption></figure><h3>GraphQL Advantages</h3><blockquote><em>“What you want is what you get</em>”</blockquote><h4><strong>No Over-Fetching or Under-Fetching of data:</strong></h4><p>With GraphQL, developers can fetch only what they actually need, nothing less, nothing more. No need to fetch data from large objects with a fixed structure that contain unnecessary data (Over-Fetching), or fetch relational data with multiple requests because an endpoint doesn’t return all the information required by a client (Under-Fetching).</p><h4><strong>Reduce response time and network bandwidth:</strong></h4><p>GraphQL allows making a single query request to get exactly the data we need, including relationships, from multiple resources. This means fewer network requests and<strong> </strong>round trips<strong>, </strong>which saves time and bandwidth. It also helps to save waterfall network requests, where you need to resolve dependent resources on previous requests.</p><h4><strong>Reducing complexity through abstraction:</strong></h4><p>All of the requests are sent to a single endpoint. The client side doesn’t need to worry about endpoints and the response codes.</p><h4><strong>Auto-generating API documentation:</strong></h4><p>GraphQL Playground allows API developers and consumers to explore the relationships between types across your schema in addition to getting details about individual types. GraphQL keeps documentation in sync with API changes.</p><h4><strong>Strongly-typed Schema:</strong></h4><p><a href="https://graphql.org/learn/schema/">Types</a> supported by the API are specified in the schema using GraphQL Schema Definition Language (SDL), which makes GraphQL less error-prone, and more validated. GraphQL has a better relative type safeness compared to REST with documentation because the types described are enforced by the GraphQL engine.</p><h3>When Should We Use GraphQL?</h3><p>Common use cases of GraphQL API:</p><ul><li>Applications for low bandwidth devices such as mobile phones and IoT devices, where bandwidth usage matters.</li><li>Applications where nested data needs to be fetched in a single call.</li><li>Applications that retrieves data from multiple, different APIs.</li><li>GraphQL as an abstraction on an existing API.</li><li><strong>GraphQL is particularly great for front-end development</strong> (for example: Angular, React, Mobile), since using a GraphQL client library helps eliminate many of the inconveniences and shortcomings that are experienced with REST APIs.</li></ul><h3>When shouldn’t We Use GraphQL?</h3><p>Despite GraphQL’s advantages, GraphQL has implementation challenges, and is not necessarily the right solution for any use-case.</p><h4>Existing REST service</h4><p>If you have a stable RESTful service, there’s probably not a strong case to overwrite it all. What you might want to do is implement additional abilities using GraphQL.</p><p>One of the great benefits of GraphQL is that it’s not designed as a wholesale replacement for alternative API solutions. Instead, GraphQL can be implemented incrementally to complement or enhance existing APIs.</p><h4><strong>Overkill for small applications</strong></h4><p>While GraphQL is the right solution for multiple micro-services, a simple REST API can be better in case you have a service that exposes a really simple API\not API-centric. REST can also be a valuable approach for connecting resource-driven apps that don’t need the flexible queries offered by GraphQL. Take into consideration future needs, it might change if your app is intended to grow.</p><h4>Server to Server</h4><p>When the client of your API is another service (server-to-server) it may not make much sense to choose GraphQL. The investment cost might not be worth the value as most of GraphQL advantages refer to front-end clients.</p><h3>Challenges That Are Worth Taking Into Consideration:</h3><h4><strong>Web </strong><a href="https://graphql.org/learn/caching/"><strong>caching</strong></a><strong> complexity</strong></h4><p>Caching helps reduce the amount of traffic to a server by keeping frequently accessed information close to the client.</p><p>In an endpoint-based API (like REST), clients can use HTTP caching to easily avoid re-fetching resources, and for identifying when two resources are the same. The URL in these APIs is a <strong>globally unique identifier</strong> that the client can leverage to build a cache. In GraphQL, though, there’s only one endpoint with many different queries, and it’s much harder to implement a caching mechanism.</p><p>GraphQL caching at the database or client level can be implemented with the Apollo clients that have caching mechanisms built in. However, GraphQL doesn’t rely on the HTTP caching methods, which enable storing the content of a request.</p><p>The problem can be partially solved by using <a href="https://blog.apollographql.com/persisted-graphql-queries-with-apollo-client-119fd7e6bba5">persisted GraphQL queries</a> that assist in producing a JSON file for mapping queries and identifiers.</p><h4>Server-side Batching</h4><p>GraphQL executes a separate resolver function for every field, whereas REST has one resolver per endpoint. IT means that GraphQL runs the risk of making additional round trips to the database than are necessary for requests (<a href="https://medium.com/slite/avoiding-n-1-requests-in-graphql-including-within-subscriptions-f9d7867a257d">n+1 requests</a>).</p><p>This is commonly solved by a batching technique, where multiple requests for data from a backend are collected over a short period of time and then dispatched in a single request to an underlying database or microservice by using a tool like Facebook’s <a href="https://github.com/facebook/dataloader">DataLoader</a>.</p><h4><strong>Error Handling</strong></h4><p>GraphQL is protocol agnostic, it means that when using GraphQL over HTTP, you’ll always get 200 OK response, whether the query was OK or not. You will have to parse the response message to know if there’s an error.<br>Read more - <a href="https://www.apollographql.com/docs/apollo-server/data/errors/">Handling errors with Apollo</a> , <a href="https://graphql.org/learn/serving-over-http/#response">GraphQL response</a>.</p><h4><strong>File Uploading</strong></h4><p>The GraphQL specification does not address file uploading. You won’t have to deal with this limitation in case of REST, as there you can POST or PUT whatever content you want to.</p><p>File uploading can be implemented in several options:</p><ul><li>Using Base64 encoding. But it will make the request larger and expensive to encode/decode.</li><li>Making a separate API endpoint just for this purpose.</li><li>Using a library like Apollo for implementing the GraphQL multipart request specification.</li></ul><h4><strong>Versioning</strong></h4><p>One of the major challenges for API developers is handling updates in API contracts. In REST architecture, developers create new versions due to breaking changes in resources over time. A versioning strategy allows clients to continue using the existing API and migrate their applications to the newer API when they are ready.</p><p>With GraphQL, there is no need to maintain versions- the resource URL or address remains the same. You can add new fields and deprecate older fields. The advantage of GraphQL here is that the API consumer receives a deprecation warning when querying a deprecated field, and if he attempts to use a field or parameter you’ve removed he will get an immediate error from the server about the specific field.</p><p>Deprecating some part of an API simply at a functional level, totally changes the API consumer assumptions of what the data model is and how it behaves. Versioning helps deprecating APIs because it gives developers time to upgrade.</p><h4>Security</h4><p>GraphQL gives enormous power to API consumers, which expose your API to security threats.</p><p>The most common is deeply nested queries that result in expensive computations and large JSON payloads that can disrupt your network quality. To protect your API from this kind of attack, limit query depth to block maliciously deep queries.</p><p>There are several known strategies to mitigate additional security risks, read more about this topic- <a href="https://www.howtographql.com/advanced/4-security/">GraphQL security</a>, <a href="https://medium.com/swlh/protecting-your-graphql-api-from-security-vulnerabilities-e8afdfa6fbe4">Protecting Your GraphQL API From Security Vulnerabilities</a>.</p><h4><strong>Developer experience &amp; capacity</strong></h4><p>At Trax we started to use GraphQL over the last year, but still not all developers are experienced with GraphQL best practice. When starting a new project, one of our criteria to choose whether or not using GraphQL is if we have the time and resources. This is because GraphQL implementation can take some time and make tasks more complex for developers who didn’t work with it before. As part of GraphQL adoption we did several coding workshops, with hands-on exercises, to expose our developers to GraphQL.</p><h3>Decision table</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*2dLqCJvqX-qmfWJ15keLVQ.png" /><figcaption>Image from <a href="https://marmelab.com/blog/2017/09/08/dive-into-graphql-part-v-should-you-use-graphql.html">marmelab</a></figcaption></figure><p><strong>If you have a majority of B answers, it’s a good idea to investigate more about GraphQL for your project.</strong></p><p><strong>Otherwise, REST might be good enough for your case and there’s no need adding a new layer of complexity to your architecture.</strong></p><h3>Conclusion</h3><p>GraphQL, <strong>when implemented properly</strong>, is a powerful technology that offers an elegant methodology for<strong> </strong>data retrieval and<strong> </strong>impacts both API’s provider and consumers. Having said that, choosing an API design style will have impact on the whole API lifecycle - From design and implementation to developers experience.</p><p>Therefore, when choosing to work with GraphQL, choose it because it solves a problem in a given context, knowing its strengths and weaknesses.</p><blockquote><a href="https://nordicapis.com/when-to-use-what-rest-graphql-webhooks-grpc/"><strong>You shouldn’t think of GraphQL as “better” than REST or the “next step”, but rather as an alternative option for a “new relationship between client and data”.</strong></a></blockquote><h3>References</h3><p><a href="https://nordicapis.com/when-to-use-what-rest-graphql-webhooks-grpc/">When to Use What: REST, GraphQL, Webhooks, &amp; gRPC</a></p><p><a href="https://marmelab.com/blog/2017/09/08/dive-into-graphql-part-v-should-you-use-graphql.html">Dive Into GraphQL Part V: Should You Use GraphQL?</a></p><p><a href="https://medium.com/@__xuorig__/why-we-dont-see-many-public-graphql-apis-ad972bcb201e">Why We Don’t See Many Public GraphQL APIs</a></p><p>______________________________________________________________</p><p>Did this article spark your interest? We have lots of exciting things going on at Trax! If you’re interested in learning more about how you can join the revolution of digitizing the physical world of retail, click <a href="https://traxretail.com/careers/">here</a>, and don’t forget to clap!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d85283150f91" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-at-trax/graphql-beyond-the-buzzword-when-to-use-graphql-d85283150f91">GraphQL beyond the buzzword; When to use GraphQL</a> was originally published in <a href="https://medium.com/tech-at-trax">Tech at Trax</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>