<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tamas Foldi on Medium]]></title>
        <description><![CDATA[Stories by Tamas Foldi on Medium]]></description>
        <link>https://medium.com/@tfoldi?source=rss-a586bc473bbf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*IT9GI8-8i31yb_wQq_Y-BQ.jpeg</url>
            <title>Stories by Tamas Foldi on Medium</title>
            <link>https://medium.com/@tfoldi?source=rss-a586bc473bbf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 09:45:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tfoldi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ROS 2 Live Depth Cam / Point Cloud Visualization with Rerun]]></title>
            <link>https://medium.com/@tfoldi/ros-2-live-depth-cam-point-cloud-visualization-with-rerun-66534561557a?source=rss-a586bc473bbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/66534561557a</guid>
            <category><![CDATA[realsense]]></category>
            <category><![CDATA[re-runs]]></category>
            <category><![CDATA[ros2]]></category>
            <category><![CDATA[robotics]]></category>
            <dc:creator><![CDATA[Tamas Foldi]]></dc:creator>
            <pubDate>Thu, 03 Oct 2024 16:31:20 GMT</pubDate>
            <atom:updated>2024-10-03T18:12:41.528Z</atom:updated>
            <content:encoded><![CDATA[<p>When working with robots, or really anything that interacts with the physical world, visualizing what has happened and what is happening in real-time becomes essential. While several observability tools focus solely on telemetry (Grafana, Datadog, etc.), the harsh reality with physical systems is that you <strong>need to see them</strong> in their environment alongside their metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qCk1VRD6mCZCy3gj5JmZtA.png" /><figcaption>Visualizing my Intel RealSense d435i color and depth camera using Rerun 0.18.2</figcaption></figure><p>In the ROS 2 ecosystem, <strong>rviz2</strong> and <strong>Foxglove</strong> are the two de facto tools for visualization. Rviz2 is native to ROS 2, working seamlessly out of the box, though it has some limitations when it comes to reconstructing more complex visual scenes. Foxglove, a newer solution targeting the same ROS 2 community, comes with tight ROS integration, too. However, it can also operate standalone using its own WebSocket protocol or MCAP file format with its ROS-inspired message types.</p><h4>Rerun enters the game</h4><p><a href="https://rerun.io/">Rerun</a> seems to take a different approach, not focusing primarily on ROS 2 integration. Instead, it began by building core data structures to log and visualize multimodal data from the ground up, without mimicking existing types as Foxglove did. This approach has both pros and cons: while it allows any application to easily log data to the Rerun platform, it doesn’t natively support ROS 2 systems without implementing conversion rules or adding extra metadata.</p><p>There are two methods to visualize ROS 2 data in Rerun:</p><ol><li>a DIY solution: subscribe to the topics you want to visualize and log them in Rerun (like in <a href="https://rerun.io/docs/howto/ros2-nav-turtlebot">this</a> example)</li><li>Use Rerun’s <a href="https://github.com/rerun-io/cpp-example-ros2-bridge">example ROS 2 bridge</a> and set up conversion/ingestion rules there.</li></ol><h4>Using Rerun ROS 2 bridge</h4><p>The Rerun ROS 2 bridge is one of many Rerun examples showcasing a potential, more general integration solution. It subscribes to all topics with supported types (Image, PointCloud2, Odometry, Pose, etc.), converts the messages into Rerun archetypes, and logs the data to Rerun.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*jjS8B8zlmOWK3eOIw_JnGg.gif" /><figcaption>Rerun ROS 2 Bridge example, image by Rerun (<a href="https://github.com/rerun-io/cpp-example-ros2-bridge?tab=readme-ov-file">https://github.com/rerun-io/cpp-example-ros2-bridge</a>)</figcaption></figure><p>The mapping between ROS 2 frames and Rerun entity paths is configured in a parameter file, where additional transformations or settings can also be specified.</p><h4>Logging data from Intel RealSense D435i</h4><p>To demonstrate how the integration works, I chose a relatively simple and common task: visualizing topics from my depth camera. The camera publishes two feeds — one color and one depth image, along with their intrinsic parameters — and a point cloud representing points in 3D space.</p><p>I used the default configuration to launch the camera, without any additional transformations or custom settings.</p><pre>ros2 launch realsense2_camera rs_launch.py pointcloud.enable:=true</pre><p>This will produce the following topics which are need to be mapped to Rerun entities:</p><pre>❯ ros2 node info /camera/camera <br>[..]<br> Publishers:<br> /camera/camera/color/camera_info: sensor_msgs/msg/CameraInfo<br> /camera/camera/color/image_raw: sensor_msgs/msg/Image<br> /camera/camera/depth/camera_info: sensor_msgs/msg/CameraInfo<br> /camera/camera/depth/color/points: sensor_msgs/msg/PointCloud2<br> /camera/camera/depth/image_rect_raw: sensor_msgs/msg/Image<br> /camera/camera/depth/metadata: realsense2_camera_msgs/msg/Metadata<br> /camera/camera/extrinsics/depth_to_color: realsense2_camera_msgs/msg/Extrinsics<br> /tf_static: tf2_msgs/msg/TFMessage</pre><p>The mapping is handled through standard parameter files, specifying which frame_id will be logged under which entity path. The challenge is that ROS 2 messages only provide the frame_id, while Rerun requires all data to be logged with the full entity path.</p><pre>tf:<br>  update_rate: 0.0  # set to 0 to log raw tf data only (i.e., without interoplation)<br>  tree:<br>    base_link:<br>      camera_link:<br>        camera_depth_frame:<br>          camera_depth_optical_frame:<br>            points:<br>        camera_color_frame:<br>          camera_color_optical_frame:<br>topic_options:<br>  /camera/camera/depth/color/points:<br>    colormap: rgb<br>    colormap_field: rgb<br>    entity_path: /base_link/camera_link/camera_depth_frame/points<br>  /camera/camera/color/image_raw:<br>    entity_path: /base_link/camera_link/camera_color_optical_frame<br>  /camera/camera/color/camera_info:<br>    entity_path: /base_link/camera_link/camera_color_optical_frame<br>  /camera/camera/depth/image_rect_raw:<br>    entity_path: /base_link/camera_link/camera_depth_optical_frame<br>  /camera/camera/depth/camera_info:<br>    entity_path: /base_link/camera_link/camera_depth_optical_frame</pre><p>In addition to the mapping, I added an RGB color map for the points in the point cloud. There’s also an option to apply a turbo colormap based on any point cloud values (such as the z-coordinate).</p><p>Since my camera wasn’t facing upward, I needed to define a base_link to camera_link transformation as well.</p><pre>extra_transform3ds:<br>  - entity_path: &quot;base_link/camera_link&quot;<br>    transform: [1,0,0,0,0,0,-1,0,0,1,0,1,0.0,1.0,0.5]<br>    from_parent: true</pre><p>After starting the node with:</p><pre>ros2 launch rerun_bridge realsense.launch</pre><p>the viewer started to stream the messages into the UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0hW8Wosolfswewy8UeNUlA.gif" /><figcaption>Streaming ROS 2 Data with Rerun — visualizing image, depth image, and point cloud in one screen</figcaption></figure><p>Everything worked, except for some flakiness due to the default blueprint (dashboard layout) values. Here are the changes I made:</p><h4>Blueprint settings</h4><p>In my 3D view, I wanted only the pinhole projection from the color camera, not along with the depth camera. To achieve this, I removed the camera_depth_frame from the entities listed in the <em>Entity path filter</em> on the view&#39;s property page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xudsJCXRd-UMI11MLoD5kg.png" /><figcaption>Remove the camera_depth_optical_frame from the 3D view</figcaption></figure><p>I also set the ImagePlanDistance to a fixed value, which controls how far the pinhole image projection is visualized in 3D space. This value isn&#39;t available in the ROS 2 messages but is necessary for the visualization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/288/1*-g29OXGhB5dIdSY-jMsiOg.png" /><figcaption>Set the ImagePlanDistance to a reasonable value, 15cm in my case</figcaption></figure><p>If you’re streaming live data, it may be wise to <a href="https://rerun.io/docs/howto/limit-ram">set memory limits</a>, so older data is deleted once the memory threshold is reached. By default, it takes 75% of the available memory, so be careful.</p><h4>Summary</h4><p>Rerun is a super neat tool to visualize multimodal data, but unlike rviz2, it requires some configuration to work with ROS 2. While there’s no official integration with the framework, you can set up your visual observability pipeline using the provided examples in just 10–15 minutes.</p><p>For me, it was worth it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66534561557a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hiding injected modules and TracePid on ARM64 Linux]]></title>
            <link>https://medium.com/@tfoldi/hiding-injected-modules-and-tracepid-on-arm64-linux-49818db6d182?source=rss-a586bc473bbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/49818db6d182</guid>
            <category><![CDATA[obfuscation]]></category>
            <category><![CDATA[reverse-engineering]]></category>
            <category><![CDATA[linux-kernel-module]]></category>
            <dc:creator><![CDATA[Tamas Foldi]]></dc:creator>
            <pubDate>Sat, 14 Sep 2024 12:23:01 GMT</pubDate>
            <atom:updated>2024-09-14T18:10:12.269Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to use kernel modules to hide your tracers and debuggers</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2chdjb7MuQJUYIZi" /><figcaption>Hiding GDB and IDA via Linux kernel modules — Image generated by the author</figcaption></figure><p>When I was young (several decades ago), I used to take everything apart just to learn how it worked. Radios, alarm clocks — nothing was safe. I was curious about their inner workings, and dissecting them down to the smallest parts seemed like the best way to understand them. I still believe in this approach today: if you want to master something, you have to understand how it works at its most fundamental level. My first encounters with PCs were no different. After playing games like <em>Alley Cat</em> and <em>Prince of Persia</em>, the first programs I wrote were kernel drivers and system applications.</p><p>Since then, my focus has shifted more toward data-intensive applications, but I am still a systems programmer at heart. Even today, I want to know how the applications I use work. Whether you’re a data engineer, robotics engineer, or so-called data scientist, you must understand concepts like vectorization, I/O scheduling, or the Thread-Per-Core model. If you don’t, you’re just a <em>script kiddie</em> — not a real <em>hacker</em>.</p><h3>The Problem Statement: how to debug apps with anti-debug protection</h3><p>To understand what a third-party application does, the best approach is to examine its source code. Fortunately, most of the tools we use are open source, which makes this process easier (who else reads other projects’ source code just for fun?). However, many proprietary applications are closed source, which complicates things — but it’s not impossible. With tools like <a href="https://github.com/radareorg/radare2">radare2</a> or <a href="https://frida.re/">Frida</a>, we can peek inside and see the magic at work, unless, of course, the developers have implemented anti-tampering protections to make it harder for us.</p><p>On Linux, the most common anti-debug measures are:</p><ul><li>Reading <strong>/proc/self/status</strong> and inspecting the <strong>TracerPid</strong> field. If TracerPid is not zero, the code is being debugged.</li><li>Checking <strong>/proc/self/maps</strong> for additional libraries.</li><li>Using <strong>ptrace(PTRACE_TRACEME)</strong> to detect tracing.</li><li>Using <strong>prctl(PR_SET_DUMPABLE)</strong> to prevent debugging.</li><li>Monitoring and <strong>handling </strong><strong>SIGTRAP</strong> signals.</li><li>Detecting memory modifications through <strong>self-checksumming</strong>.</li><li>Identifying debugger stepping via <strong>timing anomalies</strong>.</li><li>Looking for <strong>dynamic linker environment variables</strong> like LD_PRELOAD.</li><li>Other running processes names.</li></ul><p>The good news is that with minimal kernel code modifications, we can bypass most of these. By creating a small kernel module that inserts itself between the user-space call and the original kernel code, we can modify return values on the fly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y8OLKi6KPmlgr_nxdrAVtQ.png" /><figcaption>Basic syscall hooking concept. App calls kernel, in kernel space, the syscall table points to our code while we call the orig syscall with some modifications.</figcaption></figure><h3>Hooking functions in Linux Kernel</h3><p>So how can we modify running code in the kernel? There are several methods, each with its own pros and cons. However, some techniques have become obsolete due to enhanced security and anti-tampering measures — for example, the kallsyms function was removed in kernel version 5.6.</p><p>As <a href="https://github.com/h33p">h33p</a> wrote <a href="https://github.com/h33p/kallsyms-mod">here</a>:</p><blockquote>kallsyms_lookup_name is a wonderful interface that is useful for quickly hacking kernel code without recompiling most of default kernels. Unfortunately, it is also very easy to abuse it, and deincentivizes hardware manufacturers from open-sourcing their drivers and adding them to mainline kernel tree. This is one of the reasons, why the function was unexported in Linux release 5.7.</blockquote><p>Fear not — there are still plenty of ways to hook into exported kernel functions, not just syscalls. Alexey Lozovsky wrote a <a href="https://www.apriorit.com/dev-blog/544-hooking-linux-functions-1">great post</a> on the different options and also made a POC library available <a href="https://github.com/ilammy/ftrace-hook">here</a> using Linux’s <a href="https://www.kernel.org/doc/html/v5.0/trace/ftrace.html">ftrace</a> functionality.</p><p>This ftrace-based function hooking works by dynamically intercepting kernel functions and allowing custom code execution around them. Here’s how it works:</p><ol><li><strong>Resolve the kernel function address</strong> by<strong> </strong>finding the memory address of the original function (specified by its name) using lookup_name(). This address is needed to set up ftrace filters.</li><li><strong>Create a thunk function</strong>. When this thunk function is called, it modifies the instruction pointer to execute our custom function instead of the original one.</li><li><strong>Register the hook</strong> with ftrace by setting up ftrace_ops and setting a filter on the original symbol or instruction pointer. ftrace will invoke our thunk function whenever the original symbol or pointer is called. You can think about this like setting up a breakpoint on the original function: when the kernel wants to jump there the thunk function gets called. Then the thunk function changes the instruction pointer and jumps to our custom function.</li><li>To <strong>avoid recursive calls</strong> (where the hooked function calls itself), the code skips over ftrace handling by directly modifying the instruction pointer.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zqgQ1x7t9wKbL9gr7yBZ0Q.png" /><figcaption>Installing a hook function via ftrace. fh_ftrace_thunk function will be invoked anytime when hook-&gt;address is called.</figcaption></figure><p>The only issue is that his code targets x86_64 architectures, while some of the embedded systems I use run on ARM64.</p><h4>ftrace based hooking on ARM64 platform</h4><p>While Alexey’s code works fine on Intel a few changes are required to make it work on ARM64 architectures. First and foremost, ARM64 uses different registers compared to x86_64. For example, the program counter on ARM64 is stored in the pc register, while on x86_64 it’s stored in the ip register. The hooking function needs to be updated to handle this difference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NpnN4gvodFI-n_Wbqr_zuA.png" /><figcaption>Thunk function supporting multiple architectures: changing directly the pr or ip registers to jump to our custom function.</figcaption></figure><p>Additionally, the FTRACE_OPS_FL_SAVE_REGS flag is not supported on ARM64. Therefore, we can either omit that flag or use FTRACE_OPS_FL_SAVE_REGS_IF_SUPPORTED for compatibility.</p><p>Lastly, instead of using ftrace_set_filter_ip, we should switch to ftrace_set_filter. Since ARM64 doesn’t have an ip register, using a symbol name-based approach is more platform-agnostic (or at least filter ip was throwing EINVAL while filter did work properly for me).</p><p>The modified code for basic syscall hooking on ARM can be found <a href="https://github.com/ilammy/ftrace-hook/tree/24e651ee5a0125ac583752273a35f65448e92dc1">here</a>.</p><h3>Hiding TracePid and virtual memory areas (maps)</h3><p>With this ftrace-based approach, our next task is to modify specific kernel functions to fix TracePid and hide memory maps.</p><h4>Hiding TracePid</h4><p><a href="https://github.com/LWSS">LWSS</a> published an ftrace based approach in his <a href="https://github.com/LWSS/TracerHid">TraceHid</a> repository. While his hooking code is x86_64 based, the hooked functions can be used as-is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*StoaINr36UYtpA_wTanWwQ.png" /><figcaption>Simple code to fix TracePid to 0 value</figcaption></figure><p>After hooking the proc_pid_status function (save it as orig_proc_pid_status and add hooked_proc_pid_status as a hook) we can changetask-&gt;ptrace to zero, then call the original function, then set it back to its original value. This will do the trick:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9XNoCDxhTPM_Wu7xgnS5ow.png" /><figcaption>After loading our module TracePid becomes 0.</figcaption></figure><h4>Hiding memory maps in /proc/$PID/self</h4><p>Each row in /proc/$PID/maps describes a region of contiguous virtual memory in a process or thread. In case we are preloading libraries via LD_PRELOAD or similar applications can assume they are being traced or modified:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wx0Flx7Y_ltApGo3e3lnDA.png" /><figcaption>dynamic library injection can be easily captured via proc/maps</figcaption></figure><p>Similar to proc_pid_status we should first hook into the original kernel function that returns the contents of /proc/self/maps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WfPQ4ng_rfAhEWx86-fGDA.png" /><figcaption>Remove all entries from maps that are listed in hide_maps_array</figcaption></figure><p>To make the kernel module more user-friendly (for example, allowing dynamic configuration of what to hide), one of the simplest methods is to define a sysctl entry. Sysctl values can be easily mapped to kernel-level variables, or even better, can trigger callback functions when the values are changed at the user level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1LbA48ubasFHU3SexEIWLw.png" /><figcaption>Dynamic filtering of the entries can be controlled through undebug.hide_maps sysctl</figcaption></figure><h3>Summary</h3><p>By using ktrace or similar methods (such as syscall hooking), we can easily modify the kernel’s default behavior. Existing kernel modules can also be adapted from x86_64 by simply changing instruction pointer references to the program counter for ARM64 compatibility.</p><p>The example code of this undebug kernel module can be found here: <a href="https://github.com/tfoldi/ko-undebug/tree/master">https://github.com/tfoldi/ko-undebug/tree/master</a></p><h4>Credits</h4><p>Most of the code is based on <a href="https://github.com/ilammy">ilammy/Alexey Lozovsky’s</a> <a href="https://github.com/ilammy/ftrace-hook">ftrace-hook repository</a>. The <em>TraceHid</em> code comes from LWSS’s <a href="https://github.com/LWSS/TracerHid">TracerHid repository</a>. Some anti-debug techniques are from legion.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49818db6d182" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Nvidia LPDNet vs Meta EgoBlur —blurring license plates with deep learning]]></title>
            <link>https://medium.com/starschema-blog/nvidia-lpdnet-vs-meta-egoblur-blurring-license-plates-with-deep-learning-d7cc2d770685?source=rss-a586bc473bbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7cc2d770685</guid>
            <category><![CDATA[pii]]></category>
            <category><![CDATA[license-plates]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Tamas Foldi]]></dc:creator>
            <pubDate>Mon, 13 Nov 2023 14:30:07 GMT</pubDate>
            <atom:updated>2023-11-28T11:05:32.672Z</atom:updated>
            <content:encoded><![CDATA[<h3>Nvidia LPDNet vs Meta EgoBlur — blurring license plates with deep learning</h3><p>Collecting data from cameras and sensors is all fun and games until legal and compliance matters enter the game. But that’s why we have superheroes who can protect us by masking or blurring out personal identifiable information (PII) like license plates or people’s faces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aM2w49XIgGL3s89uyRhPcw.png" /><figcaption>NVIDIA LPDNet and its new rival, Meta’s EgoBlur</figcaption></figure><p>Much like in various other aspects of life, you can build something from scratch or use pre-trained models available for free. Since most of us consider anonymizing datasets as a non-core task, instead of spending days building the next generation of license plate detectors, we use what’s available on the market.</p><h4>The Use Case</h4><p>Let’s imagine that we store camera feeds from a fleet of cars to use the video files for offline analysis, training or simulations. Due to privacy reasons, we need to blur any faces and license plates to comply with PII requirements. Looks like an easy CV task: some basic OpenCV code from the internet and we’re done.</p><p>Well, not really.</p><p>The issue is that external cameras can be dirty (one look at my car will reveal as much), visibility can be low and it could be raining or snowing outside, so our algorithm should be rock-solid in all circumstances.</p><p>But what are our options?</p><h4>Haar Cascades</h4><p>Perhaps the most trivial solution would be to use <em>Haar Cascades</em>, one of the most well-known solutions for real-time object detection. These algorithms rely on simple features and can detect objects quickly on resource-constrained devices. However, they are less sophisticated for capturing complex patterns in data than deep learning models, and you cannot easily train/retrain them with your own data sets.</p><p>Let’s try the default license plate model from OpenCV. While it was trained on Russian license plates, it can also detect ones from other countries:</p><pre>import cv2<br><br>faceCascade = cv2.CascadeClassifier(&#39;haarcascade_russian_plate_number.xml&#39;)<br><br>img = cv2.imread(&quot;input.jpg&quot;)<br>gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)<br><br>faces = faceCascade.detectMultiScale(gray,scaleFactor=1.05, <br>  minNeighbors = 5, minSize=(25,25))</pre><p>Applying it to my car’s image produces promising output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*Fu45fGLKjI1moyv_NvndyQ.jpeg" /><figcaption>Well, it found the license plate—two, actually. While Haar Cascades are fast, they often need to be retrained on larger datasets and require manual tuning.</figcaption></figure><p>Not great, not terrible. It’s okay. It found the plate. But even for a crisp image like this, we got another bounding box. Now, let’s try it out in a more realistic scenario using the left repeater cam’s image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*fQYyfJOT_bm19z-x7TUIdg.jpeg" /><figcaption>While we humans (assuming you’re also human) can find the license plate, the same Haar Cascade model failed this time.</figcaption></figure><p>No luck, it didn’t find the car’s plate.</p><p>If the image has some noise or the license plate is small, not frontal, we need a more heavyweight solution.</p><h4>Deep Learning FTW</h4><p>Convolutional Neural Networks (CNN) are the standard when it comes to object detection. With the recent advancements in deep learning, the availability of large training datasets and fast GPUs have enabled CNNs to outperform traditional algorithms in accuracy and speed for image recognition tasks. The most widely used models/frameworks for object detection that employ CNNs are:</p><ul><li><strong>YOLO</strong> (You Only Look Once): Known for its speed and efficiency in real-time detection.</li><li><strong>SSD</strong> (Single Shot Multibox Detector): Balances speed and accuracy by using a single neural network for detection tasks.</li><li><strong>Faster R-CNN</strong>: A more accurate model that introduces Region Proposal Networks for generating object proposals.</li><li><strong>RetinaNet</strong>: Uses a focal loss function to handle the class imbalance problem, which is helpful for detecting objects in varied scales.</li></ul><p>You can find license plate and face detection algorithms with any of these networks. Not surprisingly, the two most common models, <strong>LPDNet </strong>(YOLOv4) and <strong>EgoBlur</strong> (Faster R-CNN) are also built on one of these frameworks.</p><p>But are they good? And if so, which one is better for our use case?</p><h3><strong>NVIDIA LPDNet</strong></h3><p>NVIDIA offers two pre-trained license plate detector (LPD) models based on <strong>DetectNet_v2</strong> and <strong>YOLOv4-tiny</strong>. Both have versions trained on NVIDIA’s US license plate and CCPD datasets. DetectNet_v2 uses a ResNet18 feature extractor with GridBox detection, requiring post-processing like DBSCAN or NMS for final outputs. YOLOv4-tiny employs <strong>cspdarknet_tiny</strong> as its feature extractor. The US models are trained on over 45,000 images, while the CCPD models use about 172,000 images from a Chinese city’s streets, as detailed in an <a href="https://openaccess.thecvf.com/content_ECCV_2018/html/Zhenbo_Xu_Towards_End-to-End_License_ECCV_2018_paper.html">ECCV 2018 paper</a>. The training focused on minimizing localization and confidence loss, making it an ideal choice for a use case like ours. The LPDNet is generally optimized for fast inference, making it a feasible choice for edge devices (I was able to make it work on my old Jetson Nano).</p><p>Since YOLOv4-tiny models are newer and offer higher accuracy, I decided to start with that.</p><p><a href="https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/lpdnet">LPDNet</a> is available from the NVIDIA GPU Cloud (<a href="https://www.nvidia.com/en-eu/gpu-cloud/">NGC</a>) Catalog, as part of the the <a href="https://developer.nvidia.com/tao-toolkit">TAO Toolkit</a>. From the model&#39;s webpage, you can directly access the model files in TensorRT (TLT) format, prepared exclusively for NVIDIA GPUs. By <a href="https://www.nvidia.com/en-us/data-center/products/nvidia-ai-enterprise/eula/">definition</a>, the model is free to use in commercial applications, except for a few edge cases (competing with NVIDIA, etc.).</p><p>The NVIDIA ecosystem is largely a mess, so you might need to spend some time getting it to work if you’re using a non-standard configuration. Maybe the easiest way to work with the model is through <a href="https://developer.nvidia.com/deepstream-sdk">DeepStream SDK</a>, which is a set of plugins for inferencing on top of the <a href="https://gstreamer.freedesktop.org/">gstreamer</a> framework. I used the DeepStream SDK 6.0 container on my Jetson Nano and the 6.3 container on a cloud dGPU instance, and, after installing a few missing packages, things started to work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s40H_y4IwJO6jvOScxuNxA.png" /><figcaption>Where Haar Cascades failed, LPDNet v2.1 gave us 99% confidence. Overall, the model was accurate.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eDeEHIchBc5CH-HMAnGlDA.png" /><figcaption>I ran into false positives occasionally, even with 80%+ confidence. However, the accuracy was still in the expected range.</figcaption></figure><p>The results were great, LPDNet was able to detect LPs from low-res, low-quality feeds. (scroll down to the comparison video).</p><p>But what about EgoBlur? Can Meta build something better?</p><h3>Meta EgoBlur</h3><p>The <a href="https://github.com/facebookresearch/EgoBlur">EgoBlu</a>r model was recently released by Facebook Research, and it promises to do exactly what we need: removing license plates and faces from image and video streams.</p><p>The concepts are explained in the <a href="https://arxiv.org/abs/2308.13093">EgoBlur: Responsible Innovation in Aria</a> paper, but in short, Egoblur is based on the Faster RCNN model with a ResNext backbone. The two models (Face and LP) are trained using Meta’s publicly available <a href="https://ai.meta.com/tools/detectron2/">Detectron2</a> and <a href="https://ai.meta.com/blog/d2go-brings-detectron2-to-mobile/">Detectron2go</a> libraries, are approximately 400 MB and have ~104 million parameters each. It’s licensed under Apache-2.0, so it’s completely free.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uffdv2q13njgls4k7JUajQ.png" /><figcaption>EgoBlur is part of Project Aria, an AR Glass and SDK from Meta. EgoBlur is supposed to ensure that recorded videos and images from egocentric devices (like glasses) will be free of PII.</figcaption></figure><p>The Meta/Aria team aims to achieve performance comparable to (or better than) other publicly available methods for face and license plate detection on cameras, especially on AR and VR devices. According to my tests, the accuracy indeed is quite convincing, slightly better than LPDNet’s.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*shH66SrPZRYUme2hOMWIgw.png" /><figcaption>In addition to license plates, EgoBlur can also blur faces. All plates were identified even with a 90% confidence threshold.</figcaption></figure><p>Installing the model is also a pleasant experience: the GitHub repo comes with a conda environment file, making the project simple to deploy. EgoBlur can run on both GPUs and CPUs, and the GPU is not limited to NVIDIA chips only. However, due to some issues within PyTorch v2.1, I was unable to use it on Apple M2 silicon with metal performance shaders (MPS device).</p><p>Okay, so which model is the best?</p><h3>Side-by-Side Comparison</h3><p>Seeing is believing, so here’s a sample output from a video recording. I downloaded the video file straight from the car’s camera system, with all the noises and distortions you can expect from an on-board cam.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F9JK4_yZN2iU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D9JK4_yZN2iU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F9JK4_yZN2iU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e953eef6bff87ed6e787f7164640d3f3/href">https://medium.com/media/e953eef6bff87ed6e787f7164640d3f3/href</a></iframe><p>Both models provide accurate output, but EgoBlur slightly outperforms LPDNet in terms of false positives. Both output videos were recorded without object tracking across frames, and adding that will further improve the quality (license plates rarely disappear from one frame to another). DeepStream comes with an out-of-the-box tracker (<a href="https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvtracker.html">gst-nvtracker</a>), while in the case of EgoBlur, this is something we need to add to the pipeline ourselves.</p><p>From an accuracy point of view, the winner is EgoBlur according to the jury’s (my) subjective decision.</p><h4>Inference Performance</h4><p>Accuracy is great, but the price per minute of anonymization also matters a lot. LPDNet had blazingly fast inference, even on edge devices:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hbgsY_NEPgspfrlsgDQL7g.png" /><figcaption>LPDNet performs more than 1200 FPS on T4 and 40FPS on Nano (image from <a href="https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/lpdnet">https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/lpdnet</a>).</figcaption></figure><p>On the other hand, <strong>EgoBlur is slow</strong>. On their website, the FAQ answers a question on inference speed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ba5n9uoc0-BLTiQN4kYnQA.png" /><figcaption>EgoBlur claims 2 FPS for full HD image blurring. (Image from <a href="https://www.projectaria.com/tools/egoblur/">https://www.projectaria.com/tools/egoblur/</a>)</figcaption></figure><p>Not super fast. I also ran tests on T4 and V100 dGPUs, as well as on my Jetson Nano, and got similar results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NOPb9cLJ2aVadM0QyjTPaA.png" /><figcaption>Quick and dirty performance test results on my environments with 1280x720 images.</figcaption></figure><p>Here, LPDNet wins by orders of magnitude.</p><h3>Summary</h3><p>The verdict is straightforward: if you need real-time or fast inference with acceptable accuracy, and have NVIDIA silicons at hand, then go with LPDNet. You won’t be disappointed. If quality and accuracy are more important than speed, or your startup recently bought a big block of A100s, then EgoNet is a decent option, just don’t expect it to be fast.</p><p>REACH OUT TO US <a href="https://starschema.com/contact">HERE</a> TO LEARN MORE:</p><p><a href="https://starschema.com/contact">Contact</a></p><p>READ MORE STORIES FROM STARSCHEMA:</p><ul><li><a href="https://medium.com/starschema-blog/collecting-can-bus-data-with-ros2-and-qt-11cd4e4e0918">Collecting CAN Bus Data with ROS2 and Qt</a></li><li><a href="https://medium.com/starschema-blog/offline-camera-calibration-in-ros-2-45e81df12555">Offline Camera Calibration in ROS 2</a></li><li><a href="https://medium.com/starschema-blog/mqtt-on-steroids-running-emqx-enterprise-on-kubernetes-d1d92d97f9fa">MQTT on Steroids: Running EMQX Enterprise on Kubernetes</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7cc2d770685" width="1" height="1" alt=""><hr><p><a href="https://medium.com/starschema-blog/nvidia-lpdnet-vs-meta-egoblur-blurring-license-plates-with-deep-learning-d7cc2d770685">Nvidia LPDNet vs Meta EgoBlur —blurring license plates with deep learning</a> was originally published in <a href="https://medium.com/starschema-blog">HCLTech-Starschema Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Offline Camera Calibration in ROS 2]]></title>
            <link>https://medium.com/starschema-blog/offline-camera-calibration-in-ros-2-45e81df12555?source=rss-a586bc473bbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/45e81df12555</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[ros2]]></category>
            <dc:creator><![CDATA[Tamas Foldi]]></dc:creator>
            <pubDate>Fri, 03 Nov 2023 12:10:37 GMT</pubDate>
            <atom:updated>2023-11-03T12:42:46.603Z</atom:updated>
            <content:encoded><![CDATA[<p>In applications like self-driving cars, robots and augmented reality systems, where precise and reliable data is needed for positioning and overall perception, calibration is crucial. “Garbage data in, garbage data out,” as people say in today’s data world — and you can easily imagine the potential consequences of “garbage out” in the case of an autonomous vehicle or drone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VVmRcxluUQ5YAR4d_HcvrA.png" /><figcaption>The good old checkboard pattern</figcaption></figure><p>Preparing a computer vision pipeline almost invariably starts with camera calibration. When we calibrate a camera, we’re collecting information on its own features and how it sees the world. This includes knowing how far it can see and at what angle — for example, we cannot make good use of an image produced by a fisheye-like lens without knowing just how it distorts the image.</p><p>So, let me show you how I went about calibrating a camera in ROS 2 and how you can do the same.</p><h3>Online Calibration</h3><p>First, we have to collect the ROS 2 package for the calibration:</p><pre># on linux with apt<br>sudo apt install ros-&lt;ros2-distro&gt;-camera-calibration<br><br># on mac with robostack<br>mamba install ros-&lt;ros2-distro&gt;-camera-calibration </pre><p>Then, we have to ensure we have a camera driver compatible with our camera. I tend to reply in the following packages for these systems:</p><ul><li><a href="https://index.ros.org/r/usb_cam/#galactic">usb_cam</a> on Linux using V4L2. It does not work on Mac or Windows.</li><li><a href="https://github.com/clydemcqueen/opencv_cam">opencv_cam</a> on Mac (or Windows) using OpenCV to capture images from a camera or files.</li><li><a href="https://github.com/dusty-nv/ros_deep_learning/tree/9b5a668ba2cb24708d35eb13f734f343be223bd0#video_source-node">video_source</a> on Nvidia Jetson devices (supports both cameras and flat files).</li></ul><p>We also need a pattern to use during the calibration. You can generate one easily here: <a href="https://calib.io/pages/camera-calibration-pattern-generator">https://calib.io/pages/camera-calibration-pattern-generator</a>. To fit your pattern on an A4/US letter page, I suggest an 8x10 board with 15mm squares. Print it on thick paper to ensure it doesn’t bend during the calibration process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Juvl0tliR-Y4yu8frlhZoA.png" /><figcaption>An 8x10 checkerboard with 15mm squares is a good option.</figcaption></figure><p>To start calibration in one console, start the following command:</p><pre>ros2 run camera_calibration cameracalibrator --size 7x9 --square 0.015 \<br>  --ros-args -r image:=/image_raw</pre><p>The size option here denotes interior corners (e.g. a standard chessboard is 7x7), so for an 8x10 checkerboard, we go with 7x9.</p><p>On the other console, start your camera capture:</p><pre># in case of opencv_cam driver<br>ros2 run opencv_cam opencv_cam_main</pre><p>You should now see the calibration window and begin the calibration process. Next, move the pattern to all screen corners and tilt in every direction. When enough information is gathered, press the calibrate button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bW3FA6xozKT2XAljKhdmLQ.png" /><figcaption>Try to remember what you do here when switching to offline mode.</figcaption></figure><p>Now you can see your camera calibration data in the console. You can simply save it in a file with ini extension or press the save button in the app to save the same in a tarball with both ini and yaml format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kHnKxd-8CIjUXZ4kU9yK5w.png" /><figcaption>An example of calibration output for my MacBook camera. Save the contents from image till the end to an ini file.</figcaption></figure><h3>Offline Calibration</h3><p>The process will bethe same as above, but instead of using your camera image, ingest video files to the /video_raw topic. Most camera drivers can do this, but I prefer opencv_cam due to its Mac support.</p><p>First, record a video of you dancing in front of your camera:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0fy11fxCBHw3I0Pz7828Ew.png" /><figcaption>Calibrating my car’s cameras is always a fun.</figcaption></figure><p>Then, load the video files using opencv_cam</p><pre>ros2 run opencv_cam opencv_cam_main --ros-args -p file:=true \<br>  -p filename:=/../TeslaCam/RecentClips/2023-11-01_15-07-10-front.mp4</pre><h3>Viewing the Results</h3><p>You can also use the same opencv_cam command to replay your recordings, but this time, you should also specify the camera_info_path argument for feeding data to the /camera_info topic. This topic uses the <a href="https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/CameraInfo.html">CameraInfo</a> message to tell upstream systems how to interpret the images from video frames.</p><p>The easiest way to visualize the difference between the calibrated and non-calibrated images is to open two image windows in Foxglove, one with a calibration topic and one without it. The image on the left is the raw image, while the right is the one we can use for further processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*czcSob0cIisF3MuW72Wwew.png" /><figcaption>You can easily compare calibrated images and video streams side by side before and after calibration.</figcaption></figure><h3>Summary</h3><p>The calibration process I described here is a necessary step in a computer vision pipeline, especially if you will use the images for localization or mapping. ROS 2 supports this project out of the box with a calibrator, camera info messages, services and visualization tools.</p><p>REACH OUT TO US <a href="https://starschema.com/contact">HERE</a> TO LEARN MORE:</p><p><a href="https://starschema.com/contact">Contact</a></p><p>READ MORE STORIES FROM STARSCHEMA:</p><ul><li><a href="https://medium.com/starschema-blog/collecting-can-bus-data-with-ros2-and-qt-11cd4e4e0918">Collecting CAN Bus Data with ROS2 and Qt</a></li><li><a href="https://medium.com/starschema-blog/mqtt-on-steroids-running-emqx-enterprise-on-kubernetes-d1d92d97f9fa">MQTT on Steroids: Running EMQX Enterprise on Kubernetes</a></li><li><a href="https://medium.com/starschema-blog/avoid-these-3-mistakes-when-working-with-a-synapse-dedicated-sql-pool-621986fd7c80">Avoid These 3 Mistakes When Working with a Synapse Dedicated SQL Pool</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45e81df12555" width="1" height="1" alt=""><hr><p><a href="https://medium.com/starschema-blog/offline-camera-calibration-in-ros-2-45e81df12555">Offline Camera Calibration in ROS 2</a> was originally published in <a href="https://medium.com/starschema-blog">HCLTech-Starschema Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Collecting CAN Bus Data with ROS2 and Qt]]></title>
            <link>https://medium.com/starschema-blog/collecting-can-bus-data-with-ros2-and-qt-11cd4e4e0918?source=rss-a586bc473bbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/11cd4e4e0918</guid>
            <category><![CDATA[can-bus]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[tesla]]></category>
            <category><![CDATA[foxglove]]></category>
            <category><![CDATA[ros2]]></category>
            <dc:creator><![CDATA[Tamas Foldi]]></dc:creator>
            <pubDate>Wed, 25 Oct 2023 14:20:30 GMT</pubDate>
            <atom:updated>2023-11-03T11:46:40.010Z</atom:updated>
            <content:encoded><![CDATA[<h3>Collecting CAN Bus Data with ROS 2 and Qt</h3><p>One obvious way to learn something new is to use stuff you already have. This was my thinking when I decided to invest my time in learning sensor fusion at the edge — more precisely, performing sensing and perception in ROS 2 to see how autonomous vehicles work. I drive a Tesla as a company car, and it’s outfitted with tons of cameras and sensors, so I dusted off my Raspberries and got to wiring.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gh7p5_ovsroUFojzKyDdKA.png" /><figcaption>Looking at CAN frames and signals in my car was the first step in the effort to collect all required data.</figcaption></figure><p>To meaningfully perform perception, we need sensing, and for sensing, we need sensors. AVs typically use the following telemetry and sensor data to see and understand the vehicle and its surroundings:</p><ul><li>Car telemetry like speed and distance to support odometry. These usually come from the CAN bus, the car’s internal communication backbone;</li><li>GPS position signals, also usually coming through CAN bus;</li><li>Cameras like dash cams;</li><li>LIDAR and RADAR point clouds.</li></ul><p>We need to fuse all these sensors to gain full awareness of our and surrounding objects’ position. Thankfully, ROS 2 has a lot of prebuilt packages that can do the heavy lifting, which helps me focus more on the collection and analysis rather than reinventing and reimplementing sensor fusing algorithms.</p><h3>Collecting CAN Bus Signals</h3><p>I bought a <a href="https://www.csselectronics.com/products/can-bus-logger-interface-cl1000">CL1000</a> adapter to log the CAN bus signals. To connect it to my Tesla 3, I ordered an <a href="https://www.aliexpress.com/item/1005003069207525.html">OBD2 16PIN-26BIN</a> adapter along with an <a href="https://www.csselectronics.com/products/obd2-db9-adapter-cable">OBD2</a> adapter. Wiring them up wasn’t hard, but as it’s mentioned everywhere, do not forget to turn off your car before this step, otherwise, it can harm your vehicle. CSS Electronics also has a nice <a href="https://www.csselectronics.com/pages/tesla-data-dashboard-telematics-can-bus-grafana">blog post</a> on how to connect their logger.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x1XimJ9nm2neU4DSvnbK_A.png" /><figcaption>I know, I need to clean my car more frequently. Anyway, the logger simply connected to the Tesla’s CAN bus.</figcaption></figure><p>Out of the box, the logger supports Qt-based CAN bus analyzer tools like SavvyCAN, so I was able to quickly check if the connection was OK:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MFpfTWHnFi8tFX5PkC5pRA.png" /><figcaption>I was able to see CAN signals in SavvyCAN in real time.</figcaption></figure><p>The CL1000 already comes with a Qt5 <a href="https://doc.qt.io/qt-6/qtcanbus-backends.html">QtCanBus</a> plugin that makes it easy to use with any Qt application. The advantage of this <a href="https://doc.qt.io/qt-6/qtcanbus-backends.html">QtCanBus</a>-based connection was that I could use my Mac for debugging and reverse engineering, as <a href="https://www.kernel.org/doc/html/latest/networking/can.html">socketcan</a> (the standard can interface) primarily works with Linux.</p><h4>Adding Qt CanBus Support for ROS 2</h4><p>The next step on my list was to feed the data from the car to an ROS 2 topic, decode it and dispatch it as standard messages like Twist, NavSatFix or DiagnoticArray to make them available for standard sensor fusing libraries.</p><p>I didn’t find any ROS 2 library that captures and decodes CAN frames the way I wanted, so I ended up building my own set of packages.</p><p>I also couldn’t find a standard CAN bus message type in ROS 2, but ROS 1 Industrial’s can_msgs/Frame from <a href="https://github.com/ros-industrial/ros_canopen/blob/melodic-devel/can_msgs/msg/Frame.msg">https://github.com/ros-industrial/ros_canopen</a>/ was love at first sight, so I used it as <a href="https://github.com/tfoldi/ros2_qtcanbus/blob/main/ros2_qtcanbus_msgs/msg/QCanBusFrame.msg">QCanBusFrame.msg</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UuWcyy80CpKxWhxW42uuVQ.png" /><figcaption>ros2_qtcanbus_msgs/QCanBusFrame, same as can_msgs/Frame from ROS1 Industrial</figcaption></figure><p>To build the publisher node, I used libros2qt, a small library that provides a ROS 2 executor that enables running QApplication and ROS nodes from the same process.</p><p>The processing node’s code is fairly straightforward: when a frame becomes available from QtCanDevice, the code publishes it as a QCanBusFrame message to the /from_can_bus topic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GfL_mJItVY5-xGMqAip1MQ.png" /><figcaption>Publishing frames from QtCanBus callback</figcaption></figure><h4>Decoding Signals with DBC Files</h4><p>The next step is to make sense of the data collected from the vehicle. So far, we have the frameId (do not confuse this with ROS’s frame_id) and 8 bytes of data (or less) per frame.</p><p>The solution is to use database definitions like DBC files. There’s a really good write-up on what DBC files are and how they work <a href="https://www.csselectronics.com/pages/can-dbc-file-database-intro">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2wNabbiW7NYkqrD_u9zvog.png" /><figcaption>DBC file entry for a frame and a signal, image from <a href="https://www.csselectronics.com/pages/can-dbc-file-database-intro">https://www.csselectronics.com/pages/can-dbc-file-database-intro</a></figcaption></figure><p>I went ahead and picked <a href="https://github.com/joshwardell/model3dbc/blob/master/Model3CAN.dbc">Josh Wardell’s Model3.dbc</a> file, with additional records and collections from Greg Zimmerman’s <a href="https://raw.githubusercontent.com/thezim/DBCTools/master/Samples/tesla_model3.dbc">GitHub repo</a> (here), which had more up-to-date signal definitions.</p><p>After I had a DBC file to decode the data and a node publishing CAN bus data to a ROS topic, the next task was decoding. I chose to go with a Python node this time instead of C++, as I was somewhat familiar with the <a href="https://cantools.readthedocs.io/en/latest/">cantools</a> library.</p><p>The process is:</p><ol><li>ros2_qtcanbus/qtcanbus_sender node sends messages to /from_can_bus topic in format:</li></ol><pre>id: 599<br>data: [43, 88, 34, 4, 2, 64, 17, 1]</pre><p>2. ros2_candecode/candecode_node decodes the message using DBC database.</p><p>For message #599 from above, the DBC frame definition is:</p><pre>BO_ 599 ID257DIspeed: 8 VehicleBus<br> SG_ DI_speedChecksum : 0|8@1+ (1,0) [0|255] &quot;&quot;  Receiver<br> SG_ DI_speedCounter : 8|4@1+ (1,0) [0|15] &quot;&quot;  Receiver<br> SG_ DI_uiSpeed : 24|9@1+ (1,0) [0|510] &quot;&quot;  Receiver<br> SG_ DI_uiSpeedHighSpeed : 34|9@1+ (1,0) [0|510] &quot;&quot;  Receiver<br> SG_ DI_uiSpeedUnits : 33|1@1+ (1,0) [0|1] &quot;&quot;  Receiver<br> SG_ DI_vehicleSpeed : 12|12@1+ (0.08,-40) [-40|285] &quot;kph&quot;  Receiver<br><br>VAL_ 599 DI_uiSpeedUnits 1 &quot;DI_SPEED_KPH&quot; 0 &quot;DI_SPEED_MPH&quot; ;</pre><p>Then, decoding the data translates to:</p><pre>{&#39;DI_speedChecksum&#39;: 221, <br> &#39;DI_speedCounter&#39;: 13, <br> &#39;DI_vehicleSpeed&#39;: 46.96000000000001, <br> &#39;DI_uiSpeed&#39;: 48, <br> &#39;DI_uiSpeedUnits&#39;: 1, <br> &#39;DI_uiSpeedHighSpeed&#39;: 0}</pre><p>3. If we enable the decode_choice option, numbers will be replaced with text values according to the DBC database.</p><pre># replaces the number value in the message with the matching string value<br>{...&#39;DI_uiSpeedUnits&#39;: &#39;DI_SPEED_KPH&#39;, ...} </pre><p>4. All frames will be published to /diagnostics topic with the type DiagnosticArray. One array will be one CAN frame with all signals as KeyValue. While I do not like the idea of storing values as strings, this seemed the most standard message type for the decoded messages.</p><p>5. In the case of GPS coordinates or speed/IMU datasets, the package supports translating these messages to Twist and NavSatFix messages for sensor fusing, like for the localization of the car.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gdK0T9hEQbbJViurh0t-tg.png" /><figcaption>Default config file with parameters for my ros2_candecode package</figcaption></figure><p>After running the two nodes (ros2_qtcanbus and ros2_candecode), all the configured topics should be accessible with the respective decoded images.</p><p>Now, let’s have a look at what we’ve collected.</p><h3>Visualizing CAN Bus Data with Foxglove</h3><p>Don’t get your hopes up too much just yet, there isn’t much we can do with the data we have — the fancier sensors (cameras, LIDARs) are still down the line. Still, seeing is believing, so let’s look inside:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q8iidvX4YTojHbAxbQ9ySQ.gif" /><figcaption>Speed, pedal position, and steering wheel angle with a neat map — all from our Tesla’s CAN bus in <a href="http://studio.foxglove.dev/">Foxglove Studio</a>.</figcaption></figure><p>By using <a href="http://studio.foxglove.dev/">Foxglove Studio</a> (or with Rviz2) we can see the freshly populated topics, including hundreds of sensors from different systems in our very own car.</p><h3>Summary</h3><p>We’ve taken the first step to collect the necessary data to understand how autonomous vehicles work. With not-so-expensive cables and CAN loggers, you too can easily start your ROS 2 and AV journey, with Foxglove providing a great tool to see where exactly you are in your physical journey.</p><p>Find the source code of the packages here: <a href="https://github.com/tfoldi/ros2_qtcanbus/">https://github.com/tfoldi/ros2_qtcanbus/</a></p><p>REACH OUT TO US <a href="https://starschema.com/contact">HERE</a> TO LEARN MORE:</p><p><a href="https://starschema.com/contact">Contact</a></p><p>READ MORE STORIES FROM STARSCHEMA:</p><ul><li><a href="https://medium.com/starschema-blog/mqtt-on-steroids-running-emqx-enterprise-on-kubernetes-d1d92d97f9fa">MQTT on Steroids: Running EMQX Enterprise on Kubernetes</a></li><li><a href="https://medium.com/starschema-blog/avoid-these-3-mistakes-when-working-with-a-synapse-dedicated-sql-pool-621986fd7c80">Avoid These 3 Mistakes When Working with a Synapse Dedicated SQL Pool</a></li><li><a href="https://medium.com/starschema-blog/find-the-balance-between-cloud-cost-and-efficiency-889958c39908">Find the Balance between Cloud Cost and Efficiency</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11cd4e4e0918" width="1" height="1" alt=""><hr><p><a href="https://medium.com/starschema-blog/collecting-can-bus-data-with-ros2-and-qt-11cd4e4e0918">Collecting CAN Bus Data with ROS2 and Qt</a> was originally published in <a href="https://medium.com/starschema-blog">HCLTech-Starschema Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Schedule and Invoke Notebooks as Web Services using Jupyter API]]></title>
            <link>https://medium.com/data-science/schedule-and-invoke-notebooks-as-web-services-using-jupyter-api-fb71c1e8a008?source=rss-a586bc473bbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb71c1e8a008</guid>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Tamas Foldi]]></dc:creator>
            <pubDate>Thu, 10 Aug 2023 15:37:13 GMT</pubDate>
            <atom:updated>2023-08-10T15:37:13.302Z</atom:updated>
            <content:encoded><![CDATA[<p>Thanks to serverless cloud services like GCP CloudRunner and Cloud Functions, we don’t need to manage costly virtual machines or servers to deploy our notebooks and execute them periodically. With Jupyter API, you can lift and shift your notebooks to the cloud, turn them into web services and integrate them with scheduling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*hn35U_DUunDE1ketwpklcQ.png" /><figcaption>Python notebook scheduled in the cloud, generated by MidJourney, instructed by the author</figcaption></figure><p>However, the most commonly used approach (unless you’re using a cloud-native services like Vertex AI or SageMaker) is to convert notebooks to Python code using nbconvert and adding the code to a freshly bootstrapped Tornado or Flask custom web application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JDOBuE6jqBO_4Wa0rqc5bQ.png" /><figcaption>Traditional containerization of a Python Notebook without, image by the author</figcaption></figure><p>This includes some coding and external libraries, but the good news is that we can leave our code in our Jupyter development container and trigger it directly from there using the <a href="https://jupyter-server.readthedocs.io/en/latest/developers/rest-api.html">Jupyter Rest API</a>.</p><h3>Accessing a Notebook over Web API</h3><p>Before we jump into the details of how to use Jupyter API, I’ll demonstrate how the architecture will work. First, let’s take a simple notebook we can use for testing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i5GDPWr-LxxEwe_-6ZoizA.png" /><figcaption><a href="https://github.com/tfoldi/jupyterapi_nbrunner/blob/main/tests/test_notebook/JupyterAPI_Test.ipynb">Simple test notebook</a> returning with “15&quot; if all works well.</figcaption></figure><p>To run it locally using Jupyter, the easiest way is to run it in a Jupyter Lab container:</p><pre># download the test workbook<br>wget https://raw.githubusercontent.com/tfoldi/vizallas/main/notebooks/JupyterAPI_Test.ipynb<br><br># Spawn a new Jupyter lab instance with token auth (and without XSRF)<br>docker run -it --rm -p 8888:8888 \<br>  -e JUPYTER_TOKEN=ab30dd71a2ac8f9abe7160d4d5520d9a19dbdb48abcdabcd \<br>  --name testnb -v &quot;${PWD}&quot;:/home/jovyan/work jupyter/base-notebook \<br>  jupyter lab --ServerApp.disable_check_xsrf=true</pre><p>Once the service starts, you’ll be able to access the notebook at <a href="http://127.0.0.1:8888/lab/tree/work">http://127.0.0.1:8888/lab/tree/work</a> using the token passed in the JUPYTER_TOKEN environment variable.</p><h4>Calling the Notebook from the command line</h4><p>From the command line, you can download <a href="https://github.com/tfoldi/jupyterapi_nbrunner/blob/main/main.py">this</a> small script (it requires requests and websocket-client packages) or run it through a Docker container:</p><pre># check the IP address of our previously started &quot;testnb&quot; container<br>docker inspect testnb | grep IPAddress<br>            &quot;SecondaryIPAddresses&quot;: null,<br>            &quot;IPAddress&quot;: &quot;172.17.0.2&quot;,<br>                    &quot;IPAddress&quot;: &quot;172.17.0.2&quot;,<br><br># Invoke our notebook. Replace the IP below with yours from previous step.<br>docker run -it --rm \<br>  -e JUPYTER_TOKEN=ab30dd71a2ac8f9abe7160d4d5520d9a19dbdb48abcdabcd \<br>  tfoldi/jupyterapi_nbrunner 172.17.0.2:8888 /work/JupyterAPI_Test.ipynb<br><br>Creating a new kernel at http://172.17.0.2:8888/api/kernels<br>Sending execution requests for each cell<br>{&#39;data&#39;: {&#39;text/plain&#39;: &#39;15&#39;}, &#39;execution_count&#39;: 3, &#39;metadata&#39;: {}}<br>Processing finished. Closing websocket connection<br>Deleting kernel</pre><p>This script connects to our newly created JupyterLab server, executes our notebook, returns the last cell’s result, then exits. The entire procedure occurs over web protocols without requiring any modifications to the notebook code or additional libraries.</p><h4>Under the hood</h4><p>Unfortunately, there’s no single endpoint in Jupyter API to execute a notebook end-to-end. First, we have to initialize a new kernel (or use an existing one), retrieve the notebook metadata, get all code cells and send an execute_request for each one of them.</p><p>To retrieve the results, we need to listen for incoming messages in the WebSocket channel. Since there’s no “end of all code executions” message, we have to manually keep track of how many code blocks we sent for execution and how many of them have actually been completed by counting all messages of type execute_reply. Once everything has finished executing, we can stop the kernel or leave it in an idle stage for future executions.</p><p>The following diagram illustrates the complete flow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P4JnLVgS8FL0WlmHFN7pLg.png" /><figcaption>Steps to execute a Jupyter Notebook over the Rest API. Notebook-level actions use Rest API, while cell-level invocations are on WebSockets. Image by the author.</figcaption></figure><p>To remain authenticated, we have to pass the Authorization header for all HTTP and WebSocket calls.</p><p>If this feels like a somewhat excessive amount of steps just to execute a notebook, I hear you. I’m sure it would be useful to implement a higher-level function inside Jupyter Server to reduce the complexity.</p><p>The complete <a href="https://github.com/tfoldi/jupyterapi_nbrunner/blob/main/main.py">script is here</a>, ready to be used in your applications.</p><h3>Schedule our workbook on GCP for Free (almost)</h3><p>While there are many options for hosting a notebook, the most cost-effective way is to leverage Google Cloud’s Cloud Run service. With Cloud Run, you only pay for the exact runtime of your job, making it a cost-efficient choice for infrequently triggered tasks without extra packages or additional SaaS vendors (other than Google) — and, again, without writing a single line of code.</p><p>The architecture and invocation flow will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-xu_f0jZRJk4lYGR3WCPbA.png" /><figcaption>We’re going to use serverless services only to keep the costs down. Image by the author.</figcaption></figure><p>First, we have to deploy our notebook in GCP Cloud Run. There are multiple ways to add a file to a Cloud Run service, but perhaps the easiest one is to copy our notebooks to a Docker container.</p><pre># Simple dockerfile to host notebooks on a Jupyter Server<br>FROM jupyter/base-notebook<br>COPY JupyterAPI_Test.ipynb /home/jovyan/workspaces/</pre><p>To build and make the container available in Cloud Run, we can simply specify the --source option with gcloud run deploy, pointing it to a directory where our notebooks and Dockerfile are located.</p><pre># get the source code of the Jupyter notebook and the Dockerfile<br>git clone https://github.com/tfoldi/jupyterapi_nbrunner.git<br><br># Deploy the test notebook in a jupyter/base-notebook container <br># The Dockerfile and JupyterAPI_Test.ipynb files in the tests/test_notebook <br># folder<br>gcloud run deploy test-notebook --region europe-west3 --platform managed \<br>  --allow-unauthenticated --port=8888 \<br>  --source tests/test_notebook \<br>  --set-env-vars=JUPYTER_TOKEN=ab30dd71a2ac8f9abe7160d4d5520d9a19dbdb48abcdabcd <br>  <br>[...]<br>Service [test-notebook] revision [test-notebook-00001-mef] has been deployed and is serving 100 percent of traffic.<br>Service URL: https://test-notebook-fcaopesrva-ey.a.run.app</pre><p>JupyterLab will be available at the Service URL. Google Cloud Run will provide the SSL certificates and the mechanisms to start or suspend the container depending on requests hitting the deployment.</p><p>To trigger our newly deployed notebook from the Cloud Scheduler, we have to create a Cloud Function tied to a PubSub topic. The following command will deploy main.py and requirements.txt from <a href="https://github.com/tfoldi/jupyterapi_nbrunner/">this repository</a>. The main.py is the same script we used previously to trigger our code from the command line.</p><pre># make sure you are in the same directory where you cloned the <br># contents of https://github.com/tfoldi/jupyterapi_nbrunner.git <br><br>gcloud functions deploy nbtrigger --entry-point main --runtime python311 \<br>  --trigger-resource t_nbtrigger --trigger-event google.pubsub.topic.publish \<br>  --timeout 540s --region europe-west3 \<br>  --set-env-vars=JUPYTER_TOKEN=ab30dd71a2ac8f9abe7160d4d5520d9a19dbdb48abcdabcd</pre><p>Let’s test our new Cloud Function by sending a message to the t_nbtrigger topic with the appropriate parameters, just like we did in the command line:</p><pre>gcloud pubsub topics publish t_nbtrigger \<br>  --message=&quot;test-notebook-fcaopesrva-ey.a.run.app:443 <br>       /workspaces/JupyterAPI_Test.ipynb --use-https&quot;</pre><p>If you check the nbtrigger Cloud Function’s logs, you may notice that emitting a record to the topic successfully triggered the execution of the notebook we specified:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EnlZ9BF6QaLpc2g9GbCw9Q.png" /><figcaption>The logs show the successful execution of our notebook. Image by the author.</figcaption></figure><p>The final step is to create a schedule that runs at specified times. In this case, we are about to run our notebook every hour:</p><pre>gcloud scheduler jobs create pubsub j_hourly_nbtrigger \<br>  --schedule &quot;0 * * * *&quot; --topic t_nbtrigger --location europe-west3 \<br>  --message-body &quot;test-notebook-fcaopesrva-ey.a.run.app:443 /workspaces/JupyterAPI_Test.ipynb --use-https --verbose&quot;   </pre><p>You’re all set — you just scheduled your first Jupyter Notebook in a serverless way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2OSTarw4cPzU11frRmR6yA.png" /><figcaption>CloudRun automatically shuts down our container after the job execution. The “idle” state is also free of charge in case we do not specify min-instances.</figcaption></figure><p>Our Notebook will consume only a few cents per day, making this deployment method one of the most cost-effective in Google Cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wudc1inBOqXMdDbFco9t6g.png" /><figcaption>After few days of execution the costs are around three cents.</figcaption></figure><h3>Conclusion</h3><p>We used to rely on converting our Jupyter Notebooks to Python code to make them available to cloud-native tools, or on more complex and expensive services like Vertex AI or SageMaker. However, by utilizing the Jupyter Rest API and deploying your Notebooks along with its “development environment,” you can skip the extra steps and enable web service calls or scheduling for your Notebooks.</p><p>While this approach is not necessarily appropriate for large-scale projects with compute-heavy long-running notebooks, it is perfectly fine for your home automation or hobby projects — without (over)spending on infrastructure.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb71c1e8a008" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/schedule-and-invoke-notebooks-as-web-services-using-jupyter-api-fb71c1e8a008">Schedule and Invoke Notebooks as Web Services using Jupyter API</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MQTT on Steroids: Running EMQX Enterprise on Kubernetes]]></title>
            <link>https://medium.com/starschema-blog/mqtt-on-steroids-running-emqx-enterprise-on-kubernetes-d1d92d97f9fa?source=rss-a586bc473bbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/d1d92d97f9fa</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[mqtt]]></category>
            <category><![CDATA[pulumi]]></category>
            <category><![CDATA[emq-x]]></category>
            <dc:creator><![CDATA[Tamas Foldi]]></dc:creator>
            <pubDate>Tue, 14 Feb 2023 11:34:47 GMT</pubDate>
            <atom:updated>2023-02-14T11:34:47.380Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*BO5C5Gfx0JhFUwNsvouV9w.png" /><figcaption>Image generated by Dall-E.</figcaption></figure><p>Working a lot in the IT-OT integration space with demanding clients, we at Starschema have been using <a href="https://www.emqx.io/">EMQX</a> as a standard MQTT message broker due to its robust features, scalability and high-performance capabilities.</p><p>EMQX has proven to be a pretty neat broker, so to support the broader community, we’ve open-sourced our deployment scripts for it in <a href="https://github.com/starschema/antares-idl">Antares iDL</a>, our modern data stack deployment system. In this blog, I will show you how to deploy it in your cloud or an on-prem Kubernetes cluster with a single click (well, more like a single command line).</p><h3>Introducing EMQX, Your Open-Source MQTT Broker</h3><p>EMQX is an open-source IoT (internet of things) message broker that supports MQTT, MQTT-SN, CoAP, Sparkplug B and LwM2M protocols. Its core advantages are:</p><ol><li><strong>Scalability</strong>: EMQX can handle millions of concurrent connections and messages. It’s written in Erlang, a language designed for scalability with low latency and high availability.</li><li><strong>High availability</strong>: EMQX supports horizontal clustering and message persistence to ensure high availability.</li><li><strong>Customizability</strong>: EMQX offers a plugin architecture that enables custom extensions and enhancements.</li></ol><p>We prefer EMQX in our projects for the following reasons:</p><ol><li><strong>Open source</strong>: EMQX is open-source and free to use, with a large community of developers contributing to its development and evolution. There’s also an Enterprise version for larger deployments and commercial support.</li><li><strong>High-performance</strong>: It’s designed for high-throughput, low-latency message processing. In the IIoT (industrial internet of things) space, it’s common to expect messages from millions of devices or process millions of messages.</li><li><strong>Built-in processing engine</strong>: EMQX supports routing, filtering and alerting out-of-the-box. It can also send messages to external systems like databases or other message queues.</li><li>And it <strong>works on-premises</strong>, as well in the <strong>cloud or hybrid deployments</strong>.</li></ol><h3>Deploying in Kubernetes</h3><p>In our Antares iDL platform, we use only cloud-native services or Kubernetes deployments. For EMQX, we took the latter method and deployed it in Kubernetes, which provides several advantages, including:</p><ol><li><strong>Scalability</strong>: Kubernetes makes it easy to scale EMQX horizontally by adding or removing nodes as needed. EMQX has its own Kubernetes operator to take care of scaling.</li><li><strong>Resilience</strong>: Kubernetes provides automatic failover and self-healing capabilities, ensuring high availability for EMQX.</li><li><strong>Portable deployment</strong>: Kubernetes allows EMQX to be deployed on-premises, in the cloud or in hybrid environments, which provides greater flexibility and portability. The same deployment automation can work everywhere.</li></ol><h4>Deploying with Command Line Only</h4><p>In order to install EMQX, first we need to install cert-manager, a Kubernetes-native service for managing SSL certificates with Let’s Encrypt and other providers:</p><figure><img alt="helm repo add jetstack https://charts.jetstack.io helm repo update helm install \ cert-manager jetstack/cert-manager \ — namespace cert-manager \ — create-namespace \ — version v1.11.0 \ — set installCRDs=true" src="https://cdn-images-1.medium.com/max/1024/1*26esqpIag-_7Oxbm_UKCKQ.png" /><figcaption>Install cert-manager with Helm. Make sure you install custom resources (CRDs) as well.</figcaption></figure><p>When it completes, you can proceed to install EMQX’s operator. This operator will manage and operate EMQX clusters on our Kubernetes cluster and automate tasks such as deployment, scaling, upgrades and backups. It also provides ensures high availability through automatic failover.</p><figure><img alt="helm repo add emqx https://repos.emqx.io/charts helm repo update helm install emqx-operator emqx/emqx-operator \ — namespace emqx-operator-system — create-namespace" src="https://cdn-images-1.medium.com/max/1024/1*xivQKuZNHCqpdy_OvKUMuw.png" /><figcaption>Install EMQX Operator through their official Helm chart.</figcaption></figure><p>While other MQTT brokers (like HiveMQ) instantly provision a new cluster after Helm install, in EMQX you have to define your own cluster as a CRD. You can go with a plain EMQX broker or with the Enterprise Edition. In the latter case, the following Kubernetes resource will do the trick:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6HOssQuOyGTC3K15jVcA5Q.png" /><figcaption>The resource kind is EmqxEnterprise in case of the Enterprise Edition.</figcaption></figure><p>If all is well, you should be able to see your EMQX pods and services. The admin port is exposed by the generated Service port 18083. The default username is admin and the password is public.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7nLvLQokbef8A-rLFKt5ig.png" /><figcaption>Fresh new installation of EMQX Enterprise Edition. Without a trial license, you’re limited to 10 connections per second. The open-source version does not have this limitation.</figcaption></figure><h4>Deploying with Pulumi and Antares iDL</h4><p>Internally, we prefer using <a href="https://www.pulumi.com/">Pulumi</a> for deploying infrastructure. It offers several advantages over Terraform or manual, Helm-based installations. The biggest on is that you can write the infra code in modern programming languages (TypeScript, Go and Python, if that qualifies as modern), with advanced templating and control structures. On top of this, Pulumi’s cross-platform architecture eliminates the need to write separate code for different cloud providers, making it easier to manage infrastructure across multiple clouds.</p><p>To install cert-manager, we use the following Pulumi code, which relies on Release resource:</p><figure><img alt="from pulumi_kubernetes.helm.v3 import Release, ReleaseArgs, RepositoryOptsArgs def deploy(): cert_manager_release = Release( “cert-manager”, ReleaseArgs( chart=”cert-manager”, name=”cert-manager”, repository_opts=RepositoryOptsArgs( repo=”https://charts.jetstack.io/”, ), namespace=”cert-manager”, create_namespace=True, values={“installCRDs”: “true”}, ), )" src="https://cdn-images-1.medium.com/max/1024/1*JnX4ojao3Op9jiLlHrghBQ.png" /><figcaption>The cert-manager will have its own namespace.</figcaption></figure><p>Then, we install the EMQX Kubernetes operator:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mwhendy5c9-_1ZLuA1PdRg.png" /><figcaption>Install and configure EMQX with Pulumi.</figcaption></figure><p>To install EMQX in a single command line, make sure you enable cert-manager and emqx in your Pulumi stack config.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z-HYFYDv9dJWWhTVPeY3Jw.png" /></figure><p>Now, simply execute pulumi up, and everything will be there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZCUKA5K2GZqF2FdR7bldPg.png" /><figcaption>Output from Pulumi Up.</figcaption></figure><p>That’s pretty much it. If you want to try installing it with Pulumi, the code is <a href="https://github.com/starschema/antares-idl/blob/main/deployment/kubernetes/emqx.py">here</a>, while you can find the entire Antares iDL project page <a href="https://github.com/starschema/antares-idl/">here</a>.</p><h3>Using EMQX — Testing the Installation</h3><p>The most convenient way to test your installation is to use <a href="https://github.com/txn2/kubefwd">kubefwd</a>. After creating the necessary port forward, you can access the admin interface at emqx-ee host’s 18083 port by default.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IVYtW_YuSINqTaruWjRveQ.png" /><figcaption>EMQX’s admin interface gives you the option to understand your deployment as well as to interact with plugins, set up rules, etc.</figcaption></figure><p>You can also check it from command line using any mqtt consoles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M8IdmoVNuVQVVlc0puJCpw.png" /><figcaption>Subscribing to “#” will show all incoming messages in the system.</figcaption></figure><h3>Summary</h3><p>I’ve shown how to deploy EMQX in a Kubernetes cluster with a single command line using Pulumi and our <a href="https://github.com/starschema/antares-idl">Antares iDL</a> project. It’s definitely worth a look, as it has extensive infrastructure as code templates for the modern data stack components (Airbyte, Airflow, Fivetran, dbt, Dagster, Kafka, etc.) complete with some IIoT-relevant components. And, as always, reach out using the contact information below if we can help.</p><p><strong>REACH OUT TO STARSCHEMA </strong><a href="https://starschema.com/contact"><strong>HERE</strong></a>:</p><p><a href="https://starschema.com/contact">Contact</a></p><p><strong>READ MORE FROM STARSCHEMA:</strong></p><ul><li><a href="https://medium.com/starschema-blog/iac-101-building-reliable-infrastructure-with-the-power-of-code-f82ab3fa985d">IaC 101: Building Reliable Infrastructure with the Power of Code</a></li><li><a href="https://medium.com/starschema-blog/ingestions-in-dbt-how-to-load-data-from-rest-apis-with-snowpark-a2c27c4b5315">Ingestions in DBT: How to Load Data from REST APIs with Snowpark</a></li><li><a href="https://medium.com/starschema-blog/building-an-open-source-data-platform-consider-these-tools-cc4743a79f86">Hand-Picked Tools for Building an Open-Source Data Platform</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1d92d97f9fa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/starschema-blog/mqtt-on-steroids-running-emqx-enterprise-on-kubernetes-d1d92d97f9fa">MQTT on Steroids: Running EMQX Enterprise on Kubernetes</a> was originally published in <a href="https://medium.com/starschema-blog">HCLTech-Starschema Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ingestions in DBT: How to Load Data from REST APIs with Snowpark]]></title>
            <link>https://medium.com/starschema-blog/ingestions-in-dbt-how-to-load-data-from-rest-apis-with-snowpark-a2c27c4b5315?source=rss-a586bc473bbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2c27c4b5315</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[modern-data-stack]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[snowpark]]></category>
            <dc:creator><![CDATA[Tamas Foldi]]></dc:creator>
            <pubDate>Mon, 17 Oct 2022 14:29:16 GMT</pubDate>
            <atom:updated>2022-10-17T22:34:49.320Z</atom:updated>
            <content:encoded><![CDATA[<p>We all know the <em>status quo</em> in the modern data stack: Ingestion tools for ingestions and transformation tools for transformations. Use Fivetran, Airbyte or Meltano to get the data in, then run dbt models to transform them. But do we always need ingestion tools? Why should we subscribe to another SaaS platform or deploy an additional software component if we only deal with standard file ingestions and Web APIs?</p><p>Dbt can do the entire ELT process alone, and I will show you how.</p><figure><img alt="Snowboarded in a Snowpark sliding on a dbt ramp" src="https://cdn-images-1.medium.com/max/1024/1*CwEKdancROAmYylgcyAiVQ.png" /><figcaption>Image generated by <a href="http://dall-e.com/">Dall-E AI</a></figcaption></figure><h3>Keep it Simple, Stupid</h3><p>While we always want to use the best tool for the job, sometimes, in the data world, good enough, boring, and simple beats the perfect combination of 10+ components. If dbt-core and Snowflake can manage our entire data pipeline, why introduce additional complexity? With dbt alone, we can have all of our ingestion and transformation codes, job history, logs, and schedules in a single location — like in dbt cloud — even if we have to do a little bit — just a little — preparation and coding.</p><h3>Ingesting Github data to Snowflake with dbt</h3><p>Let’s consider an easy use case: I want to visualize the <a href="https://github.com/dbt-labs/dbt-core/commits">dbt-core</a> project’s traction looking at the number of commits over its lifetime, using dbt and Snowflake only. Since dbt — and especially dbt cloud — by design, don’t allow you to run code on their platforms, the only option is to invoke HTTP calls on the backend database level. In one of my previous posts, I shared my <a href="https://medium.com/starschema-blog/starsnow-http-client-for-snowflake-sql-e1b329293fc6">StarSnow</a> project, which calls external HTTP endpoints from SQL as external functions — precisely what we need here.</p><p>In our case, we need to interface with the <a href="https://docs.github.com/en/rest/commits/commits">commits endpoint of the GitHub API</a>. The only input parameter we have to pass is the repository&#39;s name — dbt-core in our example. The GitHub API will then return the commit information in JSON format, one semi-structured entry per commit.</p><p>The process flow is illustrated in the diagram below. The steps we have to execute:</p><ol><li>Call StarSnow external function from our dbt model with the GitHub API specific request object (URL, HTTP method, headers, etc.)</li><li>Snowflake will retrieve the response object for our StarSnow request</li><li>dbt wraps the results in a CREATE TABLE statement and materializes the result</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WtcBizfDABMhb0SaIK1p2A.png" /><figcaption>End-to-end architecture for flat file and web api ingestions with dbt only — no ingestion tools required.</figcaption></figure><p>It is worth mentioning that for any flat file ingestions, we can simply use SnowPipe — it’s well-integrated with dbt. And honestly, with web services and flat file ingestions, we can cover many use cases.</p><p>Now let’s make things happen. First, install StarSnow, then create a model that invokes the commit endpoint, as shown here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MSZycNQTQ9UGvjV7OwdacA.png" /><figcaption>In my dbt project I made separate folders under models for ingestions and transformations</figcaption></figure><p>In the<a href="https://github.com/tfoldi/dbt-github-api-ingestion/blob/master/models/ingestions/stg_commits.sql">stg_commits.sql</a> model I issue a select statement to read from the StarSnow external function. Dbt will turn this into a CREATE TABLE statement and store the results in stg_commits table. The table will contain one row and one column only with a VARIANT type. This field will store the data returned by the GitHub API in JSON format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2PdJlfcbZPw_wJpiA4ZfgQ.png" /><figcaption>We have one single row with one single column for all commits. This is the response from the API call, as-is.</figcaption></figure><p>Now we can build our model that flattens this JSON object, moving each commit to separate rows. Since Snowflake is great at flattening and pivoting semi-structured data, we can combine the lateral flatten and the colon operator to format the response object to our desired table format: one row per commit with the necessary fields pivoted to individual columns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lDTv1Q3Nup_sUlCCep7LUw.png" /><figcaption>Our commit model will split and flatten the API response.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OJclaJoUVpXW-WF55JE-nQ.png" /><figcaption>The model results: we just ingested and transformed our first Web API with dbt cloud only</figcaption></figure><p>The results look great, well, except for the number of rows. It seems we have only 30 rows. What went wrong?</p><h3>Snowpark enters the game.</h3><p>Well, life is not just single GET requests. We forgot to implement pagination, which is needed if we want more than a few dozen records from our source systems. Implementing pagination might suck in SQL statements, so wouldn’t it be great to use python code to implement these control structures?</p><h4>Dbt Python models</h4><p>The good news is that starting from version 1.3 we can <a href="https://docs.getdbt.com/docs/building-a-dbt-project/building-models/python-models">use Python programming language to build </a>models in dbt. Our model will have .py file extension instead of .sql, and will deploy and will be execute as a Snowpark function. In Snowflake’s case, a dbt python model must implement a single model function with dbt and database session objects as parameters and return a data frame as the execution result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UKhn_v7XgD5rjR-GMQxb5g.png" /><figcaption>Example dbt python model returning with a single row table</figcaption></figure><p>All we have to do is to rewrite stg_commits.sql from SQL to python code and add pagination logic. Luckily, Snowpark code can also easily invoke external functions and other UDFs using call_udf function, allowing to use StarSnow HTTP library natively.</p><p>All-in-all, the following model will do what we need:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4z1Co9QvywQ-ur6EcWFneg.png" /><figcaption>Our <a href="https://github.com/tfoldi/dbt-github-api-ingestion/blob/master/models/ingestions/stg_paginated_commits.py">stg_paginated_commits.py</a> dbt python model implements pagination using StarSnow and Snowpark</figcaption></figure><p>The code above does just what you expect:</p><ol><li>Define a few constants (dbt v1.3 does not allow jinja templates for python models yet)</li><li>Start an iteration to capture pages one by one (for cycle)</li><li>Define a data frame with the URL to invoke and request parameters</li><li>Execute STARSNOW_REQUEST with parameters defined in the data frame</li><li>Store the results in ret variable. Use union_all to concatenate pages.</li><li>Break from the iteration in case there is no next page mentioned in the response</li></ol><p>Now let’s try to execute it end-to-end:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oa03LgoxJlAuFQ-hwkmUKw.png" /><figcaption>My first Snowpark web API model on dbt cloud</figcaption></figure><p>It works like a charm. Finally, I can see the commit trends of dbt-core in my Snowsight UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mVlob9NlUAovLXuSlPkcbA.png" /><figcaption>Something happened during the Summer of 2018</figcaption></figure><p>If you want to try this demo, all codes are pushed to the <a href="https://github.com/tfoldi/dbt-github-api-ingestion">https://github.com/tfoldi/dbt-github-api-ingestion</a> repository. Don’t forget to deploy StarSnow to your AWS account before you run the models. It should work out of the box for Amazon, but hasn’t been tested on GCP or Azure — as always, surprises could happen.</p><h3>Next Steps</h3><p>Well, the possibilities are endless. For instance, with a little effort, we could port Singer taps to Snowpark, allowing dbt to interface seamlessly with hundreds of source systems . Imagine the possibilities.</p><p>Also, while this post heavily relies on StarSnow and Snowflake, there is no reason why you should not be able to do the same on Amazon Redshift with external functions or Google Big Query with remote functions.</p><p>Solutions like this allow the community to easily transform dbt into a full-fledged ELT solution. I’m sure it will happen.</p><p><strong>REACH OUT TO STARSCHEMA </strong><a href="https://starschema.com/contact"><strong>HERE</strong></a>:</p><p><a href="https://starschema.com/contact">Contact</a></p><p><strong>READ MORE FROM STARSCHEMA:</strong></p><ul><li><a href="https://medium.com/starschema-blog/dagster-airbyte-dbt-how-software-defigned-assets-change-the-way-we-orchestrate-ac70bb29d640">Dagster + Airbyte + dbt: How Software-Defined Assets Change the Way We Orchestrate</a></li><li><a href="https://medium.com/starschema-blog/starsnow-http-client-for-snowflake-sql-e1b329293fc6">StarSnow: HTTP Client for Snowflake SQL</a></li><li><a href="https://medium.com/starschema-blog/extending-airbyte-creating-a-source-connector-for-wrike-8e6c1337365a">Extending Airbyte: Creating a Source Connector for Wrike</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2c27c4b5315" width="1" height="1" alt=""><hr><p><a href="https://medium.com/starschema-blog/ingestions-in-dbt-how-to-load-data-from-rest-apis-with-snowpark-a2c27c4b5315">Ingestions in DBT: How to Load Data from REST APIs with Snowpark</a> was originally published in <a href="https://medium.com/starschema-blog">HCLTech-Starschema Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dagster + Airbyte + dbt: How Software-Defined Assets Change the Way We Orchestrate]]></title>
            <link>https://medium.com/starschema-blog/dagster-airbyte-dbt-how-software-defigned-assets-change-the-way-we-orchestrate-ac70bb29d640?source=rss-a586bc473bbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac70bb29d640</guid>
            <category><![CDATA[airbyte]]></category>
            <category><![CDATA[dagster]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Tamas Foldi]]></dc:creator>
            <pubDate>Wed, 05 Oct 2022 18:01:12 GMT</pubDate>
            <atom:updated>2022-10-06T17:28:25.868Z</atom:updated>
            <content:encoded><![CDATA[<p>A few weeks ago, I heard an interview with Dagster founder Nick Schrock on the <a href="https://www.dataengineeringpodcast.com/dagster-software-defined-assets-data-orchestration-episode-309/">Data Engineering Podcast</a>. Although I tried Dagster way back in its early stage, listening to Nick’s approach to Software-Defined Assets (SDA) in the data orchestration field intrigued me enough to give the new, 1.0 version a try. Why we should care, how does it compare to Airflow, and why will I be using only Dagster for my Airbyte and dbt jobs — you will know more if you continue reading.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ttLxEzlgEsk6cLoTS1nurw.png" /></figure><h3>Software-Defined What?</h3><p>Assets. Before we discuss what exactly these Software-Defined Assets are in the ETL scheduling/orchestration space, let’s see how things were done in the past. Your code was stored in different locations (ingestion tools, dbt models, Python notebooks, etc.) and your orchestration tool of choice executed or invoked these objects in “tasks.” These tasks were organized into graphs: dependencies and relationships to say how they should be executed relative to each another.</p><p>In your orchestration tool, you had full visibility over task executions, but on the other hand, you had no or little overview of the created/refreshed objects made by your tasks. However, in the data world, this is what really matters. When did you refresh a table or a table partition, what other tables should be refreshed if an upstream table changes, how many rows are loaded: just a few examples of questions you want to answer, regardless of what underlying tool or technology loads that asset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SgYpySWDUok0eHY0cYBlWg.png" /><figcaption>Airflow DAG chaos. While you can see the tasks and dependencies, you probably have no clue about the status of the tables behind the tasks. (Credit: Apache Airflow)</figcaption></figure><p>This is where Dagster differentiates itself. Instead of talking about Tasks and Graphs, you have Assets and Materializations. According to D<a href="https://docs.dagster.io/concepts/assets/software-defined-assets">agster’s documentation</a>:</p><ul><li>An <strong>asset</strong> is an object in persistent storage, such as a table, file or persisted machine learning model.</li><li>A <strong>software-defined asset</strong> is a Dagster object that couples an asset to the function and upstream assets that are used to produce its contents. In practice, you define <strong>what</strong> you materialize and <strong>how</strong>.</li></ul><p>Like in other tools (such as Airflow), the definition of “what” and “how” is defined in Python code. Therefore, you can generate assets with reusable code: libraries to sync metadata from other tools to build up dependency graphs and functions to materialize them. Think about a single function that produces Software-Defined Assets for all of your dbt models, defining the underlying tables, their metadata, the SQL code behind the code and the dbt command to invoke for the model run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HgsC9uHMvKusPMG7h_VYmQ.png" /><figcaption>An example set of Assets, viewed by the dependencies across different tools and applications</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HrUs-P-XiWfloTPRv8GX8Q.png" /><figcaption>Example Asset. The definition and lineage information was added by the Dagster-dbt library.</figcaption></figure><p>In short:</p><blockquote>Software-Defined Assets enable a declarative approach to data management, in which code is the source of truth on what data assets should exist and how those assets are computed.</blockquote><p>Let’s see this in practice.</p><h3>Installing Dagster Locally</h3><p>Running Dagster and its graphical UI (Dagit) is pretty easy. First, you have to install the core modules and the desired plugins with pip:</p><pre>pip install dagster dagit dagster-dbt dagster-airbyte dagster-snowflake dagster-postgres</pre><p>However, before you can see your assets, lineage or operations, you first need a repository with some stuff.</p><h3>Adding Airbyte Extracts and Load (EL) Connection to Dagster</h3><p>There are two ways to import our Airbyte connections. One is to use build_airbyte_assets, which connects to a remote Airbyte instance, or you can use load_assets_from_airbyte_project, which relies on a local Octavia CLI export. For simplicity, I used the build_airbyte_asset version, which does not require any external dependency (other than installing dagster-airbyte package). However, this version requires us to specify all tables in our connection:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/008a818165b18751fb818c5ebbd2a7e7/href">https://medium.com/media/008a818165b18751fb818c5ebbd2a7e7/href</a></iframe><p>What happens above is:</p><ol><li>We define a resource object (in this case an Airbyte resource). Resource objects contain metadata (connection information in most cases), used by repository discovery or materialization operations. Unfortunately, you still have to pass these configuration items when you trigger jobs manually on Dagit (this is something I don’t like).</li><li>We generate our assets using build_airbyte_assets function. In my scenario, the table names are hard-coded, but they can be stored in YAML files too (or I could use load_assets_from_airbyte_project).</li><li>We define our repository. All I have to do is to return my assets (or other objects) defined from a function decorated with @repository.</li></ol><p>Now we can finally start Dagit and see what we did:</p><pre>dagit -f dagster_airbyte_example.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*92gJJwww_fq1RjupqSw7aQ.png" /><figcaption>My first two tables! How cool is this.</figcaption></figure><p>I can try to materialize these tables by shift+clicking on “Materialize all.” We need shift+click because we have to add runtime configuration for the materialization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fwicGQaOMJgNtPMH66K7dw.png" /><figcaption>Materialization configuration page. You can add per execution configurations from Dagit.</figcaption></figure><p>A convenient “Scaffold missing config” button will show you the mandatory configuration items. If all looks good, you should be able to start Materializing your Airbyte connection:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-2y4LU0khUlIUEJMrsx2eQ.gif" /><figcaption>The end-to-end flow of running executing materialization.</figcaption></figure><p>It’s all nice and dandy, except I <em>always</em> have to provide the resource configuration for manual executions. Even if I hard-coded the resource config to my airbyte.py file, I had to specify it <em>again</em>. This is not needed when executing from schedules but was annoying enough to mention it here.</p><p>In the end, I can see when my assets were last refreshed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zo-RhH5vShTvvCX_1yAYlw.gif" /><figcaption>You can access Airbyte logs directly from Dagster.</figcaption></figure><p>Looks good, so let’s set up dbt as well.</p><h3>Importing a dbt Project to Dagster</h3><p>This will be easy, too. Just like with Airbyte, we need an asset loader helper function: load_assets_from_dbt_project in our case.</p><pre>from dagster_dbt import dbt_cli_resource, load_assets_from_dbt_project<br>from dagster import with_resources</pre><pre>dbt_assets = with_resources(<br>    load_assets_from_dbt_project(<br>        project_dir=&quot;path/to/your/dbt/project&quot;<br>    ),<br>    {&quot;dbt&quot;: dbt_cli_resource},<br>)</pre><p>By default, Dagster only invokes dbt run, which excludes seeds and snapshots. If you want to include those assets as well, add the following parameter to load_assets_from_dbt_project:</p><pre>use_build_command=True,</pre><p>That’s it. Let’s put it alltogether with the Airbyte code (into the same Python file) to see our Airbyte source tables and dbt models together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RdaI5RXVlOIXFOrhu_NxJQ.png" /><figcaption>Finally, we can see our Airbyte and dbt models together. However, they are not connected.</figcaption></figure><p>This is good, but not what we expected. It would be great to see our Airbyte tables and dbt tables connected as dependencies. To understand why, let’s look into the assets tab:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0wQ5sgCmQA0J-KyzBQytBw.png" /></figure><p>We can see two versions of the src_contacts table. One is under / and the other is under wrike. The issue here is that we need to define where these tables are and make their schema part of their asset keys. In my case, all Airbyte tables should go under stage/source_name schema, while all models should go under practice schema. I can quickly change the code to specify these schemas as part of the asset keys:</p><pre>dbt_assets = with_resources(<br>        project_dir=load_assets_from_dbt_project(<br><strong>        key_prefix=[&quot;practice&quot;],<br>        source_key_prefix=[&quot;stage&quot;],</strong><br>    ),<br>    {&quot;dbt&quot;: dbt_cli_resource},<br>)</pre><pre>ab_assets = with_resources(<br>    build_airbyte_assets(<br>        connection_id=&quot;connection_id&quot;,<br>        destination_tables=[&quot;table1&quot;],<br><strong>        asset_key_prefix=[&quot;stage&quot;, &quot;wrike&quot;],</strong><br>    ),<br>    {&quot;airbyte&quot;: airbyte_resource},<br>)</pre><p>This time, if I check my lineage on Dagit, dependencies are looking more decent:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7KOeY12toC2ZJs5p_1HNlg.gif" /><figcaption>Dependencies between Airbyte assets and dbt assets are looking cool.</figcaption></figure><p>Trust me, this is awesome. Seeing your end-to-end data pipeline across multiple tools with such simplicity is game-changing.</p><h3>Next Steps</h3><p>This is just one differentiating feature of Dagster; you can still expect all the orchestration must-haves like job definitions (sets of assets and operations to execute), ops (traditional tasks), schedules, sensors and a lot more. Dagster also nicely supports Kubernetes deployments — its Helm package really just works out-of-the-box.</p><h4>User Security</h4><p>One thing I should highlight here is the authentication and authorization part. Just like Airbyte, Dagster does not support user authentication and authorization in its open-source version. This is not a huge issue for us: if you deploy it in EKS or GKS, it’s easy to configure an authentication frontend load balancer to filter access to the main Dagit web service. Dagit also supports read-only and read-write modes, so you can run two separate Dagit pods with two configurations and redirect authenticated user traffic according to their access level (RW or RO).</p><p>While Dagster is still in an early phase, I already enjoy working with it in production internally and at our clients’ deployments. Maybe you too should give it a try.</p><p><strong>REACH OUT TO STARSCHEMA </strong><a href="https://starschema.com/contact"><strong>HERE</strong></a>:</p><p><a href="https://starschema.com/contact">Contact</a></p><p><strong>READ MORE FROM STARSCHEMA:</strong></p><ul><li><a href="https://medium.com/starschema-blog/extending-airbyte-creating-a-source-connector-for-wrike-8e6c1337365a">Extending Airbyte: Creating a Source Connector for Wrike</a></li><li><a href="https://medium.com/starschema-blog/fivetran-acquires-hvr-youre-in-for-a-treat-8ddfcbe4b9d7">Fivetran Acquires HVR: You’re in for a Treat</a></li><li><a href="https://medium.com/starschema-blog/advanced-server-side-filtering-for-tables-in-appsmith-1c8bd595559c">Advanced Server-side Filtering for Tables in Appsmith</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac70bb29d640" width="1" height="1" alt=""><hr><p><a href="https://medium.com/starschema-blog/dagster-airbyte-dbt-how-software-defigned-assets-change-the-way-we-orchestrate-ac70bb29d640">Dagster + Airbyte + dbt: How Software-Defined Assets Change the Way We Orchestrate</a> was originally published in <a href="https://medium.com/starschema-blog">HCLTech-Starschema Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Extending Airbyte: Creating a Source Connector for Wrike]]></title>
            <link>https://medium.com/starschema-blog/extending-airbyte-creating-a-source-connector-for-wrike-8e6c1337365a?source=rss-a586bc473bbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e6c1337365a</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[airbyte]]></category>
            <category><![CDATA[data-integration]]></category>
            <category><![CDATA[wrike]]></category>
            <category><![CDATA[modern-data-stack]]></category>
            <dc:creator><![CDATA[Tamas Foldi]]></dc:creator>
            <pubDate>Mon, 15 Aug 2022 16:54:30 GMT</pubDate>
            <atom:updated>2022-08-15T16:54:30.161Z</atom:updated>
            <content:encoded><![CDATA[<p>The data ingestion tool business is anything but new, and there are many solutions on the market, all with different histories and superpowers. <a href="https://airbyte.com/"><strong>Airbyte</strong></a><strong> claims to be one of the most extensible</strong>: using their programming language-agnostic open protocols and CDKs, one can make connectors in matter of hours, if not minutes.</p><p>In this post, I will fact-check Airbyte’s claims by building a new connector from scratch for my Wrike data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*souygI7Sf4GZjkjDRY516g.png" /></figure><h3>Airbyte’s Superpowers</h3><p>Again, if you start designing a data platform, you’ll quickly realize that ingestion is a crowded market. Old players, dating back to 1995, are still in the game — GoldenGate is 27 years old — while new players like Fivetran, Stitch, Airbyte and Dataddo emerged with the modern data stack hype as market disruptors. <br>Let’s see how Airbyte is trying to stand out from this crowd:</p><ul><li><strong>The core product is open-source</strong>, including all source and destination integrations and a web UI (without built-in authentication). It makes Airbyte one of the few usable data integration products that can run on your own infrastructure, in your own cloud VPC and on-prem too.<br>Yes, you can run your production Airbyte on your own Kubernetes cluster for free.</li><li><strong>Connectors utilize Airbyte’s own, open standards</strong>. You can pick any of your favorite programming languages to write a connector — no need to stick to slow languages like Python. A connector written in Rust or C++ could be simply faster than interpreted or JIT-ed languages. On the other hand, there are higher-level APIs (CDKs) in Python and Java for quick development.</li><li>An interesting design decision I actually like is that <strong>all connectors are separate Docker containers</strong>. This is great because you do not have to deal with the dependency hell of connectors requiring different libraries. You can isolate your own code from the rest of the system, update it without the fear that it will break something else. Plus, connectors in containers help with elasticity: things can run parallel in different workers on your Kubernetes cluster.</li><li>And last, but not least: <strong>Airbyte claims that they will support your connectors after publishing</strong>. While you can always help, the responsibility is theirs after you share your code with them.</li></ul><p>Good stuff. But let’s see how it fares for my use case.</p><h3>The Use Case</h3><p>I work at a decent-sized consulting tech company, and our presales team uses <a href="https://www.wrike.com/">Wrike</a> to manage sales support requests from the account teams. We’re constantly working on dozen of customer demos and presentations, and Wrike has been a trendemous help in keeping us organized. To visualize our impact, I wanted to collect this information from Wrike, store it in Snowflake and build a cool-looking dashboard on top it.</p><p>Wrike is quite developer-friendly: their <a href="https://developers.wrike.com/">API</a> is beautiful and well documented. Everything is provided to help you get started.</p><h3>Building the Wrike Source Connector</h3><p><a href="https://docs.airbyte.com/quickstart/deploy-airbyte">Installing Airbyte</a> couldn’t be easier — just clone their Git repo and ramp it up with Docker Compose. To start building a new connector, use their connector generator:</p><pre>cd airbyte-integrations/connector-templates/generator <br>./generate.sh</pre><p>I’ve selected Python HTTP API Source and called it as source-wrike. The generator will create a set of sample files in anairbyte-integrations/connectors/source-wrike folder, so we just need to customize them.</p><h4>Specifying metadata</h4><p>All integration components have some sort of input variables defined in source_wrike/spec.yaml. In my case, I needed three inputs: the API key, the wrike instance name and the start date for replication. My spec.yaml looked like <a href="https://gist.github.com/tfoldi/a256ec2507f0b8a98e2c3d660d24747b#file-spec-yaml">this</a>.</p><h4>Defining source and check_connection</h4><p>All sources should implement at least two functions: one for testing the connectivity to the source and another to specify what streams (tables) are available from that source.</p><p>For web-based APIs a simple request-based invocation should be sufficient to test if the API key and Wrike instance are properly configured:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e75f51fa2c9a90d9f9ae06488be07549/href">https://medium.com/media/e75f51fa2c9a90d9f9ae06488be07549/href</a></iframe><p>To test the connection, create a config.json with the same properties defined in spec.yaml, then run the following:</p><pre>❯ <strong>python main.py check --config secrets/config.json</strong><br>{&quot;type&quot;: &quot;LOG&quot;, &quot;log&quot;: {&quot;level&quot;: &quot;INFO&quot;, &quot;message&quot;: &quot;Ping response code: 200&quot;}}<br>{&quot;type&quot;: &quot;LOG&quot;, &quot;log&quot;: {&quot;level&quot;: &quot;INFO&quot;, &quot;message&quot;: &quot;Check succeeded&quot;}}<br>{&quot;type&quot;: &quot;CONNECTION_STATUS&quot;, &quot;connectionStatus&quot;: {&quot;status&quot;: &quot;SUCCEEDED&quot;}}</pre><p>Looks rad so far. Now, let’s define the output streams inside the same SourceWrike class:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f4c5083397948dc84d6990922dc1cd88/href">https://medium.com/media/f4c5083397948dc84d6990922dc1cd88/href</a></iframe><p>Airbyte has two Authenticator classes to handle HTTP API authentications: a TokenAuthenticator to deal with Bearer tokens and one for OAuth 2.0 authentication flows. Theoretically, you can just pass this authenticator object to your stream objects — this will be the next step — and Airbyte will do the rest. You don’t have to worry about setting headers, managing secrets and so on. Pretty convenient.</p><h4>Defining streams</h4><p>In the previous steps, I defined the stream objects (Tasks, Folders, etc.), but now we have to actually implement them. Since the API endpoints are really similar, the pagination is the same and the URL base is the same, I made a neat superclass for the actual streams:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d4a92de2651d2091a33b0a9952bee05/href">https://medium.com/media/0d4a92de2651d2091a33b0a9952bee05/href</a></iframe><p>There’s a few things going on, so let me explain:</p><ul><li>In __init__ we set the URL base. This is what Airbyte will use when making the HTTP calls — yes, <strong>Airbyte will issue the HTTP request</strong>, we just need to instruct what we need. All caching, throttling, error handling is done by the CDK, not by us.</li><li>In next_page_token, we parse the next page token from the response. This will be passed to all subsequent function calls by Airbyte.</li><li>The request_params will return the parameters that need to be added to the request. At the very least, the next_page_token should be added in case we have one.</li><li>The parse_response is the actual code that parses the response and returns with the data in JSON format.</li><li>The path function (or property) returns with path needs to be concatenated with the base_url. In my case, the default path name is the same as the class name — for example, Tasks will use thetasks API endpoint.</li></ul><p>What is important here is to define base_url, path and request_params. Airbyte will make the HTTP request and call parse_response.</p><p>Now, we can go ahead and define the stream classes:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/efd5af874ef4e9806b9bfedcf738300e/href">https://medium.com/media/efd5af874ef4e9806b9bfedcf738300e/href</a></iframe><p>In case of Tasks, I added a plus parameter to retrieve more fields. All I had to was to override request_params.</p><h4>Defining static schema</h4><p>The most tedious part is to define the output schema. You need to place one JSON file for each stream as source_wrike/schema/&lt;stream_name&gt;.json. No magic here, all properties need their data type(s). Later, Airbyte will use this information to build database tables and flatten JSON hierarchies when asked.</p><p>For the Comments stream, my comments.json is:</p><pre>{<br>  &quot;$schema&quot;: &quot;<a href="http://json-schema.org/draft-07/schema#">http://json-schema.org/draft-07/schema#</a>&quot;,<br>  &quot;type&quot;: &quot;object&quot;,<br>  &quot;properties&quot;: {<br>    &quot;id&quot;: {<br>      &quot;type&quot;: [&quot;null&quot;, &quot;string&quot;]<br>    },<br>    &quot;authorId&quot;: {<br>      &quot;type&quot;: [&quot;null&quot;, &quot;string&quot;]<br>    },<br>    &quot;text&quot;: {<br>      &quot;type&quot;: [&quot;null&quot;, &quot;string&quot;]<br>    },<br>    &quot;taskId&quot;: {<br>      &quot;type&quot;: [&quot;null&quot;, &quot;string&quot;]<br>    },<br>    &quot;createdDate&quot;: {<br>      &quot;type&quot;: [&quot;null&quot;, &quot;string&quot;],<br>      &quot;format&quot;: &quot;date-time&quot;<br>    },<br>    &quot;updatedDate&quot;: {<br>      &quot;type&quot;: [&quot;null&quot;, &quot;string&quot;],<br>      &quot;format&quot;: &quot;date-time&quot;<br>    }<br>  }<br>}</pre><p>To test the schema discovery, run the following command:</p><pre>python main.py discover --config sample_files/config.json</pre><p>You should see all of your streams’ schema. Save this file, as next up is the most annoying part of the development: we need to create a configured_catalog.json file. This is the combination of the stream schema with additional properties on how to retrieve files like supported_sync_modes or destination_sync_mode.</p><pre>{<br>  &quot;streams&quot;: [<br>    {<br>      &quot;stream&quot;: {<br>        &quot;name&quot;: &quot;tasks&quot;,<br>        &quot;json_schema&quot;: {<br>           &quot;$schema&quot;: &quot;<a href="http://json-schema.org/draft-07/schema#">http://json-schema.org/draft-07/schema#</a>&quot;,<br>             &quot;type&quot;: &quot;object&quot;,<br>             &quot;properties&quot;: {<br>               &quot;id&quot;: {<br>                 &quot;type&quot;: &quot;string&quot;<br>               },<br>               // other properties, as defined in schema<br>             },<br>           &quot;supported_sync_modes&quot;: [ &quot;full_refresh&quot; ],<br>           &quot;source_defined_primary_key&quot;: [[ &quot;id&quot; ]]<br>         },<br>         &quot;sync_mode&quot;: &quot;full_refresh&quot;,<br>         &quot;destination_sync_mode&quot;: &quot;overwrite&quot;<br>      },<br>      // next stream</pre><p>Save this file as sample_files/configured_catalog.json and run the following command to <em>actually</em> read records from the API:</p><pre>python main.py read --config sample_files/config.json --catalog sample_files/configured_catalog.json</pre><p>If it looks good — and it should — we should be able to build a nice Docker container from it:</p><pre>docker build . -t airbyte/source-wrike:dev</pre><h4>More trouble: iterations, slices</h4><p>During development, I realized that the comments endpoint limits the query to seven days. This means I need to iterate from the initial replication time until today in seven-day chunks and retrieve the data for that period only. Airbyte has a concept for slices: it is a nifty method for defining your own partitioning logic for your stream. It can be used to improve performance by parallelization or handle situations like mine. I ended up implementing stream_slices and request_params with the following logic:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e33c0adeaf2d322d29326ef0f9dc08ca/href">https://medium.com/media/e33c0adeaf2d322d29326ef0f9dc08ca/href</a></iframe><p>The request_param function will be called as many times as the number of records emitted by stream_slices with the current slice data.</p><p>The source code of the connector is <a href="https://github.com/tfoldi/airbyte/blob/source-wrike/airbyte-integrations/connectors/source-wrike/source_wrike/source.py">here</a> in case you want to have a look.</p><h3>Using My New Connector from the UI</h3><p>After local development, I decided to deploy Airbyte in Kubernetes (well, I deploy everything there, so this shouldn’t come as a surprise). To deploy it, I only had to issue the following:</p><pre>kubectl apply -k kube/overlays/stable</pre><p>Then, I logged in by using the superb <a href="https://kubefwd.com/">kubefwd</a>. Please note that the open-source version does not include authentication — it is advised to put a reverse proxy in front of it, adding the necessary SAML or OAuth 2.0 authentication.</p><h4>Adding the Connector from the UI</h4><p>Head to Settings, then hit + New Connector to configure the source connector as Docker container.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DnuZMafVTU2YDJOUNRGUkQ.png" /></figure><p>Next, let’s head over to Sources and select +New Source. In the dropdown, we should already see our newly created connector:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aUfOvzvui_eZfgclFBNW5g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ifEWp2OhaKjm0dw6dUEzZQ.png" /></figure><p>After filling outthe configuration forms, we are ready to roll.</p><h3>Results</h3><p>After configuring a destination — Snowflake, in my case — we can start the sync process and obtain the results on the target side. If you enabled Normalized tabular data in the settings, Airbyte will create flattened tables from your JSON structures, using dbt internally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OcZvLXuhukbIztcAruWTtg.png" /></figure><p>In case of a Snowflake destination, you can directly access the JSON results as VARIANTs in _airbyte_raw_&lt;stream&gt; tables or as normalized tables like src_folders or src_tasks. If a stream had a nested object, Airbyte will also create a subtable for our convenience:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H5i9v7sB_0CUstR917J8QA.png" /><figcaption>The “folders_project” table is the “project” nested object in the folders stream.</figcaption></figure><p>Everything is in the stage area — job done.</p><h3>Summary</h3><p>After an hour or two, I immediately felt fluent in Airbyte, and during this time I was able to make my first connector using their Python CDK. The documentations were great: they have <a href="https://docs.airbyte.com/connector-development/tutorials/cdk-speedrun">short</a> and <a href="https://airbyte.com/tutorials/extract-data-from-the-webflow-api">more detailed</a> ones for all kinds of audiences. I also like the Docker-based approach, as managing connectors centrally using Docker repositories feels convenient and safe.</p><p><strong>REACH OUT TO STARSCHEMA </strong><a href="https://starschema.com/contact"><strong>HERE</strong></a>:</p><p><a href="https://starschema.com/contact">Contact</a></p><p><strong>READ MORE FROM STARSCHEMA:</strong></p><ul><li><a href="https://medium.com/starschema-blog/starsnow-http-client-for-snowflake-sql-e1b329293fc6">StarSnow: HTTP Client for Snowflake SQL</a></li><li><a href="https://medium.com/starschema-blog/fivetran-acquires-hvr-youre-in-for-a-treat-8ddfcbe4b9d7">Fivetran Acquires HVR: You’re in for a Treat</a></li><li><a href="https://medium.com/starschema-blog/advanced-server-side-filtering-for-tables-in-appsmith-1c8bd595559c">Advanced Server-side Filtering for Tables in Appsmith</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e6c1337365a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/starschema-blog/extending-airbyte-creating-a-source-connector-for-wrike-8e6c1337365a">Extending Airbyte: Creating a Source Connector for Wrike</a> was originally published in <a href="https://medium.com/starschema-blog">HCLTech-Starschema Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>