<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Oleksandr Zhyhalo on Medium]]></title>
        <description><![CDATA[Stories by Oleksandr Zhyhalo on Medium]]></description>
        <link>https://medium.com/@oleksandr.zhyhalo?source=rss-f94fe415217b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*IvjKYg0poliMCD9C2UPqdg.jpeg</url>
            <title>Stories by Oleksandr Zhyhalo on Medium</title>
            <link>https://medium.com/@oleksandr.zhyhalo?source=rss-f94fe415217b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 03:04:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@oleksandr.zhyhalo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Robotic Hives to S3: Searching 61 Million Log Lines a Day]]></title>
            <link>https://medium.com/@oleksandr.zhyhalo/from-robotic-hives-to-s3-searching-61-million-log-lines-a-day-44a5ea7d419d?source=rss-f94fe415217b------2</link>
            <guid isPermaLink="false">https://medium.com/p/44a5ea7d419d</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[logging-and-monitoring]]></category>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Oleksandr Zhyhalo]]></dc:creator>
            <pubDate>Wed, 29 Jul 2026 08:35:24 GMT</pubDate>
            <atom:updated>2026-08-11T16:40:53.956Z</atom:updated>
            <content:encoded><![CDATA[<h3>From Robotic Beehives to S3: Searching 61 Million Log Lines a Day</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AlBGaL4PvUH4K8lxw3kP8A.png" /></figure><p>At <a href="https://beewise.ag/">Beewise</a> we build and operate robotic beehives: machines packed with robotics, sensors and cameras, doing precision work on live bee colonies far from the nearest engineer. The fleet is around 2,000 devices, running on solar power, talking over cellular, producing about 61 million log lines a day. Each device has to tell us what happened, and that record has to survive the trip from an off-grid field to our systems.</p><p>This is the story of that pipeline: the constraints, the tools, and the point where off-the-shelf ran out and I built my own. That missing layer became <strong>Rootprint</strong>.</p><h3>Devices are not a SaaS</h3><p>With a normal web service, observability forgives you. You spot a gap in your logging, ship a fix, and an hour later the new telemetry flows. Devices don’t work that way. A fleet update is a planned, careful rollout, and some devices will be offline when it lands. You decide what to collect and how to collect it ahead of time, then live with that decision for months.</p><p>Then there is the environment itself:</p><ul><li><strong>Power and connectivity are unstable.</strong> The hives sit off-grid on solar power. Outages are the normal operating mode, not an exception.</li><li><strong>Disk is limited.</strong> A device can’t buffer logs forever while it waits for the network to come back.</li><li><strong>Bandwidth costs money. </strong>The hives talk over cellular, and cellular data across a whole fleet, around the clock, is a real cost to consider.</li></ul><h3>What the pipeline had to guarantee</h3><p>Those constraints translate into requirements. Yours may differ, but these were mine:</p><ol><li><strong>No lost telemetry during outages.</strong> Logs must be stored on the device and retried until the receiving system confirms that they were delivered.</li><li><strong>Maximum compression before anything leaves the device.</strong> Preferably use something like zstd; for text like logs it has one of the best ratios available.</li></ol><p>Buffering, retries, compression, and encryption all need to happen on the device before the data goes anywhere else.</p><h3>Full-text search was the main requirement</h3><p>Search requirements depend on what kind of data you send. Ours was unstructured logging: free-form text from many subsystems, written over years, with no clean set of fields to extract.</p><p>Systems like Grafana Loki index labels (small structured tags such as region or service) and expect you to narrow by label before you search. Our logs had very limited labels to offer. I needed the opposite model: type any word or phrase and find it across everything, the way a search engine works.</p><p>With 2,000 devices in the field, finding the logs for one specific hive must remain fast even as the total amount of stored data continues to grow.The pipeline, stage by stage</p><h3>1. Collect and transform on the device</h3><p>Something needs to run on every hive and handle the hard work:</p><ul><li>collect logs from different subsystems;</li><li>normalize their format;</li><li>compress them;</li><li>store them temporarily;</li><li>send them;</li><li>retry when delivery fails.</li></ul><p><strong>Normalization</strong> (cleaning and structuring the raw logs) can live in two places: a central gateway on the receiving end, or on each device before sending. Given our constraints, I chose on-device. <br>That choice also comes with a cost:</p><ol><li>Changing the normalization logic means updating configs across the fleet.</li><li>During a rollout, we also need to support old and new formats at the same time. An off-grid device may take a long time to receive an update, or it may not update at all.</li></ol><p>The two serious candidates were <a href="https://vector.dev/">Vector</a> and <a href="https://fluentbit.io/">Fluent Bit</a>. Both are good tools. <br>The big advantage of Vector is VRL(Vector Remap Language). VRL lets you express complex log transformations in the agent config, which is what on-device normalization needs. Buffering, retries and format conversion come out of the box.</p><p>As for binary size, which was also important — you can compile Vector from source with only the components you use. Combined with <a href="https://upx.github.io/">UPX</a> (an executable packer), my build came out around 6.5 MB.</p><h3>2. Index and store</h3><p>Collected, transformed, compressed, delivered. Now the logs need a home: something that indexes them for search and stores them for the long term. My prerequisites were self-hosted and open source, which narrowed the field:</p><ul><li><strong>Grafana Loki. </strong>Already in our stack for applications observability, so it was the default candidate. Two problems. Using device IDs and other changing dimensions as Loki labels would move us toward the high-cardinality model Loki explicitly discourages. Keeping them out of labels, meanwhile, would leave us scanning log content at query time rather than using a full-text index.</li><li><strong>Elasticsearch / OpenSearch.</strong> A solid choice, and it would have worked. It is also a large stack to operate, with cluster management and tuning out of proportion to the task. Overkill for us.</li><li><strong>VictoriaLogs</strong> provides fast full-text search, works well with high-cardinality fields such as device IDs, and includes both a web interface and Grafana integration. The main limitation for us was storage. It relies on attached filesystem storage.</li><li><strong>ClickHouse-based tools (HyperDX, SigNoz).</strong> Capable, and worth a look if you want metrics and traces in the same box. But they keep logs in a database cluster, which means the storage, memory and operational load grow with your data.</li></ul><p>Then I found <a href="https://quickwit.io/">Quickwit</a>.</p><h3>Quickwit: search that lives on S3</h3><p>Quickwit is a search engine written in Rust, built for logs and other append-only data. Three design decisions made it the fit:</p><ul><li><strong>The index lives on object storage.</strong> Quickwit writes data as immutable, compressed chunks called splits and puts them straight on AWS S3 (object storage: cheap, durable, close to unlimited). Search nodes fetch the required split data and cache frequently used index structures locally.</li><li><strong>It searches S3 fast.</strong> Each split carries a “hotcache”, a small header the searcher reads first, letting it open a split in under 60 ms even on high-latency storage. That is how you get sub-second full-text queries over terabytes sitting in S3.</li><li><strong>Real full-text search. </strong>Under the hood it is a proper inverted index (built on <a href="https://github.com/quickwit-oss/tantivy">tantivy</a>, a Rust implementation of Lucene) plus columnar storage for aggregations.</li></ul><p>For me the combination was the killer feature: terabytes of logs, retained for two years at a fraction of the usual cost, and still searchable in under a second. Quickwit’s own <a href="https://quickwit.io/docs">docs</a> and benchmarks are worth a read if you’re evaluating it.</p><h3>What Quickwit doesn’t do</h3><p>Adopt it with open eyes. There are some obvious restrictions:</p><ol><li><strong>No authentication. </strong>The API is wide open to anyone who can reach it. You cannot expose it to the internet, which means devices can’t send logs to it and people can’t query it from outside your network.</li><li><strong>No product UI. </strong>There is a basic web interface for poking at raw data, and that’s it. No histograms, no saved searches, no comfortable way to navigate a huge volume of logs.</li><li><strong>Append-only.</strong> Documents can’t be updated, and deletes exist only as slow batch jobs. For logs this costs you nothing, but Quickwit is the wrong tool for data that changes.</li><li><strong>Not real-time. </strong>New logs become searchable when a chunk commits, by default within about a minute. Fine for log search, wrong for millisecond-fresh dashboards.</li></ol><h3>3. Query the logs. Or: the part where I built my own</h3><p>The first proof of concept worked around the exposure problem. Vector on each device uploaded log batches to an S3 bucket; S3 notifications plus an SQS queue told Quickwit about new files, and Quickwit ingested them.</p><p>Two problems surfaced:</p><ol><li><strong>Cost, again. </strong>For logs to feel live, devices upload often, every 30 or 60 seconds. AWS charges $0.005 per 1,000 S3 PUT requests, which sounds like nothing until you do the math: a thousand devices uploading every 30 seconds produce about 86 million PUTs a month. That is roughly $864 per month just for PUT requests, before paying for the actual storage.</li><li><strong>The search experience. </strong>For a UI, there is a Quickwit plugin for Grafana, but a Grafana plugin is still a plugin. At our log volumes it was hard to work with and harder to navigate.</li></ol><p>Quickwit solved the indexing and storage problem. It did not solve secure ingestion from thousands of devices or the day-to-day search experience for engineers.</p><p>So I built the missing layer myself, with Quickwit as the engine underneath. Each Quickwit limitation became a feature of the new system:</p><ol><li><strong>Devices send logs to my service</strong>, a token-based authentication gateway that forwards them to Quickwit for indexing. No S3 PUT storm, and Quickwit itself is never exposed.</li><li><strong>People get SSO </strong>and proper authentication.</li><li><strong>The UI has what a log tool needs</strong>: histogram, quick filters, aggregations, live tail, fast navigation.</li></ol><p>That layer became <a href="https://rootprint.io/">Rootprint</a>: an open-source system for securely ingesting, searching and navigating device logs, with Quickwit providing the indexing engine and S3 providing the durable storage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LchJofxK1mTbuAUsEU_STw.png" /></figure><h3>The pipeline, end to end</h3><pre>hive ── Vector: collect → normalize → compress (zstd) → buffer &amp; retry<br>  │<br>  ▼<br>Rootprint ── token auth gateway, forwards logs for indexing<br>  │<br>  ▼<br>Quickwit ── indexes, stores compressed on S3<br>  │<br>  ▼<br>Rootprint full-text search: by device ID, by phrase, across terabytes</pre><h3>The numbers</h3><p>After more than four months in production, the pipeline has processed:</p><ul><li>2,000 devices;</li><li>approximately 7.3 billion log lines at the moment;</li><li>about 61.3 million log lines per day;</li><li>around 716 log lines per second on average;</li><li>roughly 31,000 log lines per device per day.</li></ul><p>At the current rate, two years of retention will contain around <strong>45 billion searchable log lines</strong>.</p><h4><strong>Infrastructure</strong></h4><p>The entire Rootprint and Quickwit indexing and search stack currently runs on a single EC2 instance with:</p><ul><li>4 vCPUs and 8 GB of RAM</li><li>S3 as the primary index storage</li><li>an Amazon RDS t3.medium instance for the metastore</li></ul><p>The centralized ingestion rate is not extraordinary by itself. The difficult part is reliably moving that data from <strong>2,000</strong> intermittently connected, solar-powered devices and keeping it searchable without running a large storage cluster.</p><h4>Search performance</h4><p>Current search latency is:</p><ul><li>p50: 200 ms</li><li>p95: 582 ms</li><li>p99: approximately 1 second</li></ul><p>These measurements come from production searches over logs stored on S3.</p><h4>Storage efficiency</h4><p>The log data achieves a compression ratio of approximately <strong>5.5×</strong>. The compressed index remains on S3, while search engine fetches and caches only the data required to execute each query.</p><p>For a system serving compressed full-text indexes directly from object storage, the combination of cost, resource usage and query latency has been impressive.</p><blockquote>Rootprint is free and open-source. You can explore the project, run it yourself, or contribute <a href="https://github.com/rootprint/rootprint">here</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44a5ea7d419d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLMs in your terminal: Guide for ZSH]]></title>
            <link>https://medium.com/@oleksandr.zhyhalo/llms-in-your-terminal-guide-for-zsh-a90b4f6f20fe?source=rss-f94fe415217b------2</link>
            <guid isPermaLink="false">https://medium.com/p/a90b4f6f20fe</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[cli]]></category>
            <category><![CDATA[zsh]]></category>
            <category><![CDATA[claude]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[Oleksandr Zhyhalo]]></dc:creator>
            <pubDate>Tue, 25 Feb 2025 23:16:53 GMT</pubDate>
            <atom:updated>2025-02-25T23:16:53.530Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s world of pre-built solutions, there’s something satisfying about crafting your tools. While applications like Warp offer terminal AI integration, creating a custom solution to interact with LLMs directly from your command line provides flexibility.</p><p>This guide will walk you through establishing your own terminal-based LLM interface, providing essential configuration details for each step of the process.</p><p>These instructions are designed for ZSH, however they will give you a solid foundation to integrate it with any shell.</p><h3>Accessing LLMs Programmatically</h3><p>To start with, we require a way to interact with a model. Unfortunately, programmatic LLM access is not free, even if you have a subscription to ChatGPT or Claude. You will still have to pay for API usage.</p><p>But, for personal use, pricing is quite low — for Anthropic’s latest model Claude 3.7 Sonnet you would pay $15/MTok, and Claude 3.5 Haiku costs only $4/MTok.</p><p>For Anthropic Claude, you can set up your key <a href="https://console.anthropic.com/">here</a>. <br>And for ChatGPT <a href="https://platform.openai.com/">here</a>.</p><h3>Interacting with a models</h3><p>Here you can use different approaches, starting from custom curl requests, your own library, or ready-to-use tools. For this tutorial, I will go with <a href="https://github.com/simonw/llm">https://github.com/simonw/llm</a>.</p><p>This is a CLI tool that allows you to interact with various LLMs of your choice and can even work with LLMs deployed locally.</p><p>Installation:</p><pre>pip install llm</pre><p>or if you are using brew</p><pre>brew install llm</pre><h3>OpenAI configuration</h3><p>Let’s add API leys. You can save your OpenAI API key like this:</p><pre>llm keys set openai</pre><p>This will prompt your for a key:</p><pre>Enter key: &lt;paste here&gt;</pre><p>That it! You can verify everything is working by running a prompt:</p><pre>llm &quot;What is zsh?&quot;</pre><h3>Anthropic (Claude) configuration</h3><p>Configuring Claude is almost the same process, except we have to do an extra step and install a plugin for it.</p><p>Install the Anthropic plugin:</p><pre>llm install llm-anthropic</pre><p>Now add your API key:</p><pre>llm keys set anthropic</pre><p>Verify by:</p><pre>llm -m claude-3.7-sonnet &#39;Fun facts about Linus Torwalds&#39;<br># or to use differnt mode<br>llm -m claude-3.5-haiku &#39;Fun facts about Linus Torwalds&#39;</pre><p>However, the problem is that LLMs output answers using Markdown format. This can lead to a not-very human-readable format with different symbols. See for yourself!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*mVVmRGQYqjzviVRB2v6j2Q.png" /></figure><p>Not a great one, right?</p><p>Luckily, there are tools to fix it, one of them is <strong>Glow</strong>. Glow allows you to render MD markup directly in the terminal. So let&#39;s install it and use it.</p><p><strong>macOS or Linux:</strong></p><pre>brew install glow</pre><p><strong>Debian or Ubuntu:</strong></p><pre>sudo mkdir -p /etc/apt/keyrings<br>curl -fsSL https://repo.charm.sh/apt/gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/charm.gpg<br>echo &quot;deb [signed-by=/etc/apt/keyrings/charm.gpg] https://repo.charm.sh/apt/ * *&quot; | sudo tee /etc/apt/sources.list.d/charm.list<br>sudo apt update &amp;&amp; sudo apt install glow</pre><p>Let’s test it again:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*Zczoee-nHbVlxV51rBvJHg.png" /></figure><p>And we are good to go!</p><h3>Shell/ZSH Configuration</h3><p>The last step is to create a small script that allows us to interact more conveniently with our LLMs. Go to your .zshrcand add the following at the end of the file:</p><pre>SHOW_LLM_PROMPT=0<br>DEFAULT_LLM_MODEL=&quot;claude-3.7-sonnet&quot;  # Set your default model here<br><br>askllm() {<br>  print -n &quot;Ask LLM: &quot;<br>  read -r question || return<br>  <br>  if [[ -n &quot;$question&quot; ]]; then<br>    print &quot;Asking LLM...&quot;<br>    llm -m &quot;${LLM_MODEL:-$DEFAULT_LLM_MODEL}&quot; &quot;$question&quot; | glow -<br>  else<br>    print &quot;Question cannot be empty.&quot; &gt;&amp;2<br>    return 1<br>  fi<br>}<br><br>llm_precmd() {<br>  (( SHOW_LLM_PROMPT )) || return<br>  SHOW_LLM_PROMPT=0<br>  askllm<br>}<br><br>autoload -U add-zsh-hook<br>add-zsh-hook precmd llm_precmd<br><br>trigger_llm_prompt() {<br>  SHOW_LLM_PROMPT=1<br>  zle .accept-line<br>}<br><br>zle -N trigger_llm_prompt<br>bindkey &#39;^G&#39; trigger_llm_prompt</pre><p>What it does:</p><ol><li>You press <strong>Ctrl+G</strong> after typing text in the terminal</li><li>This sets an internal flag and simulates pressing the Enter</li><li>Before showing the next prompt, ZSH runs a check:</li></ol><ul><li>If the flag is active, it launches the LLM prompt</li><li>Your question goes to the LLM API via the llm command-line tool</li><li>Responses appear formatted using glow for better readability</li></ul><p><strong>Model choice is customizable via </strong>DEFAULT_LLM_MODEL<strong> variable. For ChatGPT set </strong>gpt-4o<strong> or any other, you can find out the full models list supported with:</strong></p><pre>llm models</pre><p>Now restart your terminal, or run source .zshrc.</p><p><em>Note: by default, it’s binded to Ctrl+G shortcut, but feel free to change it as you see fit.</em></p><h3>Final result</h3><p>Hit Ctrl+Gtype your question:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*9brE5ZnTAI1emQUcFgB2Ng.png" /></figure><blockquote>You can ask any questions and follow me on <a href="https://www.linkedin.com/in/oleksandr-zhyhalo/">LinkedIn</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a90b4f6f20fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Step By Step Guide to Building a Robust IoT Infrastructure with AWS: Part 1]]></title>
            <link>https://awstip.com/a-step-by-step-guide-to-building-a-robust-iot-infrastructure-with-aws-part-1-e2a3004e303f?source=rss-f94fe415217b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2a3004e303f</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[smart-home]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Oleksandr Zhyhalo]]></dc:creator>
            <pubDate>Thu, 11 Apr 2024 09:39:51 GMT</pubDate>
            <atom:updated>2024-04-13T17:11:37.928Z</atom:updated>
            <content:encoded><![CDATA[<h4>The IoT revolution already happened. You’ve just missed it.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uLon7B-fid_NPB9jFBWW8Q.jpeg" /></figure><h3>Introduction</h3><p>The Internet of Things (IoT) integrates internet-connected devices into our daily lives and businesses, presenting vast opportunities yet significant challenges.</p><p>We use these devices every day, but do you ever wonder what is happening behind the scenes? How to communicate with IoT devices? How to make them secure? How to collect, manage, and analyze data from millions of devices on the fly?</p><h3>The scope</h3><p>This series is designed to guide you through building a scalable, secure, and world-ready IoT infrastructure focused on the most common IoT domain: Smart Home. By choosing this specific type of application, we’ll explore implementations and solve the complex challenges in modern smart home ecosystems.</p><p>We will work not only with AWS IoT but also with a variety of AWS services: storage, database, computing, and others.</p><p><strong>Why a Smart Home?</strong></p><p>A Smart Home system is a quintessential IoT application that integrates various home devices connected through the internet. This system simplifies and improves the living experience through automation and personalization. Here’s why it’s an ideal framework for this series:</p><ul><li>Scalability: Infrastructure for a smart home ecosystem needs to efficiently manage a growing number of connected devices.</li><li>Real-Time Data Processing: These devices operate in an environment that demands immediate data processing and response.</li><li>Security: Given the personal nature of the smart home ecosystem, security is crucial to protect user data and prevent unauthorized access.</li></ul><h3>Overview of AWS IoT Services</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*TPBA2__Z1Zfmsr-vrjo6jQ.png" /><figcaption>Examples of AWS IoT services</figcaption></figure><p>AWS offers an extensive set of tools that assist in managing an IoT system, from device connectivity to complex data analysis. These services are foundational for building scalable, secure, and efficient IoT solutions:</p><ul><li><strong>AWS IoT Core:</strong> Manages secure device-to-cloud and cloud-to-device communication, and supports billions of devices and trillions of messages.</li><li><strong>AWS IoT Device Management:</strong> Helps to onboard, organize, and manage IoT devices at scale.</li><li><strong>AWS IoT Analytics:</strong> Analyzes large streams of data from devices efficiently.</li><li><strong>AWS IoT Greengrass:</strong> Brings local computing, messaging, data caching, sync, and ML inference capabilities to edge devices.</li><li><strong>AWS IoT Events:</strong> Respond to changes detected by IoT sensors and applications using automated processes.</li></ul><p>This is just a glimpse of useful AWS IoT services for any IoT infrastructure. We will explore more of them as we go with this series.</p><h3>MQTT Significance in IoT</h3><p>MQTT (Message Queuing Telemetry Transport) is a lightweight, publish-subscribe network protocol that enables IoT devices to exchange messages efficiently. This protocol is well-suited for IoT communication due to its minimal bandwidth usage, which is ideal for environments where network connectivity is limited or expensive.</p><h4>Why Use MQTT?</h4><ul><li>Efficiency: MQTT’s lightweight packet structure conserves bandwidth and ensures low latency in message delivery, which is essential for the real-time responsiveness required in many IoT scenarios.</li><li>Reliability: It offers various levels of Quality of Service (QoS) for message delivery.</li><li>Scalability: MQTT scales easily from a single device to a hundred thousands, making it great for projects ranging from home automation to industrial IoT applications.</li><li>Security: It supports secure communication with TLS/SSL to encrypt messages between devices and the broker, ensuring data privacy and integrity.</li><li>Last Will and Testament (LWT): This feature is particularly useful in IoT applications for alerting when a device disconnects ungracefully, providing a mechanism for managing client sessions.</li></ul><p>By leveraging MQTT, IoT devices can maintain constant and efficient communication with their network, respond quickly to real-time situations, and save IoT device energy and network bandwidth.</p><h3>What are we building today?</h3><p>Building an IoT infrastructure effectively begins with fundamentals and gradually expanding the scope of your project. In this first part of the series, we will focus on the very basics:</p><ul><li>Prepare AWS environment for device provisioning.</li><li>Provision our Thing(Device).</li><li>Create a role for our device.</li><li>Establish a connection with AWS IoT MQTT Broker.</li></ul><p>Prerequisites:</p><ul><li>AWS Account</li><li>CloudShell (Optional, to be used as our IoT Device)</li></ul><p><strong>Script for MQTT Connection:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1dc38e8835665b11b4b48ca7c4f1a485/href">https://medium.com/media/1dc38e8835665b11b4b48ca7c4f1a485/href</a></iframe><p>The script is designed to establish a secure connection to AWS IoT Core through the MQTT protocol and send JSON-formatted messages. We make use of the AWSIoTPythonSDK to set up the MQTT client connection. The messages are sent to hello/sensor_01 topic by default. Message example:</p><pre>{<br>   &quot;message&quot; : &quot;Hello from sensor_01&quot;<br>}</pre><p>The middleware is not the main point of this series, so we will only cover the basic aspects necessary to simulate real-world scenarios. This approach will provide sufficient context without going too deeply into the middleware.</p><h3>Connecting to AWS IoT Core</h3><p>Connecting an IoT device to AWS IoT Core involves several steps, from setting up your device in AWS IoT Core, generating and configuring the necessary certificates and policies, to coding your device to communicate with AWS IoT Core using MQTT.</p><h4>Step 1: Setting Up Policy</h4><p>Before we get our device connected, we need to set up a policy that defines where it’s allowed to send its messages.</p><ol><li>Navigate to “Secure” &gt; “Policies” within the AWS IoT Core console.</li><li>Create a new policy by clicking the “Create Policy” button. Policy content:</li></ol><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: &quot;iot:Publish&quot;,<br>      &quot;Resource&quot;: &quot;arn:aws:iot:your-region:your-account-id:topic/hello/${iot:Connection.Thing.ThingName}&quot;<br>    },<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: &quot;iot:Connect&quot;,<br>      &quot;Resource&quot;: &quot;arn:aws:iot:your-region:your-account-id:client/${iot:Connection.Thing.ThingName}&quot;<br>    }<br>  ]<br>}</pre><p>This policy permits connections to the IoT Service and allows messages to be sent to the MQTT topic hello/&lt;device_name&gt;. It restricts the IoT device from publishing to any other topics by using variables.</p><p><em>We will cover policies and how to work with them in the future.</em></p><h4>Step 2: Create a thing</h4><p>To onboard a device, follow these steps</p><ol><li>Navigate to “Manage” &gt; “Things” in the AWS IoT Core console.</li><li>Click on “Create Things”.</li><li>Thing name: sensor_01. Leave all other parameters as default for now.</li><li>For certificates, choose Auto-generate a new certificate</li><li>Select the policy you created earlier.</li><li>Click “Create Thing”.</li></ol><p>To finalize the setup process, AWS will prompt you to download the necessary files for establishing the connection:</p><ul><li>Device certificate file</li><li>Private key file</li><li>Public key file</li><li>Amazon Root CA</li></ul><h4>Step 3: Running our “IoT Device”</h4><p>To run the script, you have several options: your own Linux system, Windows Subsystem for Linux (WSL), or AWS CloudShell(recommended)</p><p>Start by installing the SDK needed to communicate with AWS IoT Core:</p><pre>pip install AWSIoTPythonSDK</pre><p>Next, create a folder to organize your files:</p><pre>mkdir sensor_01<br>cd sensor_01</pre><p>Upload your device certificate, private key, and Amazon CA. If you are using AWS CloudShell, you can upload files via “Actions” &gt; “Upload file”.</p><p>For convenience, rename and move your files into the sensor_01 folder. <br>In the end, your folder content should look similar to this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*4lbg3w0bijBERVFg3cJHdA.png" /></figure><p>The last thing we need is the endpoint. It can be found in the settings of the IoT Core dashboard. It looks something like: abc123defg.iot.region.amazonaws.com.</p><p>Now let’s run the script:</p><pre>python mqtt_client.py \<br>--cert device-certificate.pem.crt \<br>--key private.pem.key \<br>--root-ca AmazonRootCA1.pem \<br>--endpoint &quot;abc123defg.iot.region.amazonaws.com&quot; \<br>--port 8883 \<br>--topic &quot;hello/sensor_01&quot; \<br>--client-id &quot;sensor_01&quot;</pre><h4>Step 4: Connection Verification</h4><p>After starting the script, it will connect to AWS IoT Core and start sending messages. Python script output:</p><pre>2024-04-10 16:34:48,333 - INFO - Published data to topic hello/sensor_01: {&quot;message&quot;: &quot;Hello from sensor_01&quot;}</pre><p>To track this activity on the IoT Core side, you can use the MQTT test client in AWS IoT Core:</p><ol><li>In the AWS IoT Core console, find and open the MQTT test client.</li><li>Use the ‘Subscribe to a topic’ tab to set up a subscription.</li><li>Input hello/+ in the topic filter to start monitoring messages from your device.</li><li>Observe the messages sent by your device.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*emu69laJaoHz1iEclIbIVw.png" /></figure><h4>Step 5: IoT Logs</h4><p>To activate logging, go to the “Settings” of AWS IoT Core and scroll to the “Logs” section. Here, you can manage logs by creating a role, and then set the Log Level to “Informational.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mlCQrtzQTy5fzletRVSMSg.png" /></figure><p>Inspect logs in the CloudWatch Log group named AWSIoTLogsV2 to see the messages.</p><h3>Troubleshooting Common Errors</h3><ul><li>Certificate Mismatch: Ensure the certificate on your IoT device matches the one registered in AWS IoT Core.</li><li>Incorrect Policy Settings: Double-check your policy settings if your device can’t publish to a topic. Make sure the policy explicitly allows actions (iot:Publish, iot:Connect) on the correct topics.</li></ul><p><strong>Error messages example in </strong><strong>AWSIoTLogsV2 logs group:</strong></p><pre>{<br>    &quot;timestamp&quot;: &quot;2024-04-10 18:44:03.885&quot;,<br>    &quot;logLevel&quot;: &quot;INFO&quot;,<br>    ...<br>    &quot;reason&quot;: &quot;AUTHORIZATION_FAILURE&quot;,<br>    &quot;details&quot;: &quot;Authorization Failure&quot;<br>}</pre><h3>Conclusion</h3><p>We have covered the basics of connecting a device to AWS IoT Core. This includes setting up policies, provisioning the device, and ensuring secure communication. But we barely scratched the surface.</p><p><strong>In the next part, we will simulate a temperature and humidity sensor, and talk about methods for collecting, storing, and analyzing telemetry data from IoT devices.</strong></p><blockquote>You can ask any questions and follow me on <a href="https://www.linkedin.com/in/oleksandr-zhyhalo/">LinkedIn</a>. Script used in this seriers is avalible on my <a href="https://github.com/oleksandr-zhyhalo/smart-home-project-middleware/blob/main/scripts/hello_world_mqtt_client.py">Github</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2a3004e303f" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/a-step-by-step-guide-to-building-a-robust-iot-infrastructure-with-aws-part-1-e2a3004e303f">A Step By Step Guide to Building a Robust IoT Infrastructure with AWS: Part 1</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>