<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[YouScan - Medium]]></title>
        <description><![CDATA[We’re the Product team at YouScan. Engineers, PMs, and designers sharing insights on architecture, ML, data processing, and team practices as we build social listening at scale for global brands. It works for us and might work for you too, enjoy :) - Medium]]></description>
        <link>https://medium.com/youscan?source=rss----36c52163afe6---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>YouScan - Medium</title>
            <link>https://medium.com/youscan?source=rss----36c52163afe6---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 05:00:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/youscan" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Runbook That Runs Itself: Building AI Agents for On-Call]]></title>
            <link>https://medium.com/youscan/the-runbook-that-runs-itself-building-ai-agents-for-on-call-84dd3309f1b8?source=rss----36c52163afe6---4</link>
            <guid isPermaLink="false">https://medium.com/p/84dd3309f1b8</guid>
            <category><![CDATA[claude-skills]]></category>
            <category><![CDATA[troubleshooting]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[on-call]]></category>
            <dc:creator><![CDATA[Yaroslav Tykhonchuk]]></dc:creator>
            <pubDate>Tue, 18 Nov 2025 10:03:22 GMT</pubDate>
            <atom:updated>2025-11-18T10:13:23.188Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How we turned a complex troubleshooting scenario into an autonomous AI diagnostic system — and why you should too</em></p><p>This article explains how we designed a hierarchical AI agent that autonomously troubleshoots our history collection pipeline. It summarizes complex telemetry data, like metrics, logs, and internal API data, into clear and actionable insights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X9188C5wan99tsypPcTLSQ.png" /><figcaption>Multi-agent troubleshooter</figcaption></figure><h3>The Message That Escalates</h3><blockquote>“Our history collection has been running for 15 hours, and it’s still at 25%. This is blocking our quarterly report. What’s going on?”</blockquote><p>The message is from one of our biggest clients. Our on-call engineer — a couple of months into the job — stares at the screen. The mental checklist starts racing: <em>Which service is this? What logs should I check? What dashboard shows… wait, which Grafana panel was it?</em></p><p>An on-call engineer shouldn’t just know where to find the relevant information — logs, dashboards, metrics — but also be able to<strong> connect the dots, which is usually the hard part.</strong> In distributed systems, the answer could be hiding in any of a dozen places. And the client is waiting.</p><p>That’s when it hit me: <em>What if the troubleshooting guide could troubleshoot itself?</em></p><h3>The Problem: Too Many Moving Parts</h3><p>History collection at <a href="https://youscan.io/">YouScan</a> lets clients retrieve social media mentions going back to 2020. Simple enough on the surface. But under the hood, it’s a delicate dance of distributed systems:</p><p><strong>Phase 1: Search<br></strong>Find relevant mentions in our massive 2.48 petabyte Elasticsearch History Cluster. Performance depends on:</p><ul><li>Current cluster load (how many parallel collections are running?)</li><li>Data age (hot nodes with SSDs for recent data, warm nodes with HDDs for older data)</li><li>Query complexity (wildcards, distance calculations)</li><li>Cluster health (unassigned shards? node failures?)</li></ul><p><strong>Phase 2: Save</strong> <br>Process those mentions through the enrichment pipeline.<br>The issue is typically in:</p><ul><li>External service health (sentiment analysis, entity extraction, image recognition, and many more)</li><li>Pipeline backpressure (are pipeline blocks full?)</li></ul><p>A traditional runbook would say: “Check the collections API, then check Grafana dashboard A, then query these logs, then cross-reference with dashboard B…”</p><p>But which order? What’s the priority? How do you correlate findings across different data sources?</p><h3>The Solution: Claude Skills as your Observability expert</h3><p><a href="https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills">Claude Skills</a> are project-specific AI agents that can use tools, access APIs, and execute complex workflows autonomously. Think of them as automated procedures that actually understand what they’re doing.</p><p>For troubleshooting our history collection, this was perfect. We needed something that could:</p><ol><li><strong>Investigate multiple dimensions in parallel</strong> (don’t wait for logs before checking metrics)</li><li><strong>Synthesize findings across data sources</strong> (correlate cluster load with error patterns)</li><li><strong>Prioritize recommendations</strong> (what needs immediate action vs. what can wait)</li><li><strong>Explain its reasoning</strong> (teach new engineers <em>why</em> we’re looking at specific things)</li></ol><p>The key insight? <strong>Hierarchical multi-agent architecture.</strong></p><h3>Building the Troubleshooter: Agents All the Way Down</h3><p>Instead of one monolithic agent trying to do everything (and bloating the context window with every log line), we built a coordinated team:</p><pre>Main Orchestrator (SKILL.md)<br>├── Search Phase Coordinator (search-subagent.md)<br>│   ├── Collections Analysis Agent (API queries)<br>│   ├── Cluster Health Agent (Grafana metrics)<br>│   └── Error Logs Agent (Elasticsearch logs)<br>└── Save Phase Coordinator (save-subagent.md)<br>    ├── Slow Save Analysis Agent (Elasticsearch logs)<br>    └── External Services Agent (Grafana error rates)</pre><p>Here is the current version of our main orchestrator: 👉 <a href="https://gist.github.com/yaroslav-tykhonchuk/d3fc6b3724a656507f8966bdb05d0ee0">SKILL.MD </a>👈</p><p>These agents run <strong>in parallel</strong>. While one queries the collections API, another pulls Grafana metrics, and a third analyzes logs. Each returns a <strong>synthesized summary</strong>, not raw data dumps.</p><p>This parallel architecture using subagents does two critical things:</p><ol><li><strong>Speed</strong>: We get results in minutes, not sequential investigation steps</li><li><strong>Context management</strong>: Each agent’s context window stays clean and focused. They each solve one problem well, then pass their <strong>conclusions</strong> up the chain.</li></ol><p>The coordinators (Search and Save) consolidate these synthesized findings into phase-specific reports. <strong>The main orchestrator then correlates both phases to answer</strong>: Is Search the bottleneck? Is Save the bottleneck? Are both degraded? Is one phase blocking the other?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AAKGSMxtAOzE-Nsv5y-H8g.png" /><figcaption>Simple ask Claude code directly from the terminal</figcaption></figure><p>When we ran the skill on our actual incident, here’s what it found in <strong>under 3 minutes</strong>:</p><pre>Executive Summary:<br>History collection health: Degraded (Search bottleneck)<br>Primary issue: 208 active collections (HIGH), 1 power users<br>Cluster load: 17.4 avg (elevated), search latency: 2800ms (2.5x baseline)<br>External services: All healthy<br>Severity: High</pre><pre>Prioritized Actions:<br><br>CRITICAL (immediate):<br>1. Check collections from *** account, launched by *** user.<br>2. Stop 5 lowest-priority collections targeting 2022 data.<br><br>HIGH (urgent):<br>1. Monitor cluster load after reduction<br><br>MEDIUM (short-term):<br>1. Implement per-account limit: max 20 concurrent collections</pre><p>That’s <strong>actionable intelligence</strong> with <strong>specific user emails</strong> and <strong>specific collection IDs to abort</strong>. No “check the logs and see if anything looks weird.”</p><h4>Structured Output is Critical</h4><p>Each sub-agent returns <strong>structured findings</strong>, not raw data. This keeps context windows manageable and makes consolidation tractable:</p><pre>Agent Output Format:<br>  summary: &quot;2-3 sentence assessment&quot;<br>  key_findings: [&quot;bullet&quot;, &quot;point&quot;, &quot;list&quot;]<br>  root_cause: &quot;What&#39;s actually wrong&quot;<br>  severity: &quot;LOW/MEDIUM/HIGH/CRITICAL&quot;<br>  recommendations: [&quot;Specific&quot;, &quot;Actionable&quot;, &quot;Prioritized&quot;]</pre><p>Context is critical for the effective work of agents; check <a href="https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents">this article from Anthropic</a> for strategies on how to manage it.</p><h4>Teach, Don’t Just Tell</h4><p>Our skill doesn’t just say “abort these collections.” It explains:</p><ul><li><em>Why</em> they’re problematic (warm nodes, 2+ years old, user has 190 concurrent)</li><li><em>What</em> impact they’re having (cluster load at 17.4, search latency 2.5x baseline)</li><li><em>How</em> to prevent recurrence (implement per-account limits, stagger warm node collections)</li></ul><p>This turns every incident into a learning opportunity for new engineers.</p><h3>The Bigger Picture</h3><p>History collection is just one troubleshooting scenario; we can have a separate Claude skill for anything.</p><p>But here’s where it gets really interesting: <strong>agents don’t have to wait for you to ask</strong>.</p><p>Imagine your history troubleshooter running automatically whenever cluster load exceeds thresholds. Or your API latency agent triggering when p95 response time crosses 500ms. They investigate, synthesize findings, and either:</p><ul><li>Auto-resolve simple issues (restart a stuck service, clear a cache)</li><li>Present diagnosed findings to engineers (skipping the investigation phase entirely)</li><li>Escalate with full context already gathered</li></ul><p>The skill becomes <strong>proactive ops intelligence</strong> rather than reactive troubleshooting.</p><p>For experienced engineers, skills are time-savers and force multipliers. For new engineers, they’re <strong>accelerated learning</strong> — seeing expert troubleshooting patterns in action, with explanations. And for everyone, they’re <strong>tireless observers</strong> that can monitor, diagnose, and triage 24/7.</p><p>The skill <strong>is</strong> your runbook. But instead of a static document that goes stale, it’s a living diagnostic system that actually performs the investigation — on demand or continuously.</p><h3>Your Turn</h3><p><strong>What’s your team’s most complex on-call scenario?</strong> The one that requires checking 5 different systems and cross-referencing 3 dashboards?</p><p><strong>Have you experimented with AI-assisted incident response?</strong> What worked? What didn’t?</p><p><strong>If you could automate one part of your troubleshooting workflow, what would it be?</strong> The initial triage? The log analysis? The correlation step?</p><p>AI agents are no longer just for the vibes and green-field projects. They can be a crucial part of the Observability and Troubleshooting.</p><p><em>Yaroslav Tykhonchuk is a Software Engineer at YouScan, where he spends his days making distributed systems behave. Follow </em><a href="https://medium.com/youscan">https://medium.com/youscan</a> <em>for more engineering stories.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84dd3309f1b8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/youscan/the-runbook-that-runs-itself-building-ai-agents-for-on-call-84dd3309f1b8">The Runbook That Runs Itself: Building AI Agents for On-Call</a> was originally published in <a href="https://medium.com/youscan">YouScan</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Are you afraid of hackathons?]]></title>
            <link>https://medium.com/youscan/are-you-afraid-of-hackathons-85a99081d8cf?source=rss----36c52163afe6---4</link>
            <guid isPermaLink="false">https://medium.com/p/85a99081d8cf</guid>
            <category><![CDATA[hackathons]]></category>
            <category><![CDATA[experiment]]></category>
            <category><![CDATA[organization]]></category>
            <category><![CDATA[ideas]]></category>
            <category><![CDATA[experience]]></category>
            <dc:creator><![CDATA[Anastasia Kibets]]></dc:creator>
            <pubDate>Wed, 21 Aug 2024 10:30:23 GMT</pubDate>
            <atom:updated>2024-08-22T14:08:18.273Z</atom:updated>
            <content:encoded><![CDATA[<p>A hackathon is a great opportunity for the team to switch gears, take a break from the routine, and do something innovative or something that they don’t usually get around to.</p><p>We held our first hackathon at YouScan in 2016, which has become a tradition. Now, we hold hackathons almost every year, and during this time, we have had both technical hackathons and hackathons dedicated to GPT — and whatnot :)</p><p>As the Head of Product, I had the task of organizing our latest hackathon. One thing I’ve learned is that a hackathon’s success relies on good planning. Below, I’ll share the insights, tips, and challenges we faced along the way. I hope this will inspire you to host your own hackathon — and help you keep your sanity intact while doing so 🙃</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RIxyuvdRQaG7VW5WUA5k-w.png" /><figcaption><em>The winners of the 2019 hackathon and their winning project — Insight Wall</em></figcaption></figure><h3>Why do you need a hackathon?</h3><p>This is a key question to think about in advance: do we want to do something innovative, do we want to improve some part of the product, or is it important for us to quickly learn a new technology? This determines the topic of the hackathon and the evaluation criteria.</p><p>For example, last year, we wanted to start using GPT models, so we organized a thematic hackathon. Participants tried different ways to use GPT in YouScan, and we chose the most promising idea. Now, we have <a href="https://youscan.io/insights-copilot/">a smart assistant</a>.</p><p>This year, we wanted to take a break from the flow of important tasks, so we decided to let the team work on any customer or business problems that would be interesting to solve.</p><h3>Who can participate?</h3><p>At YouScan, hackathons are voluntary, but we do our best to get more people from different teams to join because it’s more fun. Our hackathons are attended not only by developers, designers, and product managers but also by CS, Sales, and anyone else who is interested. That’s great because this diversity helps us look at problems and solutions from different angles and perspectives.</p><p>We still have the majority of participants from the development team, but we’re working on it, and we already know some hacks to attract more people. For example, for teams that work with clients, it’s very important to know the dates as early as possible to plan a minimum of meetings and calls for those days.</p><p>In my experience, what puts people off the most is their lack of understanding of how they can contribute to the hackathon if they don’t code or design interfaces. Therefore, my advice is to tell them how else they can help their team win the hackathon: test prototypes with customers, do analytics, define use cases, develop a market entry strategy, etc.</p><h3>Organizer’s checklist</h3><ul><li>Choose dates, set a timeline, and announce it all in advance. The sooner, the better; before anyone plans a vacation.</li><li>Decide on the topic and evaluation criteria. We have a time-tested set of criteria that we further adapt to each hackathon:<br>1. Business value 1–5 points<br>2. Readiness and feasibility of implementation 1–5 points<br>3. Quality of the prototype 1–5 points</li><li>Select a jury that will be able to evaluate the teams according to the selected criteria.</li><li>Indicate restrictions on the size and composition of the teams. Such restrictions are especially relevant for small teams so that you don’t end up with two giant teams competing. Pay attention to bottlenecks. For example, if you have few backend developers, limit the number of backenders on one team.</li><li>Create a space where you can start sketching out ideas for the hackathon in advance. We use a table in Notion.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4KZ8iVNj02i7n2JU" /></figure><ul><li>Choose prizes.</li><li>Create a channel where you will post news about the hackathon and where participants can collaborate and exchange ideas.</li></ul><h3>Timeline</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-6QEZVcpJ-CPUOw7N2TXLA.png" /></figure><h4>Kickoff</h4><p>On the first day, we gather the entire team for an introduction. We encourage even those who are still unsure or just want to observe the process.</p><p>💡 I strongly advise holding a light brainstorming session during the intro in any format convenient for you. For example, ask everyone to sketch as many ideas for your hackathon topic as possible on a whiteboard. This will help people get involved in the hackathon and guarantee that more people will participate.</p><p>In the afternoon, we hold a pitching session where everyone can share their idea and get their colleagues interested in joining the team. We collect a list of ideas for pitching in the Notion table.</p><p>Then, we give people time to think and communicate, and on the morning of the second day, we form teams. Organizing such events in distributed teams is a painful question for many people. We don’t regulate this point, but I’ve noticed that those who go to the office and those who work remotely often join different teams. It seems more convenient for everyone, and we don’t mind.</p><p>💡 Ask the teams to choose catchy names. It’s more fun and easier to cast votes later.</p><h4>Working on projects</h4><p>We allocate 2.5 days to work on projects.</p><p>💡 Encourage teams to seek advice from experts within the company. For example, jury members can tell you about gaps according to some evaluation criteria.</p><h4>The final of the hackathon</h4><p>We hold project presentations and voting on the last day, around noon. Voting usually consists of two parts: the jury and the audience vote.</p><p>To determine the winner, we use a system similar to the Eurovision Song Contest: we convert the scores of the audience and the jury to a scale from 0 to 12 points based on a ranking, where the highest score is 12 points, the second place is 10 points, the third place is 8 points, etc. The final scores of the audience and the jury are summed up to determine the winner. The process is a bit complicated but fair, so if you come up with a better way, please contact me 👋</p><h3>Is there a life after a hackathon?</h3><p>It’s very difficult to make something ready to launch within 2–3 days of a hackathon. That’s why we used to face the fact that very hot ideas after the hackathon either didn’t appear in the product at all or appeared years later. After all, everyone returned to their current tasks after the hackathon.</p><p>The last time, as an experiment, we set aside an extra week to finish everything we could and launch it to our customers, and we liked it.</p><p>To be honest, we certainly haven’t finished much in a week; most projects took 2 weeks to complete, and some had to be postponed. But we have already added 5 out of 7 improvements to the product.</p><p>There are also disadvantages to this approach. Usually, we work on something we have thought about in advance and discussed several times. After several iterations, we have come up with the best option and considered the nuances. At a hackathon, the process is more chaotic, and time is limited, so we had a rough time finalizing our projects. At first glance, it seemed easy to release, but many curious and controversial things came up in the process.</p><p>Next time, I plan to take a break for a couple of days after the hackathon to discuss the projects with the teams, plan the scopes and detailed work plans, consider the impact on other functionalities, and so on.</p><h3>Final words</h3><p>For us, hackathons are a great combination of interesting and productive experiences. I’d be happy if our ideas were useful to someone! And we are already preparing for the next hackathon, so stay tuned for new hacks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85a99081d8cf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/youscan/are-you-afraid-of-hackathons-85a99081d8cf">Are you afraid of hackathons?</a> was originally published in <a href="https://medium.com/youscan">YouScan</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Batching for cloud queues: spend 10x less]]></title>
            <link>https://medium.com/youscan/batching-for-cloud-queues-spend-10x-less-7ec5ac231443?source=rss----36c52163afe6---4</link>
            <guid isPermaLink="false">https://medium.com/p/7ec5ac231443</guid>
            <category><![CDATA[cloud-cost-optimization]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[batching]]></category>
            <category><![CDATA[queue]]></category>
            <dc:creator><![CDATA[Yaroslav Tykhonchuk]]></dc:creator>
            <pubDate>Thu, 16 May 2024 10:38:10 GMT</pubDate>
            <atom:updated>2024-05-16T12:37:38.237Z</atom:updated>
            <content:encoded><![CDATA[<p>Cash flow consists of two parts: money in and money out. While increasing earnings is a priority, cutting expenses without compromising product quality is also important.</p><p>This story is about how we reduced our bills for Azure Queues operations by <strong>more than ten times</strong>, which is around 45k $ per year in our scale. It might not impress you a lot, but in Ukraine, where our technical team is based, you could hire a strong developer for that money.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7bE9toERkoTZRKrsaXMVNg.png" /><figcaption>From 140$ a day to 14$ a day</figcaption></figure><p>Our story aims to inspire companies to monitor their soaring cloud expenses. We’re excited to share our own journey and the <a href="https://github.com/youscan/azure-batch-queue">Azure Batch Queue</a> open-source library we developed, as we didn’t find any alternatives in the market.</p><h3>Who can benefit?</h3><p>The library is valuable for organizations with significant queue workloads, typically exceeding millions of daily operations. Anything less would not be noticeable on your cloud bill.</p><p>Azure Queues, particularly, are very cheap — <a href="https://azure.microsoft.com/en-us/pricing/details/storage/queues/">10000 operations cost $0.0004</a>.</p><p>How come did we pay so much for queues? In the social listening industry, <a href="https://youscan.io/">YouScan</a> collects and analyzes thousands of social media and news sources, resulting in billions of daily operations.</p><h3>Idea</h3><p>Processing one message involves three billed operations: write, read, and complete.</p><blockquote>Batching multiple items together and sending them in one message reduces the number of operations (and costs) proportionally.</blockquote><p>So now, when you send a queue message with 50 items, you pay just for 3 operations instead of 150 operations (50 writes, reads, completes).</p><h3>Requirements</h3><p>The idea is clear. Let’s outline the key requirements for our batch queue:</p><ul><li>Ensure no item is lost;</li><li>Minimize duplicate processing;</li><li>Support large messages;</li><li>Handle poison messages effectively;</li><li>Easy switch from Azure.Storage.Queues</li></ul><h3>Implementation</h3><p>Before diving into our batch queue&#39;s unique requirements, let’s first discuss two crucial aspects that apply to both batch and single-item queues: supporting large messages and effectively handling poison messages.</p><h4>Support large messages</h4><p>The limit for the Azure queue message is 64KB for XML and 48KB for Base64. If the item we send is larger than allowed, we offload the content of the message to the Azure Blob and send only reference to the data through the queue. So it doesn’t matter whether the client sends a huge item or a batch with hundreds of items in a single message — we will deliver it nonetheless.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g_MlHs0Lj85H9SBGEKpSZA.png" /></figure><h4>Handle poison messages effectively</h4><p>Message processing can fail for various reasons, like the downtime of the external service, non-backward compatible contract change, or corrupted message data. When that happens, we send the message back to the queue because, in most cases, the problem was due to timeouts from other services. But we don’t allow poison messages to degrade pipeline throughput with never-ending retries. If the message fails a configured number of times, it will be sent to a separate quarantine queue. An engineer can later inspect the messages and situation and, after resolving it, send messages back to processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hW1qoCWUExfV72t1aVWb-w.png" /></figure><p>The complete pipeline of the message queue looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_uko_J_vIrCxRUImzLDBwA.png" /></figure><h3>Batch queue implementation</h3><p>New challenges arise when sending a batch of multiple records in a single message. How do we let consumers complete every record individually? How do we minimize duplicate processing? And most importantly, how to ensure that we deliver all the items? Let’s explore how we address these challenges.</p><h4>Never lose a single item</h4><p>In typical single-item message scenarios, Azure Queue ensures <a href="https://www.cloudcomputingpatterns.org/at_least_once_delivery/">at-least-once</a> delivery through message persistence and visibility timeout.</p><p><strong>Message Persistence</strong>: Messages in Azure Queue are stored durably until explicitly deleted. This ensures that messages are not lost even during system failures or restarts.</p><p><strong>Visibility Timeout</strong>: After dequeuing, a message is temporarily hidden from other consumers for a specified visibility timeout. If a consumer fails to process the message within this time, it reappears in the queue for another attempt.</p><p>However, when dealing with batch queues, the challenge is to guarantee at-least-once delivery for every individual item within the batch. We achieve this by completing each item independently. <strong>Consumers do not even know that the batch exists</strong>. They receive multiple items and process them separately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CM7WKPhjj4ZnBdOaNmxlaQ.png" /><figcaption>Send items to processing separately without any knowledge of the batch</figcaption></figure><p>Under the hood, we maintain an in-memory structure to track the status change of every item, and the batch is considered processed <strong>only</strong> when all its items have been completed or failed. Once all items in a batch are processed successfully, we delete the original message from the queue. However, if any items fail processing, we return the updated message with only the failed items back to the queue for reprocessing. Even during a system crash and failure to commit processed items, we will not lose anything because a message with all records will return to the queue after the visibility timeout.</p><p>The simplified сonsumer looks like this in the code. Notice again that there is no knowledge about the batch.</p><pre>var items = await batchQueue.Receive();<br><br>foreach (var item in items)<br>{<br>    await sendToPipeline(item);<br>}</pre><p>When later in the pipeline, we need to mark the item as completed, we do the following:</p><pre>item.Complete()</pre><p>The same for failed items:</p><pre>item.Fail()</pre><h4>Minimize duplicate processing</h4><p>So far, we have discussed two possible states that every batch item can reach: completed or failed. However, in data flow processing, items can also become stuck for various reasons, such as when an outside service is pending or experiencing issues. With retries in place, the execution can take a long time, and the entire batch&#39;s completion could be delayed significantly, impacting the performance of other tasks and the overall system. Moreover, when the message visibility time expires, the whole batch returns to the queue and must be processed again.</p><p>For this reason, we set a timer that will trigger just before the message&#39;s visibility timeout expires. This means only failed and unprocessed items will return to the queue, and there is no need to reprocess successfully completed items.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h7rYlG-0eQdyqd3t14RgYA.png" /><figcaption>Return failed and unprocessed items back to the queue.</figcaption></figure><h3>Efficiency in processing</h3><p>So far, so good; we send a bunch of items in a single message, and we make sure they are never lost or stuck. Let’s see what else we can improve.</p><h4>Compression</h4><p>In our system, we send data to queues in JSON format, and with tens or hundreds of items in a single queue message, the duplication of fields and values is significant. That’s why BatchQueue is shipped with a GZip compression serializer, which helped us significantly reduce the size of the message.</p><h4>Recyclable memory</h4><p>The other thing that can greatly improve application performance lies in the land of memory and GC. Billions of messages need billions of serializations that require extensive memory allocation and deallocation. This can lead to frequent garbage collection and memory fragmentation.</p><p>Microsoft’s <a href="https://github.com/microsoft/Microsoft.IO.RecyclableMemoryStream">Recyclable Memory Stream</a> library addresses this issue by implementing a memory stream that recycles memory blocks instead of allocating new ones. It maintains a pool of memory blocks of varying sizes, and when a stream is disposed of, the memory block is returned to the pool for later reuse. This reduces the frequency of allocating new memory blocks, incurs far fewer gen 2 GCs, and reduces memory fragmentation.</p><h3>Results</h3><p>It’s crucial to continuously monitor infrastructure expenses and seize opportunities for optimization when they arise. Cost reduction projects offer clear goals and immediate measurability, making them highly rewarding endeavors.</p><p>We had a great time writing Azure Batch Queue and witnessed a tenfold decrease in queue costs. It has already transferred hundreds of billions of records and saved us some money. We hope more companies will start challenging their cloud expenses. Cheers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ec5ac231443" width="1" height="1" alt=""><hr><p><a href="https://medium.com/youscan/batching-for-cloud-queues-spend-10x-less-7ec5ac231443">Batching for cloud queues: spend 10x less</a> was originally published in <a href="https://medium.com/youscan">YouScan</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[On call: how not to burn]]></title>
            <link>https://medium.com/youscan/on-call-how-not-to-burn-7737be8cb390?source=rss----36c52163afe6---4</link>
            <guid isPermaLink="false">https://medium.com/p/7737be8cb390</guid>
            <category><![CDATA[on-call-rotation]]></category>
            <category><![CDATA[on-call]]></category>
            <category><![CDATA[engineering-culture]]></category>
            <category><![CDATA[burnout]]></category>
            <dc:creator><![CDATA[Yaroslav Tykhonchuk]]></dc:creator>
            <pubDate>Fri, 12 Apr 2024 13:33:36 GMT</pubDate>
            <atom:updated>2024-04-14T05:47:17.467Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YARskH8sBWOc9cTMxcpawA.png" /><figcaption>My art version of “On-call Engineer on Fire”</figcaption></figure><p>I quit my previous job significantly because of horrible on-call shifts, never-ending pager duty alerts, and continuously waking up in the middle of the night. You could argue that I am a new generation of soft smoothie drinkers who don’t know what a hard job is. Well, you are right. I prefer solving problems during the day through well-defined processes and engineering culture. In the end, the job should be fun, right?</p><p>At <a href="https://youscan.io/">YouScan Social Listening</a>, an on-call engineer looks at the alerts and responds during business hours, sometimes a bit later, but never at night. Let’s see why it works for us.</p><h3>Our proven strategies for peaceful nights and productive days</h3><h4>Weekly incident reviews</h4><p>We don’t have a lot of incidents so we can review every one of them. We don’t usually write postmortems, but the on-call engineer says why the incident happened and what was done to prevent it in the future.</p><p>Room for improvement: some incidents happen more than once, but the fix is costly, so we are postponing it. That’s probably a bad thing to do.</p><h4>Transparency</h4><p>Do not try to hide when you did something wrong or the code change went terribly. Show it to the other developers and tell them why it happened and what you did to fix it. It will make both you and the team stronger.</p><p>Also, when something important gets broken, we don’t try to sweep it under the rug — we tell it to the whole company. They deserve to know right away and update the customers if needed. In case of a fire, we update our <a href="https://youscan.statuspage.io/">status page</a> as well.</p><h4>Metrics, dashboards, logs</h4><p>Boring — yes. Do all companies do it well? I doubt that.</p><p>Good metrics and logs make it easy to identify the problem quickly. After the fix, when the indicator that triggered the alert stabilizes, and the logs look good, you know that the incident was resolved.</p><p>We collect and process around 600 mln unique posts daily from social media and news platforms worldwide, so queue ingestion throughput is critical for us. Here is an example of the dashboard we created to track main queues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fHswWDgexlASK-Yrp-A2dA.png" /><figcaption>Dashboard where the Product team tracks its main queues</figcaption></figure><h4>Alerts</h4><p>Alerts are created to catch the incident before clients notice it first. You should trust the alerts and minimize the false positives. If everyone gets used to the alert, it’s very easy to miss something important. We constantly review alerts but still have room for improvement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-RK9hGrIIg5oJzvykAKTBg.png" /><figcaption>#alerts-disaster channel in our Slack</figcaption></figure><h4>Catching more zzzs</h4><p>We don’t wake up engineers at night unless the world is falling… (Excited? Check the open roles <a href="https://youscan.io/careers/">here</a>)</p><p>It’s a conscious choice. We are not launching spaceships or doing heart surgery. Can something minor go unnoticed for a couple of hours from time to time — yes, it can. Is it something that clients notice? Very rarely. But the benefits of good sleep for all engineers are huge. We have a pretty big product, but only three engineers rotate on-call bi-weekly. Without the proper rest, we would constantly get tired.</p><h4>It’s the developer’s time</h4><p>On-call gives you an excellent time to get familiar with the entire codebase and gain broad knowledge. It’s also a time when you don’t need to add more and more new features, creating more and more liability and tech debt. It’s this Zen time when you can finally go and fix things that have been broken for a while. Or refactor that ugly place in the code stuck in your head. You own your time fully, and you decide what to do.</p><h4>Appreciation</h4><p>What a beautiful feeling it is when you find that nasty bug or badly written test that caused the incident and fix it for good. And then you show it you your team on regular meetups and feel proud that you did a good job.</p><p>Or when you answer the client’s request in minutes, not hours or days. They appreciate your work, and you do too.</p><p>This is what our client wrote us the other day when we added a useful feature instantly:</p><blockquote>I had a question regarding the api and whether I could query for the ´resourceType´ that got resolved super quickly. Just wanted to say thanks for the quick implementation of that feature.</blockquote><h3>Problems that we have</h3><h4>Busy Weeks: numerous questions from clients and support</h4><p>Our on-call engineer doesn’t just respond to the incidents, he/she also answers questions and addresses<strong><em> </em></strong>bugs uncovered in our product<em> </em>by the Support team. Some weeks can go crazy with requests, and combined with the incidents, it may be hard for the on-call engineer.</p><h4>Stealth mode</h4><p>We call the stealth mode a situation where other developers release big changes without notifying teammates. It doesn’t happen much, but it is very frustrating when the simple rollback doesn’t help, and you need to dig deeper when the original troublemaker is unavailable.</p><h3>Summary</h3><p>We are using on-call time to improve our lives. We improve the clients’ lives when we fix bugs immediately when they are found, and we improve our lives as developers when we ensure that incidents will not happen again. And we sleep at night…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7737be8cb390" width="1" height="1" alt=""><hr><p><a href="https://medium.com/youscan/on-call-how-not-to-burn-7737be8cb390">On call: how not to burn</a> was originally published in <a href="https://medium.com/youscan">YouScan</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What it’s like to be a developer at YouScan]]></title>
            <link>https://medium.com/youscan/what-its-like-to-be-a-developer-at-youscan-90fd1bc46043?source=rss----36c52163afe6---4</link>
            <guid isPermaLink="false">https://medium.com/p/90fd1bc46043</guid>
            <category><![CDATA[culture]]></category>
            <category><![CDATA[workplace]]></category>
            <dc:creator><![CDATA[Yaroslav Tykhonchuk]]></dc:creator>
            <pubDate>Mon, 31 Jan 2022 08:57:13 GMT</pubDate>
            <atom:updated>2022-02-01T09:40:25.021Z</atom:updated>
            <content:encoded><![CDATA[<h4>From popular tools to code review, deployment to daily life, here’s a look at the developer experience at YouScan from an insider.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hqxDOUzSLo9wcnRa6b1RrA.png" /><figcaption>Mob programming</figcaption></figure><p><em>Yaroslav Tykhonchuk, Software engineer</em></p><h4>What are the most common tools that developers use?</h4><ul><li>JetBrains Rider, WebStorm, PyCharm, or VS Code are favorite IDEs/editors depending on the team;</li><li>Git, GitHub for code management and reviews;</li><li>Jenkins and GitHub Actions for CI/CD;</li><li>Kibana and ElasticSearch for logging, Grafana for dashboards;</li><li>Notion for knowledge base and as a product management tool;</li></ul><p>Check more on our <a href="https://tech-radar.youscan.io/">tech radar</a>. Aside from that, our developers use whatever tools they are most comfortable with.</p><h4>Which languages do developers code in?</h4><p>Our backend stack is mainly written in C#. Our user-facing UI is written in TypeScript and uses React. The Data Science team develops in Python.</p><h4>What is the development process like? (the lifecycle of a piece of committed code)</h4><p>We use <a href="https://basecamp.com/shapeup">Shape Up</a> methodology from Basecamp to shape, bet, and build our product, meaning that we have 6-week cycles to develop product features following a 2-week cool-down period when each developer decides what to do on their own. When the team has agreed on what to do during the cycle and has split into teams (individual, pair, or group) coding work starts.</p><p>The following is highly specific to developers that work in Product Squad. Our Data Science, Platform, Data teams have a similar but different development lifecycle.</p><ul><li>Pull a <em>master </em>branch from <em>youscan</em> repository. If you want to share your work after, checkout to a new branch.</li><li>Complete a discrete unit of work (fixing a bug, getting started on a feature). Decide if it’s going to be live or not — we have a set of feature flags that show and hide code in production. Write unit tests, or extend existing ones.</li><li>If you were working on <em>master</em> just push the branch and wait for the build to complete a new artifact, otherwise open a pull request, and request a reviewer (just someone you want to share a piece of new code with or someone with more context on the part of the codebase you maintain).</li><li>When tests have passed successfully and your artifact is ready, hit the deploy button to ship a specific service to <em>staging</em> or <em>production.</em></li><li>We ship 5–10 times a day depending on the service and have multiple channels in slack (#alerts-exceptions, #alerts-fatality), grafana, and kibana dashboards to monitor deployment after. If the change goes bad, I figure out how to fix forward or revert my change.</li><li>Optional: Socialize the change. This might mean telling your team or customer support agents, or telling a colleague in your triage channel, depending on the size of the change.</li></ul><h4>What is code review like?</h4><p>Code review is not required at YouScan. Normally you ask for a review when you have something to share with your teammates or want to ask someone with bigger expertise to verify your changes.</p><h4>How is testing done, and what kind of tests are run?</h4><p>We don’t have QA engineers on purpose so every developer is responsible for their piece of code along with tests. Some practice TDD, some write tests after, the only thing that matters is that new/changed functionality was covered by tests.</p><h4>How is code deployed?</h4><p>When the artifact is ready and the developer pushes the deploy button, K8s or Swarm (depending on service) pulls a new image and extracts it into place. All of that happens in just a few seconds.</p><p>If something significant was deployed we share it to our #product-announcements slack channel so that everyone at the company can put their favorite emojis and recognize your work.</p><h4>What is the average day-in-the-life of someone on one of the development teams?</h4><p>I guess it depends on your team and your location. Some engineers are fully remote, some partially. Right now we have teammates drinking coconuts in Thailand, Bali, and Brazil near their laptops, while most of us enjoy the beautiful view from our Kyiv office during the workday.</p><p>Development teams can set their own working styles, but they are mostly centered around development time. We don’t have daily stand-ups, everyone just writes their status (progress, interesting findings, problems that have occurred) to the team channel, and is good to go coding. Btw, all channels in Slack are public, making the work of every team very transparent.</p><p>Teams generally interact in person or over Slack to discuss issues, coordinate development or ask questions. We have one weekly sync to deeply discuss the progress, talk through the created bugs and incidents that may have happened a week before and share a pair of jokes.</p><h4>What makes YouScan a special place to be a developer?</h4><p>At YouScan you get to be part of an incredibly supportive and highly professional team motivated to build a world-class product. YouScan has a broad stack and very interesting problems. Our coworkers are experts in their fields (and nice, easy-going people!), and we get to be curious and learn a lot.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90fd1bc46043" width="1" height="1" alt=""><hr><p><a href="https://medium.com/youscan/what-its-like-to-be-a-developer-at-youscan-90fd1bc46043">What it’s like to be a developer at YouScan</a> was originally published in <a href="https://medium.com/youscan">YouScan</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>