<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ohad Edelstein on Medium]]></title>
        <description><![CDATA[Stories by Ohad Edelstein on Medium]]></description>
        <link>https://medium.com/@ohadedelstein?source=rss-14bbf574ca6d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*GDPbFoQtnRUCIscJ.</url>
            <title>Stories by Ohad Edelstein on Medium</title>
            <link>https://medium.com/@ohadedelstein?source=rss-14bbf574ca6d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:56:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ohadedelstein/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Microservices Made Me Paranoid: Confessions of a Former Monolith Developer]]></title>
            <link>https://medium.com/@ohadedelstein/https-medium-com-ohadedelstein-are-you-a-software-psychopath-467525bd6a79?source=rss-14bbf574ca6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/467525bd6a79</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Ohad Edelstein]]></dc:creator>
            <pubDate>Sun, 23 Feb 2025 19:27:07 GMT</pubDate>
            <atom:updated>2025-02-23T19:27:07.907Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qcE6UibFMBXTNwxb.jpg" /><figcaption>Love Michael C. Hall’s “Dexter”, but Bale’s “<a href="https://en.wikipedia.org/wiki/Patrick_Bateman">Patrick Bateman</a>” will always be my first psychopath</figcaption></figure><p>In software development, we don’t truly understand system fragility until something breaks catastrophically in production. Sure, we all think about failure scenarios — that’s Software Engineering 101. But I’m not talking about the basic “what if” scenarios we discuss in code reviews. You know, the kind where someone asks “What if the database is down?” and we smugly point to our retry mechanism.</p><p>No, I’m talking about the kind of deep, uncomfortable contemplation that happens when you’ve spent countless nights staring at logs, trying to understand why your perfectly tested service decided to throw a tantrum in production. The kind of thinking that makes other developers slowly back away when you start explaining all the ways your system could fail.</p><p>Feeling uncomfortable? Good! It means you’re not yet jaded enough. But don’t worry — I wasn’t either. This is the story of how I transformed from an optimistic developer into someone who lies awake thinking about edge cases.</p><h3>Microservices Will Make You Question Everything</h3><p>Once you’ve gone down the microservice path, you spend a considerable amount of your time as a developer thinking about all the ways your code is going to misbehave in production, and you obsessively plan for its inevitable failure. You’ve seen things in production logs that would make a junior developer cry. You know that success scenarios work… until they don’t.</p><p>Network hiccups? Expected. Load spikes? Planned for. Latency issues? Naturally. Hardware failures? Of course. Third-party library breaking changes? Just another Tuesday. Open source infrastructure quirks? Part of life. Other developers’ “creative” code? The spice of life.</p><p>It means you truly own your code, including all its potential failure modes.</p><h4>Why You’re Probably Still an Optimist</h4><p>In a typical scenario, when your code lives inside a cozy monolith, you find yourself in the comfortable “what if” mindset. Why? Because of The Voice. You know the one — it whispers “don’t worry too much, if the system goes down, your little feature will be the least of anyone’s problems. Just ship it, you have deadlines!”</p><p>So you follow all the established patterns:<br>- Write the unit tests (green!)<br>- Add some regression tests (green!)<br>- Test in staging (looks good!)<br>- Set up basic monitoring (heartbeat’s steady!)<br>- Ship it and move on</p><p>This approach works fine when your code lives in a stable, battle-tested system. I know because I lived this life, comfortable in our monolith’s embrace.</p><h4>The Microservice Awakening</h4><p>When you go microservice, you become responsible for a system that can not only fail but can take others down with it. At first, you’re excited about the independence, the scalability, the pure architectural beauty of it all.</p><p>But then came the task that changed everything: extracting critical business logic into a microservice while keeping the monolith dependent on it. No pressure, right?</p><h4>The Freedom That Kills</h4><p>Here’s what really messed with my head: the sudden freedom. You see, a microservice gives you the liberty to break free from all those constraints the monolith enforces. Suddenly, you can upgrade that decaying infrastructure. Maybe bump up to the latest Spring Boot version. Update those ancient third-party libraries. Or — dare I say it — upgrade the Python version (because let’s be honest, what monolith in 2025 isn’t still stuck on Java 8?).</p><p>Your testing pipeline? It doesn’t have to follow the company’s “happy flow” anymore — you know, that battle-tested-but-slower-than-molasses deployment process that keeps the monolith functioning. Because isn’t that why we extracted this logic in the first place? To move fast and break things?</p><p>And that’s exactly when it hits you. The same code that worked perfectly in the monolith now keeps you up at night:<br>- Database connection pooling looks naive<br>- API timeout calculations feel arbitrary<br>- Logs that seemed informative are now useless<br>- Error handling that felt robust seems laughably optimistic</p><p>You find yourself obsessing over questions like:<br>- What happens when (not if) this service fails?<br>- How will the monolith cope?<br>- Can it cope?<br>- What’s my fallback strategy?</p><p>Suddenly, you’re not just writing code — you’re building a system that needs to be resilient, observable, and gracefully degradable. Your “what if” scenarios transform into “when this fails” preparations.</p><h4>The Transformation</h4><p>This isn’t about becoming pessimistic — it’s about becoming realistic. Every system will fail. The question is: will it fail gracefully, with clear error messages and fallback behaviors, or will it drag down adjacent services in a cascading failure that has ops teams pulling their hair out at 3 AM?</p><p>The code didn’t change, but you did. You started thinking about failure modes not as edge cases, but as inevitable scenarios that need to be handled elegantly. You became productively paranoid, and your systems became more resilient as a result.</p><h4>The Self-Assessment</h4><p>Want to test if you’ve made the transition? Ask yourself:<br>- Do you have clear fallback behaviors for every external dependency?<br>- Can your service fail independently without taking down others?<br>- Do your logs tell a story, or just spam errors?<br>- Does your monitoring tell you why something failed, or just that it failed?</p><p>If you’re not satisfied with your answers, welcome to the club. We’ve got work to do.</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>I’d love to hear your thoughts and experiences. Have you made a similar transition? What pushed you over the edge? Share your stories in the comments below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=467525bd6a79" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Fidelity TestThe Surprising Simplicity of Mimicking Thought]]></title>
            <link>https://medium.com/@ohadedelstein/the-fidelity-testthe-surprising-simplicity-of-mimicking-thought-6cbf25125d74?source=rss-14bbf574ca6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/6cbf25125d74</guid>
            <category><![CDATA[human-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[agi]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Ohad Edelstein]]></dc:creator>
            <pubDate>Tue, 18 Feb 2025 16:16:34 GMT</pubDate>
            <atom:updated>2025-02-18T16:16:34.213Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Surprising Simplicity of Mimicking Thought</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*Z5CRy_yLzzII6W3Nr-JPyQ.png" /><figcaption>The Fidelity Test</figcaption></figure><p>As a software engineer working daily with AI, I’m struck by a profound realization: our assumptions about the complexity of intelligence might have been fundamentally misplaced. Each interaction with tools like Cursor forces me to confront an increasingly apparent truth — what we consider intelligence might be far more replicable than we imagined.</p><p>Working with Cursor has become a daily exercise in questioning my assumptions. When I describe a feature in natural language, provide some context about our codebase, and watch as Cursor grasps my intent and generates sophisticated solutions, I’m witnessing a system that exhibits three key properties we once thought were uniquely human: the ability to transfer knowledge across domains, adapt to novel contexts, and engage in abstract reasoning about implementation details. This suggests that these capabilities, while impressive, might emerge from sophisticated pattern recognition rather than requiring some special form of consciousness.</p><p>This observation leads to a provocative question: what if the distinction between “simulating” intelligence and “being” intelligent is more arbitrary than we assumed? We’re discovering that tasks we once thought required deep understanding can be accomplished through statistical pattern matching at scale. The implications are both unsettling and fascinating.</p><p>Consider how we interact with these systems. When Cursor understands my coding requirements, suggests improvements, and generates solutions, does it matter whether it has “true” consciousness or self-awareness? The outputs are increasingly indistinguishable from what we’d expect from a human colleague. Modern AI systems can engage in sophisticated problem-solving, creative generation, and even apparent metacognition — reflecting on their own limitations and reasoning processes. This suggests that what we perceive as consciousness might be an emergent property of sufficiently sophisticated information processing systems rather than a binary trait that systems either have or don’t have.</p><p>What’s particularly striking is how this mirrors long-standing philosophical debates about consciousness and free will. Philosophers and neuroscientists have long questioned whether our sense of self, consciousness, and free will might be elaborate constructions of our pattern-recognizing brains. The success of large language models in replicating intelligent behavior without these supposed prerequisites adds weight to this perspective.</p><p>The real surprise isn’t that AI can perform these tasks — it’s that performing them doesn’t require the kind of deep, mysterious intelligence we imagined. Watching Cursor refactor complex code, suggest architectural improvements, and even explain its own reasoning process has shown that intelligence, or at least its observable manifestations, might be more mechanistic and replicable than our intuitions suggested. Each advancement in AI capability seems to chip away at our assumptions about what makes human intelligence unique.</p><p>The traditional view of AGI requires solving hard problems like causal reasoning, common sense understanding, and autonomous goal-setting. But our experience with large language models suggests these capabilities might emerge from scale and architecture improvements rather than requiring fundamental breakthroughs. The system’s ability to reason about code, debug issues, and propose architectural solutions demonstrates a form of causal thinking that, while not perfect, challenges our assumptions about what’s required for “true” intelligence.</p><p>This isn’t to say we’ve solved all the mysteries of intelligence or consciousness. The one frontier that remains truly elusive is genuine self-reflection — the ability to form a persistent, evolving self-model and genuinely question one’s own nature and existence. This theme was brilliantly explored in the first season of Westworld (2016), where artificial beings’ path to consciousness was marked not by their ability to think or solve problems, but by their capacity to question their own reality and form genuine autobiographical awareness. While our current AI systems can engage in apparent metacognition, they lack this deeper form of self-reflection and persistent identity. But perhaps even this final frontier isn’t as insurmountable as we imagine. If what we consider intelligence can be effectively replicated through sophisticated pattern matching and response generation, then maybe we’re not waiting for some fundamental breakthrough. Maybe we’re just iterating toward increasingly sophisticated versions of what we already have.</p><p>The implications are profound. As our AI systems become more sophisticated, the line between simulated and “real” intelligence continues to blur. We might be approaching a point where the distinction becomes meaningless — not because we’ve created true consciousness, but because we’ve discovered that consciousness itself might be more elusive than we imagined.</p><p>Rather than pursuing AGI as a distant, monolithic goal requiring fundamental breakthroughs, we might instead focus on incrementally expanding the capabilities of existing systems. The fact that AI tools like Cursor can already engage in sophisticated software engineering, creative writing, and complex problem-solving suggests that general intelligence might be more accessible than we imagined — not because it’s simple, but because our conception of intelligence itself needed revision.</p><p>This perspective doesn’t diminish the remarkable nature of human intelligence. Rather, it suggests that we might be closer to understanding and replicating it than we thought — not because the task is simple, but because intelligence itself might be more mechanical than our intuitions led us to believe.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6cbf25125d74" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond Buzzwords: The Peril of Mislabeling Tech and How to Get It Right]]></title>
            <link>https://medium.com/@ohadedelstein/beyond-buzzwords-the-peril-of-mislabeling-tech-and-how-to-get-it-right-c9f31e3c3857?source=rss-14bbf574ca6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9f31e3c3857</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[technology-trends]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <dc:creator><![CDATA[Ohad Edelstein]]></dc:creator>
            <pubDate>Sun, 16 Jun 2024 11:39:19 GMT</pubDate>
            <atom:updated>2024-06-16T11:39:19.480Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/590/0*lPKH7JZnzEG3i-AL.jpg" /><figcaption>Seeing past the shine to the substance can save your (professional) life</figcaption></figure><h3><strong>Introduction:</strong></h3><p>In the tech world, buzzwords and broad labels are as common as code. It’s easy to throw around terms like “database,” “file system,” or “virtual machine” without a second thought. But when these labels are applied incorrectly to technologies, it can create a ripple effect of confusion and costly mistakes, especially when it comes to building and integrating systems. Let’s unravel some common tech misnomers and then delve deep into a specific example to illustrate the very real dangers of mislabeling.</p><h4><strong>Common Tech Misnomers: A Few Examples</strong></h4><ul><li><strong>Elasticsearch:</strong> While Elasticsearch can store and retrieve data, its core functionality lies in search and analytics. It’s like a supercharged search engine, built to sift through massive datasets and uncover hidden insights. Thinking of it primarily as a database can downplay its true strengths in searching, indexing, and real-time data analysis. However, it’s important to note that Elasticsearch offers some features that might seem contradictory to traditional databases. For instance, it has the ability to delete the original document after it’s been analyzed, focusing solely on the extracted information for search purposes. This can be a great way to reduce storage usage, but it goes against the core principle of a database, which is to persistently store the actual data itself. Here, we see how mislabeling can lead to misunderstandings. Elasticsearch excels at specific tasks, but it’s not a one-size-fits-all replacement for traditional databases.</li><li><strong>AWS S3:</strong> It might seem like a “file system” on the surface, but AWS S3 is fundamentally an object storage service. Instead of a traditional hierarchical structure, it uses a flat address space where each object (think photos, videos, or documents) is assigned a unique identifier. This design prioritizes durability, scalability, and the ability to handle vast amounts of unstructured data.</li><li><strong>Docker:</strong> While often grouped with virtual machines, Docker takes a different approach to isolation. It utilizes containers, lightweight packages that encapsulate applications and their dependencies. Unlike VMs, which emulate entire operating systems, containers share the host’s kernel, making them significantly faster and more resource-efficient. This is a major advantage, allowing developers to spin up and scale applications much quicker than with VMs. However, this very benefit can also be a hidden danger if mislabeled as a VM. Because containers share the kernel, they inherit the security vulnerabilities of the host machine. Additionally, if one container crashes, it can potentially bring down other containers sharing the same kernel. This is a stark contrast to VMs, which provide a more isolated environment, where a failing VM won’t impact others on the same host. So, while Docker offers advantages in speed and efficiency, it’s crucial to understand its limitations, especially when it comes to security and isolation.</li></ul><h4><strong>The S3 Trap: When a Misnomer Becomes a Nightmare</strong></h4><p>Let’s take a closer look at the dangers of mislabeling by examining AWS S3, often erroneously called a “file system.” This seemingly innocuous misnomer can lead to a cascade of problems:</p><ul><li><strong>False Assumptions about Structure:</strong> If you design your application expecting S3 to behave like a file system with nested folders, you’re in for a rude awakening. S3’s flat structure means you’ll need to rethink your data organization strategies, potentially leading to inefficient storage and retrieval.</li><li><strong>Unrealistic Latency Expectations:</strong> S3 is optimized for durability and high throughput, not the low-latency access you’d expect from a local file system. Assuming otherwise can lead to performance issues, especially for applications that require frequent, small data reads or writes.</li><li><strong>Consistency Conundrum:</strong> S3 offers eventual consistency, meaning there might be a delay before changes to your data are reflected across all of S3’s servers. If your application depends on immediate consistency, like a financial transaction system, this can cause serious problems if not properly accounted for.</li><li><strong>Concurrency Chaos:</strong> S3 lacks the locking mechanisms you’d find in a file system. This means if multiple processes try to modify the same object simultaneously, you could end up with data corruption or race conditions. You’ll need to implement your own concurrency controls, adding complexity to your application.</li><li><strong>Permission Perplexity:</strong> S3’s access control model is based on policies rather than traditional file permissions. Misunderstanding this distinction can leave your data vulnerable to unauthorized access or make it difficult to manage permissions effectively.</li></ul><h4><strong>The Importance of Deep Tech Knowledge</strong></h4><p>As these examples illustrate, mislabeling technology can lead to a world of hurt. To avoid these pitfalls, it’s crucial to go beyond the buzzwords and marketing hype. Dive into official documentation, understand the underlying architecture, and grasp the nuances of each technology you work with. By investing in this knowledge, you equip yourself to make informed decisions, design optimized systems, and build products that truly stand out.</p><h4><strong>Conclusion</strong></h4><p>In the fast-paced tech world, where new tools and technologies emerge constantly, it’s easy to get swept up in the hype. But remember, a clear understanding of the tools you’re working with is essential. Don’t let misnomers lead you astray. Take the time to educate yourself, ask questions, and challenge assumptions. By doing so, you’ll build a solid foundation for your projects, avoid costly mistakes, and ultimately create innovative solutions that deliver real value.</p><p>Thank you for reading. Let’s continue to strive for clarity and understanding in our tech discussions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9f31e3c3857" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Do We Keep ‘Killing the Messenger’?]]></title>
            <link>https://medium.com/@ohadedelstein/why-do-we-keep-killing-the-messenger-cb3d26392e14?source=rss-14bbf574ca6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb3d26392e14</guid>
            <category><![CDATA[management-and-leadership]]></category>
            <category><![CDATA[workplace-culture]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[communication]]></category>
            <dc:creator><![CDATA[Ohad Edelstein]]></dc:creator>
            <pubDate>Fri, 26 May 2023 12:01:47 GMT</pubDate>
            <atom:updated>2023-05-26T12:01:47.098Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QOudmDNCgZqRIAxJBtEFgw.jpeg" /><figcaption>one flew over the cuckoo’s nest</figcaption></figure><h4>Understanding Our Tendency to Blame the Bearer: Navigating Emotional Responses, Scapegoating, and Cognitive Dissonance</h4><p>In colloquial terms, “<em>killing the messenger</em>” refers to the act of blaming the bearer of bad news, rather than dealing with the root cause of the problem at hand. While we no longer physically harm the messenger in our modern society, the metaphor lives on. But why does this happen? Why do we continuously fall into the trap of displacing our frustration onto those who are simply delivering the message?</p><h4>Human Nature and Emotional Responses</h4><p>The first reason is rooted in our instinctive emotional responses. When faced with negative news, our immediate reaction is typically one of defense or denial. We may experience fear, anger, or sadness, and our instinct is to lash out at what we perceive as the source of our discomfort — in this case, the messenger.</p><h4>Scapegoating</h4><p>The second reason is connected to the concept of scapegoating. Scapegoating is a psychological defense mechanism where we project our faults or mistakes onto someone else, thus absolving ourselves of responsibility. When bad news is delivered, it’s easier to target the bearer of the news, rather than tackling the issue or admitting our role in the problem.</p><h4><strong>Cognitive Dissonance</strong></h4><p>Lastly, cognitive dissonance can also play a role. This refers to the discomfort we experience when we hold two contradictory beliefs simultaneously, or when our actions do not align with our beliefs. When faced with bad news that challenges our existing beliefs or perceptions, we might react negatively towards the messenger to alleviate the discomfort.</p><h4>False Problem Resolution</h4><p>Interestingly, another reason we ‘kill the messenger’ is a misguided perception that by doing so, we might make the problem disappear. It’s an illusory sense of problem resolution — if we eliminate the bearer of the bad news, we can pretend the issue doesn’t exist. But in truth, this tactic merely blinds us to reality. The problem persists, often growing worse in the absence of attention and resolution.</p><h4>The Implications</h4><p>Despite being a defense mechanism, ‘killing the messenger’ has negative implications for both the individual and society. On a personal level, it hinders our ability to face and overcome challenges. It limits personal growth and the development of resilience.</p><p>In a broader context, it stifles open communication and transparency. Fear of blame discourages people from speaking up or sharing information, leading to a culture where problems are hidden instead of resolved.</p><h4>Conclusion: An Evolution Needed</h4><p>Understanding the psychology behind ‘killing the messenger’ is the first step towards changing this detrimental behavior. The next time we find ourselves blaming the bearer of bad news, it would be beneficial to pause and reflect. Are we upset at the person delivering the news, or the situation they’re describing?</p><p>As we navigate through an increasingly interconnected and complex world, it is vital that we cultivate an environment that encourages open, honest communication. This starts with ensuring that we don’t shoot the messenger but instead focus on addressing the problem at hand. Only then can we begin to truly evolve and progress, facing reality rather than turning a blind eye to it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb3d26392e14" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Window Functions in Apache Spark]]></title>
            <link>https://medium.com/@ohadedelstein/understanding-window-functions-in-apache-spark-b4d6fdf32ce1?source=rss-14bbf574ca6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4d6fdf32ce1</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[transformation]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Ohad Edelstein]]></dc:creator>
            <pubDate>Fri, 26 May 2023 10:29:14 GMT</pubDate>
            <atom:updated>2023-05-26T10:45:18.854Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9rmQiVkcvgqyv2_ptOP7Dg.png" /><figcaption>sparks in a square windows</figcaption></figure><h4>Unraveling the Mechanism of partitionBy(), orderBy() and first() Combination in Apache Spark</h4><p>Working with vast datasets requires a nuanced understanding of Apache Spark’s operations to ensure the reliability of data transformations. This article focuses on window functions, a powerful feature provided by Spark. Window functions are utilized to perform computations across a group of rows related to the current row.</p><p>Consider the following Scala code block to illustrate:</p><pre>import spark.implicits._<br>import org.apache.spark.sql.expressions.Window<br>val data = Seq(<br> (&quot;James&quot;,null,&quot;M&quot;, 1),<br> (&quot;Anna&quot;,&quot;TX&quot;,&quot;F&quot;,1),<br> (&quot;Julia&quot;,null,null,3),<br> (&quot;James&quot;,&quot;CA&quot;,&quot;M&quot;, 3),<br> (&quot;James&quot;,&quot;NY&quot;,&quot;M&quot;, 2)<br>)<br><br>val columns = Seq(&quot;name&quot;,&quot;state&quot;,&quot;gender&quot;,&quot;number&quot;)<br>val df = data.toDF(columns:_*)</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d9a8f00ae131ae0ffc5ea079799d34b0/href">https://medium.com/media/d9a8f00ae131ae0ffc5ea079799d34b0/href</a></iframe><pre>val windowSpec = Window.partitionBy(col(&quot;name&quot;)).orderBy(col(&quot;number&quot;))<br>val df_agg = df.select(<br> col(&quot;name&quot;),<br> col(&quot;state&quot;),<br> first(&quot;state&quot;, ignoreNulls = true).over(windowSpec).as(&quot;state_agg&quot;),<br> col(&quot;number&quot;))<br>.dropDuplicates(Seq(&quot;name&quot;))<br>df_agg.show()</pre><p>The df_agg DataFrame after running the code is:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a20bfc5de9e37802e489e74c60ae00a9/href">https://medium.com/media/a20bfc5de9e37802e489e74c60ae00a9/href</a></iframe><p>You might be wondering why we received null for James when we specifically coded to get the first non-null state, and the first non-null value for James is NY. Let’s unravel this puzzle.</p><h3>Unraveling the Code</h3><p>The orderBy clause in the window specification plays a crucial role here. The ‘windowSpec’ object partitions the DataFrame by the ‘name’ column and sorts it by the ‘number’ column. In other words, the data is grouped by name and sorted by number within each group.</p><p>The DataFrame then undergoes a transformation, selecting the ‘name’ and the first non-null ‘state’ within each partition of the window. We employ ‘ignoreNulls = true’ to avoid selecting null ‘state’ values. Subsequently, duplicate rows, based on the ‘name’ column, are dropped.</p><p>The orderBy clause determines the order in which rows in a partition are processed. In this scenario, we’ve sorted the records by the ‘number’ column.</p><h4>Step by Step Unfolding of the Code</h4><p><strong>Step 1: Partition by name <br></strong>Our df gets divided into partitions based on the ‘name’ column. Here’s what it looks like, James is the one who will be impacted, so let’s focus only on him:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a3609f6ccac480c648eb89cdf7d2933/href">https://medium.com/media/5a3609f6ccac480c648eb89cdf7d2933/href</a></iframe><p><strong>Step 2: Order by number within each partition</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/90d1e92d3509091f6e2b316b17a7f6b1/href">https://medium.com/media/90d1e92d3509091f6e2b316b17a7f6b1/href</a></iframe><p><strong>Step 3: apply window function by number within each partition</strong></p><p>Initially, the window only “allows” the code to see the first row of James. So:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c87be5183cf2637120f456603cf97a49/href">https://medium.com/media/c87be5183cf2637120f456603cf97a49/href</a></iframe><p>Here, the “state” is null. As we’ve specified `ignoreNulls = true`, the function returns null because `first(_, ignoreNulls = true)` indicates that it will try to avoid returning null, but if it can’t, it will have to. Hence:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/13c559caa9e6a98ae4553135850a4840/href">https://medium.com/media/13c559caa9e6a98ae4553135850a4840/href</a></iframe><p>This moves onto the next record in the partition:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5d5034498f6b0d4cc2f17791d8343907/href">https://medium.com/media/5d5034498f6b0d4cc2f17791d8343907/href</a></iframe><p>Now, two rows are visible, and the `first` function can choose the first non-null value and assign it to ‘state_agg’:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7beca85db5d48eb160ede1c64d47e62/href">https://medium.com/media/b7beca85db5d48eb160ede1c64d47e62/href</a></iframe><p>Finally, the processing reaches the last record in the partition:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/70ecf55327ca5e75b2b7a47e5eedacd9/href">https://medium.com/media/70ecf55327ca5e75b2b7a47e5eedacd9/href</a></iframe><p>Now, three rows are visible, and the first non-null value among these is from row 2, ‘NY’, which is assigned to the final ‘state_agg’:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f682f63bd833b311d41f40e1dc5b2af8/href">https://medium.com/media/f682f63bd833b311d41f40e1dc5b2af8/href</a></iframe><h3>Wrapping Up</h3><p>The state of the intermediate df after all of this, results in:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31647d92cef863c6e654de8635e83906/href">https://medium.com/media/31647d92cef863c6e654de8635e83906/href</a></iframe><p>The final `dropDuplicates()` command keeps the first unique match as it iterates over the df, resulting in:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a20bfc5de9e37802e489e74c60ae00a9/href">https://medium.com/media/a20bfc5de9e37802e489e74c60ae00a9/href</a></iframe><p>I hope this step-by-step unfolding of the code helps clarify the intricate mechanism of this important Apache Spark functionality and you now understand the rationale behind our initial unexpected results. <br>Window functions are an indispensable tool in the data craftsman’s toolbox when it comes to handling large datasets and executing complex computations. Being well-versed in their optimal utilization is key to unlocking their full potential.</p><p>Thank you for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4d6fdf32ce1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Don’t Fall for Apache Spark’s ‘First’ and ‘Last’ Trickery, It Will Cost You!]]></title>
            <link>https://medium.com/@ohadedelstein/first-and-last-are-not-your-friends-in-apache-spark-heres-why-841acda7ba4b?source=rss-14bbf574ca6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/841acda7ba4b</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Ohad Edelstein]]></dc:creator>
            <pubDate>Sun, 21 May 2023 09:33:52 GMT</pubDate>
            <atom:updated>2023-05-26T11:15:50.760Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Zpvec5j_BDOjz-tH6vq3Q.jpeg" /></figure><h4>Examining Apache Spark’s ‘First’ &amp; ‘Last’ Methods: Understanding Their Pitfalls and Proposing a More Reliable Approach</h4><h4>1. Introduction</h4><p>Once again, I was faced with the all-too-familiar error. It was like an old nemesis that kept resurfacing, catching unsuspecting coders off guard. Having encountered it on multiple occasions, I decided it was high time to bring this issue to the forefront and raise awareness about it. The culprits? ‘First’ and ‘last’ methods in Apache Spark.</p><h4>2. Understanding ‘First’ and ‘Last’ in Spark</h4><p>In Apache Spark, ‘first’ and ‘last’ are used to retrieve the first and the last element from a group in a DataFrame. This seems straightforward, right? However, the reality is far from it. The confusion lies in how Spark handles data distributed across various partitions.</p><h4>3. The Problem with ‘First’ and ‘Last’</h4><p>Here’s the catch: ‘first’ and ‘last’ are not truly representative of the first and last elements of a group. They are, instead, indicative of the first and last data points from an arbitrary partition that Spark chooses to process. This unpredictability is what makes these methods problematic. They may not necessarily return the same results every time, especially when the data is shuffled or the partitions are processed in a different order. The names are, in a way, misleading — because ‘first’ isn’t always the true first, and ‘last’ isn’t always the true last.</p><h4>4. Spotlight on a Scenario</h4><p>In one case, ‘last’ was used to track the last action a user performed on an application, with data grouped by user id. Here’s a sample of how this was implemented:</p><pre>val df = spark.read.schema(schema).json(events)<br>val grouped = df.groupBy(&quot;userId&quot;)<br>val lastAction = grouped.agg(last(&quot;action&quot;))<br>lastAction.show()</pre><p>However, the action returned was not the true last action, leading to a misunderstanding of user behavior. The arbitrary nature of ‘last’ could cause significant confusion in understanding user’s actions in the application.</p><h4>5. A Suggested Approach: The ‘Any’ Method</h4><p>So, what’s a better approach? Consider using ‘any’. ‘Any’ is a method that, like ‘first’ and ‘last’, returns an element from a group. However, it is honest about its randomness. It clearly states that it will return any arbitrary value from the group, avoiding any false impressions.</p><p>‘Any’ is a more reliable choice. It does not give the illusion of being deterministic like ‘first’ or ‘last’, making it less likely to lead to mistakes.</p><h4>6. Tips for Avoiding the ‘First’ and ‘Last’ Pitfall</h4><p>To avoid falling into the ‘first’ and ‘last’ trap, it’s crucial to be aware of this issue. You need to stop using ‘first’ and ‘last’ forever! Even if you understand their nature and think you’re fine with their non-deterministic behavior, still resist using them. Because one day, you might forget about that piece of code or you may move on from the company, leaving the code to be maintained or modified by someone else.</p><p>Bringing ‘first’ or ‘last’ into the picture can invite a host of uncertainties, especially when new code is being added to the same area. This can potentially spin off into numerous outcomes, some benign but many harmful. The involvement of ‘first’ and ‘last’ in code can lead to the creation of new bugs, amplification of existing ones, or even waste substantial development time, as was our experience.</p><p>For instance, one of my developers thought he’d found a bug with ‘first’. Given our wariness around ‘first’, he ended up spending three days refactoring the code to make it “correct”, only to discover that the original code was actually designed to accommodate the non-deterministic nature of ‘first’. The complexity of the code and the paranoia incited by the presence of ‘first’ resulted in a significant loss of time on a refactor that ultimately wasn’t required.</p><p>In the end, we abandoned that refactor, realizing that chasing the actual ‘first’ element could have serious performance implications. We chose instead to replace ‘first’ with ‘any’, ensuring we wouldn’t stumble over this pitfall in the future.</p><p>So, always be vigilant. Check your code and replace ‘first’ and ‘last’ with ‘any’ when you’re dealing with distributed data in Spark. Remember to always verify the results of your data processing to ensure accuracy.</p><h4>7. Conclusion</h4><p>In conclusion, while ‘first’ and ‘last’ might seem like convenient methods to use in Apache Spark, they can lead to unpredictable and misleading results due to their inherent behavior with distributed data. It’s time we shed light on this issue and encourage the use of ‘any’, a more honest and reliable method. Let’s share this knowledge and improve our coding practices for better, more accurate data handling in Apache Spark.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=841acda7ba4b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Intellij Tricked Me To Think Mockito Is Broken]]></title>
            <link>https://medium.com/@ohadedelstein/my-doh-moment-with-mockito-8a0ffac681e1?source=rss-14bbf574ca6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a0ffac681e1</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[mockito]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[intellij]]></category>
            <dc:creator><![CDATA[Ohad Edelstein]]></dc:creator>
            <pubDate>Sat, 11 May 2019 11:03:54 GMT</pubDate>
            <atom:updated>2019-05-24T12:36:41.269Z</atom:updated>
            <content:encoded><![CDATA[<h4>My Little DOH! Moment</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vy9kx-POwLe34oJ9uPX4aw.jpeg" /></figure><p>Did you ever debug a piece of code, and said to yourself: “<strong><em>this shouldn’t happen!”</em></strong>? <br>Well, me too, so when it happened to me and I learned why it happened, I thought it’s worth sharing.</p><p>In my project I have two classes, let&#39;s for simplicity call them Bouncer &amp; EntranceChecker, Now I know that Bouncer is basically EntranceChecker, but bare with me, it’s not really important to the story.</p><p>Bouncer holds EntranceChecker and every time a Person tried to enter, <br>Bouncer calls EntranceChecker’s isInTheGuestList method, and if the person in the list Bouncer return not to block, else it blocks.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b34a8927aac2240a233a3ed28054fa2/href">https://medium.com/media/8b34a8927aac2240a233a3ed28054fa2/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4ecc7a29bb672e2d78d99be10617532/href">https://medium.com/media/e4ecc7a29bb672e2d78d99be10617532/href</a></iframe><p>Like any good programmer I wrote Bouncer a unit test to check it works.<br>I used Mockito to Mock EntranceChecker.<br>When it receives a Person that is in the list (“mockInListPerson”) it should return true, and otherwise false (“mockNotInListPerson”).</p><p>Looks simple enough, right? should pass both tests, right? Want to guess what happened when I tried to run it?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/271bf728da37a6b96c36bf2025ba605b/href">https://medium.com/media/271bf728da37a6b96c36bf2025ba605b/href</a></iframe><p>Got this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*LLTKyqutqZ_xKWm7oMklVA.png" /></figure><p>When I tried to debug it, I saw that it happens when the init() code runs <br>and it tried to execute the line below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qFynqKKdf_73uJTthLfNRg.png" /></figure><p>It <strong>enters(!)</strong> the EntranceChecker’s isInTheGuestList method, and throws NPE on the line below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MI9B7m4hZNYMOjIkTWBrrA.png" /></figure><p>It turned out The NPE was thrown because names wasn’t initialized. <br>Which is true, I didn’t initialize it, it’s shouldn’t have been running in the first place! it’s mocked! that’s the all idea of mock! <br>Haa I have a bug in Mockito!!! it’s broken! I broke Mockito!</p><p>After way more hours than I am proud to say, I calmed down and started to think rationally again, and figured it out.<br>EntranceChecker, was simply missing a small little <strong><em>public </em></strong>in front of isInTheGuestList.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c09dd237ccef388822fb33bd29c229b1/href">https://medium.com/media/c09dd237ccef388822fb33bd29c229b1/href</a></iframe><p>The moment I did it, Mockito came back to life and managed to mock my method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/1*W0ql7eHFB3fkya5Xm1AN4A.png" /></figure><h4><strong>Why Was This The Problem?</strong></h4><p>I don’t want to go into too many details on how mockito works, especially when Reinhard Seiler explains this so good in his blog: <a href="http://blog.rseiler.at/2014/06/explanation-how-proxy-based-mock.html">explanation how proxy based mock work</a>, I strongly recommend you read it if you are writing tests and want to know how the tools you use work.</p><p>I’ll just say that when you ask Mockito to mock a class for you, it basically wraps it in a wrapper class and answers what you ask of him.</p><p>The problem is, that it cannot access private method and<strong> package private methods </strong>fall under that same restriction. <br>Once I turned the method from <strong>package private methods </strong>to<strong> public, </strong>Mockito could “see” it again and mock it. when it failed to see it, the line of code simply tried to call the real method and crashed.</p><p>When I went back to Intellij and looked at the metod, I remembered why I did it in the first place. Intellij suggested it to me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*HlRgG8xrx3oaPlNnYCxASg.png" /></figure><p>Doing that, didn’t break any thing and so I assumed I was OK.<br>But Mockito is not part of that package, and so when I asked Mockito to mock the class it was blind to that method and failed.</p><p>It’s a weird behavior on the part of Intellij not to be able to detect this.<br>When I tried to pass Mockito a private method, it knew to prevent me from doing it:</p><pre>isTheGuestList(Person) has private access in EntranceChecker</pre><h4>Bottom Line:</h4><p>I find Intellij suggest most helpful and try to accept them whenever I can, but this time it was the thing that gave me an headache, for more hours than I feel comfortable admitting.</p><p>I can offer excuses why it took me so long to find this simple thing:</p><ul><li>I just upgraded the Intellij version and I was worried it might have broken the mockito annotation or something. so I reverted it back, just to find out it wasn’t the issue.</li><li>I tried to rebuild the all project — more then once — and run other tests with similar code.</li><li>It was very late, and I was tired.</li></ul><p>Bottom line, after I found it, and felt really stupid that I didn’t think about it in the first place.<br>Because I am a strong believer in the practice of sharing one stupidity with others, I decided to make it as public as possible.<br>In the hope that, you found my stupidity funny enough that you share it with your friend and colleagues.<br>And maybe one day you or one of your colleagues will face similar issue, have a vague memory that you already seen this problem, fix it in seconds and go on in your merry way.</p><p>Thanks for reading, <br>Please share!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a0ffac681e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[You Are Logging All Wrong]]></title>
            <link>https://medium.com/hackernoon/you-are-logging-all-wrong-341d4246a6dd?source=rss-14bbf574ca6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/341d4246a6dd</guid>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[best-practices]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[error-handling]]></category>
            <dc:creator><![CDATA[Ohad Edelstein]]></dc:creator>
            <pubDate>Wed, 03 Oct 2018 11:18:14 GMT</pubDate>
            <atom:updated>2019-04-26T06:54:01.516Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tFiA2gs8W89XfFIC" /></figure><p>This post is based on an email one of the Principal Engineers at my company sent to the R&amp;D mailing list a few months ago.</p><p>The scenario he described in the email is so obvious, and yet I can honestly say — that when I looked back at commits I did — I found that I did it myself from time to time. I liked it so much that I had so share it.</p><h4>Two things to note</h4><ol><li>It talks about <a href="https://hackernoon.com/tagged/logging">logging</a> in <a href="https://hackernoon.com/tagged/java">Java</a> but you will see that the concept is easily imported to other languages.</li><li>We wrapped log4j with a LogWrapper object we wrote, that among other things checks to see what is the current log level in the system before it continues to log the messages it got. So, when you see <em>logger.debug</em> — it is of type LogWrapper, and when you see <em>m_logger</em> it’s the wrapped logger.</li></ol><p>This is an example of some code that logs messages the <strong>wrong</strong> way:</p><pre>List&lt;Stuff&gt; veryLongList = …</pre><pre>…</pre><pre>// explicit invoke object&#39;s toString()<br>logger.debug(“my long list is “ + veryLongList.toString());</pre><pre>logger.debug(“my extra long list is %s”, veryLongList.toString());</pre><pre>// implicit invoke object&#39;s toString()<br>logger.debug(“my long list is “ + veryLongList);</pre><p>All three of these versions are inefficient and <strong>error prone</strong> and should be avoided.</p><p>The reason is the call to <em>toString(). </em>it forces the creation of a possibly expensive String object that is then passed to the LogWrapper.debug() method.</p><p>To understand the problem with this, look at how the <em>debug</em> method is implemented:</p><pre>public void debug(Object message) {<br>   if (ThreadedLogLevelManager.<em>isDebugEnabled</em>(m_logger)) {<br>      m_logger.log(<em>s_className</em>, Level.<em>DEBUG</em>, message, null);<br>   }<br>}</pre><pre>public void debug(Object message, Object arg) {<br>   if (ThreadedLogLevelManager.<em>isDebugEnabled</em>(m_logger)) {<br>      m_logger.log(<em>s_className</em>, Level.<em>DEBUG</em>, message, arg);<br>   }<br>}</pre><pre>...</pre><pre>public void debug(Object message, Object... args) {<br>   if (ThreadedLogLevelManager.<em>isDebugEnabled</em>(m_logger)) {<br>      m_logger.log(<em>s_className</em>, Level.<em>DEBUG</em>, message, args);<br>   }<br>}</pre><p>You can see that we first check and see the debug mode is enabled for this piece of code and only then goes to call the m_logger that does the logging.</p><blockquote>Usually the m_logger won’t even be called since debug logging is generally disabled in production.</blockquote><p>The m_logger itself then goes and turn the message into a string.<br>You can see below that the way it converts it into string is also an important check that we as developers sometime forget to do:</p><pre>protected final String convertToString(Object message) {<br>    return message == null ? (String)message : message.toString();<br>}</pre><p>This protects you from getting a NullPointerException.</p><h4>The Right Way</h4><p>The way to do such logs is simply:</p><pre>logger.debug(“my long list is %s”, veryLongList);</pre><p>This has 3 benefits:</p><ol><li>LogWrapper will call toString() only when logging is enabled.</li><li>If the object in question is null, it will be logged as the “null” string rather than raising an exception.</li><li>The code is cleaner.</li></ol><p>This may sound obvious but that are about 100 such cases in the code.</p><p>On a personal note, be minded in what you log. don’t log just for the sake of logging.</p><p>No better then Winston Churchill to sum it up most accurately:</p><blockquote>“This report, by its very length, defends itself against the risk of being read.”</blockquote><p>P.S. my good friend <a href="https://medium.com/u/24e471391af">elad shmidov</a>, while giving me notes on the post mentioned that:</p><blockquote>Python has something similar about this in it’s logging <a href="https://docs.python.org/2/howto/logging.html">best practice</a> under the optimization section</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JRTnL1a3oQfdas44DSZdXg.png" /></figure><p>P.S. 2 Another good friend/Editor/critic <a href="https://medium.com/u/c8259072fdfa">Guy Cepelevich</a>, shared how they are doing it in Ruby:</p><pre>// define the log method<br>def log<br>    logger.debug yield if Environment.type == DEVELOPMENT<br>end</pre><pre>// use it<br>log {&quot;this is what i want to log: &quot; + String(element_list)}</pre><p>The content of the curly brackets will be evaluated only when Environment.type == DEVELOPMENT</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=341d4246a6dd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/you-are-logging-all-wrong-341d4246a6dd">You Are Logging All Wrong</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Re2 Shattered My Bottleneck]]></title>
            <link>https://medium.com/hackernoon/how-re2-shattered-my-bottleneck-e18d77afbf6e?source=rss-14bbf574ca6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/e18d77afbf6e</guid>
            <category><![CDATA[regex]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[benchmark]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Ohad Edelstein]]></dc:creator>
            <pubDate>Tue, 13 Mar 2018 22:20:09 GMT</pubDate>
            <atom:updated>2019-04-26T06:54:02.658Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*wpSJQxhqN0RtTakw.jpg" /></figure><p>Almost over night one of the main data pipeline systems had become a major bottleneck for the company, and a solution was needed, Fast!</p><p>Luckily Re2 was there to the rescue. With very little work, I managed to reduce my system runtime by roughly 15%.</p><p>In this post I will share how to find the bottlenecks using Gprof2dot beautiful image rendering, and of course, what Re2 is and how to use it.</p><blockquote>This article addresses a pain I had in a Python framework, but because there are Re2 implementations to all major languages you should still find value reading this article.</blockquote><h3><strong>The strategy to relieve bottlenecks is pretty straightforward</strong></h3><ul><li><strong>Step one:</strong> Recognize were it hurts the most</li><li><strong>Step two:</strong> Reduce the pain</li><li><strong>Step three:</strong> Iterate back to step one until you get the desired result</li></ul><p>Sounds simple enough, right? Well it is, in theory.<br>Usually, the implementation is where it gets you.</p><h3><strong>Step one: Recognize where it hurts the most</strong></h3><p>Like anyone who ever saw a guy talking about A.A. meetings on TV, will say:<strong><em> “The first step in dealing with your problem is recognizing that it exists”</em></strong>, in our case that meant measurements and finding the bottlenecks.</p><p>The system in question is a pretty straightforward ETL system written in Python, so I decided to start by looking at the transform part first.</p><p>I spent a couple of hours extracting the main part of the code into a single independent process, and when it was done I started a <a href="http://jupyter.org/">Jupyter notebook</a>. <br> Using cProfile built-in python package and gprof2dot, I ran the following code to test it -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B36kGrXZbgox5k-K.png" /></figure><p>This is the image Gprof2dot produced:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SoJF1oYL_hkRTJHt.png" /></figure><p><strong>A short explanation if this is the first time you encountered the Gprof2dot library</strong>: The image that I got was a profile that measured our code, and determined, for each major block, how much time our code visited that block and how much time in total each block contributed to the overall running time.</p><p>The dark blue ones are the “cold” areas, the interesting parts in our investigation.<br> The path starting from the red box, and “cooling down” all the way to the leaves are the interesting paths to investigate.</p><p>Each box has the following format:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*-7niAaZEcQOUYcj-.png" /></figure><p>In the next image below, I looked at the leaf at the end of the path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/0*gPaUQKzE7xNAjzRK.png" /></figure><p>If you look at the upper box, you can see that it’s a method called doesTextFit in class GenericTextInTextSearch. The system spent 32.05% of it’s runtime in that code, but only 4.94% of it is in the actual code of the method, the other ~95% of the time was spent in code called from doesTextFit.<br> The 2,221,386 is the amount of times this code was called.<br> The more interesting part is that doesTextFit calls another method, where it spent ~26% of the time.<br> That was the method I decided to focus my attention on.</p><blockquote>It might happen in the leaf box, that the <strong><em>self time</em></strong> — in the parenthesis — and the <strong><em>total time</em></strong> — above the self time — don’t add up.<br>This happens when the execution time is too short, causing the round-off errors to be large. Don’t worry about it.</blockquote><p><strong><em>“Method ‘Search of’ _sre.SRE_Pattern object“</em></strong> is from Python built-in library — the <a href="https://hackernoon.com/tagged/regex">Regex</a> library <strong><em>re</em></strong>.<br> It turned out that the system that had a lot of moving parts, spent ~26% of the time on a single method on a single library call.</p><p>So now I set out to see if I could find a replacement for the Python built-in method.</p><p>It turns out there is. And I was completely surprised by it.</p><h3><strong>Step two: Reduce the pain</strong></h3><p>I came across these two articles:</p><ul><li><a href="http://www.benfrederickson.com/python-catastrophic-regular-expressions-and-the-gil/">Python, Catastrophic Regular Expressions</a></li><li><a href="https://swtch.com/~rsc/regexp/regexp1.html">Regular Expression Matching Can Be Simple And Fast</a></li></ul><p>They basically told me, I was living a lie — I always thought that Regex is the most efficient way to match a string. It turns out that it might have started like that, but somewhere along the way we sacrificed — in all the major programing languages — <a href="https://hackernoon.com/tagged/performance">performance</a> for features.</p><p>The salvation was going back to the fundamentals: State Machine Based Regex!</p><h3><strong>What’s wrong with the built in Regex engine?</strong></h3><p><strong><em>TL;DR </em></strong>Backtracking.</p><p>When the built-in Regex tries to match a string, it follows a greedy algorithm pattern. Trying to match the full string first, and when it fails trying to match minus one characters on the full string and the leftover char separately. Then on the full string minus two characters, and on the two leftover characters separately, and so on and so forth. If there is no match, the engine is destined to fail, it simply doesn’t know that before it iterates through all the different combinations.</p><p>This process is time and space consuming.</p><h3><strong>So what’s the alternative?</strong></h3><p>Introducing Ken Thompson. you can read more about him in his <a href="https://en.wikipedia.org/wiki/Ken_Thompson">wiki page</a>, but here is the TL;DR for the list of his achievements:</p><ul><li>Unix</li><li>B (programming language) — ancestor to C</li><li>UTF-8</li><li>Go (programming language)</li><li>State Machine Based Regex!</li></ul><p>The last bullet is best represented in this comparison graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/0*cDTk-r0UdodJxnZZ.png" /></figure><p>The left one is for Perl, but represents the major programing languages, the right one is the Thompson method. Make sure you notice that the left one is in <strong>seconds</strong> and the right one in <strong>microseconds(!)</strong>.</p><p>How did he do that? state machine.</p><p>Think of a regular expression as a state machine built from the expression you want to match.</p><p>Each char the engine encounters, propels it to make the next decision, if by the end of its journey it reaches a “final” state. Then the engine found a match.</p><p>If not, then there is no match.</p><p>The space complexity of a solution like that, is the amount of different states the regular expression has. And the time complexity is the length of the text being checked.</p><p>With that in mind, I went out to see if I could find an implementation of the Thompson engine, and I found the good people of Google with their <a href="https://github.com/google/re2">Re2 implementation</a> in C++.</p><p>To integrate Google Re2 in Python, I had to use a third party wrapper <a href="https://pypi.python.org/pypi/re2/">py-re2</a> , but for Java there is <a href="https://github.com/google/re2j">Google Re2/J project</a> or <a href="http://www.brics.dk/automaton/">Brics</a>, we in Taboola, use Brics in our Java code, you can read comparison of the two projects in this great blog: <a href="https://blog.rapid7.com/2016/03/15/brics-vs-re2j/">brics-vs-re2j</a>.</p><p>The integration in the code is that simple:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*Ju8iu2qmJ6KtuIsO.png" /></figure><p>It is important to remember that not all the functionality that exists in re, exists in re2. So what I did, and what you should do if you plan to use this solution, is to analyze all your Regex calls, and make sure that you get the expected result.</p><p>Although the graph above promised a performance boost of hundreds of percent, the performance boost I managed to get was just a ~40%-50% boost. I didn’t complain, it was not too bad for a few days work.</p><h3><strong>Step three: Iterate back to step one till you get the desired result</strong></h3><p>I never got to this part, and hopefully, if you use this engine you won’t either…</p><h3><strong>No Silver Bullet</strong></h3><p>The specific circumstances of the system i described here, is that the majority of the Regex comparisons finish without a match.<br> So it worked really well to have a new Regex engine that returns “no match” faster than the regular — backtracking — one. You will need to test it in your unique scenario to see how much performance improvement you will get.</p><p>Second thing to check is the Regex you use, if it needs Backtracking to complete it’s comparison it won’t work, and you will get Segfault.<br> But, it shouldn’t be too hard to rewrite them to a simpler syntax.</p><p>Third thing to monitor, is the state explosion problem when building deterministic automata, that can happen in some of the Re2 implementations — In Java implementation for example: Brics library for example but not in Google Re2. The cost is in performance.</p><p>The last thing you need to check is the return values, there might be a case that the expected outcome has changed from what you had before. For example to get a null were before you got a False.</p><p>The bottom line is that you will need to do the same evaluation process that you would have done before upgrading to a new version of a library. Things might change, so you need to be carefull.</p><p>Having said that, even if it is only a few percent improvement, the fact that it is so easy to integrate makes it a “money on the floor” scenario.</p><p><strong>Good luck</strong>, and feel free to contact me for help and guidance.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e18d77afbf6e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/how-re2-shattered-my-bottleneck-e18d77afbf6e">How Re2 Shattered My Bottleneck</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When The Heartbeat Stopped]]></title>
            <link>https://medium.com/@ohadedelstein/when-the-heartbeat-stopped-c2d17cc36b1c?source=rss-14bbf574ca6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2d17cc36b1c</guid>
            <category><![CDATA[miscarriage]]></category>
            <category><![CDATA[pregnancy]]></category>
            <category><![CDATA[death]]></category>
            <dc:creator><![CDATA[Ohad Edelstein]]></dc:creator>
            <pubDate>Sat, 05 Aug 2017 17:37:45 GMT</pubDate>
            <atom:updated>2018-10-06T07:16:26.501Z</atom:updated>
            <content:encoded><![CDATA[<p>It happened two years ago to the day. I was in the main hall of the apartment. Alone, packing the last of the stuff before friends and family arrive to help move the boxes to our new apartment. It was a hot summer morning but the apartment was cold.</p><p>Then the front door of the house opened and my wife came through it. the hall was dark, so when she opened the door she brought in the light.</p><p>I took me just a fraction of a second, looking at her, to understand.<br>She was wearing her big sunglasses but i could tell right away, she was crying. The heartbeat of our first unborn child has stopped.</p><p>We found out my wife is pregnant about a month before, it was suppose to be our first. we were married then for eight months. and my wife stopped taking here peals just the month before and so we were surprised by how fast we got pregnant.</p><p>She was having her morning sickness for a while. Then two days before her regular checkup, shw woke up, looked at me, smiled and said: “I don’t have morning sickness any more”.</p><p>That triggered a memory from something I guess I read or heared. It wasn’t of course the only sign. She suddenly had a lot of here energy back, helping me with the packing, she had her appetite back and looked a lot less tired. But that moment in bed, was the moment that i said to myself in the back of my mind: “We lost the baby”.</p><p>I had a vague recollection that the morning sickness are the result of the women’s body trying to cope with the foreign body that has started to grown in it, with all the new blood vessels that are needed to form around the new body. and that after the first trimester that feeling usually goes away and the two bodies learn how to live with each other.</p><p>So seeing and hearing my wife at the morning of the almost nine week of pregnancy, brought all thous half facts to my mind and wouldn’t let go.</p><p>My biggest regret from does days was letting my wife know about all my fears. I am a big believer that open communications and trust are the bedrock of a good marriage.<br>But knowing what I know today, I wish that I could have given here does two more blissfully ignorant days, thinking she is still bearing a healthy living baby.</p><p>I should have talked to a friend instead, unburden my fears elsewhere.<br>But we promised our selfs that we won’t tell anyone until we pass the first trimester. <br>Instead i got to make her sad, worried and annoyed, just were you want your hormonal wife to be at our first attempt to bring a life to the world.<br>She was mad at me that she couldn’t rely on me to silence my inner demons and those terrifying “what if’s”.</p><p>We don’t know why it happened. we had two more incidents after that one. Same issue, almost to day.</p><p>We went through lots of examination and doctors and found noting wrong. But as one of the specialists we we saw said: “something is wrong, because if all was OK you would have three healthy babies by now.”</p><p>We are now at our fourth attempt, week twenty. The longest one we ever been. I want to say it’s behind us.</p><p>We went every week during the first trimester to make sure we have a pulse. I can’t say that we are without worried, we had a few days during the trimester that my wife didn’t have nausea. Usually by the afternoon she was tired and nausea and she — and me secretly — calmed down. I kept saying that everything is ok, and that the stress she is feeling is not good for the baby. I was worried too, but this time I knew better,</p><p>We coped with our fears.</p><p>A close friend of my wife asked me how am i coping this time around? I told here that i am ok, i am keeping my self in a positive mindset. its sad to say, but having been through the experience number of times reduce the level of anxiety we feel.</p><p>We know what’s the worst possible outcome could be and we know that we will be ok even if we will have to go through it again.<br>It sounds like a cliché but we got stronger because of it.</p><p>This post is a sort of me coping with what happened. <br>I felt that i need to write it down, partly for my self, partly to inform other couples like us that didn’t know how common this thing is.<br>When we got to talk about this with friends and family we kept hearing: “oh it happened to my sister/friend/daughter” or “its pretty common, the statistic is that <strong>one out of three has a miscarriage in the first trimester</strong>.</p><p>My advice to you, find a close friend. not your parents, they are anxious as it is.</p><p>No, you need a friend that cares but can also distance him self and be rational with advice and support. that will be there for you if things don’t go as plan. And tell him, don’t wait.</p><p>A pregnancy is something new to both of you and you need that good friend, preferably one that already has a child to hear you and simply be there.</p><p>Thanks for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2d17cc36b1c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>