<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Joseph Monti on Medium]]></title>
        <description><![CDATA[Stories by Joseph Monti on Medium]]></description>
        <link>https://medium.com/@joemonti?source=rss-9ba11025fb37------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*UOYmyIGygMCjFLwCWlntjQ.jpeg</url>
            <title>Stories by Joseph Monti on Medium</title>
            <link>https://medium.com/@joemonti?source=rss-9ba11025fb37------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 19:34:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@joemonti/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AI is Making Code Disposable]]></title>
            <link>https://joemonti.medium.com/ai-is-making-code-disposable-3092d5226f81?source=rss-9ba11025fb37------2</link>
            <guid isPermaLink="false">https://medium.com/p/3092d5226f81</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[vibe-coding]]></category>
            <category><![CDATA[spec-driven-development]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Joseph Monti]]></dc:creator>
            <pubDate>Mon, 22 Sep 2025 14:03:58 GMT</pubDate>
            <atom:updated>2025-09-22T14:03:58.922Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rj3NNzlndazaB4R9ypp-TA.png" /><figcaption>A lovely Gemini representation of AI making code disposable</figcaption></figure><p>There is a point in the lifecycle of a commodity where the cost of producing the thing becomes so inexpensive that for consumers it is cheaper to buy new products than it is to repair and maintain the old ones. This is the result of <strong>planned obsolescence</strong>.</p><p>Early builders are craftspersons — they are artisans and masters of their trade. They own and support the lifecycle of the products or goods they produce — from design to maintenance. They are built with passion, and quality is paramount. But this is expensive and not very profitable, so aspects of the process are automated, outsourced and optimized to drive down the costs and increase profits. As direct result, <strong>quality </strong>and <strong>lifetime </strong>of the products <strong>degrade</strong>.</p><p>There are, however, some important <strong>positive </strong>effects. Because this process typically reduces investment costs, it also lowers the barrier of entry and ultimately fosters <strong>innovation </strong>and <strong>competition </strong>in the space. No longer is producing the thing the hardest part — it is coordinating a global supply chains, designing products that consumers want to buy, and convincing them to buy them. Just think of print-on-demand services like Vista Print, Printful, and others. Anyone with an eye for design can, in a few clicks, design and launch a web store of custom-designed t-shirts, mugs, and other products. I used to order my geek-wear from <a href="https://en.wikipedia.org/wiki/ThinkGeek">Think Geek</a>, but now I can find geek-inspired clothing anywhere, from Amazon and Etsy to small geek-inspired retailers advertising across my Instagram feed.</p><p>This is <strong>planned obsolescence</strong>. It happens to the products we use, like dishwashers and toys, and this is exactly what is happening to software as a result of AI being integrated into the development process.</p><p>Like the builders of consumer goods in early stages of production, software builders are craftspersons — they too are artisans and masters of their trade. They own and support the lifecycle of the software they create, from design to maintenance. Their software is built with passion and quality is paramount. But with the advances in AI leading to the availability of tools like ChatGPT, Cursor, and Claude Code, it is becoming cheaper to write new code than it is to maintain old code. <strong>Code </strong>is now <strong>disposable</strong>.</p><p>Keep in mind — we are in the very early stages of being able to treat code as disposable. At the time of writing (Sept 2025), AI can only reliably author code on a very <strong>small scale</strong> compared to the entirety of a fully functional, performant, reliable, and secure<strong> </strong>software system. But even on that small scale, AI can produce a lot of code fast — and I would say faster than Software Engineers are setup today to really take advantage of. We lack adequate software and feature specifications (see <a href="https://thenewstack.io/spec-driven-development-the-key-to-scalable-ai-agents/">Spec-driven development</a>), testing and validation methods, and code review processes at the scale AI is capable of maximizing utilization.</p><p>There are 5 general types of software work at which AI can excel —</p><ol><li>Functions and Algorithms</li><li>Refactoring</li><li>Unit tests</li><li>Configuration and infrastructure code</li><li>Readme files and documentation</li></ol><p>This may look like the entire software stack, but those are only a some of the smallest pieces of the puzzle in building software, and AI needs strong oversight from a Software Engineer in order to be successful. There are still a number of much more thought-intensive tasks that Software Engineers need to perform in order to make AI successful at those tasks listed above. These include —</p><ol><li>Architecture and software design</li><li>Software and feature requirements</li><li>Task breakdown and management</li><li>Feature and software validation and testing</li><li>Infrastructure setup and management</li><li>Deployment and monitoring</li></ol><p>You could say that a Software Engineer should <strong>trust but validate</strong> the AI generated code. Although I have always maintained that you should never trust even your <strong>own code</strong> until you run it for yourself. This even more so true for AI generated code, where the trust doesn’t come until <em>after </em>it is validated.</p><p>Remember the LLMs are essentially giving you the most statistically likely code to the prompt and context given. With a human developer, you can give them the benefit of trust that they are responsible and acting with good intentions. With AI, it is acting on statistics. And as we know from Mark Twain — “There are three kinds of lies: lies, damned lies, and statistics.”</p><p>In the age of AI-driven development, the role of a Software Engineer is evolving. For those of us who have spent years or decades agonizing over whether to use tabs or spaces, OOP or FP, TTD or not TTD — those decisions are becoming less critical when code is disposable. If you change your mind, you can ask the AI to change the code. Instead, worry about different things like whether you want to architect your software as Monolith or as Microservices or how you craft your <a href="https://agents.md/">AGENTS.md</a> file.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3092d5226f81" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Value of Abstracting Behavior as Data]]></title>
            <link>https://joemonti.medium.com/the-value-of-abstracting-behavior-as-data-0d5e8aa2ce1f?source=rss-9ba11025fb37------2</link>
            <guid isPermaLink="false">https://medium.com/p/0d5e8aa2ce1f</guid>
            <category><![CDATA[configuration]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[dsl]]></category>
            <dc:creator><![CDATA[Joseph Monti]]></dc:creator>
            <pubDate>Thu, 03 Oct 2024 13:43:17 GMT</pubDate>
            <atom:updated>2024-10-03T13:55:31.673Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1IElsMUyXxzbh4Dj3m5Lew.png" /></figure><p>Warren Buffett famously said, “If you don’t find a way to make money while you sleep, you will work until you die.” This idea is equally true in software, but I’ll give it a slight twist —If you don’t make it possible to create new value from old code, your company will fold.</p><p>The most valuable asset of a Software Engineer is their time, and that time is finite. Stealing time at the beginning or end of typical working hours is just a small percentage increase, and not sustainable. And hiring new Engineers brings diminishing returns. In order to achieve a meaningful increase in value, you need to find value multipliers in your code that delivers new value without delivering new code. It’s all about leverage, in a similar way to how Warren Buffett and investors grow value. You must “borrow” code you wrote in the past to grow new value.</p><p>Code is only the tip of the iceberg in the total time it takes to deliver code. Delivering code is a multi-step, lengthy process that includes planning, design, review, testing, deployment, monitoring, and more — a.k.a. the <a href="https://aws.amazon.com/what-is/sdlc/">SDLC</a> (Software Development Lifecycle). Many of these steps only exist because code is complicated and difficult to predict. This uncertainty has the chance to annoy a few users. Or it has the chance to disrupt millions of people, costing millions of dollars as we saw with the CrowdStrike/Windows outage caused by a <a href="https://www.washingtonpost.com/business/2024/07/24/crowdstrike-microsoft-crash-bug-report/">software bug</a>. Because of this, the software development community has developed practices to manage this uncertainty and risk to deliver high-quality and reliable software.</p><p>The more code you have to write for new features, the slower you will be able to deliver, adapt, and bring value to your business. If your code can deliver additional value, such as supporting more customers, new use cases, or new features, without having to deliver new code, then it will save you and your company its most precious resource — time. And that time can be spent delivering more value and more growth elsewhere.</p><p>How do you deliver more value from code without writing more code? You write code that can adapt by defining additional functionality or behavior through data and/or configuration.</p><p>Another way to achieve additional value from code is by minimizing the breadth or depth of changes required for additional functionality. This can be a tremendous time-saver by reducing the number of functions, files, or systems that need to be updated for new changes, particularly in distributed or multi-tiered applications. However, the focus of this article is on purely data-driven functionality that requires no code changes, at least in general purpose programming languages.</p><p>Now, don’t think that data-driven functionality completely absolves you from the SDLC. Your data, or configuration, must also follow a process to ensure quality and reliability. But a data change is typically much less risky than a code change, as there is a much smaller blast radius of potential impact because the code is already in-place and presumably well-tested. But in the end, it is still a change to the system and must be treated as such. Data changes should preferably be done through version control and release processes, but can be done much more quickly than a typical code change. I will go into more detail toward the end of the article on how this process can be managed and some potential dangers to avoid.</p><p>These are two primary ways of abstracting behavior into data —</p><ol><li>Abstracting commonalities</li><li>Create <a href="https://medium.com/wearewaes/making-your-life-easier-with-domain-specific-languages-dsl-1838d351d35">DSLs</a> (Domain-specific Languages)</li></ol><p>These offer flexibility in functionality without requiring code changes and accompanying overhead. Instead, focused but flexible data definitions allow for additional use cases to be supported, without requiring code changes.</p><h4>Abstracting Commonalities</h4><p>Let’s take a scenario where we are integrating third-party login to our website, and solve it by abstracting commonalities. Keep in mind that there are existing solutions to this problem, like <a href="https://aws.amazon.com/cognito/">Amazon Cognito</a> or <a href="https://auth0.com/">Auth0</a>, but let’s assume you need to implement it yourself. For the initial implementation we will just have a few providers to integrate, say Google and Apple, but we know we will want to add more in the future like Twitter/X, LinkedIn, Pinterest, etc.</p><p>A naïve approach would be to create a service and set of APIs for each provider. You would create a Google service, with APIs to check Google login status, start a login process, and save login credentials. Then you create an Apple service following a similar pattern. Then on your frontend you would hook up a Google login button to your Google APIs, and an Apple button to your Apple APIs. You may be clever and notice some commonalities, and create some software abstractions, like creating a provider interface and having your Google and Apple services inherit those. Then you can create a common API. But we can take this even further. We can implement this whole flow, to the point where new providers require zero additional code, and in fact no code will even mention the word Google or Apple across either our backend or frontend.</p><p>Fortunately or us, we know that these providers generally utilize the identity protocol <a href="https://oauth.net/2/">OAuth2</a> for authorization, and this standard identifies a common pattern and process that we can abstract into data! We can generalize those patterns into a single OAuth service provider which puts provider-specific details into configuration (a.k.a. data). These details include Client IDs/Secrets (we be sure to store secrets in a secure way!), and authorization and token endpoints.</p><p>We will then need a generalized way to store login credentials (refresh tokens) and user metadata (user id, email, etc), but that can be stored in a standard data model in our database independent of provider (again, be sure to store user tokens in a secure way!).</p><p>Lastly, we will need a way for the frontend to know what providers are available, along with some details such as their name, icon, and anything else you might need to stylize the login buttons. You can store these details in the same provider information used for authorization, and create an API to retrieve a list of available providers.</p><p>Once we deploy this code, then apart from bugfixes and improvements, we will not have to change it again to add new providers. Granted, you sometimes run into cases where a provider might operate slightly different, even if it claims to support OAuth 2.0, or to support additional protocols, but largely new providers only require a configuration update. No code changes; no major code testing.</p><p>Third-party login can be critical to your business, as it allows more users to access your product and grow its value and revenue. A login provider can be a very individual preference or requirement of a user, so the more options you can support, the more potential customers you can reach. You don’t want the reason a user turns away from your product to be that they can’t easily, aren’t comfortable, or aren’t able to login. And providers are continually changing, so this is something that will require continual updates. With this abstraction to data, you can even deploy new providers with no Engineering involvement. Obviously provider configuration changes need to be tested, but the barrier to adding providers is much lower and they can be deployed much more reliably.</p><h4>Create DSLs</h4><p>A DSL is similar to a configuration, but is generally more flexible, although not nearly as flexible as a full-fledged general-purpose programming language.</p><p>Common examples of DSLs are SQL, HTML, CSS, and Regular Expressions. Your DSLs don’t need to be nearly as powerful or complicated, in fact you can make them as powerful or complicated as you need.</p><p>The advantage over a simple configuration is that you can construct more complex behavior out of simple and well-constrained rules and boundaries. This makes them powerful for abstracting behavior in what is essentially data, while still providing much simpler and more reliable updates than changes involving general-purpose programming languages.</p><p>The challenge of DSLs is that they can be much more complicated to implement than a simple configuration. You often times need to implement a lightweight interpreter or engine to parse and execute the DSLs.</p><p>In the past I had created a DSL to define data-driven data transformations. In my use case, I had several large-scale data sources and needed to perform transformations into many different formats including into a standardized format and then into several client-specific output formats. Rather than writing custom SQL for each transformation, I abstracted this into data definitions and field mappings. And so to integrate new inputs or produce new outputs, I had built a web-based UI to view, create, and update the definitions and mappings into a database. This saved a tremendous amount of time and complexity over managing many hand-written SQL files during the many years this was in operation. Since this was side contract work, this time was incredibly valuable and paid for itself many times over. Once I had built the system, the only time I would need to spend on the project was to configure and adjust data definitions and mappings in the UI, and run/monitor the jobs.</p><p>The DSL was primarily in the field mapping logic, where transformations could require a large variety of operations, from case statements to string functions to key-value lookups in an external data store (MongoDB). I had implemented my own small DSL as opposed to something like generating SQL for safety, reliability and ease of use. The transformations were run on Hadoop MapReduce, which used the data definitions and field mapping DSL engine to perform the transformations to generate the desired outputs.</p><p>Another example of a DSL I had created in the past was for the robotics company I worked for, Smart Robots, called SRML (Smart Robots Markup Language). The DSL was an XML definition of robotic controls, allowing for simple “programs” to be created in XML definitions. Software modules for as navigation, sensors, and other hardware components registered SRML handler functions, and a parser invoked the corresponding functionality. This was a powerful tool for students to experiment with and learn robotics.</p><h4>Things to Watch Out For</h4><p>In the overall strategy of bringing a product from idea to success, these types of abstractions play a critical role in building software that can drive business opportunity and success. But there are some things to watch out for.</p><p>These capabilities are not necessary, and are often detrimental in the early iterations or alpha/beta stages of a product. In these stages, getting a working MVP as quickly as possible to early users is most important, and shortcuts and/or tradeoffs must be made. That means getting it done quick and probably messy. The things you will learn from those first users of the product or feature are going to be more valuable than the future benefits of scalable and abstracted functionality. Early stages of a project typically don’t have a well-defined problem with well-understood constraints that is necessary for good data-based abstractions. But once you do start to prove your idea and need to grow, this will be the time to invest into abstractions that allow you to scale quickly and grow your users, clients, and use cases.</p><p>The next big thing to watch out for is exactly how you make data or configuration changes. While these changes are much less risky that code changes, you must define and follow a process to ensure the changes are made safely and correctly. There are generally two ways to apply these changes — directly applied or version-controlled changes. The biggest difference is that the former is quicker and the latter is safer. And the former is made directly on the target environment (ex: Production), while the latter is made in all release environments (ex: Development, Staging, and Production).</p><p>Directly applied changes can be done in as low-level as directly modifying the database tables or JSON/YAML configuration files, or high-level as a UI that can render, validate, and apply changes. The advantage is that changes can be made by non-Engineers, typically the people who have the most contextual knowledge. This allows the people who know what needs to be done to be the ones making the changes, cutting out people and steps, which makes changes much quicker to apply.</p><p>The biggest drawback is that it is very difficult to test, validate, and manage. That is because the changes are made directly to the production environment and the people most able to do deep debugging may not be the people making the changes.</p><p>This further disconnects the people who wrote the code to support it from the people who are making the data/configuration changes. So if (when) there are problems with the changes, it is much more difficult for the team who wrote the code to support it to be able to debug it. They may not be familiar with the changes, but more importantly they cannot easily reproduce the issues in lower environments because the changes were applied directly on production.</p><p>There are options to help this, such as having a process to back-propagate production configuration to lower environments, and this can be automated or manual, but that can be a challenge.</p><p>The alternative that I typically prefer is storing configuration in version control. This allows for more safe and controlled changes, with all the advantages of version control such as peer review and change history. It also allows you to connect with CI/CD pipelines to deploy the changes, including validation, transforming, and publishing the definitions to their targets.</p><h4>Conclusion</h4><p>In these examples, new users, clients, and use cases could be supported easily and quickly with no code changes. This allows you to focus more on the things that bring in those new users, clients, and use cases, rather than delaying and struggling to support them.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0d5e8aa2ce1f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Language Models Are Not the Foundation of AGI]]></title>
            <link>https://joemonti.medium.com/language-models-are-not-the-foundation-of-agi-522c55344681?source=rss-9ba11025fb37------2</link>
            <guid isPermaLink="false">https://medium.com/p/522c55344681</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[agi]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Joseph Monti]]></dc:creator>
            <pubDate>Fri, 07 Jun 2024 13:32:30 GMT</pubDate>
            <atom:updated>2024-06-07T13:32:30.067Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FSMAHvkXcEp7e4wa01V0yg.png" /></figure><blockquote>AGI will not be an emergent property of language — language will be an emergent property of AGI.</blockquote><p>The definition of AGI, or Artificial General Intelligence, is complex and highly debated. Short of a definition, I’ll provide some key requirements —</p><ol><li>Ability to understand the physical world — For an AI this is the virtual world or environment in which it exists as well as the physical world in which we exist.</li><li>Ability to remember and retrieve information — In an AGI this must be a continual and self-managed process, very different from how AI is currently trained and augmented through context windows.</li><li>Ability to reason — This goes beyond information storage and retrieval and allows the machine to analyze information, draw abstract relationships, and form conclusions.</li><li>Ability to plan — This applies the ability to remember and reason, as well as other cognitive abilities, to create sequences of actions or tasks. AI needs to not only be able to share or execute a plan, but it must internally be able to model or simulate the effectiveness of multiple plans and choose the best one before acting.</li></ol><p>These requirements were borrowed from the AI Scientist Yann LeCun in a recent Lex Friedman podcast [<a href="https://www.youtube.com/watch?v=5t1vTLU7s40">https://www.youtube.com/watch?v=5t1vTLU7s40</a>]. By Yann LeCun’s own words, “LLMs can do none of those or they can only do them in a very primitive way…” While I have a fraction of Yann LeCun’s knowledge of AI and LLMs, it meshes with my understanding of language models and intelligence.</p><p>Another interesting feature of human intelligence that has allowed us to adapt and advance our species is our ability to create and use tools. Of all human-like abilities has the most potential to be dangerous to humanity when wielded by a super-human AI. Look what happens when humans wield nuclear weapons. This is one of the most critical areas of AI safety. The fortunate thing is that we are the ones that must give them the tools or the ability to create or use them. And the AI will be bound by hardware we control… for now.</p><p>Back to where we started, and I want to reiterate this: <em>AGI will not be an emergent property of language — language will be an emergent property of AGI.</em></p><p>The LLM’s style of intelligence is a brute-force appearance of intelligence. It has been gathered from enormous volumes of text — nearly the entire corpus of electronically recorded and available text. From it, LLMs are able to convincingly emulate the intelligence exhibited in text on which it was trained.</p><p>To put it in fashion terms — they have read every fashion magazine, and can wear stylish outfits. But they will always be a season behind and will never set trends. (I recently watched Ugly Betty…)</p><p>Yes we can make LLMs smarter. We will bolt on special capabilities to work around their shortcomings (i.e. Wolfram Alpha plugin for ChatGPT [<a href="https://www.wolfram.com/wolfram-plugin-chatgpt/">link</a>]). We will enhance and optimize their speed, context window, and number of parameters. But it will still be a brute-forced approach. And I think is unlikely and impractical to Engineer AGI in this way.</p><p>More likely AGI will start relatively unintelligent, but exhibit a balanced set of the above four requirements of AGI. And through improvements in software and hardware, higher-level intelligence will be emerge. Eventually language will emerge as well as speech, visual, motor I/O.</p><p>This ties back to a blog post I wrote in 2008, Where to Simulate Behavior, which unfortunately (or maybe fortunately) I was unable to find a link to. In it, my point is that in order to simulate behavior of a person, for instance, you could encode all of a person’s behavior into software. Similarly, you could simulate a person’s major components such as organs, bones, fluids, etc. But rather than simulate the whole or its specialize parts, you can simulate reusable components and through clever structuring of those components create emergent behaviors all the way up to a person’s behavior, given enough compute and precision.</p><p>You could go all the way down to simulate quarks or atoms, and then construct those into molecules, and so on up to a person. Or you can start somewhere in the middle like the cells and work your way up. The point is that there are lower-level, more general-purpose, and reusable components, where the challenge lies not only in writing the code for the components but also in constructing those components to achieve higher-level behavior.</p><p>When you look at the human brain there are reusable components everywhere, particularly in the <a href="https://en.wikipedia.org/wiki/Neocortex">neocortex </a>which is the primary source of our intelligence. Not just the neurons, but in the repeated structures the neurons form (e.g. layers and <a href="https://en.wikipedia.org/wiki/Cortical_column">cortical columns</a>). In the brain there are other components besides the neocortex that play critical roles in intelligence, but the primary source is a generally uniform, repeated, and highly adaptable system.</p><p>To simulate human-style intelligence we don’t have to simulate atoms or even neurons. Instead we can simulate reusable components that give rise to memory, learning, planning, etc. The general challenge is there are many levels at which we can attempt to create intelligence, at each level there are a variety of challenges. The path through LLMs is one that I believe is too high a level. We need to step back and continue exploring lower-level paths.</p><p>AI researchers have been making great strides each decade as they explore how to simulate intelligence up and down the hierarchy (technically down then up) — starting from emulating neurons in neural networks. Then creating deeper structures of neurons in Deep Neural Networks. Experimenting with the boundaries of how wide or deep layers can be and how they can be organized. Learning how different types of activation functions stabilize the network between layers (<a href="https://en.wikipedia.org/wiki/Sigmoid_function">Sigmoid</a>, <a href="https://builtin.com/machine-learning/relu-activation-function">ReLU</a>, etc). Then developing specialization in network structures such as for visual recognition with Convolution Neural Networks (CNNs) or sequential data in Recurrent Neural Networks (RNN).</p><p>Today the best technique for language emulation is LLMs, which were born of RNNs, encoder/decoder networks, and a key ingredient called <a href="https://en.wikipedia.org/wiki/Attention_(machine_learning)">Attention</a>. The problem LLMs have, which all prior major specialized network architectures share, is that they are still very much single-purpose AI. They take a huge amount of resources and data to train, are trained at the pace of the researchers, and can perform that one task on which they are trained. Granted, that one task is getting more and more complex and capable.</p><p>LLMs, while the most advanced AI being used widely today, is most certainly not going to produce AGI. In the mean time we have powerful AI tools that are becoming more and more useful across more and more domains from creative work to programming. And the AI community is learning a lot about how to build higher-level behavior from lower-level components.</p><p>Let’s keep exploring. It is just a matter of time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=522c55344681" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Future of AI-Driven Software Development]]></title>
            <link>https://joemonti.medium.com/the-future-of-ai-driven-software-development-0dec24759a71?source=rss-9ba11025fb37------2</link>
            <guid isPermaLink="false">https://medium.com/p/0dec24759a71</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[future-technology]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Joseph Monti]]></dc:creator>
            <pubDate>Fri, 08 Mar 2024 14:50:27 GMT</pubDate>
            <atom:updated>2024-03-08T14:50:27.405Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jhsouBXjOn0ushD_v8H5Mw.png" /></figure><p>LLMs were a breakthrough in AI capability, bringing far superior language understanding and generation for not only natural language but software language. This makes it the biggest single leap forward in computing since the invention of programming languages, giving software developers the ability to describe a piece of software in nature language.</p><p>AI is far from perfect at writing software, but in this article I will present a path forward for AI to drive software development further and further. Eventually software development will be so changed that it won’t at all resemble software development as we think of it and do it today.</p><p>The important thing to consider with current LLM models is that they are purely text generation systems. And standard publicly available LLM models are general-purpose to natural language interaction. Think if your only interaction with the world outside of your self is being able to receive and send messages through a web-based chat interface or API. You probably couldn’t be very productive.</p><p>Therefore the key advances needed for LLMs to become more useful will be to build them into domain-specific applications, like for software development. Not only for specific domains, but for specific problems within that domain, like optimizing performance or interacting with Git repositories, i.e. the things a typical human Software Engineer must be able to do.</p><p>Of course, there is the possibility that AI will reach <a href="https://en.wikipedia.org/wiki/Technological_singularity">Singularity</a>, surpassing human-level intelligence at some point in our journey, which will throw a wrench in my planned stages for AI-driven software development. But this article isn’t about that, or its societal implications. My focus will be on using AI as a tool for productivity that will empower new generations of software builders.</p><h3>Stage 1: The AI “Toddler Engineer”</h3><p>In its current state, AI/LLMs are about advanced as what I would call a “Toddler Engineer.” They can “speak” in code, but they are far from being able to work independently, and can’t do most of the things a human Junior Engineer can do. You can’t send AI a link to a Jira ticket, no matter how thoroughly defined it is, and expect it to be able to commit code to a repository, make changes from MR feedback, and coordinate a release.</p><p>AI can, however, write a small function at the level of maybe a Junior Engineer. If you’re an expert prompt artisan, and get lucky, maybe you can get a decent function on your first try, but typically complex problems require iterations of improvements before you have a reasonably well-working function.</p><p>It’s even better at anticipating what you are about to write, as you are writing it in your IDE (<a href="https://github.com/features/copilot">GitHub Copilot</a>). This is probably my favorite part of LLMs today. The biggest advantage is that the AI can use your existing software as context. A lot of software development is maintaining existing patterns in your code or within a framework. One of the best use cases is for writing Unit Tests. The AI knows the function you are testing, is fairly good at interpreting what the function you’re testing does, and has a large base of training examples.</p><p>This makes the LLM feel like a super-powered auto-complete. Auto-complete in IDEs previously could only infer the current variable or function you are trying to type against the set of available variables/functions. It was very rigid and very limited. AI however has a much more robust model of what software looks like, and at its core is a text prediction engine. With the robust model of software, combined with the context of your application, it is able to predict with great accuracy much further into the future of what you are about to write.</p><p>One of the main advantage of current LLMs is that they have a wide breadth of knowledge, across many languages, frameworks, and libraries. This gives Software Engineers a wider breadth of ability by extension, allowing them to be productive in more areas. An Engineer who has never written a line of Go can use an LLM to understand and write Go code reasonably well.</p><p>For me personally, I am much more efficient writing code on my own in a language or framework I’m familiar with than I am using an LLM to assist me in a language or framework that I’m not. But where I’m not familiar, I can work better, faster, and in a wider variety of technologies through the use of LLMs. Using LLMs is far more effective than scouring the Internet for documentation, tutorials, or Stack Overflow.</p><p>Using an LLM for working in text is akin to using a calculator for numbers. If you can do it in your head you can be really quick, but if you can’t then using a calculator will be quicker than writing it out by hand.</p><h3>Stage 2: Problem-Specific Applications of AI</h3><p>Now that AI can understand and write software, we can begin to integrate it into solving specific problems within Software Engineering. This will have the effect of making AI a more well-rounded Engineer. It will require building <a href="https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/">RAG systems</a> or fine-tuned LLMs for specific use-cases, as well as combining architectures of LLMs or other AI models to build complex capabilities. This can include stages for ingesting and organizing data for a subject, applying fine-tuned expert system LLMs, validating outputs and testing hypothesis through techniques like <a href="https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback">RLHF</a>, and translating predictions to a human user.</p><p>One important area ripe for use of AI is in software optimization. Software Engineers spend a lot of time tuning their software to run faster. Whether it is load testing, inspecting traces, or analyzing query plans. AI can analyze more data and improve the feedback loop of testing to make improvements more quickly than a human can.</p><p>An interesting example of this is the <a href="https://www.modular.com/max/mojo">Mojo</a> Language feature for <a href="https://docs.modular.com/mojo/faq#how-does-mojo-autotuning-work">auto-tuning</a>, which can automatically tune your program’s parameters using AI. Hyperparameter tuning is standard in AI model building, but this extends it as a language feature for a wide variety of parameters within your application. This can optimize your application for specific hardware or real-world scenarios much more quickly than you can by hand.</p><p>Another example is a paper published by AWS — <a href="https://www.amazon.science/publications/panda-performance-debugging-for-databases-using-llm-agents">Panda: Performance debugging for databases using LLM agents</a>. This proposes an architecture and method for training a collection of AI models to diagnose and suggest query optimizations. Being AWS, this type of system could be fully integrated into their range of database services to provide recommendations automatically. Eventually, the AI will be able to automatically tune queries on-the-fly, and not only queries, but the underlying tables and indexes.</p><p>Optimization will be a powerful tool in the toolbox of an AI, as it allows the AI to validate and iterate on its own software on its own. But it is not the only tool it will need to go further in software development. Software testing is another area suitable for AI automation that will also benefit the AI writing its own software.</p><p>You can think of the process of solving specific problems in software development as the AI becoming a more well-rounded Engineer. It’s also not always going to require building new ways of doing things, like new software testing tools. It may just be automating the process of allowing the AI to use those software testing tools directly, rather than through the proxy of a human.</p><p>What might also happen, which may be even more interesting, is if there is a more general purpose solution to allow AIs to interact with the software development process though execution, optimization, testing, and deployment.</p><h3>Stage 3: The AI “Junior Engineer”</h3><p>It’s also not until AI reaches the Junior Engineer level that it can really be put to work on more interesting and complex problems, where it will surely surpass the ability of a Junior Engineer, if not Senior+, likely faster than we expect. If not in its ability to understand and build complex systems, then in its sheer ability to produce volumes of code and iterate on it.</p><p>This is AIs true strength — its ability to brute-force a problem with its access to large amounts of compute, memory, and data, well beyond that of a human. It doesn’t have to be elegant, or simple, it just has to produce results and not make mistakes.</p><p>The key distinguisher between “Toddler” and “Junior” really is independence. It not only needs to be able to write code, but execute it, verify it, deploy it. And more importantly, we need to trust it. Trust is critical for LLMs to earn —which it just doesn’t have now. It <a href="https://towardsdatascience.com/llm-hallucinations-ec831dcd7786">hallucinates</a> all to easily, spewing complete lies or inaccuracies with utmost confidence. It has no ability today to (a) validate the code it writes is correct or (b) ask for help when it is not 100% confident. However, with the ability to access and use the full suite of tools that a Software Engineer uses, it will be able to earn back trust and build more reliable code, without having to have a real Software Engineer do that for them.</p><h3>Stage 4: The Death of Frameworks</h3><p>One of the most well-known and widely adopted principles of Software Development is <a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself">DRY (Don’t Repeat Yourself)</a>. And frameworks exist almost solely so that you DRY. If you had to write code in every project to translate operations on your data objects to queries on your database, to request or receive REST API requests, or to interact with the browser DOM, you would go insane — and you wouldn’t get much actual business logic written. That is why Engineers love and write frameworks — its one of our favorite topics, and one of our favorite thing to argue about.</p><p>Frameworks abstract out the repeated, low-level, and sometimes complex steps needed to build applications so Engineers can focus on the high-level behavior and functionality of the application, i.e. the business logic. And ultimately, allow everyone to get more work done more quickly and reliably.</p><p>The underlying foundation of frameworks, the DRY principle, is not just for speed, but reliability and maintainability as well. When you repeat yourself, often times by copying code, those copies drift and become inconsistent over time. Maybe you copy a regular expression to validate an email address (<a href="https://blog.bounceless.io/the-art-of-email-validation-mastering-regular-expressions/">how</a>, and <a href="https://michaellong.medium.com/please-do-not-use-regex-to-validate-email-addresses-e90f14898c18">why not to</a>) from one part of your code to another. Then you find a case where your signup page won’t allow someone to sign up with a + in the email address, so you fix it in that one file and now the user can signup. Then three months later that user tries to change their email address and gets the same error because the user profile page used a different copy of the email regex. Now you need to make and deploy another fix, and hope nobody else at your company copied that regex somewhere else. This is precisely why you use a library or framework, in this case one that includes a well-tested and reliable email validation function.</p><p>Frameworks are not all positive, however. When you build abstractions, you add layers of code between the code you write and the browser, operating system, and hardware. Each layer adds compute time to execute, slowing down your application. And those layers of the framework are built to be general purpose, where 90% of the time they are more than performant. But the remainder 10% of the time you suffer noticeable performance loss due your misuse or misunderstanding of the framework. Sometimes it’s just by the sheer amount of framework code necessary to execute your application, so you accept it or write lower-level versions of your code.</p><p>A common example where framework complexity can lead to performance issues is in an ORM framework for translating data object operations to SQL queries. Sometimes the ORM generated query is complex or poorly optimized, so you may be forced to skip the ORM layer and directly write SQL in order to achieve the best performance. It’s more difficult to write and maintain, but will sometimes be necessary.</p><p>Another is in frontend frameworks like React, which is notorious for performance problems due to unnecessary re-rendering of components or creating extremely large DOM structures. These require you to have a strong understanding of the framework to use it correctly, especially when approach more complex or large projects.</p><p>Frameworks are also notoriously difficult to keep updated over time. Minor updates can be trivial, but major updates can be as difficult as integrating an entirely new framework. This leaves a lot of software projects stuck on a particularly old version, if they are lucky to survive long enough, leaving possible security flaws and untapped potential for performance improvements from the newer versions. This requires sometimes completely rewriting a piece of software just to get off a particularly old version, sometimes to an entirely new framework or language.</p><p>Beyond framework and library versions, developer preferences also change over time based on what is the most “modern” approach to a problem. This is reflected in the libraries and frameworks developers adopt, or switch to.</p><p>Redux. Hooks. Higher order components. Component libraries. If you work frequently in React, these thing will bring either tears of joy or anger.</p><p>Monolith. Serverless. Spring Framework. NestJS. Django. If you work frequently in backend technologies, <em>these </em>things will bring either tears of joy or anger.</p><p>Now that AI can so easily write and optimize code, what if we threw away all of our frameworks? We humans don’t want to repeat ourselves, but that is because we are lazy and slow. AI on the other hand is not; it is neither lazy nor slow. As long as the AI has enough compute, it will not tire. As long as its algorithms are good and has enough compute, it will be fast. Let the AI repeat itself — as long as it can optimize its code and detect, merge common patterns that yield the best results, and maintain that code over time. This will allow the AI to converge on the optimal framework, where it becomes beneficial. Or not. Maybe it’s just a mass of code. It shouldn’t really matter to us as long as it works and continues to work.</p><p>AI may even collaborate with other AI on optimal libraries, frameworks, or ways of organizing code. Maybe it will even have arguments about whose is better.</p><h3>Stage 5: The Death of Languages</h3><p>You know your favorite language? I know mine (it’s Python if you’re wondering). Well, forget it. It’s irrelevant. It is irrelevant because AI will be able to reliably translate any language to any other language.</p><p>While AI can translate languages today, it can’t reliably — primarily because it can’t reliably write software to begin with. But when it <em>can</em>, and it will, then language-to-language translation will be trivial. And it won’t matter what language a program is sourced in. AI will be able to translate it to any other more efficient language for execution if it needs to, like C, Go, or Rust. Eventually the AI will be able to compile and optimize any language directly to machine code, skipping a C/Go/Rust translation, but I expect language-to-language translation to be a more likely first step.</p><p>What this really means, however, is actually the opposite of the death of languages — it means the re-birth of languages. No longer must you adhere to strict company policies (cough, TypeScript) or whatever the hippest language is of the day (cough, Go). Collaborate with AI to define your program in whatever language you like, then have AI translate it to the final execution language. Write it in Python, Lisp, Clojure, heck even Perl for all I care (Perl would have been blasphemy to me a few years ago). Although the reality is you’ll probably write in some kind of pseudo-code from which the AI will infer and translate to an actual programming language.</p><p>Ultimately, there is still a benefit to there being code that is written, done in collaboration between human and machine, in a software repository like it is today, and iterated on in much the same way. The process of writing software can be with minimal human interaction — just defining requirements through natural-language prompts. Or it can be as much human interaction as the human writing some code for the higher level parts of the program, then setting the AI off to figure out the lower-level parts like calling APIs, validating data, or interacting with databases. Either way, the depth that a Software Engineer must go into the software will become less and less.</p><p>Having code written, partially or primarily by AI, most importantly allows humans (or other AIs) to continue to review and approve code so that 100% of the code control is not handed over to AIs. I expect this will be a big point of contention in the software world, especially now that we don’t get to argue about languages and frameworks.</p><h3>Stage 6: The Death of Software</h3><p>At this final stage it gets a little difficult to predict, and is likely very far in the future, but eventually the idea of software will be irrelevant. AI will likely operate all computing platforms, likely in order to solely run AI. What we think of as software will just be models or some abstract data representation of the behavior of an application.</p><p>The transition from the death of languages to the death of software will be a gradual one. The key driver will be AI optimizing the process of translating a user’s definition of what the software should do to the order of bits that make the hardware perform that functionality. This may be a model that is essentially a mathematical and/or numerical representation of a process that behaves as intended. The model will then run on some kind of engine or runtime, likely designed by the AI.</p><p>The process of writing software will be a distant memory. Not only will humans not need to write software, but they won’t be able to comprehend all but the most trivial or highest-level parts of an application. The artisanal purists of software development will poeticize about writing software, people in the future will probably still learn about it, much like we still learn about Latin. But it will be purely academic, or for fun.</p><h3>Conclusion</h3><p>In this futuristic, maybe optimistic, view of the path of AI-driven software development, the role of the Software Engineer gets iteratively optimized to its essence — I think its most interesting essence — designing things that do cool things. I don’t write software just to write software, I write software to do cool things, or do things that make my company money. Don’t get me wrong, it’s interesting, challenging, and I enjoy it. But in the end it is a means to some outcome.</p><p>The fact that I have to painstakingly articulate the precise process a computer must perform is preposterous. I have to worry about memory allocation, variable names, code quality, data quality, side effects, and errors, just to name a few. I say, let an AI take care of those parts. Let me think about cool things technology can do to make our lives more fun, interesting, efficient, and safe. Because <em>that </em>is what technology and software really does.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0dec24759a71" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ChatGPT and the AI Revolution]]></title>
            <link>https://joemonti.medium.com/chatgpt-and-the-ai-revolution-4a5103682f03?source=rss-9ba11025fb37------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a5103682f03</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Joseph Monti]]></dc:creator>
            <pubDate>Tue, 07 Feb 2023 13:31:16 GMT</pubDate>
            <atom:updated>2023-02-07T13:31:16.632Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l6gIkkMS-r9enTJQSLutag.png" /><figcaption>Dall-E 2 image generated by “A confused robot sitting at a desk with a pen in their hand, staring at a blank piece of paper with a question mark”</figcaption></figure><p>I know I’m late to the ChatGPT bandwagon. I wanted to let the buzz die down and really try and understand what it is and how it works.</p><p>ChatGPT, the chatbot released by OpenAI, is an amazing example of making AI accessible and tangible to the masses. It’s powerful, useful, and inspiring. And unlike a lot of AI in the past that has been overshadowed by its dangerous consequences such as facial recognition in mass surveillance, ChatGPT and the technology on which it is built has more potential for positive impact than any AI before it.</p><p>Let me be clear, ChatGPT is not the savior of AI. Technologically, ChatGPT is not a major advancement in AI. No new AI algorithms were invented for it. It did not learn on its own, nor is it capable of creating new AI on its own. It is certainly not the type of <a href="https://www.ibm.com/topics/strong-ai">Strong-AI</a> (ChatGPT is the perfect example of why unfortunately the Turing Test is not a reliable gauge of human-level intelligence) capable of overthrowing humanity.</p><p>It uses AI/ML technologies that have been around for years. It relies on a large amount of painstakingly crafted human-labeled training data (not including the massive amount of general text its underlying language models are trained on). And it can do just one thing — generate snippets of text in a conversational manor. It does it really well though, and is the first chatbot that really feels like you’re talking to a human. <a href="https://towardsdatascience.com/how-chatgpt-works-the-models-behind-the-bot-1ce5fca96286">This article</a> does a great job diving deep into the underlying technology of ChatGPT.</p><p>What ChatGPT did best is bring together three critical ingredients making a powerful showcase of what can be done with AI today and how it can integrate with and empower users of its service in a positive way -</p><ol><li>A number of cutting edge AI/ML techniques, including advanced language models and learning models</li><li>Extremely talented Scientists and Engineers</li><li>A product that is accessible to anyone</li></ol><p>With the whirlwind of buzz around ChatGPT came overwhelming speculation on how bad it could be for society, which I think assumes ChatGPT is more capable than it is and the arguments will largely be irrelevant in the long-term.</p><p>One interesting impact is in academia. But I think the majority of challenges can be managed through guidelines and controls on its use, much like how Wikipedia is treated (really, avoided) and how plagiarism is checked for and handled. There are already ways to <a href="https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text/">detect AI-produced text</a>. And while plagiarism in academia won’t ever be completely avoided and not require effort to combat, I think AI will ultimately help us more in this area than it will hurt.</p><p>The bigger issue is that it is difficult to validate the truth of ChatGPT text. OpenAI has specific mechanisms in it to attempt to produce accurate responses, but it is <a href="https://analyticsindiamag.com/freaky-chatgpt-fails-that-caught-our-eyes/">far from 100% accurate</a>. OpenAI is very <a href="https://openai.com/blog/chatgpt/">open about its limitations</a> (see Limitations section). I think the major issue is that its outputs are <em>modeled</em> to be truthful — i.e. based on statistics. What it really needs is to be concretely and provably true, which is not possible by statistically-based validation on its own.</p><p>Humans have the same challenge when sharing information with each other. Granted, humans can be intentionally subversive with each other, but even well intentioned information exchange can go awry. Thankfully, over our history we have developed and refined a generally reliable method — and that is through citations or references. Even Wikipedia, which on its own is an un-trusted source of information in academia, provides a wealth of useful <a href="https://en.wikipedia.org/wiki/Wikipedia:Citing_sources">references</a> to the sources of the information contained within most articles. Citations build trust and are a mark of good writing, lending not only credibility but providing readers with a way to dive deeper on a particular topic.</p><p>Without being provably accurate, such as through citations, ChatGPT can be misleading use without having some knowledge of the topic for which it is being used. That doesn’t mean it is not valuable, far from it, but it means it is a tool. And with any tool requires having a good understanding its use and limitations.</p><p>The provable accuracy of AI-authored content is also important in media, and one where citations might not be the solution. How many people read a headline and jump to conclusions without even reading the article, let alone attempt to pursue references to assess it for accuracy? At least ChatGPT makes an effort to be truthful, which cannot be said for many in mainstream and social media. Ultimately I don’t think ChatGPT is going to cause widespread media disinformation. Although its underlying technology could be used to combat it (similar to OpenAI’s classifier of AI-written text referenced above), whether it came from AI or humans.</p><h3>The Dawn of the AI Revolution</h3><p>The most inspiring aspect of ChatGPT, including Dall-E, is not the services themselves but the power of AI in augmenting human capability and creativity in a very accessible way through a natural language interface.</p><p>Even if you’re not an artist, you can create custom graphics with these tools. If you’re not a writer, you can generate copy tailored for your product or website. If you’re not a developer, you can generate working code to solve your problem at hand. They’re not the highest quality, but are better than general non-experts can produce. All you need to do is describe what you want in human natural language. There is certainly an art to composing good prompts to get the best outcomes, much like we had to learn how to search Google effectively.</p><p>What this means is you can be more capable and productive with these tools. Much like <a href="https://www.youtube.com/watch?v=0iTq0FLDII4">Ray Kurzweil’s idea of extending our brain’s capacity with the cloud</a>. While we don’t yet have a neural link to the cloud as Ray Kurzweil imagines, with these new AIs anyone can use them and take advantage of them. We can interact with them in our own natural language, not the language of technology. You don’t need to learn complex systems, be an expert in the topic you’re using them for, or even understand the underlying technology on which they are built.</p><p>Those who <em>do </em>understand the underlying technology and are able to adapt it to their domains have a huge opportunity. ChatGPT is tailored to conversational text generation. Dall-E is tailored to descriptive image generation. But there are so many more areas these technologies (<a href="https://en.wikipedia.org/wiki/GPT-3">GPTs</a>, <a href="https://en.wikipedia.org/wiki/Reinforcement_learning">Reinforcement Learning</a>, <a href="https://en.wikipedia.org/wiki/Stable_Diffusion">Stable Diffusion</a>, etc) can impact.</p><p>To start with, OpenAI has APIs to their services that can be plugged in to applications to augment them with AI capabilities. A great example of this is <a href="https://docs.github.com/en/copilot/overview-of-github-copilot/about-github-copilot">Github Copilot</a>. It uses the code generation capabilities behind ChatGPT (<a href="https://openai.com/blog/openai-codex/">OpenAI Codex</a>) but in an environment closer to where your actual development happens (i.e. in your IDE).</p><p>While current AI is nowhere near able to write code independently, I can see them being adopted on Software Engineering teams. Maybe entry-level Engineers will be encouraged or required to use AI-assisted development. Maybe the Engineer’s role will become more formalized into effectively defining the unit tests, while the AI writes the code to pass those tests.</p><p>There are also open-source <a href="https://www.altamira.ai/blog/open-source-gpt-alternative-solutions/">GPT alternatives</a> to OpenAIs GPT model(s) capable of powering natural language applications. These can be more well integrated with other AI technology and/or traditional technology to build new applications. And they can be further customized to fit their specific domain. However, this is a more resource-intensive route.</p><p>Maybe a gaming company will build an AI to generate immersive environments personalized to the individual players. Every time you play can be a unique experience tailored your own preferences or play style. And that wouldn’t mean game designers would be out of jobs. They would be critical to the training, validation, and fine-tuning of the AI models.</p><p>Ultimately we are a long way from AI that can “think” on its own. AI systems are still very limited and very single-purposed. And that’s a good thing. It means there is opportunity. It means humans are still needed. It means we have a long time before AI will be our demise.</p><p>Most importantly, the AI today is most powerful when it can work with us, in our own language, and empower us.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a5103682f03" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[It’s Time We Treat Our Data Like an API]]></title>
            <link>https://joemonti.medium.com/its-time-we-treat-our-data-like-an-api-2a5723b3830b?source=rss-9ba11025fb37------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a5723b3830b</guid>
            <category><![CDATA[api]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[api-development]]></category>
            <category><![CDATA[data-lake]]></category>
            <dc:creator><![CDATA[Joseph Monti]]></dc:creator>
            <pubDate>Wed, 30 Nov 2022 06:20:18 GMT</pubDate>
            <atom:updated>2022-11-30T06:20:18.831Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R-dKPUck7Nbsz00yxXdoLA.png" /></figure><p>Modern API conventions and practices ensure integrating APIs can be done quickly, easily, and with confidence. Many of these conventions and practices can be applied to data management practices within a data team or organization to gain similar benefits. In this article I will discuss the conventions and practices adopted by API Engineers and how they can be utilized by Data Engineers to improve usage, adoption, and reliability of data across your organization.</p><p>When I am talking about APIs I am generally referring to Web Service APIs ranging in both technologies, such as REST APIs or private gRPC APIs, and use cases, such as Web Frontends or internal microservices. While these technologies and use cases have their differences, the API conventions and best practices they follow are the same. You will see that these can be equally applied to data, whether it is CSVs or Parquets in a Datalake or tables in an database, to gain similar benefits.</p><p>Conventions and best practices for building APIs have a long and evolved history. Early APIs, or bad APIs, had/have little to no documentation, exhibit unexpected behavior, and can break compatibility without notice. Modern practices, or good APIs, however address these challenges to make integrating APIs easier, faster, and more error-resistant.</p><p>Surrounding and supporting these API conventions and practices are a number of tools (<a href="https://www.postman.com/">Postman</a>, <a href="https://swagger.io/">Swagger</a>), open specifications (<a href="https://www.openapis.org/">OpenAPI</a>), and services (<a href="https://aws.amazon.com/api-gateway/">AWS API Gateway</a> and its <a href="https://aws.amazon.com/blogs/devops/deploy-and-manage-openapi-swagger-restful-apis-with-the-aws-cloud-development-kit/">OpenAPI integration</a>). These are critical in maintaining accurate, consistent, and functional APIs. They not only save time and effort, but provide a functional application of the API definition/documentation to the actual API.</p><p>In the world of data, when all our data was in one database or data warehouse, we had protocols like JDBC/ODBC and SQL schema definitions to provide access to and details on the data housed within it. But with the shift to a distributed data ecosystem across a datalake and microservices, the rules we follow for how we define and change data over time are even more important to have standards and practices to follow. And it is conventions and standards from API design that will provide a sound, stable, and strong future for data.</p><h3>API Contracts</h3><p>The primary way that APIs ensure they can be used with ease and confidence is by having a well defined API specification and treating that specification as a contract. That contract is a detailed description of exactly what an API is and does, and most importantly is a promise that the API not break compatibility with it — that is, it will not change in a way that breaks your use of the API without notice as described in the contract.</p><p>The specification, or contract, defines the endpoints, arguments, data model, and behavior of the API. This has a number of benefits across every aspect of working with an API, including -</p><ul><li>Planning</li><li>Building</li><li>Overall development speed</li><li>Maintenance and Features</li></ul><h3>Planning</h3><p>The first and most important step in building an API should be designing the API specification, or contract. This allows developers to <strong>plan </strong>both building the API and integrating (aka using) the API. The API developer can plan what endpoints to create, what the API tests should be, and what functionality is required to build the API. The users of the API can plan how they integrate with the API. Most importantly, all of this can be done without requiring a functional API, as long as the API adheres to the API contract.</p><h4>How can this be applied to data?</h4><p>This is primarily done through a schema definition. In the world of Hadooop, this might be the <a href="https://blog.jetbrains.com/big-data-tools/2022/07/01/why-we-need-hive-metastore/">Hive metastore</a>. In the world of Databricks, this would be the <a href="https://docs.databricks.com/data-governance/unity-catalog/index.html">Unity Catalog</a>. In the world of AWS, this might be the <a href="https://docs.aws.amazon.com/glue/latest/dg/components-overview.html#data-catalog-intro">AWS Glue Catalog</a>.</p><p>What has not emerged, however, is a vendor-agnostic data catalog, or an open specification like <a href="https://www.openapis.org/">OpenAPI </a>does for APIs. This is a critical area of innovation needed in data data ecosystem.</p><h3>Building</h3><p>Developers can mock the API, and <strong>build </strong>against the mock API. An API mock is a static, predefined response isolated from and independent of the real API. An API can be mocked by storing static responses in the codebase or through mocking services (<a href="https://learning.postman.com/docs/designing-and-developing-your-api/mocking-data/setting-up-mock/">Postman mocking</a>). This allows developers to build against the API even if the API isn’t yet implemented, or without relying on access to the API or the real-world behavior of the API. Certain parts of an API can be difficult to access, such as API rate limits or large result sets. However, they can be mocked to build a more complete, robust implementation more easily and quickly.</p><p>Keep in mind that there are limitations to developing against a mock API. Because it is a mock, and not the real thing, there will be differences that may impact your implementation. A mock may not behave exactly as the real thing, either in how long the requests take or how certain input is handled. The real API may have to change due to an implementation or design issue that comes up during API development that requires changing the API. Or there may have been gaps in the API specification to where the mock and the real API picked different interpretations that are not compatible.</p><p>It is for these reasons that when working with mocks —</p><ol><li>You must ensure the specification is complete and accurate to you and your team’s best ability</li><li>You must not release until validated against the real API</li><li>You must fully validate (or re-validate) your implementation after it is switched to the real API— automated tests are important</li></ol><h4>How can this be applied to data?</h4><p>Just like when building software against an API that doesn’t yet exist or is in development, data systems may need to be, and certainly can be, built without having a live, or real, dataset available. This is enabled by having a well defined and thoroughly reviewed data specification.</p><p>There are two primary ways of mocking data for building data processing systems:</p><ol><li>Fabricating the data by hand — this requires strong knowledge of the data, the relationship between the fields, and an understanding of what real data looks like.</li><li>Adapting real data for development — you may have some real data, but it either might not be complete, might not be enough, or might not be possible to transport to a development (or local) environment. In that case, you can modify it by hand or programmatically to fill holes, extend it through variations, or mask sensitive data.</li></ol><p>It is also good practice to test a data process at scale by mocking larger datasets. This can be done by generating random permutations of the mocked data.</p><p>Also, just like with APIs, mocked data has limitations and needs extra care and validation.</p><h3>Overall Development Speed</h3><p>API contracts also benefit <strong>development speed</strong>, through both the planning and building steps, but also by allowing developers or teams to work in parallel between those building the API and those using the API. Related to speed is reliability in development so that fewer defects are inadvertently introduced and require time to address. Development can be more reliable if the API is correct to its specification and the edges of the API can be adequately tested.</p><h4>How can this be applied to data?</h4><p>In a data-driven application, starting with a well-defined data definition is the most important technical aspect of the design of a new feature. It is more important than the APIs. It is more important than the UI Design. It is more important than how the data is stored (there is some dependency there).</p><p>The data definition is important in a data-driven application because the data is the glue that holds the whole application together. It’s produced by Data Science or Data Engineering. It’s queried, cached, and transported by Software Engineering. And it’s displayed by UX/Design. And by defining the data in the beginning, all of those teams can operate independently and in parallel, for the most part.</p><p>And when we are talking data-driven applications, there is one thing that is important above all-else — <strong>getting the application in the hands of users quickly</strong>, so we can learn, adapt, and improve our product and its adoption.</p><p>There are also some neat tricks you can do to prioritize certain types of development so you can iterate and release features to users while working on a skeleton of the data. But I’ll have to save the details of that for another post. It’s also enabled by following the recommendations in this post.</p><h3>Maintenance and Features</h3><p>APIs are not constant — they are improved and expanded over time through <strong>maintenance</strong> and as <strong>features</strong> are added. However, APIs should never break compatibility for its users without clear communication and adequate time to adapt. This is done by making changes additive or through API versioning.</p><p>Additive changes ensures the existing endpoints and data model does not change. It can, however, be added to with new endpoints or new fields on the data. If an API response is {&quot;id&quot;: &quot;abc-123&quot;, &quot;name&quot;: &quot;Bob&quot;} and you change it to {&quot;id&quot;: &quot;abc-123&quot;, &quot;name&quot;: &quot;Bob&quot;, &quot;occupation&quot;: &quot;Builder&quot;}, it will not break compatibly with any users of the API reading id and name fields. It will, however, break compatibility if it were changed to {&quot;id&quot;: 1234, &quot;name&quot;: &quot;Bob&quot;} or {&quot;id&quot;: &quot;abc-123&quot;, &quot;firstName&quot;: &quot;Bob&quot;} .</p><p>API versioning allows functionality that is being deprecated or changes to behavior to be supported by the API in parallel, meaning both the old versions and new versions are active and maintained simultaneously. That allows users of the API to migrate their use of the API on their own schedule. The API can then set a clear deadline for removing old versions, communicated to the users of the API.</p><h4>How can this be applied to data?</h4><p>Data is also not constant — it is also being continually improved and expanded over time. This is not as important with a single or very few use cases, but as data is opened up to more users or consumers of the data, it becomes ever more important to maintain compatibility over time and have a communication strategy for changes.</p><p>Similar to APIs, there are generally two strategies for changing data -</p><ol><li>Keeping changes additive — Either adding new datasets/tables or appending columns to the end of a dataset. This allows those reading the data to use the new data on their own time, or ignore new columns. The important part is that existing columns are not modified and new columns are not inserted in the middle — they should only be inserted at the end.</li><li>Creating new versions of data and communicating changes — Also, while users of the data are migrating, it may be necessary to keep both old and new version of the data maintained in parallel. On datasets that evolve in versions, I either keep a version number in the file/S3 path or in the table name.</li></ol><h3>Conclusion</h3><p>While the title was a bit flippant (it’s about time… ), the world of API design has had a long and arduous history, and has only recently been stabilizing around standard practices, tools, and open specifications. The data ecosystem has also had a long and arduous history, bouncing from mainframes to RDBMS to distributed frameworks to datalakes. Data has also taken a shift in how organizations truly get the most out of their data, and how they build or shift to data-driven organizations.</p><p>Through all those shifts, maintaining well-defined data models, having strategies for working across larger more distributed teams, and building an ever-evolving data platform is critical to the success of data-driven organizations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a5723b3830b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Convergence of Snowflake and Databricks]]></title>
            <link>https://joemonti.medium.com/the-convergence-of-snowflake-and-databricks-20a7ecafe722?source=rss-9ba11025fb37------2</link>
            <guid isPermaLink="false">https://medium.com/p/20a7ecafe722</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Joseph Monti]]></dc:creator>
            <pubDate>Thu, 17 Feb 2022 14:09:23 GMT</pubDate>
            <atom:updated>2022-02-17T14:09:23.770Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*cnhoghMxhMuxewr-s24cMA.jpeg" /></figure><p><a href="https://www.snowflake.com/">Snowflake</a> and <a href="https://databricks.com/">Databricks</a> have both played major roles in transforming the data industry. Historically, Snowflake has been known as an industry-leading data warehouse platform. And Databricks has been known as an industry-leading data processing and data lake platform. However, in recent years, both platforms have evolved their feature-sets, essentially converging to the same solution — known as a “lakehouse” architecture.</p><p>The Databricks blog has a fantastic article — <a href="https://databricks.com/blog/2020/01/30/what-is-a-data-lakehouse.html">What is a Lakehouse?</a> — describing the origins, motivations, and components of the lakehouse architecture. To put it simply, a lakehouse provides an end-to-end solution for large-scale data needs from ingestion and processing to analysis and reporting. A lakehouse services the needs of key business roles that manage and interact with data, including Data Engineers, Data Scientists, and Data Analysts, on one platform. These roles have traditionally gravitated to different tool-sets, but the lakehouse promises to satisfy them all.</p><p>The interesting part about the convergence of Snowflake’s and Databrick’s solutions in particular is that they each started on completely opposite ends of the lakehouse spectrum. But they are now ultimately moving toward the same goal.</p><p>Snowflake started on the consumption side of the lakehouse, providing a large-scale, high-performance SQL-based query platform for reporting and analysis. And it wasn’t particularly strong or well-known for its ingestion and data processing capabilities. However, with the introduction of <a href="https://www.snowflake.com/news/snowflake-introduces-snowpipe-continuous-automated-cost-effective-data-loading/">Snowpipe</a> and <a href="https://www.snowflake.com/snowpark/">Snowpark</a>, along with the way their system decouples the storage and compute layers, Snowflake is now a powerful end-to-end data platform. Even though Snowflake avoids the term “lakehouse,” their platform embodies all of its benefits.</p><p>Databricks on the other hand started on the data ingestion, management, and processing side of the lakehouse. Databricks was initially a managed <a href="https://spark.apache.org/">Apache Spark</a> platform, founded by the team that built Spark. Spark is an incredibly powerful data processing engine for large-scale batched and streaming data processing. Around 2019 Databricks launched <a href="https://venturebeat.com/2019/04/24/databricks-launches-delta-lake-an-open-source-data-lake-reliability-project/">Deltalake</a>, a solution to the problem of updating block/cloud-based data in-place, maintaining a consistent, reliable, and fault-tolerant dataset for a range of use cases across Data Engineering and Data Science. Since this is traditionally the type of problem solved by a data warehouse or relational-style database, Databricks’ transition to lakehouse was now underway. The ingredient they were missing, however, was on-platform query execution. This could be solved by Presto or Athena, but again those are off-platform systems. Last year, however, Databricks announced availability of their SQL engine, thus closing the loop on the end-to-end lakehouse solution they are today.</p><p>This may sound like a gushing exposition on these two platforms, but I will say that neither platform is matching the other in quality and performance across all of the aspects of the lakehouse <em>yet</em>. Snowflake is much more mature, performant, and feature-rich as an SQL engine than Databricks is now. And it’s not a surprise; SQL has been Snowflake’s bread and butter for a decade. They’ve put enormous amounts of Engineering behind their platform, and thousands of customers pushing them to their limits constantly. And Databricks is much more mature, performant, and feature-rich as a data processing platform. That has been their bread and butter for over a decade. Databricks’ foundation is built on innovation in the field of data processing and that has not stopped to this day as evident by the release of their new query engine <a href="https://databricks.com/blog/2021/06/17/announcing-photon-public-preview-the-next-generation-query-engine-on-the-databricks-lakehouse-platform.html">Photon</a> in 2021.</p><p>In both of their cases, I have no doubt that they will each close the gap as they progress. They are both incredibly strong platforms, albeit in their own way. And they are both putting their efforts in the right places to make their platforms even more powerful and beneficial for their customers.</p><p>The transition of both of these platforms in recent years marks a major shift in how businesses manage and use data. But more so it is a maturing of the ecosystem that allows businesses to build quickly, adapt quickly, and make good on the promises of big data. For a long time we have been able to cultivate huge datasets in the cloud. And on top of them we have built data transformations, streaming data pipelines, and analytic reporting systems — but these have until recently been on disparate platforms. Each having their own conceptual models, quirks, and platform-specific expertise required. Databricks and Snowflake are leading the way to a more unified and innovative future for large-scale data platforms, along with AWS and other cloud providers making similar advances in their own products which embrace the lakehouse architecture.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20a7ecafe722" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Programming Language Doesn’t Matter]]></title>
            <link>https://joemonti.medium.com/your-programming-language-doesnt-matter-f8c344ce49ba?source=rss-9ba11025fb37------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8c344ce49ba</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[programming-languages]]></category>
            <dc:creator><![CDATA[Joseph Monti]]></dc:creator>
            <pubDate>Tue, 21 Dec 2021 14:47:54 GMT</pubDate>
            <atom:updated>2021-12-21T14:47:54.667Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6KH4rpIcybabDXYm" /><figcaption>Photo by <a href="https://unsplash.com/@danlefeb?utm_source=medium&amp;utm_medium=referral">Dan LeFebvre</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>For a very long time I have had very strong opinions on exactly what language is the best language for a given project, and that it was one of the most important decisions for a project. There was a right time and place for C, a right time and place for Python, a right time and place for Scala, and a right time and place for Javascript.</p><p>Over time, at least in the realm of cloud-based software services, my perspective on programming languages has changed and I now put a much less importance on them. There are fewer and fewer reasons to pick one language over another. There are more and more languages that can get the job done well. And there is more opportunity to mix and match multiple languages within a single application.</p><p>The last point about mixing and matching languages is a big one, and was a pivotal point in my perception of programming languages and the motivation for this article. In this article I will discuss the importance of being or becoming a polyglot, and its relevance in the current software development landscape.</p><p><strong>Part One: The Skill of Learning and Using Multiple Languages</strong></p><p>In the journey of a Software Engineer, learning programming languages is a key aspect of your growth. When you start your journey, figuring out what programming language to learn first might seem like one of the hardest decisions to make. Programming languages are a contentious topic in general, and the specific question of what language to learn first is no exception. There are almost as many viewpoints on the topic as there are Software Engineers in the world. If you asked 10 Software Engineers the best programming language to start with you probably won’t get 10 different languages. You might get around 3–5. You will, however, most likely get 10 different reasons why those 3–5 languages are the best to start with. And the reasons will most likely be complex, nuanced, and you might not even understand them.</p><p>My viewpoint is that you should choose a language that gets you excited about programming and lets you do things through programming that get you excited. If you’re into robots or bringing physical things to life, then you might want to start with <a href="https://www.arduino.cc/en/Guide/Introduction">Arduino</a> and its programming language. If you want to teach computers how to learn, through AI and Machine Learning, then you might want to start with Python paired with an ML/DL framework like <a href="https://pytorch.org/get-started/locally/">PyTorch</a>. If you want to build web applications, then you might want to start with Javascript. There are a handful of languages I would consider good to start with, but hundreds of scenarios for which they might best be suited.</p><p>The reason why this question is so complicated is because it really does depend on what you want to do. Software Engineering isn’t about writing software, it’s about the things you do with that software. There is of course a joy in writing software; the joy of creating something out of nothing. But the ultimate goal is to do something with it — like making a robot navigate a maze, getting a computer to recognize your face, or building a beautiful website. For these reasons, the best place to start is finding what you want to do, and then find and learn the language that is most prevalent in that area. If you’re still lost, I’ll say it, just start with Python.</p><p>As you develop your skills as a Software Engineer your perception of languages will change. You will grow from learning how to program in a specific language, to mastering the fundamentals of translating thought to code, and eventually you will start to learn how to learn programming languages. That skill, the skill of learning how to learn programming languages, is a key leveling up point in your journey as a Software Engineer. It opens up so many career opportunities such as working on new types of software systems or working for new companies that rely on a greater or at least different variety of programming languages. It’s when you realize that the programming language doesn’t matter — it’s when you see the code behind the Matrix — that you can do anything you put your mind to.</p><p><strong>Part Two: The Changing Landscape of Software Engineering</strong></p><p>The Cloud has brought the greatest shift in how we build software since I have started building software. And it is so much more than just being able to host your software on nearly instantly provisionable hardware, and easily provisioning and managing server software like databases and caches. All of the underlying components of a software application are being deconstructed and turned into easily accessible services by the cloud providers.</p><p>Do you need to perform some operation in the background, and have it buffered by a queuing system? Literally the only code you need to write is a function to perform that operation. AWS, or any of the major cloud providers, will take care of operating the queuing system (ex: SQS), provisioning and executing your function (ex: Lambda Functions), and feeding it messages from the queue as they become available (ex: SQS event source on the Lambda Function).</p><p>Gone are the days of needing do-it-all application frameworks like Spring Framework, Grails, Play! Framework, or Ruby on Rails. These frameworks provide a great foundation for building applications, as both a way to get an application up and running quickly and having a reliable basis for building large-scale monolith applications. In the context of this article it is important to add that once you adopt an application framework you become locked into the language of that framework.</p><p>Because the application built on these frameworks can grow into a very large monolith, the challenges of having a large codebase have a major influence on the language you choose toward strongly typed languages like Java or Scala. Because there is no clean boundary between components of your application unless you are vigilant on your software design, you can end up with a mess of interconnected systems that can make growing the software and refactoring it nearly impossible. And the only real way to deal with that, other than avoiding monoliths, is to avoid weakly typed languages for those applications.</p><p>In today’s world of cloud services, however, the application frameworks are becoming obsolete and unnecessary. Their functionality has been replaced by services within The Cloud. Configuration management, web controller infrastructure, scheduling, caching, and more are all available as on-demand services within The Cloud. Additionally, many application framework features become obsolete in the context of a cloud-native application. Database connection management and pooling generally turn into API calls that are much simpler to manage, especially when using cloud-native databases like DynamoDB. Dependency injection also changes, as independent services no longer need to acquire service instances and instead are implemented as API calls.</p><p>Not only are the services of the application frameworks deconstructed into cloud services, but your application itself can become deconstructed into microservices or nanoservices within The Cloud. The scaling of your application goes from being either vertically or horizontally scaled as a single monolith unit, scaling the entirety of your application at once, to being n-dimensionally scalable, where n is the number of micro/nano services in your application. Each individual unit of your application is now independently scalable, and the scaling is on-demand and nearly infinite.</p><p>The shift away from application frameworks is made evident by Lighbend’s decision to <a href="https://www.lightbend.com/blog/on-the-future-of-play-framework">discontinue directly supporting the Play! Framework</a>, which is the predominant application framework for Scala. They instead are focusing on services which better compliment The Cloud and enhance the developer experience on it. Spring Framework is also evolving in The Cloud, shifting from a massive application framework to more focused libraries for building cloud-compatible or micro services with Spring Boot or Spring Cloud.</p><p>Without an application framework locking you into a single language, and with each component of your application being independently operated and executed by the cloud provider, you are now free to use almost any language you see fit for any component of your application. You can write an API handler in Javascript, a data pipeline in Python, and a message consumer in Go. The all operate independently, but when they have well defined interfaces on services that are individually resilient, then they can cooperate as a single high-performance, highly scalable, application. There is certainly a higher cognitive load when switching programming languages frequently, but in the context of a multi-team organization it is completely reasonable for one team to focus on a Javascript service and another team to focus on a Python service.</p><p>It is important to note that while some things get easier, some things also get harder, although the net result is a much more robust and feature-rich system requiring much fewer resources per feature-unit (an imagined unit of functionality). It gets easier because you end up writing less code and the code you write is specifically to implement your application. You write less boilerplate code and what I will call “connective tissue,” which as an example are the things that consume data from a queue or connect API authorization handlers for a controller. These things just become configurations on cloud resources.</p><p>There are two key reasons why it becomes harder to build applications in this context. One reason is there is much more “stuff” around your code. One of the biggest parts of this “stuff” is Infrastructure as Code (IaC), which is what provisions and configures your applications within The Cloud. Another kind of “stuff” is the “connective tissue” mentioned above. The second interesting reason why things get harder is that by simplifying Software Engineering through cloud services, reducing the effort per feature-unit, you invariably want to build more functionality. This in turn results in the challenges shifting from the code itself to the architecture and limits of your imagination. It’s often said that you can build more with less on The Cloud. But I find you rarely put less into it, and ultimately are able to move faster and do more, period.</p><p><strong>What Still Matters in a Programming Language</strong></p><p>Even though your language is not [as] important [as it used to be], there are still some areas that might make certain languages better suited for your team or project.</p><p>The most important is your efficiency in a given language. This is most likely going to be the case for the language you’re most comfortable with. If you are comfortable with a language, you’ll at least be able to get started faster. You’ll have your IDE setup, you’ll have a process for debugging your code, and you’ll be able to move a little quicker at least in the beginning. If you’re willing to take a little risk on something new, however, you might end up taking a little more time and effort getting started but gain efficiency in the long run. It might be less cumbersome, require less boilerplate, or generally have less code involved when working in some languages.</p><p>Another area which might favor a particular language is availability of libraries. Python for instance is often used when working with data, and has some of the most powerful data processing and machine learning libraries. Java, or any JVM-based language, generally has the widest variety of libraries. And newer languages like Go or Rust might be less likely to have a library you need, or a library might not be as well tested or as fully functional simply because it has not been used by as many people or for as long. In the end, it comes down to what libraries you need and what languages they support. If you are more flexible in the languages you use then you can be more flexible in using the available languages for the libraries you need.</p><p>These reasons are generally the most relevant reasons for cloud-based web applications. Outside of that context there can be much more compelling reasons to go with a particular language over another, whether its for game development, mobile or desktop development, or embedded systems. These use cases would have much more narrow set of options, like C++, Swift, Kotlin, etc.</p><p><strong>Conclusion</strong></p><p>While programming languages aren’t completely irrelevant, it is certainly becoming less important to pick one language over another. You might be more comfortable or efficient in a particular language, or you might need a certain library that is only supported by a certain language. However there are a number of factors that make programming language choice more flexible and ultimately less important -</p><ul><li>It’s a valuable skill to learn how to learn programming languages and be comfortable switching languages</li><li>Modern cloud-native applications require less application code</li><li>Modern cloud-native applications are made up of smaller, more distributed components</li><li>These smaller, more distributed components containing less code are not as locked into a particular language and may benefit from using different languages</li><li>Or it might not matter and you can continue with your day</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/0*xnKnjgHnla__2i2V" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8c344ce49ba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Tips for Writing Good One-Time Data Processing Scripts]]></title>
            <link>https://joemonti.medium.com/5-tips-for-writing-good-one-time-data-processing-scripts-7392e60977dd?source=rss-9ba11025fb37------2</link>
            <guid isPermaLink="false">https://medium.com/p/7392e60977dd</guid>
            <category><![CDATA[scripting]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Joseph Monti]]></dc:creator>
            <pubDate>Thu, 07 Oct 2021 06:00:07 GMT</pubDate>
            <atom:updated>2021-10-07T06:01:51.576Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/965/1*czSJb-tEF5fdpsnQ1SD_9A.png" /></figure><p>There are a lot of great tools and frameworks for writing and managing data processes. One of the things these tools and frameworks do is help make the process repeatable, reliable, and scalable. But when you’re trying to <em>get things done</em> and those aspects are not relevant, then a one-time data processing script may make sense.</p><p>To put a one-time data process into more context. It’s one-time when you’re not trying to build a repeatable process. And it’s a script when you don’t need a cluster of servers to handle the scale. Maybe you are queuing a batch of messages to a queue for a data backfill. Or maybe you are repairing a corrupted file. Whatever the case may be, you are generally reading some input file, iterating through the data, and performing some usually simple operations on the data. You can run the script locally or on a remote server, and then throw the script away. A good Engineer never throws anything away, so let’s say you throw the script in the junk drawer.</p><p>To start with, it might help to set some boundaries for what might work well as a one-time data processing script. First, in terms of scale of data, 1,000 to 1,000,000 records are reasonable to move around and process on a single server like this. Below 1,000 records these tips largely aren’t necessary. And above 1,000,000 records, processing the data on a single server might not make sense. Second, in terms of time, these tips are most helpful when the script takes 5 minutes to maybe 24 hours. Less than 5 minutes, and again these tips are largely unnecessary. And for a process that takes 24 hours, it is likely better to find a more scalable solution. These are just general and possibly extreme boundaries and not set in stone.</p><p>These tips are based on many years of writing many scripts. They encompass patterns you can follow to ensure you can reliably process your data, avoid some of the common pitfalls of working outside of a robust framework, and generally make it easier to use. Lastly, I nearly always write these types of scripts in Python, and all code examples in this article will be in Python. But of course you can use your own language of choice.</p><p>Let’s now move on to the 5 tips for helping to write these scripts. Along the way we will build a simple Python script to reverse the characters in an input file and write the output to a file.</p><h3>Tip 1: Use arguments to control the variables of your script</h3><p>This tip applies to any command line script, and ensures the script is easy to use and flexible to your inputs. These inputs generally come from arguments, or as files passed in as arguments. A script that hard-codes constants like what file to process or what database to connect to is difficult to validate. If you ever have to change those variables it can become difficult to know what it is running against. When this information is passed as an argument it is clear when you invoke the script and it is clear when you look back at your history to know what arguments were used.</p><p>For more robust and permanent scripts, I will usually read arguments using a module like Python’s argparse . However, for one-time data processing scripts I fall back to the good old sys.argv . It’s generally not worth the extra typing to support robust arguments and default values using a generally more complicated module or library.</p><p>So let’s start our script with some boilerplate code to open the input files.</p><pre>#!/usr/bin/env python<br>import sys</pre><pre>def process(in_file, output_file):<br>    # process file</pre><pre>def main(args):<br>    if len(args) != 3:<br>        print(f&#39;Usage {args[0]}&#39;)<br>        sys.exit(1)</pre><pre>    in_fname = args[1]<br>    out_fname = args[2]<br>    with open(in_fname) as in_file, \<br>          open(out_fname, &#39;w&#39;) as out_file:<br>        process(in_file, output_file)</pre><pre>if __name__ == &#39;__main__&#39;:<br>    main(sys.argv)</pre><h3>Tip 2: Log the progress of your script</h3><p>When you are running a script, especially one that takes more than a few minutes, it is important to know if it is making progress and how long it might take to complete. The easiest way to do this is to log its progress, and this typically starts by tracking the percentage complete. This can easily be done by counting the total input records and then tracking the line number as you iterate. For counting the total input records, I’ve experimented with different and the easiest and fastest is to literally open the file, iterate the lines, and increment a counter. Then you can log something of the format:</p><pre>{record_number} / {record_total} ({pct}%)</pre><p>The next big question is when to log. I have found that there is a balance to how much information you provide. If you log after every record you process, then on large input files the output can be overwhelming and not helpful. I find that once every 10 seconds is a good starting point, but that may decrease in frequency for longer-running scripts up to once every minute. Think about what it will look like and feel like when you’re eagerly waiting for your process to finish.</p><p>Exactly how you time when to log is dependent on how much effort you want to put in. The simplest method is to log every X number of records, and so you would use code like this:</p><pre>for line in file:<br>    # process the line<br>    if line_number % X:<br>        print(...)</pre><p>The trick with this approach is that it can be difficult to pick a number X that has a good balance of logging frequency. To more directly log on some time interval, you can track the time between logging like this:</p><pre>update_interval = 10 # time in seconds between updates<br>last_update = time.time()<br>for line in file:<br>    # process the line<br>    if (time.time() - last_update) &gt; update_interval:<br>        print(...)<br>        last_update = time.time()</pre><p>Lastly, if you want to be really helpful to your user (usually yourself), then you can not only track percentage complete and control how often you log updates, but you can estimate how long it might take to complete. Otherwise, it is left up to the user running the script to estimate this by hand.</p><p>You can do this within your script by tracking how many records are processed in a certain amount of time, then extrapolating the total remaining time based on remaining records. The general math for this is:</p><pre># records_remaining: record_total - record_number<br># time_last: time, in seconds, since last checkpoint<br># record_last: records processed since last checkpoint<br># time_remaining: estimated time, in seconds, to complete processing</pre><pre>time_remaining = records_remaining * time_last / records_last</pre><p>The one challenge with this is selecting an appropriate time period to use for estimating how fast it is processing. For anyone who has witnessed <a href="https://xkcd.com/612/">Windows file copy estimates</a>, you know how difficult this can be and frustrating it can be when it isn’t accurate. I don’t recommend the entire history of records (i.e. total time and total records processed so far). Generally 1000 records or 30 seconds is a good sample size to start with.</p><h3>Tip 3: Make it easy to continue the script when the script fails</h3><p>Errors happen and things fail. If you don’t do anything in your code to handle an error, the script will fail and you will have only a stacktrace to go on. You won’t know the data that caused the script to fail, and you won’t know what line number it left off on. Ideally you would have both of those things. Knowing the data that caused it to fail is important in order to debug and fix the script. Or you may be able to skip records that fail for that reason. And knowing what line number it left off on is important when resuming the script. That way you can create a new input file that starts where the failure left off or add a skip lines functionality as an argument. This avoids having to reprocess the same data, thus saving time, as well as in some cases it may be strictly important to not reprocess the same data again.</p><p>A good way to capture the error and log the information needed to debug and resume the script is to wrap it in a try/catch and log the pertinent information. Here is an example -</p><pre><em>try</em>:<br>    output = process_line(line)<br>    output_file.write(output)<br>    output_file.write(&#39;\n&#39;)<br><em>except</em> Exception <em>as</em> ex:<br>    print(f&#39;ERROR at line [{i+1}]: {line}&#39;)<br>    <em>raise</em> ex</pre><h3>Tip 4: Test your script in small batches to start with</h3><p>This can help understand timing and validate that the script processes the data correctly. There are three things that can help with testing:</p><ol><li>Extract a small sample, 10 to 100 records. You can do this with<br>$ head -n 100 input_file.csv &gt; input_file_sample.csv</li><li>Add extra debugging/logging, and increase logging frequency. This can be disabled for the full data set.</li><li>Disable the parts of the script that do something irreversible. If the script is computing some data then writing to a database, then disable the part that writes to a database. If it is constructing a JSON message to an API or Queue, disable the part that writes to the API or Queue so you can validate the JSON is correct.</li><li>Time how long the sample of records took and extrapolate it to the full data size, similar to how we computed the time_remaining above. This will help have an estimate on how long the whole data set will take to process.</li></ol><h3>Tip 5: Let it run without interruption</h3><p>Once you are ready to run your script, the most important thing is ensuring it can run uninterrupted. That means if running the script locally, not letting your laptop go to sleep. And if running the script remotely, not letting your SSH connection disconnect.</p><p>There is a more reliable solution than hoping your laptop won’t sleep or your SSH connection won’t disconnect. Those things are inevitable, so you might as well embrace them. That is where the mighty screen command comes in. The screen command allows you to connect to a remote host via SSH, start a screen session on the host, and start your script inside the screen session. This allows your SSH connection to disconnect or laptop to go to sleep, and because your script is in a screen session it will continue to run unattended. The best part is that you can reconnect SSH to your host and reattach your screen session bringing you back to your script as if you never left.</p><p>For a quick introduction to screen, see <a href="https://linuxize.com/post/how-to-use-linux-screen/">https://linuxize.com/post/how-to-use-linux-screen/</a></p><h3>Conclusion</h3><p>We walked through a number of features and best practices for writing one-time data processing scripts. They are not always necessary for every script, but become more relevant as you run scripts that run longer and take more iteration to get right. And to reiterate, these are not a replacement for robust data tools or frameworks. In fact, these tips are in many ways inspired by data frameworks like Hadoop but on a much smaller scale.</p><p>For a complete version of the code that was used in the walkthrough, you can find the <a href="https://gist.github.com/joemonti/ae28be46a033689322df3c688f15fb8f">code here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7392e60977dd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Controlling Data Velocity in Data Pipelines]]></title>
            <link>https://joemonti.medium.com/controlling-data-velocity-in-data-pipelines-3fed052a167b?source=rss-9ba11025fb37------2</link>
            <guid isPermaLink="false">https://medium.com/p/3fed052a167b</guid>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[real-time-analytics]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[kinesis]]></category>
            <dc:creator><![CDATA[Joseph Monti]]></dc:creator>
            <pubDate>Tue, 10 Aug 2021 04:43:44 GMT</pubDate>
            <atom:updated>2021-08-10T04:43:44.447Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gQv569yN7hmO2TJn52_J0A.jpeg" /></figure><p>When building data pipelines that operate on realtime data streams or events, it is important to understand and control data velocity. <strong>Data velocity</strong> is the rate of data through a data pipeline or process. There are elements that are important about data velocity for both incoming data and outgoing data that are critical in determining architecture, technology and the implementation of a data pipeline.</p><p>Incoming data velocity can vary significantly by application. An ad click stream application could have very high data velocity while a daily model retraining job could have a very low data velocity (although potentially high volume). It can also vary significantly by time of day or other user-driven behavior. There may also be no incoming data during off-hours but become high volume as users are interacting more with your application and generating data. In these cases, a robust architecture is important in order to scale to incoming data velocity and optimize its own processing.</p><p>When considering data velocity of outgoing data, there may be limitations on the destination that require the data to be buffered in order to slow down bursts of data. Or there may be optimal data sizes for loading into a data warehouse. The incoming data velocity is often times very different than the optimal outgoing data velocity. For these reasons it is important for the data pipeline to slow the data down to best optimize the outputs.</p><p>It is the responsibility of the data pipeline to not only perform its primary purpose, such as transforming the data and storing it to the correct output data target, but it must also be prepared for and control the data velocity in order for it to be a good shepherd of data.</p><p>The majority of the examples I’ll cover will be based on services running in AWS, particularly using <a href="https://aws.amazon.com/sqs/">SQS</a> or <a href="https://aws.amazon.com/kinesis/">Kinesis</a> with <a href="https://aws.amazon.com/lambda/">Lambda</a>. There are however analogous services using different combinations of services or cloud providers to achieve the same results.</p><h4>Managing Incoming <strong>Data</strong> Velocity</h4><p>Managing incoming data velocity can be done in a number of ways depending on where data is coming from and how you are processing it. The easiest way is to buffer incoming data in a queue such as SQS. This allows for messages to be queued when incoming velocity exceeds capacity. Kinesis can alternatively be used for even higher data velocity applications. Both of these services allow for reliable message consumption. The primary difference with Kinesis is it supports more robust controls and features for handling increased scale such as <a href="https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html#shard">sharding</a> and <a href="https://aws.amazon.com/about-aws/whats-new/2020/12/aws-lambda-launches-checkpointing-for-amazon-kinesis-and-amazon-dynamodb-streams/">checkpointing</a>.</p><p>In cases where the messages are coming from external systems that might not be practical to write directly to an SQS queue or Kinesis stream, a thin API can be created in API Gateway that can authenticate and validate messages and forward them onto the queue or stream for processing.</p><p>An additional advantage of SQS and Kinesis is these can be consumed by a Lambda function. This not only makes it easy to receive messages, but provides automatic scaling of the processing of messages. There may be cases where you need to explicitly limit concurrency, such as when connecting to a database like MySQL or PostgreSQL. Large bursts of traffic can overwhelm a database, consuming resources like connections and CPU and impacting other services using the database. To prevent this, Lambda can be configured with <a href="https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html">Reserved Concurrency</a>, which limits the concurrent invocations of your Lambda function. This can help control both resources used to execute your data process, such as reading metadata or augmenting the data, but also the outgoing data velocity of your application.</p><p>While using a queue to buffer incoming data allows incoming data velocity to be smoothed and prevent data loss without impacting upstream systems, it doesn’t help with the efficiency of the processing or allowing it to process more, faster.</p><p>To improve efficiency of processing data, one of the most effective controls is the amount of data that is processed on each invocation. In the case of SQS or Kinesis, this is controlled by the <em>batch size</em>, which is the number of reads to be read at a time, or the <em>batch window,</em> which is the amount of time between processing batches. These usually work in tandem, where a batch is processed when either it exceeds the batch size or the batch window. And they are typically configured on the Lambda’s event source. Lager batch sizes or batch windows allow the Lambda to be invoked fewer times for each incoming record, translating high velocity to lower velocity, reducing cost and improving performance. However, there are limits to how large the batches should be based on your specific application. Some considerations are:</p><ul><li>The Lambda may run out of memory or timeout —The Lambda should be monitored for invocation failures, invocation time and memory usage to identify batches are too large.</li><li>Larger batches may slow the data down too much — If the data is intended to feed a realtime interaction or feedback to a user, then large batches may result in too much of a delay in processing for your application.</li></ul><h4>Managing Outgoing Data Velocity</h4><p>Both of the controls discussed in managing incoming data velocity — queuing and batching — can assist in managing outgoing data velocity. The difference here is that the objective for outgoing velocity is to match the optimal characteristics of the data for the output system. There will usually be some constraints or optimal characteristics of the data that can be optimized for the consumer of your service by managing the outgoing data velocity of your service.</p><p>In one of the most common use cases, loading S3 data into a data warehouse such as Redshift or Snowflake, the number and size of files are key factors in performance of loading the data. Both warehouses perform best with data split into several files approximately 10 MB to 100 MB in size. In order to achieve this, you will want to control the batches processed by your service. A service that reads one input event and produces one output row to a file in S3, probably under 1 KB in size, will not be efficient to load into a data warehouse. On the other hand, waiting for 100,000 input records to achieve a 100MB file size might take hours or days for your service. It will therefore take some tuning and likely some concessions to balance efficiency and requirements of your application.</p><p>Other output systems, such as DynamoDB or other downstream services, might have other considerations. DynamoDB for example can largely be scaled automatically to handle large write volume. This does come at a cost and still requires code to handle backoff when write throughput or other errors are encountered. The advantage of having your service managed through an SQS queue or Kinesis stream is that you can slow down processing in your service in response to downstream issues to allow the downstream service to recover. While this is happening the queue or stream may back up, but data will remain in the queue (until expirations are reached). And when the downstream services recover, your service can work through the backlog to catch up and as long as the downtime was not too long then there is little or no risk of data loss. SQS can utilize <a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html">Dead Letter Queues</a> to make them more robust to data loss, and Kinesis has other approaches such as writing data to S3 via a <a href="https://aws.amazon.com/kinesis/data-firehose/">Firehose</a> and reprocessing the data from S3 after the system recovers.</p><h4>Conclusion</h4><p>By thinking about our data pipeline in terms of input and output velocity, we are able to reason through a number of controls in how data pipelines can be managed in order to achieve optimal execution. These controls were primarily achieved through controlling how data was passed to your pipeline and how it was executed. We discussed using SQS queues or Kinesis streams to manage how data is passed to your pipeline, and using Lambda to manage execution. And we discussed the following key controls of these service as it relates to managing data velocity.</p><ol><li>Concurrency — Allowed our application to scale in order to match and optimize for incoming data velocity</li><li>Batching — Allowed our application to process more data at once in order to optimize its own execution and to manage the outgoing data velocity to downstream services.</li></ol><p>Even though we were primarily talking about data in terms of Velocity, we were using it to control other features of data such as the size, or volume, of data as in the example of loading S3 data to a data warehouse.</p><p>Lastly, one of the most powerful tools in optimizing velocity is through monitoring of your pipeline. Monitoring can help identify bad or inefficient configuration that could not only cause catastrophic issues like a pipeline failing and unable to process data. But it can also identify when the tuning of configurations such as batch sizes are potentially dangerous or not optimal.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3fed052a167b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>