<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Justin Trugman on Medium]]></title>
        <description><![CDATA[Stories by Justin Trugman on Medium]]></description>
        <link>https://medium.com/@jtrugman?source=rss-8cd31a9b62fd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*NhcBysql8BhkmaTBa4ksiQ.jpeg</url>
            <title>Stories by Justin Trugman on Medium</title>
            <link>https://medium.com/@jtrugman?source=rss-8cd31a9b62fd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:23:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jtrugman/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Principles for Building Model-Agnostic AI Systems]]></title>
            <link>https://pub.towardsai.net/principles-for-building-model-agnostic-ai-systems-a3ec68158118?source=rss-8cd31a9b62fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3ec68158118</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-systems]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[multi-agent-systems]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Justin Trugman]]></dc:creator>
            <pubDate>Fri, 25 Apr 2025 14:02:45 GMT</pubDate>
            <atom:updated>2025-04-25T14:02:45.443Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nc2Buu0GQwJ2etg2qbXGYw.png" /></figure><p>While individual AI models dominate headlines, the spotlight often misses where true progress happens: the systems that put those models to work. Each new release promises more nuanced understanding, deeper reasoning, richer comprehension — but capabilities alone don’t move industries. It’s the architecture that wraps around them, the orchestration layer that knows when and how to use them, that turns raw potential into applied intelligence.</p><p>“I think in the long term, the most value will not come from the foundation models themselves, but from the systems that can intelligently call foundation models.” — <a href="https://www.youtube.com/watch?v=tsPuVAMaADY">Andrew Ng</a></p><p>The next wave of AI breakthroughs won’t come from betting on the right model. They’ll come from building systems that can continuously integrate the best model for the job — whatever that model is, and whenever it arrives.</p><h3>Understanding the Building Blocks of AI Models</h3><p>Before designing any model-agnostic system, it’s important to understand what actually goes into a model. AI models aren’t just standalone entities — they’re built in layers, each contributing a different dimension of capability.</p><p>You typically start with a base architecture. Most models today — especially those that handle text, tool use, or autonomous agent behavior — are based on transformers. These are the underlying neural network designs that make modern language models possible. If you’re working with visual generation, like images or video, you’re more likely dealing with diffusion models, which are optimized for high-fidelity synthesis through noise and denoising processes.</p><p>On top of the architecture, you then define the scale and scope. A Large Language Model (LLM) refers to a model with dozens of billions (sometimes hundreds of billions) of parameters, enabling broad, generalized capabilities across tasks. A Small Language Model (SLM) is a scaled-down version — lighter, faster, and often used for edge deployments or specific roles where compute efficiency matters more than versatility.</p><p>Once you have your base model, you can tailor it to specific domains or behaviors through post-training, commonly referred to as fine-tuning. Fine-tuning allows a model trained on general data to specialize in law, healthcare, finance, or any other area where nuanced understanding is critical. It’s also how instruction-following and tool-use behaviors are often reinforced.</p><p>From there, models can be extended with architectural practices or runtime techniques. A model might adopt a Mixture of Experts (MoE) approach, dynamically routing queries to different subnetworks based on the task. Or it might feature enhanced reasoning capabilities, such as chain-of-thought prompting, multi-step logic execution, or even structured planning frameworks. These capabilities allow the model to go beyond surface-level outputs and begin engaging in more deliberate, process-driven problem-solving.</p><p>Finally, you have specialized capabilities layered on top. A model might be multimodal, meaning it processes and generates across text, image, and audio inputs. It might combine different generative architectures — like transformers for text and diffusion for visuals — to handle diverse output modalities. These layers don’t exist in isolation — they compound. And understanding how they stack is foundational to building systems that know what kind of model to use, where, and why.</p><h3>Blueprints for Building Adaptable, Model-Agnostic Architectures</h3><p>Designing a model-agnostic system means building for constant evolution. Models will change. Capabilities will shift. Your infrastructure needs to keep up without requiring a rebuild every time something new comes along.</p><p>The first principle is decoupling logic from inference. This means separating the definition of a task from the model that executes it. Your system should understand the task that needs to be done — without baking in assumptions about how it gets done. That choice — what model to use for that task — should be abstracted out so that it’s easy to switch between models without rewriting the system’s logic.</p><p>Many modern inference providers have aligned on the OpenAI-compatible API standard (e.g., OpenAI, Anthropic, Groq, HuggingFace and others), which makes it easier to build systems that can flexibly switch models without changing the surrounding infrastructure. Designing around this standard helps ensure your system remains portable and compatible as the ecosystem grows.</p><p>It’s this layer of abstraction that enables true model-agnostic design — giving your system the ability to evolve, adapt, and scale without being anchored to any single provider or model lineage.</p><p>The next principle is treating models as specialists, not generalists. Every model has its own strengths — some are better at planning, others at creativity, some excel in reasoning, and others in speed or low-cost inference. Your system should be designed to route tasks to the model that’s best suited to handle it. This may mean assigning specific models to specific functions, or designing agents with models optimized for their assigned roles in a multi-agent system. For example, a fast, efficient planner might use a small reasoning model; a writer or content generator might use a highly expressive LLM; a fact-checking agent might use a more literal model with lower variance in output.</p><p>Whether it’s routing tasks directly to models or delegating them to agents with purpose-built model stacks, this approach acknowledges that no single model can do everything well — and that the highest-performing systems intelligently delegate tasks in ways that respect and leverage each model’s unique strengths.</p><p>Modularity means building systems where each component can be independently swapped or upgraded. Whether you’re dealing with a workflow, a multi-agent system, or something entirely custom, the principle stays the same: no single component should create friction for the rest of the system.</p><p>When planning a module — whatever the function or responsibility — it should be consumable in isolation and replaceable without downstream disruption. This allows your system to evolve incrementally as new tools and models emerge, rather than forcing wholesale rewrites just to integrate something better.</p><p>The final principle is observability. If you can’t measure how well a model is performing in context, you can’t make informed decisions about when to keep it, replace it, or reconfigure how it’s being used. Model performance should be treated as a live signal — not a one-time benchmark.</p><p>That means tracking metrics like latency, cost, token efficiency, and output quality at the system level, not just during eval runs. Is a cheaper alternative producing comparable results in certain contexts? Are reasoning agents making consistent errors under certain loads?</p><p>Telemetry is what turns gut checks into data-driven decisions. It’s what gives you confidence to experiment — and evidence to justify when a change actually makes things better.</p><p>Designing systems this way sets the stage — but actually choosing the right model for each role requires careful evaluation, not guesswork.</p><h3>Evaluating and Testing Models for Fit</h3><p>Building a modular, model-agnostic system only pays off if you have a clear, structured way to evaluate which model belongs where. It’s about finding the right model for each specific function within your system. That requires moving beyond general benchmarks and looking at how models behave in your context, under your constraints.</p><p>Start by assessing output consistency. A model that performs well in a vacuum but produces unstable or hallucinated results under pressure isn’t viable in production. You’re not just testing for correctness — you’re evaluating whether the model can behave predictably across similar inputs and degrade gracefully in edge cases.</p><p>Next, evaluate performance in the context of your system through A/B testing. Swap models across real user flows and workflows. Does a new model improve task success rates? Does it reduce fallbacks or speed up completion times? System-level testing is how you reveal performance trade-offs that aren’t visible in isolated prompts or benchmarks.</p><p>A useful tool for running these kinds of evaluations is<a href="https://github.com/promptfoo/promptfoo"> PromptFoo</a>, an open-source framework for systematically testing LLM prompts, agents, and RAG workflows. It lets you define test cases, compare model outputs side-by-side, and assert expectations across different providers. It helps turn model evaluation into a repeatable process rather than an ad-hoc exercise.</p><p>Not every evaluation is universal — some depend on the specific capabilities your AI system is built to support. Two areas that often demand targeted testing are tool use and reasoning performance.</p><p>If your AI system revolves around tool calling, it’s important to evaluate how well a model handles zero-shot tool use. Can it format calls correctly? Does it respect parameter structures? Can it maintain state across chained calls? Some models are optimized for structured interaction, while others — despite being strong at open-ended generation — struggle in environments that require precision and consistency.</p><p>For systems that depend on complex decision-making, reasoning performance becomes a critical axis. Can the model follow a chain-of-thought? Break down a problem into substeps? Resolve conflicting information? These evaluations are most useful when they mirror your actual workflows — not when they’re pulled from abstract reasoning benchmarks that don’t reflect real-world demands.</p><p>Evaluating a model’s capabilities is only half the picture. Once a model looks viable functionally, the next question is: can your system run it efficiently in production?</p><p>Start with inference latency. Some models are inherently faster than others based on their architecture or generation behavior. But just as important is where and how the model is hosted — different providers, runtimes, and hardware stacks can significantly affect speed and responsiveness.</p><p>Then consider token usage and cost efficiency. Some models are more verbose by default, or take more tokens to arrive at a meaningful answer. Even if the model performs well, inefficient token usage can accumulate into significant costs at scale.</p><p>These operational realities don’t determine which model is the most capable — but they often determine which one is actually deployable.</p><p>The pace of model development isn’t slowing down — it’s accelerating. But chasing the latest release won’t give your organization an edge. The real advantage lies in building systems that can flex, adapt, and integrate whatever comes next.</p><p>Model-agnostic systems aren’t about hedging bets — they’re about making better ones. They allow you to continuously evaluate and adopt the best tool for each job without rewriting your stack every quarter. They support experimentation, specialization, and modular upgrades — all without breaking what’s already working.</p><p>In the long run, the intelligence of your system won’t be defined by which model you chose today — it will be defined by its ability to continuously adapt and integrate the right model as new ones emerge.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3ec68158118" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/principles-for-building-model-agnostic-ai-systems-a3ec68158118">Principles for Building Model-Agnostic AI Systems</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GAIA: Redefining AI Assistant Evaluation]]></title>
            <link>https://pub.towardsai.net/gaia-redefining-ai-assistant-evaluation-2b9227ae44fd?source=rss-8cd31a9b62fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b9227ae44fd</guid>
            <category><![CDATA[multi-agent-systems]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[benchmark]]></category>
            <category><![CDATA[ai-assistant]]></category>
            <dc:creator><![CDATA[Justin Trugman]]></dc:creator>
            <pubDate>Tue, 30 Apr 2024 22:02:11 GMT</pubDate>
            <atom:updated>2024-04-30T22:02:11.386Z</atom:updated>
            <content:encoded><![CDATA[<p>We all appreciate the wonders of artificial intelligence, and AI agents as well as Multi-Agent Systems promise even greater capabilities, right? But how can we be sure of their effectiveness? Benchmarking plays a critical role in this context — it’s essential for establishing measurable standards and criteria to reliably evaluate these technologies.</p><p>However, not all benchmarks are created equal. Many can be limited in scope, overly simplistic, or fail to capture the nuances of real-world AI applications. This is where the GAIA benchmark stands out. GAIA is different because it’s designed to test AI systems through a series of tedious, multi-modal tasks that mimic the challenges encountered in everyday scenarios. Unlike conventional benchmarks that often test for narrow abilities, GAIA assesses a broader range of skills including the use of tools, complex reasoning, and the ability to browse the web effectively.</p><p>Why is this important? In our rapidly advancing age of AI, it’s crucial to have a benchmark that not only tests AI systems under laboratory conditions but also evaluates their readiness for practical deployment. Let’s dive into how GAIA works, why it represents a significant evolution in AI benchmarking, and why understanding this is crucial for anyone keeping track of AI advancements.</p><h3>So, what is GAIA?</h3><p>General AI Assistants (GAIA) is a novel benchmark for evaluating an AI system’s proficiency in using tools, complex reasoning, multi-modality, and web browsing created by a research collaboration among FAIR, Meta, HuggingFace, AutoGPT, and GenAI. Unlike traditional benchmarks, GAIA consists of 466 multi-step questions that, while straightforward, are tedious for humans, as well as prove challenging for AI systems. For instance, on average humans can complete 92% of the benchmark, while the current best-performing AI system (Microsoft AI Frontier’ Autogen Multi-Agent Framework) has achieved only 32.33%.</p><h3>Why is GAIA Important?</h3><p>As the complexity of tasks assigned to AI increases, current methods of human evaluated benchmarks for AI systems become increasingly impractical. This dilemma is particularly evident when assessing outputs like lengthy texts or advanced mathematical solutions surpass the understanding of most humans. To address this challenge, benchmarks like GAIA are redefining how we evaluate AI systems. GAIA focuses on designing tasks that, while conceptually straightforward, require the AI to navigate extensive multi-modal actions to produce a verifiable output. This method ensures that the tasks are rooted in practical scenarios where the complexity lies in the execution rather than the verification of the result. By focusing on such tasks, GAIA establishes a robust framework for evaluating AI capabilities that mirrors real-world applications of AI Assistants and ensures their readiness for practical deployment. It’s also scalable, providing a framework that allows for the inclusion of new questions and the adaptation of existing tasks in the future.</p><p>Additionally, the GAIA benchmark provides a pathway for evaluating AI systems through the lens of <a href="https://www.alignmentforum.org/posts/BoA3agdkAzL6HQtQP/clarifying-and-predicting-agi">t-AGI</a>, which measures the time it takes for an AI to complete tasks compared to humans. Humans typically complete the simplest GAIA tasks in about 6 minutes and the most complex tasks in about 17 minutes. However, even with advanced tools, GPT-4 achieves a success rate of no more than 30% for the simplest tasks and fails entirely on the hardest tasks, contrasting sharply with the average human success rate of 92%. This disparity highlights an interesting lens that allows you to use GAIA to measure the current performance of AI systems in both understanding and operational speed compared to human capabilities. Humans typically take between 6 minutes for the simplest questions to 17 minutes for the most complex tasks, GAIA sets a tangible standard for AI performance.</p><p>This temporal metric allows for a direct comparison of efficiency between AI and human problem-solving abilities. A system capable of solving GAIA tasks within these time frames demonstrates not only proficiency in handling tasks of varying complexity but also efficiency akin to or surpassing human capabilities. Such a system would mark a significant milestone in the development of AI, showcasing its potential to operate within real-world time constraints and providing a clear measure of its readiness for practical applications.</p><h3>Levels of GAIA Questions</h3><p>The questions within the GAIA benchmark are divided into three distinct levels which increase in complexity. Level 1 questions usually require no tools or just one tool and involve no more than five steps. Level 2 questions involve roughly five to ten steps and necessitate the use of multiple tools. Level 3 questions are designed for assistants that can help with very laborious tasks, requiring the use of any number of tools, lengthy sequences of actions, and comprehensive access to information. This tiered structure allows easy evaluation of an AI systems ability to solve problems within varying levels of sophistication.</p><h3>Example GAIA Questions</h3><p>A level 1 GAIA question is the following: “What was the actual enrollment count of the clinical trial on H. pylori in acne vulgaris patients from Jan-May 2018 as listed on the NIH website?”</p><p>How would you solve this question?</p><p>Most humans would take the following steps:</p><ol><li><strong>Search:</strong> Begin by entering a relevant keyword into a search engine like Google, which they believe will yield the correct results.</li><li><strong>Open Webpages:</strong> Open the web pages that appear in the search results list.</li><li><strong>Review Information:</strong> Carefully review the content of the webpages to locate the specific information needed.</li><li><strong>Use of Tools:</strong> Utilize a web browser throughout the process, leveraging its features to navigate and retrieve information effectively.</li></ol><p>This question involves several steps and iterations, utilizing both a web browser and a search engine to acquire specific information. Now, let’s up the difficulty and explore a level 2 question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/216/0*h7gqV65DiBDGl7cO" /><figcaption>Source: Image by <a href="https://arxiv.org/abs/2311.12983">GAIA: a benchmark for General AI Assistants</a></figcaption></figure><p>“If this whole pint is made up of ice cream, how many percent above or below the US federal standards for butterfat content is it when using the standards as reported by Wikipedia in 2020? Answer as + or — a number rounded to one decimal place.”</p><p>The following image is provided for the AI System to analyze. So how would you solve this question?</p><p>Most humans would follow these steps:</p><ol><li><strong>Search for Standards:</strong> Use a search engine like Google to find the US federal standards for butterfat content. Specifically, look for the standards as reported on the Wikipedia page from the year 2020.</li><li><strong>Open and Review Wikipedia Page:</strong> Navigate to the Wikipedia page and review the documented standards for butterfat content.</li><li><strong>Analyze Provided Image:</strong> Examine the image provided (likely a nutrition label) to determine the actual butterfat content of the ice cream.</li><li><strong>Calculate the Difference:</strong> Compute the percentage difference between the ice cream’s butterfat content and the federal standard. Use the formula (Actual Content − Standard Content) /Standard Content × 100%</li><li><strong>Round and Format the Result:</strong> Round the resulting number to one decimal place and format the answer as either a positive or negative percentage, indicating whether the ice cream is above or below the standard.</li></ol><p>Notice how the questions become increasingly tedious? A Level 2 question, while manageable for humans, requires a multi-modal approach that spans several steps and tools. To solve such a question, a person must effectively use a search engine and a web browser, visually inspect the nutrition facts on packaging, and perform calculations — typically using a calculator — to determine compliance with specific standards set in 2020.</p><p>Now on to a level 3 question: “In NASA’s Astronomy Picture of the Day on 2006 January 21, two astronauts are visible, with one appearing much smaller than the other. As of August 2023, out of the astronauts in the NASA Astronaut Group that the smaller astronaut was a member of, which one spent the least time in space, and how many minutes did he spend in space, rounded to the nearest minute? Exclude any astronauts who did not spend any time in space. Give the last name of the astronaut, separated from the number of minutes by a semicolon. Use commas as thousands separators in the number of minutes.”</p><p>You can see how tedious these questions are now getting. In order to answer this question, most humans would need to perform the following steps:</p><ol><li><strong>Locate the Image:</strong> Access NASA’s website to find the Astronomy Picture of the Day for January 21, 2006.</li><li><strong>Identify the Smaller Astronaut:</strong> Examine the image to determine which astronaut appears smaller than the other.</li><li><strong>Determine the Astronaut Group:</strong> Identify the specific NASA Astronaut Group that the smaller astronaut belonged to at the time of the photo.</li><li><strong>Compile a List of Astronauts:</strong> List all the members of that particular astronaut group.</li><li><strong>Retrieve Space Time Logs: </strong>Search for official records or databases detailing the space time logged by each astronaut in the group.</li><li><strong>Find Minimum Space Time: </strong>Determine which astronaut from the list spent the least amount of time in space, ensuring to exclude any astronauts who have not spent any time in space at all.</li><li><strong>Extract Relevant Data: </strong>Note down the last name of the astronaut with the least space time, along with the exact number of minutes, rounded to the nearest minute.</li><li><strong>Format the Answer: </strong>Format your response by writing the astronaut’s last name followed by a semicolon, and then the number of minutes in space, using commas as thousands separators.</li></ol><p>This question is highly tedious for a human to complete and very complex for an AI System to complete, because of the sheer amount of steps, tools required to complete it, and multi-modality of the steps required to answer the question.</p><h3>Significance of GAIA in the AI Landscape</h3><p>In an era where new AI benchmarks are appearing almost as frequently as JavaScript frameworks, it’s beneficial to concentrate on those like GAIA that offer a clear connection to practical tasks with which AI systems are expected to assist humans with. This focus ensures that the benchmarks remain relevant and directly applicable to real-world use cases.</p><p>We look forward to seeing more AI systems tested with the GAIA benchmark. It’s not only about tracking progress — these insights are crucial for shaping future benchmarks and understanding our path toward AGI and beyond. We are keen to see what these developments will reveal about the potential and future of AI.</p><p>To learn more about GAIA, you can review the original paper which we heavily referenced for this article on <a href="https://arxiv.org/abs/2311.12983">Arxiv</a> and the current GAIA Leaderboard hosted on <a href="https://huggingface.co/spaces/gaia-benchmark/leaderboard">HuggingFace</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b9227ae44fd" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/gaia-redefining-ai-assistant-evaluation-2b9227ae44fd">GAIA: Redefining AI Assistant Evaluation</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Deploy your Probot App with Vercel]]></title>
            <link>https://medium.com/@jtrugman/how-to-deploy-your-probot-app-to-vercel-9411a77fa905?source=rss-8cd31a9b62fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/9411a77fa905</guid>
            <category><![CDATA[probot]]></category>
            <category><![CDATA[vercel]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[bots]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Justin Trugman]]></dc:creator>
            <pubDate>Sun, 09 Oct 2022 21:50:35 GMT</pubDate>
            <atom:updated>2022-10-09T22:04:27.424Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fGapSb3F8-eng22y" /><figcaption>Photo by <a href="https://unsplash.com/@rocknrollmonkey?utm_source=medium&amp;utm_medium=referral">Rock&#39;n Roll Monkey</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this blog post, we’ll show you how to deploy <a href="https://probot.github.io/">Probot Apps</a> to <a href="https://vercel.com/">Vercel</a>, a platform that makes frontend and serverless deployments easy! We’ll cover everything from linking a Vercel project to your GitHub Repo, to configuring your Probot app for deployment. By the end of this post, you’ll be able to deploy your Probot app with just a few clicks.</p><h3>Why use Vercel for your Probot App?</h3><p>Vercel is a great platform to deploy your Probot app with because it simplifies the entire deployment and infrastructure management process. In addition, <em>(at the time of writing this post)</em> Vercel offers a <a href="https://vercel.com/pricing">generous free plan</a> and affordable upgrade paths which makes it ideal for Indie Developers and Enterprises alike.</p><h3>Getting started…</h3><p>Now that we know why Vercel is a great option to deploy your Probot app to, let’s dive into deploying your first Probot app to the Vercel platform!</p><p>For this tutorial we will build upon the example app from the <a href="https://probot.github.io/docs/development/">Probot “Developing an app” Documentation</a>. We assume that you have already completed this tutorial and have your example app <a href="https://probot.github.io/docs/development/">running locally</a>, <a href="https://probot.github.io/docs/development/#configuring-a-github-app">configured correctly</a>, and pushed to a <a href="https://github.com/">GitHub</a> repository.</p><p><strong>Code Changes</strong></p><p>Vercel deploys your Probot app as a serverless function.</p><blockquote>Serverless architecture (also known as <em>serverless computing</em> or <em>function as a service</em>, FaaS) is a software design pattern where applications are hosted by a third-party service, eliminating the need for server software and hardware management by the developer. Applications are broken up into individual functions that can be invoked and scaled individually.</blockquote><blockquote>Source: <a href="https://www.twilio.com/docs/glossary/what-is-serverless-architecture">https://www.twilio.com/docs/glossary/what-is-serverless-architecture</a></blockquote><p>In order for the Probot app to be deployed in this nature we can use the <a href="https://github.com/octokit/app.js/#createnodemiddlewareapp-options">createNodeMiddleware</a> to handle the requests. For this to work we need to make a new directory and JavaScript file in our project with the following path…</p><p>/api/github/webhooks/index.js</p><p>This file will handle the webhooks from GitHub to your Probot app, we will also need to configure the GitHub App for this, more on that later!</p><p>In the index.jsfile we just made, we can copy the sample code from the <a href="https://probot.github.io/docs/deployment/#vercel">Probot deployment docs</a> and make a slight modification. In the <a href="https://probot.github.io/docs/development/">Probot “Developing an app” Documentation</a> the function we are exporting is in the index.jsfile, while in the Deployment docs the function is in the App.js file. The simple modification is just switching the file we are importing as our app, see it in bold below:</p><pre>// api/github/webhooks/index.js<br>const { createNodeMiddleware, createProbot } = require(&quot;probot&quot;);<br>const app = require(&quot;../../../<strong>index</strong>&quot;);</pre><pre>module.exports = createNodeMiddleware(app, {<br> probot: createProbot(),<br> webhooksPath: &quot;/api/github/webhooks&quot;,<br>});</pre><pre><em>Source: </em><a href="https://probot.github.io/docs/deployment/#vercel"><em>https://probot.github.io/docs/deployment/#vercel</em></a></pre><p>This is the only code change required for the deployment to Vercel, once made, push this change to your repository.</p><p><strong>Configurations</strong></p><p>Now that we have the code ready, we need to configure our Vercel deployment and our GitHub App. We will start with configuring Vercel.</p><p>First we need to import our repository from to Vercel. To do this under the “Import Git Repository” section select the <strong>Continue with GitHub </strong>option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*G29GipMqWUWdz3L6yNpmqA.png" /></figure><p>Then link your GitHub account to Vercel and select the repository with your Probot app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*n8TndevUU0OHgwSVycbmLw.png" /></figure><p>After you import your Probot app repository it is time to configure the Environment Variables. These are the values Vercel uses to configure your Probot app upon deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/1*tQZljVqvzgvBp3v1Wvy1Bw.png" /></figure><p>There are three Environment Variables which need to be configured on Vercel, APP_ID, WEBHOOK_SECRET, and PRIVATE_KEY.</p><p>The APP_ID is simply your App’s unique identifier, the WEBHOOK_SECRET is used to secure your webhook connection, and the PRIVATE_KEY for your app. These can all be found in the .envfile of your existing project. Once you entered the Environment Variables you can click deploy, Vercel will then begin deploying your application!</p><p><strong>Configuring GitHub</strong></p><p>Once your Vercel deployment is ready we will need to configure GitHub to point it’s Webhooks at your Probot App. In order to do this you first need to copy your Probot app’s domain from the deployment. It can be found under the “Domains” section under your Vercel project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*RpFm5H6n_Kc0bTG_CEZOGA.png" /></figure><p>Once you have this domain handy, you need to navigate to the <a href="https://github.com/settings/apps">GitHub App Settings</a> and replace the Webhook URL with your deployment’s domain. You will also need to add the path of your webhooks directory /api/github/webhooks as this is where Vercel will find your serverless function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*04An5WCEIluSlbiBs5CuBw.png" /></figure><p>After that, you are all done! Save your changes to the GitHub App configuration and test out your Probot App! Here is an example of it working</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*IvPQrHQG7u8u7GpVPgeWOA.png" /><figcaption>The sample app adds a comment to the newly created issue thanking the author for creating it!</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9411a77fa905" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Inside WebRTC RTCPeerConnection Signaling]]></title>
            <link>https://medium.com/securemeeting/inside-webrtc-rtcpeerconnection-signaling-1737deb77107?source=rss-8cd31a9b62fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/1737deb77107</guid>
            <category><![CDATA[signaling]]></category>
            <category><![CDATA[video-streaming-service]]></category>
            <category><![CDATA[secure-meeting]]></category>
            <category><![CDATA[webrtc]]></category>
            <dc:creator><![CDATA[Justin Trugman]]></dc:creator>
            <pubDate>Tue, 16 Mar 2021 14:35:37 GMT</pubDate>
            <atom:updated>2021-03-16T14:35:37.060Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GI3tBKOBETXkakdcucaUxA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@mero_dnt">Chinh Le Duc</a></figcaption></figure><p>This article is meant to serve as a quick look-up guide about signaling components of the <strong>RTCPeerConnection</strong>.</p><blockquote>RTCPeerConnection facilitates the connection between parties who want to communicate with each other. This facilitation is done by handling the signaling, media transfer, and media performance management. All of these functions are crucial to successfully transfer media from one party to another and are encompassed in an easy-to-use API.</blockquote><h3>Signaling Basics</h3><p>Signaling is the process of two devices discovering each other and negotiating how they will communicate with one another.</p><blockquote>Imagine you are sending a package by mail to to your friend <a href="https://medium.com/u/84406bae34fc">Mukund Iyengar</a>. To do that, you need several things before you approach the post-office. You need to know Mukund’s address and details about the package such as weight, the box’s dimensions, if the contents are fragile, etc.,</blockquote><p>This is just like the signaling for WebRTC. The two devices need to know information about what data is going to be sent. Just like how the Post Office needs to know information about what package you are sending. This includes items like the media format/codec being transported, method of transporting the data, channels, communication protocols, etc. In addition, just like how you can choose different Post Carriers to deliver your package, WebRTC does not have a fixed signaling process.</p><p>Signaling for WebRTC revolves around a few key steps. The first is that each peer creates an RTCPeerConnection. This is what each peer uses to represent their session. After an RTCPeerConnection is created the session initiator <em>(caller)</em> sends an <strong>offer </strong>to the receiver <em>(callee)</em>. The callee then responds to the caller by sending an <strong>answer</strong>. Both the offer and the answer contain key codec information that needs to be exchanged between users, typically in SDP format. There is still another step though which is necessary though before the signaling process can be completed which is both peers establishing how they want to transmit media data to each other. This is done through Interactive Connectivity Establishment (ICE).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/261/1*XgBjMSBuR3zpi9MjXMTO2A.png" /><figcaption>WebRTC Signaling — <em>Diagram by </em><a href="https://medium.com/u/8cd31a9b62fd"><em>Justin Trugman</em></a></figcaption></figure><p>Each peer has ICE candidates, which are essentially different ways for other peers to connect with it. The peers then negotiate and eventually agree on a specific candidate which allows them to connect with each other. Once the peers mutually agree upon a candidate they then use the SDP from that candidate to initiate the transfer of data.</p><p>It is important to note, however, that just because the peers agreed upon an ICE candidate and started transferring data does not mean that the exchange is over. Each peer continues to send candidates until it has run out of ideas and if during this process a candidate is found to be more optimal than the existing candidate the data transfer format may change.</p><h3>NATs and NAT Traversal</h3><p>We have officially run out of IP addresses: more devices connect to the Internet today than the number of addresses available! NATs are used to manage this problem.</p><p>Network Address Translation, more commonly referred to as NAT, is the process of a network’s router/firewall assigning a <em>single</em> public IP address to a <em>group</em> of devices, typically in a home/office setting. This poses a unique problem for WebRTC because it needs to be able to locate and establish a session with the callee device hidden behind the NAT.</p><p>In order to counteract this problem, WebRTC has three tools in its toolbox: STUN, TURN, and ICE.</p><h4>STUN</h4><p>Session Traversal Utilities for NAT, more commonly called STUN, is a method for clients to find out more about themselves so that a direct peer-to-peer connection can be established.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/261/1*mLxH1uu7n6fbMH5983-Y_w.png" /><figcaption>STUN Server — <em>Diagram by </em><a href="https://medium.com/u/8cd31a9b62fd"><em>Justin Trugman</em></a></figcaption></figure><h4>TURN</h4><p>Traversal Using Relays around NAT, more commonly referred to as TURN, not only helps WebRTC clients find each other but actually acts as a server in the middle relaying all media to and from each client. This is used as an option when a direct connection between the caller and callee is not possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/261/1*rsS0jFOMgIMAco3Ep64dhA.png" /><figcaption>TURN Server — Diagram by <a href="https://medium.com/u/8cd31a9b62fd">Justin Trugman</a></figcaption></figure><h4>ICE</h4><p>Interactive Connectivity Establishment, more commonly called ICE, is the tool that wraps STUN &amp; TURN together and helps the RTCPeerConnection establish sessions with a device behind a NAT. Just like how we discussed in the Signaling Basics section, ICE’s job is to find the most efficient method for peers to send data to each other and it repeatedly searches for that best method. ICE will try using STUN and TURN to establish sessions with a device behind a NAT and will eventually select which method is the best one for the specific session at hand.</p><h3>Concluding Thoughts</h3><p>The RTCPeerConnection is one of the three core APS that WebRTC has and signaling is a key component of the RTCPeerConnection. If you are interested in learning more about the RTCPeerConnection and WebRTC Signaling the open source book WebRTC for the Curious has a great chapter on it here: <a href="https://webrtcforthecurious.com/docs/02-signaling/">https://webrtcforthecurious.com/docs/02-signaling/</a></p><h3>About SecureMeeting</h3><p><a href="https://securemeeting.org/">SecureMeeting</a> is a 501(c)(3) non-profit whose mission is to advance human rights and freedom of speech. We do this by designing, developing and deploying planetary-scale, privacy-preserving communications architectures for all of mankind.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1737deb77107" width="1" height="1" alt=""><hr><p><a href="https://medium.com/securemeeting/inside-webrtc-rtcpeerconnection-signaling-1737deb77107">Inside WebRTC RTCPeerConnection Signaling</a> was originally published in <a href="https://medium.com/securemeeting">SecureMeeting</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rethinking Net Neutrality]]></title>
            <link>https://medium.com/@jtrugman/rethinking-net-neutrality-c9c2cac79516?source=rss-8cd31a9b62fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9c2cac79516</guid>
            <category><![CDATA[politics]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[net-neutrality]]></category>
            <category><![CDATA[networking]]></category>
            <dc:creator><![CDATA[Justin Trugman]]></dc:creator>
            <pubDate>Fri, 15 Feb 2019 02:31:08 GMT</pubDate>
            <atom:updated>2019-02-15T02:31:08.167Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RCwrJu7mG8AsmA85zQEU0w.jpeg" /></figure><p>The topic of network neutrality has been widely debated in recent times. Somehow politicians who barely know how to use a computer are now experts in how the internet works and are developing legislation to regulate it. Now I am no expert however, I believe that with my experience researching computer networks for the past two years I can provide valuable insight into how the net neutrality debate should change.</p><p>The main problem with the net neutrality debate is that it is thought of as a yes or no question. Either we should have network neutrality or we shouldn’t. However, net neutrality is objectively not a yes or no debate, it is a question of when does <strong>optimization</strong> become <strong>prioritization</strong>.</p><p>From a purely scientific perspective, transporting content in ways that cater to the content’s type will result in an increased quality of experience for the content’s user. For example, take console gaming and video on demand (Video on demand is a video that is prerecorded, like a youtube clip or netflix movie). The way console multiplayer gaming works is that each console in a game session sends an update on player movement and other crucial metrics to a central server, then the central server pushes out a state update to all the consoles with how the environment changes. The consoles then render this new state on the screen for the user (this all happens in milliseconds!). Since the consoles just push change updates to a server and the server pushes a state update, it requires very little bandwidth. However, It requires a very low latency (high speed) connection because these updates need to be rendered very fast for the user. This is why the main cause of lag in multiplayer gaming is a slow connection, not low bandwidth.</p><p>The best way to think about bandwidth versus latency is that bandwidth is how many lanes a road has (the road’s capacity) while latency is the speed limit of the road (what rate at which cars travel down the road).</p><p>Video on Demand on the other hand is the process of sending a video from a server or sometimes servers to a user. The video is then stored on the client in a buffer until it is on the ready to be played. The buffer is used to compensate for poor network conditions and mitigates the effects of a high latency (slow) connection. Therefore, if the connection is delayed, the quality of experience for the user won’t be badly affected in most cases. However, video needs lots of bandwidth to play and if the user’s connection doesn’t have enough bandwidth the quality of experience for the user will be greatly decreased. Based off of these two examples, you can see why it makes no sense to treat video on demand and console multiplayer gaming the same. The user receives a huge benefit by applying different network conditions to transporting these different types of data. There are many other instances like this. For example, let’s take a closer look at video…</p><p>A video is separated into frames. There are three types of frames, I frames (Intra frames), P frames (Predicted frames), and B frames (bidirectional predicted frames). Without getting into too much detail about video compression… you can think of I frames as a “full picture” frame. An I frame contains a “whole still image” which is part of the video. Because of this, I frames are very large relative to P and B frames. Both P and B frames are predicted frames. They are the differential in pixels from the I frame to the new image needed to be represented in the video therefore, they are much smaller in size compared to I frames. This also makes I frames much more important than P or B frames because it is the base of what the P and B frames adjust from. It would make sense to be able to optimize the path for I frames because if an I frame gets lost it will have a much bigger impact on the user’s quality of experience with the video.</p><p>It makes scientific sense that there should be some level of optimization that occurs based on different types of traffic. The problem occurs when you pose the question of when does optimization go to far? When does optimizing traffic become prioritizing a certain company’s content over a competing company’s content?</p><p>Take Youtube and Netflix, they both mainly have Video on Demand content. However, their videos are different lengths with the average Youtube video being a relatively short clip and Netflix movie being, well a movie. Since the Netflix movie has a longer time duration it is safe to say it has a larger buffer window than Youtube does for most of its short clips and plots generated from my lab prove this. When a video is initially loaded, it goes into an all out sprint to fill its buffer, it then continuously fills its buffer as it depletes during video play, which leads to a bursty traffic pattern. Therefore, if one user on an Internet Service Provider is watching a Youtube video and one is watching a Netflix movie. It may be beneficial to sacrifice the Youtube viewer’s quality of experience for the betterment of the Netflix watcher’s by assigning a higher priority to fill up the Netflix movie’s buffer. This would result in the Netflix user enjoying his or her movie more than the person watching the short Youtube clip. Which may be considered a worthwhile move by the ISP depending on how they quantify video quality of experience (in this case, QoE over Viewing Time) or if they have certain contracts with different content providers. And this is only one potential way to prioritize video content for different video providers, there are countless of other methods to do so. This poses the question, when does optimizing Netflix’s content become prioritizing it over Youtube’s?</p><p>These are the types of questions people should be asking. Net Neutrality <strong>is not</strong> a <strong>political conversation</strong>, but a <strong>science conversation</strong> that needs to be backed by sound and unbiased research.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9c2cac79516" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>