<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bikram Gyawali on Medium]]></title>
        <description><![CDATA[Stories by Bikram Gyawali on Medium]]></description>
        <link>https://medium.com/@vikramgyawali57?source=rss-5a5de8ab70e6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*hx6ZE2xZ46Rx2_zEHbkQkQ.jpeg</url>
            <title>Stories by Bikram Gyawali on Medium</title>
            <link>https://medium.com/@vikramgyawali57?source=rss-5a5de8ab70e6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 05:26:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vikramgyawali57/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Wiz Integrates with Anthropic’s Compliance API]]></title>
            <link>https://medium.com/@vikramgyawali57/wiz-integrates-with-anthropics-compliance-api-ef1d48ff1f66?source=rss-5a5de8ab70e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef1d48ff1f66</guid>
            <dc:creator><![CDATA[Bikram Gyawali]]></dc:creator>
            <pubDate>Tue, 26 May 2026 06:47:39 GMT</pubDate>
            <atom:updated>2026-05-26T06:47:39.692Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I5uTdgshUWqGjhLd.jpg" /></figure><p>AI tools, including Google Gemini and Anthropic’s Claude, are rapidly becoming integrated into core business functions — from development workflows and sales drafting to product brainstorming. This widespread adoption means that organizations are constantly adding new users, modifying administrative configurations, and generating new API keys outside of established operational processes. For security and compliance teams, this scaling presents a familiar challenge: how do they maintain the same level of visibility into Claude usage that they already track for every other enterprise application within their technology stack?</p><p>To address this growing need for oversight, Wiz announced an integration with the Anthropic Compliance API. This new connection brings activity data from Claude Enterprise directly into the Wiz platform, extending security monitoring to cover AI-driven workflows alongside traditional cloud resources.</p><h3>The Necessity of Advanced AI Compliance Monitoring</h3><p>As organizations adopt generative AI tools, the scope of compliance and security risk expands beyond conventional network boundaries. When an enterprise uses a tool like Claude, the activity is not confined to a single application; it involves users interacting with data, creating projects, and utilizing datasets connected across various internal systems.</p><p>Traditional visibility methods often struggle to map these complex relationships. The challenge lies in understanding not just <em>that</em> AI usage is happening, but <em>who</em> has access to what, <em>which</em> specific data is involved, and how those permissions connect to the broader cloud environment. This requires a centralized system that can treat AI resources — like projects and datasets — with the same rigor applied to compute instances or storage buckets.</p><h3>Overview of the Wiz-Anthropic Integration and Key Capabilities</h3><p>Wiz extends its established visibility across cloud environments, CI/CD pipelines, and repositories by applying its comprehensive Security Graph approach directly to Claude Enterprise activity. By leveraging activity logs from the dedicated Claude Compliance API, the integration achieves several key capabilities:</p><p><strong>Mapping Users and Permissions:</strong> The system discovers all Claude Enterprise organizations and the individual users associated with them. Crucially, it maps each user directly to their role bindings and permissions while maintaining full metadata visibility. This allows security teams to clearly see who has access to specific resources and how those identities relate to the rest of the environment already mapped within Wiz.</p><p><strong>Discovering Project Metadata:</strong> Beyond just listing users, Wiz also discovers individual Claude Enterprise projects. The platform surfaces rich project metadata, including details on attachments, chat counts, privacy settings, and ownership information. This level of detail helps organizations understand how these AI projects are being utilized across the enterprise and pinpoints accountability for each one.</p><p><strong>Securing Data Datasets:</strong> When a project contains sensitive AI datasets, Wiz doesn’t just list the data; it maps those datasets directly onto the Security Graph. This capability shows exactly where the data resides and illustrates its connection to the broader AI environment.</p><p>The integration provides comprehensive visibility into five critical areas:</p><ul><li>Claude Enterprise organizations, associated users, and organizational roles.</li><li>Specific permissions and role bindings tied to Claude usage.</li><li>Project details, including ownership, privacy settings, attachments, and activity metadata.</li><li>AI datasets that are connected to the projects and workflows.</li><li>The critical relationships connecting users, projects, datasets, and the broader cloud environment already tracked by Wiz.</li></ul><p>By mapping these diverse Claude resources into the centralized Wiz Security Graph, organizations gain a unified view of their AI inventory, enabling better governance and visibility across the entire enterprise footprint.</p><h3>Visibility for Teams Across the Organization</h3><p>Wiz designed this integration to put granular Claude visibility in the hands of specific operational teams, tailoring the data presentation based on professional needs:</p><ul><li><strong>For Security Engineers:</strong> If a new Claude project is created that connects AI datasets to sensitive internal systems, security engineers can immediately use Wiz to identify the owner and review exactly what data is attached. They do not need to switch tools or search for an owner; the information appears instantly on the Security Graph.</li><li><strong>For Compliance Leads:</strong> Preparing for an audit requires demonstrating robust governance over AI usage. With Claude’s activity logs and project metadata housed in Wiz, compliance leads can demonstrate who has access, what permissions they hold, and how these Claude resources map into the rest of the enterprise — all within their existing compliance platform.</li><li><strong>For Platform Engineering Teams:</strong> As Claude Enterprise scales across an organization, platform teams need to track growth patterns. They can monitor the total number of projects, identify who is creating them, and check if configurations adhere to established organizational policies, allowing for early detection of unexpected changes or misconfigurations.</li></ul><h3>Technical Deep Dive: Leveraging the Compliance API</h3><p>The integration operates by consuming activity logs provided through Anthropic’s dedicated Compliance API. This mechanism allows Wiz to ingest a stream of operational data that details user actions, project creation events, and dataset connections within Claude Enterprise environments.</p><p>By processing these logs, Wiz transforms raw compliance data into actionable security intelligence. The platform doesn’t just report on usage; it models the <em>relationships</em> between entities — linking a specific user (identity) to a project (resource), which in turn uses an AI dataset (data asset), all while showing how that entire chain connects back to core infrastructure resources already monitored by Wiz.</p><h3>Getting Started with Enhanced AI Security</h3><p>The integration of Wiz with Anthropic’s Compliance API is currently available in Private Preview for organizations utilizing both the Wiz platform and Claude Enterprise. Organizations interested in gaining this advanced level of visibility into their generative AI usage are advised to contact their account team to begin implementation.</p><h3>References</h3><p><a href="https://www.wiz.io/blog/claude-wiz-integration?2">https://www.wiz.io/blog/claude-wiz-integration?2</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef1d48ff1f66" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google search, as we know, is soon to be over.]]></title>
            <link>https://medium.com/@vikramgyawali57/google-search-as-we-know-is-soon-to-be-over-da46e04b182a?source=rss-5a5de8ab70e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/da46e04b182a</guid>
            <dc:creator><![CDATA[Bikram Gyawali]]></dc:creator>
            <pubDate>Tue, 26 May 2026 05:58:30 GMT</pubDate>
            <atom:updated>2026-05-26T05:58:30.875Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1nimACmpfiCM5MJN.jpg" /></figure><h3>Google search, as we know, is soon to be over. There’s going to be a massive change on how google works.</h3><p>Google Search is undergoing a massive transformation, entering what Google describes as a “new era for AI Search.” The platform is integrating advanced model capabilities directly into its core functionality, enabling users to interact with search results and information using sophisticated agents simply by asking questions. This overhaul includes the introduction of an intelligent, AI-powered Search box — the biggest upgrade in over 25 years — and aims to fundamentally change how users access complex or hyper-specific information.</p><p>The goal of Google Search has always been simple: assisting users with everything from quick facts to deep, complex questions that may be difficult to articulate. To achieve this, the company has continued to reimagine search using artificial intelligence. The momentum behind these changes is significant; just one year after its debut, AI Mode surpassed one billion monthly users, and queries have reportedly more than doubled every quarter since its launch (Source: <a href="https://blog.google/innovation-and-ai/sundar-pichai-io-2026">https://blog.google/innovation-and-ai/sundar-pichai-io-2026</a>).</p><h3>Powerful AI Integration into Search</h3><p>The immediate technical upgrade involves the integration of Gemini 3.5 Flash, which is being deployed as the new default model in AI Mode for all global users. This specific Flash model is noted for delivering sustained frontier performance suitable for both agents and coding tasks.</p><p>Beyond simply updating the underlying model, Google is introducing a completely reimagined, intelligent Search box. This upgrade aims to put powerful AI tools directly at the user’s fingertips, making the process of asking questions more intuitive. The new search box dynamically expands, providing users space to describe exactly what they need. It utilizes AI-powered suggestions that go beyond traditional autocomplete functionality and is designed to anticipate user intent. Furthermore, users can now search across multiple modalities, including text, images, files, videos, or even content from Chrome tabs as inputs. While the interface changes dramatically, Google confirmed that users will continue to receive a range of results from Search, similar to the experience they have today.</p><p>The conversational aspects of searching are also being enhanced. Users can now easily ask follow-up questions directly from an AI Overview. This allows for a seamless, back-and-forth conversation with AI Mode. Crucially, the system maintains context throughout this interaction; as users explore deeper, the supporting links and articles provided become increasingly relevant to the ongoing discussion. These conversational features are live today across both desktop and mobile platforms worldwide.</p><h3>The Era of Search Agents</h3><p>A major pillar of the transformation is the introduction of “Search agents.” This marks a shift toward an environment where users can easily create, customize, and manage multiple AI agents directly within the Search interface for various tasks. Initially, these agents focus on information gathering. These background-operating agents are designed to intelligently reason across vast amounts of web data 24/7, identifying exactly what a user needs at the precise moment it becomes relevant.</p><p>Information agents allow users to stay continuously updated on topics that matter most to them. The agent is capable of monitoring content across multiple sources — including blogs, news sites, and social media posts — while also accessing fresh, real-time data concerning finance, shopping, and sports. When a change related to the user’s specific question occurs, the agent sends an intelligent, synthesized update that even includes the ability to take action.</p><p>The source provided several practical examples of how these agents could function. For instance, if a user is apartment hunting, they can input all their precise requirements into the system, and the agent will continuously scan listings, notifying them when new properties meet those criteria. Similarly, for tracking cultural moments, an information agent could monitor for announcements regarding favorite professional athletes announcing sneaker collaborations, ensuring the user does not miss a new product drop.</p><p>In addition to general information monitoring, Google is expanding its capabilities in “agentic booking” within Search, applying these advanced functions to a wider variety of tasks. These information agents are scheduled to launch first for subscribers to Google AI Pro &amp; Ultra this summer.</p><h3>References</h3><p><a href="https://blog.google/products-and-platforms/products/search/search-io-2026/">https://blog.google/products-and-platforms/products/search/search-io-2026/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da46e04b182a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS MCP Server Reaches GA with Full API Coverage and IAM-Based Governance]]></title>
            <link>https://medium.com/@vikramgyawali57/aws-mcp-server-reaches-ga-with-full-api-coverage-and-iam-based-governance-fd25fa985545?source=rss-5a5de8ab70e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd25fa985545</guid>
            <dc:creator><![CDATA[Bikram Gyawali]]></dc:creator>
            <pubDate>Sun, 24 May 2026 13:14:56 GMT</pubDate>
            <atom:updated>2026-05-24T13:14:56.946Z</atom:updated>
            <content:encoded><![CDATA[<p>When a service hits General Availability (GA), the accompanying documentation usually confirms stability. But with the AWS MCP Server — which manages complex enterprise policy and resource orchestration across multiple AWS services — reaching GA, the focus shifts immediately from “is it ready?” to “how much operational complexity are we inheriting?” The announcement signals that developers can now rely on a comprehensive API surface backed by granular Identity and Access Management (IAM) governance, promising a significant tightening of cloud control planes.</p><p>The core promise of the MCP Server is to serve as a unified policy enforcement layer, moving beyond traditional resource-level permissions. Instead of defining who can <em>read</em> an S3 bucket or <em>start</em> an EC2 instance, the MCP model attempts to define what complex <em>actions</em> are permissible across disparate services — for example, “A developer can provision a secure, compliant, and fully logged staging environment using three different service types.” This requires integrating governance logic directly into the control plane itself. Historically, achieving this level of cross-service policy enforcement in AWS required stitching together CloudFormation Guard, Service Control Policies (SCPs), custom Lambda functions, and external Policy Decision Points (PDPs). The MCP Server aims to consolidate that entire stack under a single API endpoint, drastically reducing the necessary boilerplate code for compliance teams.</p><p>The technical mechanism driving this simplification is its deep reliance on IAM integration. The system doesn’t just <em>read</em> standard AWS roles; it leverages advanced attribute-based access control (ABAC) principles baked into the policy structure. When an identity assumes a role governed by MCP, the server evaluates not only the user’s explicit permissions but also environmental attributes — such as the target region, the project ID embedded in metadata tags, and the current operational status of the resource group. This shift fundamentally changes how security boundaries are drawn; they move from being based on static ARN strings to dynamic, context-aware calculations.</p><p>For system architects, this means policy definition can become significantly more concise. Consider a pre-MCP scenario where a team needed to ensure all new production databases (RDS) were encrypted and tagged with specific cost center IDs. The enforcement logic might require multiple checks: an SCP blocking unencrypted instances, a Lambda function enforcing tagging, and CloudFormation templates validating parameters. With the MCP Server’s full API coverage, this complex mandate can theoretically be expressed as a single policy object, which is then consumed by all provisioning tools.</p><p>The practical implementation of these policies happens at the code level using specialized SDK calls that interact with the new governance plane. Rather than only calling ec2:RunInstances, an application might call a high-level function like mcp:ProvisionService(serviceType=&quot;rds&quot;, region=&quot;us-west-2&quot;, policyScope=&quot;production&quot;). The underlying API then handles the necessary sequence of checks, resource creation, and validation steps.</p><p>A developer implementing this would likely interact with the AWS SDK using structured inputs:</p><pre>{<br>  &quot;policy_action&quot;: &quot;provision_compliant_resource&quot;,<br>  &quot;scope&quot;: {<br>    &quot;service&quot;: &quot;rds&quot;,<br>    &quot;environment&quot;: &quot;staging&quot;,<br>    &quot;max_cost_center&quot;: 5000<br>  },<br>  &quot;required_attributes&quot;: [<br>    {&quot;tag&quot;: &quot;OwnerEmail&quot;, &quot;source&quot;: &quot;IAM&quot;},<br>    {&quot;tag&quot;: &quot;ComplianceLevel&quot;, &quot;value&quot;: &quot;PCI-DSS&quot;}<br>  ],<br>  &quot;output_format&quot;: &quot;json&quot;<br>}</pre><p>This structure encapsulates the operational intent, leaving the complex compliance logic — the <em>how</em> — to AWS. The benefit is clear: it standardizes policy enforcement and dramatically reduces the attack surface created by inconsistent developer implementations of security best practices. It shifts governance from being a set of separate, manually maintained guardrails into an integrated service mesh.</p><p>However, this centralization introduces significant technical risk points that warrant skepticism. The entire premise relies on AWS’s ability to correctly interpret and enforce every single custom policy written through the API. If the Policy Decision Point (PDP) itself becomes a single point of failure or exhibits unexpected side effects due to complex attribute interactions, the resulting security gap could be massive. Furthermore, the initial investment in migrating existing, decentralized governance logic into the MCP framework — and retraining engineering teams on this new paradigm — will be substantial.</p><p>The move toward hyper-centralized policy management also raises profound questions about vendor lock-in. By adopting a proprietary, high-level orchestration API like this, organizations are embedding their critical operational policies deep within AWS’s ecosystem. While the promise is simplified governance, the cost of switching to an open-source or multi-cloud governance layer becomes prohibitively high, potentially trapping enterprises that expected policy portability as part of modern cloud infrastructure.</p><p>For enterprise consumers, the real value lies not in the <em>existence</em> of GA, but in the accompanying tooling and documentation for edge cases — specifically, how the MCP Server handles race conditions when multiple policies attempt to modify the same resource concurrently, or what its explicit rate limiting is under extreme load. The success of this server will be judged less by its feature list and more by the robustness of its error handling and audit trails across diverse failure modes.</p><p>This release solidifies AWS’s push toward “policy-as-code” governance as a core operational utility, demanding that modern cloud architecture treats compliance not as an afterthought implemented by security teams, but as an intrinsic, enforceable parameter written into every application’s API contract. The market reaction suggests enterprises view this centralization less as a convenience and more as a necessary evolution to manage the sheer complexity of hybrid multi-cloud deployments</p><h3>References</h3><ul><li><a href="https://aws.amazon.com/mcp-server/">AWS Documentation: MCP Server Architecture Overview</a></li><li><a href="https://www.infoq.com/news/2026/05/aws-mcp-ga/">InfoQ Analysis: Implications of GA Status for Enterprise Architects</a></li><li><a href="https://cloudsecurityalliance.org/reports/policy-enforcement-trends/">Cloud Security Alliance Report on Policy Enforcement Trends</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd25fa985545" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Indexed Claude Code Skills (And What the Install Data Says About AI Coding in 2026)]]></title>
            <link>https://medium.com/@vikramgyawali57/how-i-indexed-2-000-claude-code-skills-and-what-the-install-data-says-about-ai-coding-in-2026-1d6a371c364d?source=rss-5a5de8ab70e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d6a371c364d</guid>
            <dc:creator><![CDATA[Bikram Gyawali]]></dc:creator>
            <pubDate>Sun, 24 May 2026 06:29:05 GMT</pubDate>
            <atom:updated>2026-05-24T06:30:06.064Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EiXt4MsW9pLffPYc.jpg" /></figure><p>When I built an index for the rapidly proliferating skills required to prompt Claude — a structured taxonomy encompassing everything from advanced RAG pipeline design to obscure framework-specific syntax like pydantic schema validation in asynchronous environments—I found that indexing alone was a brutal exercise. The resulting dataset wasn&#39;t just a list of capabilities; it was a live map of technical debt and emergent complexity, revealing that the true bottleneck isn&#39;t model performance, but human ability to architect prompt-chaining systems at scale.</p><p>The current narrative surrounding AI development often focuses on “prompt engineering” as an art form — a mastery of persuasive language designed to coax code from large models. This misses the point entirely. Prompting is merely the input layer; the <em>infrastructure</em> required to manage, validate, and execute those inputs reliably is the frontier. My project involved cataloging over 2,000 discrete coding skills necessary for modern LLM integration — skills that range from optimizing vector database query parameters (e.g., adjusting k nearest neighbors based on cosine distance) to writing specialized guardrail functions using tools like LangChain or LlamaIndex. This index revealed a massive chasm between academic understanding of AI and the messy, brittle reality of production-grade system integration.</p><h3>Beyond Prompts: The Infrastructure of AI Coding Skills Indexing</h3><p>To properly quantify this “AI coding skill infrastructure,” you must move past viewing prompts as mere text inputs. Think of them as API calls requiring complex pre-processing and post-validation pipelines. When a developer submits a request, the system doesn’t just send it to an LLM; it passes it through an orchestration layer that manages tool calling, state tracking, memory retrieval (RAG), and output schema enforcement.</p><p>My index forced me to categorize skills based on their architectural role: <em>Data Retrieval</em>, <em>Schema Validation</em>, <em>Execution Environment Management</em>, and <em>Control Flow</em>. For instance, a simple request like “Write a Python function to fetch the top 5 tweets about quantum computing” requires far more than just good prompting. It demands expertise in OAuth flow management for the Twitter API, robust error handling for rate limits (e.g., implementing an exponential backoff strategy), and ensuring the returned data adheres strictly to a JSON schema suitable for immediate consumption by a front-end framework like React.</p><p>Consider the difference between these two tasks:</p><ol><li><strong>Simple Prompting:</strong> “Write Python code that fetches tweets.”</li><li><strong>Infrastructure Engineering (Indexed Skill):</strong> “Design and implement a secure, asynchronous microservice endpoint /api/v1/tweets that accepts a topic string, utilizes tweepy v4 with an expired bearer token fallback mechanism, implements rate-limit backoff using a Redis cache for state management, and guarantees the output adheres to the following JSON schema: { &#39;topic&#39;: str, &#39;retrieved_at&#39;: datetime, &#39;data&#39;: list[dict] }.&quot;</li></ol><p>The second task is orders of magnitude more complex. The skills are not just “Python” or “API calls”; they are <em>systemic</em> knowledge points — the ability to combine specific libraries, manage external state (Redis), and guarantee compliance under failure conditions. This shift means that the most valuable AI developers in 2026 won’t be prompt artists; they will be systems architects who can reliably stitch together these disparate components.</p><h3>The Data Challenge: Indexing Reliability at Scale</h3><p>The core technical challenge I faced was establishing a consistent, machine-readable definition for “skill.” How do you index the difference between knowing <em>how</em> to use try...except blocks versus knowing <em>when</em> and <em>why</em> to use an asynchronous context manager (async with)? The answer lies in defining failure modes.</p><p>To build this index, I treated each capability as a node in a dependency graph. For example, “Implementing caching” isn’t a single skill; it requires nodes for (1) identifying cache invalidation triggers, (2) selecting the appropriate storage backend (Redis vs. Memcached), and (3) integrating the cache check into the existing request lifecycle — a process that fundamentally alters the code flow.</p><p>This level of detail necessitated deep dives into version control systems and API specifications. A core part of my methodology involved generating test cases for failure states, essentially treating the AI model itself as a component that must be rigorously tested against edge cases. If I were to formalize this testing framework in pseudo-code, it would look something like this:</p><pre>def validate_llm_output(request_params: dict) -&gt; bool:<br>    &quot;&quot;&quot;Validates output schema and execution feasibility.&quot;&quot;&quot;<br>    try:<br>        # 1. Schema validation (Pydantic check)<br>        if not ModelOutputSchema(**llm_response.json):<br>            raise ValueError(&quot;Schema mismatch.&quot;)</pre><pre>        # 2. Execution environment checks (Tool availability)<br>        if not check_tool_status(request_params[&#39;tools&#39;]):<br>            raise RuntimeError(&quot;Missing required tool dependency.&quot;)<br>        <br>        return True<br>    except (ValueError, RuntimeError) as e:<br>        logger.error(f&quot;Validation failed: {e}&quot;)<br>        return False</pre><pre># Example usage: <br># if not validate_llm_output({&quot;tools&quot;: [&quot;redis&quot;, &quot;requests&quot;]}) :<br>#     print(&quot;Cannot proceed; system dependencies are compromised.&quot;)</pre><p>This exercise highlights that the intellectual property value is shifting from the <em>solution</em> (the generated code) to the <em>system blueprint</em> (the reliable, validated architecture).</p><h3>The Skeptical Counterpoint: Complexity Inflation</h3><p>Of course, this focus on infrastructure has detractors. Critics point out that this entire indexing effort risks creating a “complexity inflation” bubble. They argue that by meticulously cataloging every possible failure mode and dependency — every try...except block, every cache invalidation rule—we are simply making the entry barrier to development prohibitively high.</p><p>This critique holds weight. The sheer volume of required knowledge is staggering. A developer proficient in basic Python might feel overwhelmed by the need to simultaneously master vector store indexing, message queue durability guarantees (e.g., Kafka), and advanced schema enforcement libraries. It suggests that instead of democratizing coding through powerful LLMs, we are simply automating the creation of incredibly complex, bespoke middleware layers, requiring a specialized subset of “AI Infrastructure Engineers.”</p><p>Furthermore, reliance on external APIs — whether for vector databases like Pinecone or retrieval mechanisms like LangChain — introduces dependency risk far beyond traditional monolithic codebases. If an indexing service changes its API signature, the entire application fails, regardless of how well-written the prompt was initially. The brittle nature of these rapidly evolving frameworks is a major limitation that index cards cannot solve; it requires institutional stability and standardized protocols that simply don’t exist yet.</p><p>The market response to this complexity — and the resulting skill gap — will define the next cycle of tech talent acquisition. Companies will not hire “AI coders”; they will desperately seek out “LLM System Integrators,” people who can navigate the chasm between theoretical model capability and operational reality, ensuring that code generated in a Jupyter notebook actually runs reliably on Kubernetes at 3 AM when the Redis cache expires unexpectedly. The real cost of AI coding skills isn’t learning how to prompt; it’s understanding failure modes across multiple, interdependent systems.</p><h3>References</h3><ul><li><a href="https://www.example-llm-framework.com/docs">Source Placeholder: Citation for LLM Orchestration Frameworks (e.g., LangChain/LlamaIndex documentation)</a></li><li>[Source Placeholder: Technical Deep Dive on Vector</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d6a371c364d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CAP Theorem: Why You Can’t Have It All in Distributed Systems ‍]]></title>
            <link>https://medium.com/@vikramgyawali57/cap-theorem-why-you-cant-have-it-all-in-distributed-systems-bf9e95caac1d?source=rss-5a5de8ab70e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf9e95caac1d</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[cap]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[backend]]></category>
            <dc:creator><![CDATA[Bikram Gyawali]]></dc:creator>
            <pubDate>Thu, 06 Mar 2025 10:29:05 GMT</pubDate>
            <atom:updated>2025-03-06T10:29:05.608Z</atom:updated>
            <content:encoded><![CDATA[<p>Alright, let’s talk about <strong>CAP theorem</strong> — one of the most fundamental concepts in distributed systems. If you’re dealing with databases, microservices, or anything that scales across multiple machines, you’ve probably heard of it. If not, well, you should. 😉</p><h3>What Even is CAP?</h3><p>CAP stands for <strong>Consistency, Availability, and Partition Tolerance</strong>. It was introduced by Eric Brewer back in 2000, and later proved by computer scientists Seth Gilbert and Nancy Lynch in 2002. The theorem basically says:<br> 👉 <em>In any distributed system, you can only guarantee two out of the three properties: Consistency, Availability, or Partition Tolerance.</em></p><p>Sounds confusing? Let’s break it down. 🧐</p><h3>The Three Pillars of CAP 🚀</h3><p>1️⃣ <strong>Consistency ©</strong> — Every node in the system sees the same data at the same time. If you read from one node and then another, you should get the same result. 💾</p><p>2️⃣ <strong>Availability (A)</strong> — Every request gets a response, even if some nodes are down. No matter what, the system keeps running. ⚡</p><p>3️⃣ <strong>Partition Tolerance (P)</strong> — The system continues to function even if there’s a network failure or a communication break between nodes. 🌍💥</p><h3>The Catch: You Can Only Pick Two! 😵</h3><p>Here’s the tricky part: <strong>a distributed system can’t guarantee all three properties at the same time</strong>. You must sacrifice one. Let’s see how different systems make this trade-off.</p><h3>CA (Consistency + Availability, No Partition Tolerance)</h3><ul><li>Every read gets the latest data (Consistency ✅)</li><li>The system is always available (Availability ✅)</li><li>But it fails if network partitions occur ❌</li><li>Example: Traditional relational databases like <strong>MySQL, PostgreSQL (when running on a single node)</strong></li></ul><h3>CP (Consistency + Partition Tolerance, No Availability)</h3><ul><li>Data is always consistent (Consistency ✅)</li><li>Survives network failures (Partition Tolerance ✅)</li><li>But some requests may fail if a node is down ❌</li><li>Example: <strong>MongoDB, HBase, Zookeeper</strong></li></ul><h3>AP (Availability + Partition Tolerance, No Consistency)</h3><ul><li>Always responsive (Availability ✅)</li><li>Works despite network failures (Partition Tolerance ✅)</li><li>But data might be inconsistent across nodes ❌</li><li>Example: <strong>Cassandra, DynamoDB, CouchDB</strong></li></ul><h3>So, What Should You Choose? 🤔</h3><p>It really depends on your use case. If you need banking transactions, <strong>CP</strong> (Consistency + Partition Tolerance) is a good choice since losing consistency is unacceptable. If you’re building a social media feed, you might prefer <strong>AP</strong> (Availability + Partition Tolerance) since some stale data is okay. If you’re running everything on a single node, <strong>CA</strong> might work.</p><p>Lets see what the global giants are choosing</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*PI7DwdxOKAFkPfUE" /><figcaption>CAP model adapted by tech giants</figcaption></figure><h3>Real-World Impact 🌎</h3><ul><li><strong>Banking apps (CP)</strong>: You don’t want inconsistent balances across servers. 🏦</li><li><strong>E-commerce (AP)</strong>: You want the store to be available even if some minor inconsistencies occur. 🛒</li><li><strong>Social media (AP)</strong>: It’s okay if a post takes a second to sync across devices. 📱</li></ul><h3>Final Thoughts</h3><p>The CAP theorem helps us understand trade-offs in distributed systems. It’s not about picking the best option — it’s about picking what works best <strong>for your system’s needs</strong>. ⚖️</p><p>Want to go deeper? Check out these resources:</p><ul><li>Brewer’s CAP Theorem: <a href="https://www.cs.princeton.edu/courses/archive/fall14/cos418/docs/Brewer2.pdf">https://www.cs.princeton.edu/courses/archive/fall14/cos418/docs/Brewer2.pdf</a></li><li>Gilbert &amp; Lynch’s Proof: <a href="https://groups.csail.mit.edu/tds/papers/Gilbert/Brewer2.pdf">https://groups.csail.mit.edu/tds/papers/Gilbert/Brewer2.pdf</a></li><li>A great blog on CAP: <a href="https://aphyr.com/posts/288-the-cap-theorem">https://aphyr.com/posts/288-the-cap-theorem</a></li></ul><p>Happy scaling! 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf9e95caac1d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding ^and ~in package.json for all Node.js Projects]]></title>
            <link>https://medium.com/@vikramgyawali57/understanding-and-in-package-json-for-all-node-js-projects-cb327ff9e4df?source=rss-5a5de8ab70e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb327ff9e4df</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[versioning]]></category>
            <category><![CDATA[dependency-management]]></category>
            <dc:creator><![CDATA[Bikram Gyawali]]></dc:creator>
            <pubDate>Tue, 25 Feb 2025 05:39:02 GMT</pubDate>
            <atom:updated>2025-02-25T05:39:02.517Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bzTZiUIkEosg2seicKGyWw.png" /><figcaption>semantic versioning of packages</figcaption></figure><p>In any Node.js project, the package.json file plays a crucial role in managing dependencies. If you’ve ever looked at this file, you’ve likely seen dependency versions prefixed with symbols like ^ or ~. These symbols define version ranges, helping developers manage updates efficiently. But what exactly do they mean, and when should you use them? Let&#39;s dive deep into their significance and use cases.</p><h3>The Purpose of Versioning in package.json</h3><p>In package.json, dependencies are typically specified in a format like:</p><pre>&quot;dependencies&quot;: {<br>  &quot;express&quot;: &quot;^4.17.1&quot;,<br>  &quot;mongoose&quot;: &quot;~6.2.1&quot;<br>}</pre><p>These version numbers follow <a href="https://semver.org/">Semantic Versioning (SemVer)</a>, which is structured as:</p><pre>MAJOR.MINOR.PATCH</pre><ul><li><strong>MAJOR</strong>: Breaking changes (incompatible API changes)</li><li><strong>MINOR</strong>: Backward-compatible new features</li><li><strong>PATCH</strong>: Backward-compatible bug fixes</li></ul><p>The ^ and ~ symbols define how npm should handle version updates when installing or updating dependencies.</p><h3>^ (Caret) Operator</h3><p>The caret (^) allows updates to the latest MINOR and PATCH versions, but not MAJOR versions. For example:</p><pre>&quot;express&quot;: &quot;^4.17.1&quot;</pre><p>This means:</p><ul><li>npm can install any version &gt;=4.17.1 but &lt;5.0.0</li><li>Allowed updates: 4.17.2, 4.18.0, etc.</li><li>Not allowed: 5.0.0</li></ul><h3>When to Use ^</h3><p>Use ^ when you want bug fixes and new features but want to avoid breaking changes from major version updates.</p><p>Example:</p><ul><li>If a library follows SemVer correctly, upgrading from 4.17.1 to 4.18.0 should not introduce breaking changes.</li><li>Ideal for stable libraries where new minor versions are safe to use.</li></ul><h3>~ (Tilde) Operator</h3><p>The tilde (~) allows updates to the latest PATCH version but restricts MINOR and MAJOR updates. Example:</p><pre>&quot;mongoose&quot;: &quot;~6.2.1&quot;</pre><p>This means:</p><ul><li>npm can install any version &gt;=6.2.1 but &lt;6.3.0</li><li>Allowed updates: 6.2.2, 6.2.3, etc.</li><li>Not allowed: 6.3.0</li></ul><h3>When to Use ~</h3><p>Use ~ when you want only bug fixes and security patches, avoiding even minor feature updates.</p><p>Example:</p><ul><li>If a library has a history of unstable minor updates, ~ ensures you only get safe patch updates.</li><li>Useful for production environments where stability is more critical than new features.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0VrGvqzMQ7aa1MlFg0py5g.png" /></figure><h3>When Not to Use ^ or ~</h3><ol><li><strong>For Maximum Stability</strong>: If you need absolute consistency, avoid ^ and ~, and specify exact versions.<br>&quot;lodash&quot;: &quot;4.17.21&quot;</li><li><strong>For Critical Dependencies</strong>: In production, if a dependency is critical and updates have caused issues in the past, we prefer fixed versions.</li><li><strong>For Legacy Projects</strong>: Older projects that may break with any update should use exact versions instead of ^ or ~.</li></ol><h3>How to Update Dependencies</h3><ul><li>To install dependencies as listed in package.json</li></ul><p>npm install</p><ul><li>To update dependencies within allowed ranges</li></ul><p>npm update</p><ul><li>To check outdated dependencies:</li></ul><p>npm outdated</p><ul><li>To upgrade to the latest versions:</li></ul><p>npm upgrade</p><ul><li>To force a specific new version:</li></ul><p>npm install express@5.0.0</p><h3>Conclusion</h3><p>Understanding ^ and ~ in package.json helps developers manage dependencies efficiently, balancing stability with updates.</p><ul><li>Use ^ for automatic minor and patch updates.</li><li>Use ~ for only patch updates.</li><li>Use exact versions for maximum stability.</li></ul><p>By following these guidelines, you can ensure that your Node.js projects remain stable while benefiting from updates and improvements in dependencies.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb327ff9e4df" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How SSL, HTTPS, and Certificates Work: A Deep Dive]]></title>
            <link>https://medium.com/@vikramgyawali57/how-ssl-https-and-certificates-work-a-deep-dive-b5666ef6ece9?source=rss-5a5de8ab70e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5666ef6ece9</guid>
            <category><![CDATA[servers]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[tls]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[ssl-certificate]]></category>
            <dc:creator><![CDATA[Bikram Gyawali]]></dc:creator>
            <pubDate>Sat, 25 Jan 2025 14:25:23 GMT</pubDate>
            <atom:updated>2025-01-25T14:26:11.478Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="AI generated image with prompt : Generate a banner image for a blog on SSL/TLS certificates" src="https://cdn-images-1.medium.com/max/1024/0*cVxUOOapouwIatvv" /></figure><p>In the digital era, security is everything. Whether you’re entering your credit card details on an e-commerce site or logging into your email account, protecting sensitive information is critical. This is where <strong>SSL (Secure Sockets Layer)</strong>, <strong>HTTPS (Hypertext Transfer Protocol Secure)</strong>, and <strong>certificates</strong> come into play. But how do they work together to secure your online interactions? Let’s unravel the mystery step by step.</p><h3>1. What is SSL?</h3><h3>Definition</h3><p>SSL (Secure Sockets Layer) is a cryptographic protocol designed to secure communication over the internet. It ensures that any data exchanged between a user’s browser and a web server is encrypted and safe from eavesdroppers or hackers.</p><h3>Purpose of SSL</h3><ul><li><strong>Encryption</strong>: Converts readable data into a secure, scrambled format.</li><li><strong>Authentication</strong>: Confirms the identity of the server.</li><li><strong>Data Integrity</strong>: Ensures data is not altered during transmission.</li></ul><h3>Evolution</h3><p>SSL was developed by <strong>Netscape</strong> in the mid-1990s.</p><ul><li><strong>SSL 1.0 (1995)</strong>: Never released due to security flaws.</li><li><strong>SSL 2.0 (1995)</strong>: Introduced publicly but deprecated in 2011 due to vulnerabilities.</li><li><strong>SSL 3.0 (1996)</strong>: More secure but replaced by TLS (Transport Layer Security) in 1999.</li></ul><blockquote><em>Today, SSL is commonly referred to as </em><strong><em>TLS</em></strong><em> (the successor of SSL), but the term “SSL” is still widely used.</em></blockquote><h3>Example</h3><p>Imagine sending a letter. SSL ensures the letter is placed in a locked envelope, and only the recipient has the key.</p><h4>Additional Reading:</h4><ul><li>Netscape SSL history: <a href="https://en.wikipedia.org/wiki/Transport_Layer_Security">Wikipedia</a></li></ul><h3>2. What is HTTPS?</h3><h3>Definition</h3><p><strong>HTTPS</strong> (Hypertext Transfer Protocol Secure) is the secure version of HTTP, which is the protocol your browser uses to communicate with websites. By adding SSL/TLS, HTTPS ensures data exchanged between your browser and the website is encrypted.</p><h3>Why HTTPS Matters</h3><ul><li><strong>Protects Privacy</strong>: Prevents hackers from snooping on sensitive information (e.g., passwords, payment details).</li><li><strong>Builds Trust</strong>: Users see a padlock icon in the address bar, signaling the website is secure.</li><li><strong>Improves SEO</strong>: Search engines like Google prioritize HTTPS websites in search rankings.</li></ul><h3>Timeline</h3><ul><li><strong>1994</strong>: Introduction of HTTP by Tim Berners-Lee.</li><li><strong>1995</strong>: HTTPS introduced by Netscape.</li><li><strong>2014</strong>: Google announces HTTPS as a ranking signal, encouraging adoption.</li></ul><h4>Fun Fact</h4><p>Before HTTPS became the standard, most websites transmitted data in plain text, making it easy for hackers to intercept.</p><h4>Additional Reading:</h4><ul><li>Google’s HTTPS push: <a href="https://webmasters.googleblog.com/2014/08/https-as-ranking-signal.html">Google Webmaster Blog</a></li></ul><h3>3. What are SSL Certificates?</h3><h3>Definition</h3><p>An <strong>SSL certificate</strong> is a digital document that verifies the identity of a website and enables encrypted communication. It’s issued by a trusted entity called a <strong>Certificate Authority (CA)</strong>.</p><h3>Components of an SSL Certificate</h3><ul><li><strong>Domain Name</strong>: Specifies the domain the certificate is issued for.</li><li><strong>Issuer Information</strong>: Details about the CA.</li><li><strong>Public Key</strong>: Used for encryption.</li><li><strong>Validity Period</strong>: Expiration date of the certificate.</li></ul><h3>Types of Certificates</h3><ol><li><strong>Domain Validation (DV)</strong>: Verifies domain ownership. Ideal for small websites.</li><li><strong>Organization Validation (OV)</strong>: Verifies the organization’s identity.</li><li><strong>Extended Validation (EV)</strong>: Displays the company name in the browser. Used by banks and e-commerce sites.</li></ol><h3>Certificate Authorities (CAs)</h3><p>CAs are trusted organizations responsible for issuing SSL certificates. Examples include:</p><ul><li>Let’s Encrypt (free certificates, launched in 2016)</li><li>DigiCert</li><li>GoDaddy</li></ul><h4>Fun Fact</h4><p>The first SSL certificate was issued by Netscape in the mid-1990s to secure its browser-server communication.</p><h4>Additional Reading:</h4><ul><li>History of Let’s Encrypt: <a href="https://letsencrypt.org/2016/">Let’s Encrypt Blog</a></li></ul><h3>4. How SSL, HTTPS, and Certificates Work Together</h3><p>Here’s a step-by-step breakdown of what happens when you visit an HTTPS website:</p><h3>Step 1: Browser Connects to the Server</h3><p>Your browser sends a request to a website (e.g., https://example.com) and asks to establish a secure connection.</p><h3>Step 2: Server Sends Its SSL Certificate</h3><p>The server responds with its SSL certificate to prove its identity.</p><h3>Step 3: Browser Verifies the Certificate</h3><p>Your browser checks if:</p><ul><li>The certificate is valid (not expired).</li><li>It matches the domain name.</li><li>It’s issued by a trusted CA.</li></ul><h3>Step 4: Handshake and Encryption</h3><p>If the certificate is valid, the browser and server perform a <strong>TLS handshake</strong>:</p><ul><li>They exchange public and private keys to establish a secure connection.</li><li>Once verified, they use a <strong>shared secret key</strong> for faster communication.</li></ul><h3>Step 5: Secure Communication Begins</h3><p>All data exchanged between the browser and server is now encrypted.</p><h4>Example</h4><p>When you enter your credit card details on an e-commerce site, HTTPS ensures the information is encrypted and cannot be intercepted.</p><h3>5. Why SSL/TLS is Secure</h3><h3>Encryption Types</h3><ol><li><strong>Asymmetric Encryption</strong>: Uses a public/private key pair. Slower but secure.</li><li><strong>Symmetric Encryption</strong>: Uses a single shared key for both encryption and decryption. Faster for bulk data.</li></ol><h3>Challenges and Solutions</h3><ul><li><strong>Expired Certificates</strong>: Renew them periodically.</li><li><strong>Untrusted Certificates</strong>: Use a certificate from a trusted CA.</li><li><strong>Man-in-the-Middle Attacks</strong>: HTTPS prevents attackers from intercepting or altering data.</li></ul><h3>6. Common Questions</h3><h3>Why is HTTPS Important?</h3><ul><li>Prevents <strong>man-in-the-middle attacks</strong>.</li><li>Protects user privacy.</li><li>Encourages trust and loyalty.</li></ul><h3>What Does the Padlock Icon Mean?</h3><p>The padlock icon in the browser address bar indicates that:</p><ol><li>The website is using HTTPS.</li><li>It has a valid SSL certificate.</li></ol><h3>Can SSL Be Hacked?</h3><p>When implemented correctly, SSL/TLS is highly secure. However:</p><ul><li>Outdated protocols like SSL 3.0 are vulnerable.</li><li>Poor configurations can introduce risks.</li></ul><h3>7. Timeline of Key Events</h3><p>Year Event 1994 HTTP introduced by Tim Berners-Lee 1995 SSL 2.0 and HTTPS launched by Netscape 1999 TLS (SSL successor) introduced 2014 Google prioritizes HTTPS in search rankings 2016 Let’s Encrypt starts offering free SSL certificates</p><h3>8. Final Thoughts</h3><p>The combination of SSL, HTTPS, and certificates is the backbone of secure online communication. They ensure that sensitive data is encrypted, authenticated, and protected from cyber threats. As the internet evolves, so do these technologies, making our digital experiences safer.</p><h4>Key Takeaway</h4><p>Always look for the padlock icon and “HTTPS” in the URL bar before entering sensitive information on a website.</p><h4>Further Reading:</h4><ul><li><a href="https://en.wikipedia.org/wiki/Transport_Layer_Security">Transport Layer Security</a></li><li><a href="https://letsencrypt.org/faq/">Let’s Encrypt FAQ</a></li><li><a href="https://webmasters.googleblog.com/2014/08/https-as-ranking-signal.html">Google Webmaster HTTPS Guide</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5666ef6ece9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Node.js Streams: A Comprehensive Guide]]></title>
            <link>https://medium.com/@vikramgyawali57/understanding-node-js-streams-a-comprehensive-guide-36abb0a8f78e?source=rss-5a5de8ab70e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/36abb0a8f78e</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[technical-interview]]></category>
            <dc:creator><![CDATA[Bikram Gyawali]]></dc:creator>
            <pubDate>Sat, 14 Dec 2024 14:23:56 GMT</pubDate>
            <atom:updated>2024-12-14T14:23:56.823Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LsZjmoo56idsRLnOOnu9kQ.png" /><figcaption>nodejs streams thumbnail from net ninja youtube channel tutorials</figcaption></figure><p>Node.js streams are a powerful and efficient way to handle data, especially when dealing with large amounts. They allow you to work with data in chunks, enabling efficient memory usage and faster processing. In this blog, we will explore what streams are, how they work, real-world use cases, and common interview questions related to Node.js streams. By the end, you’ll have a solid grasp of streams and their practical applications.</p><h3>What Are Streams in Node.js?</h3><p>Streams are objects that allow you to read data from a source or write data to a destination continuously. Instead of reading or writing the entire data set simultaneously, streams process it piece by piece, which is especially useful for large files or data sets. Streams are built on the <strong>EventEmitter</strong> API, which means they emit events like data, end, error, and more.</p><h4>Types of Streams in Node.js:</h4><ol><li><strong>Readable Streams</strong>: Used to read data from a source.</li><li><strong>Writable Streams</strong>: Used to write data to a destination.</li><li><strong>Duplex Streams</strong>: Can be both readable and writable.</li><li><strong>Transform Streams</strong>: A duplex stream where the output is computed based on the input.</li></ol><h3>How Do Streams Work?</h3><p>Streams work in two modes:</p><ol><li><strong>Flowing Mode</strong></li></ol><ul><li>Data is read automatically and emitted as soon as it’s available.</li><li>Listeners are used to handle data and events.</li></ul><p>2. Pause Mode</p><ul><li>Data must be explicitly requested using methods like stream.read()</li></ul><p>You can switch between modes using stream.pause() and stream.resume().</p><h4>Key Events in Streams</h4><ul><li><strong>data</strong>: Emitted when data is available to read.</li><li><strong>end</strong>: Emitted when there is no more data to read.</li><li><strong>error</strong>: Emitted if there is an error during the streaming process.</li><li><strong>close</strong>: Emitted when the stream is closed.</li></ul><h4>Basic Example of a Readable Stream:</h4><pre>const fs = require(&#39;fs&#39;);<br><br>const readableStream = fs.createReadStream(&#39;largeFile.txt&#39;);<br><br>readableStream.on(&#39;data&#39;, (chunk) =&gt; {<br>  console.log(`Received ${chunk.length} bytes of data.`);<br>});<br><br>readableStream.on(&#39;end&#39;, () =&gt; {<br>  console.log(&#39;No more data to read.&#39;);<br>});<br><br>readableStream.on(&#39;error&#39;, (err) =&gt; {<br>  console.error(&#39;An error occurred:&#39;, err.message);<br>});</pre><h3>Real-World Use Cases of Node.js Streams</h3><h4>1. File Handling</h4><p>Streams are ideal for reading or writing large files without loading the entire file into memory.</p><p>Example: Copying a large file using streams:</p><pre>const fs = require(&#39;fs&#39;);<br><br>const source = fs.createReadStream(&#39;source.txt&#39;);<br>const destination = fs.createWriteStream(&#39;destination.txt&#39;);<br><br>source.pipe(destination);<br><br>source.on(&#39;end&#39;, () =&gt; {<br>  console.log(&#39;File copied successfully.&#39;);<br>});</pre><h4>2. HTTP Requests and Responses</h4><p>In Node.js, HTTP requests and responses are implemented as streams. This allows you to process data as it arrives instead of waiting for the entire request or response to complete.</p><p>Example: Sending a large file to the client:</p><pre>const http = require(&#39;http&#39;);<br>const fs = require(&#39;fs&#39;);<br><br>http.createServer((req, res) =&gt; {<br>  const readableStream = fs.createReadStream(&#39;largeFile.txt&#39;);<br>  readableStream.pipe(res);<br>}).listen(3000);</pre><h4>3. Data Transformation</h4><p>Transform streams are used for modifying or processing data on the fly, such as compressing or encrypting data.</p><p>Example: Compressing a file using zlib:</p><pre>const fs = require(&#39;fs&#39;);<br>const zlib = require(&#39;zlib&#39;)<br><br>const gzip = zlib.createGzip();<br>const source = fs.createReadStream(&#39;file.txt&#39;);<br>const destination = fs.createWriteStream(&#39;file.txt.gz&#39;);<br><br>source.pipe(gzip).pipe(destination);</pre><h4>4. Real-Time Data Processing</h4><p>Streams are used for real-time applications, such as chat applications, live video streaming, or IoT data processing.</p><p>Example: Processing live data:</p><pre>const { Transform } = require(&#39;stream&#39;);<br><br>const transformStream = new Transform({<br>  transform(chunk, encoding, callback) {<br>    this.push(chunk.toString().toUpperCase());<br>    callback();<br>  }<br>});<br>process.stdin.pipe(transformStream).pipe(process.stdout);</pre><h3>Common Interview Questions on Node.js Streams</h3><ol><li><strong>What are streams in Node.js? Explain their types.</strong></li></ol><ul><li>Streams are objects that enable reading or writing data continuously.</li><li>Types: Readable, Writable, Duplex, Transform.</li></ul><p><strong>2. How do you handle backpressure in streams?</strong></p><ul><li>Backpressure occurs when the writable stream cannot process data as fast as the readable stream provides it</li><li>It is managed using stream.pipe() or by manually pausing and resuming the readable stream</li></ul><p><strong>3. What is the difference between </strong><strong>fs.createReadStream and </strong><strong>fs.readFile?</strong></p><ul><li>fs.createReadStream reads data in chunks and streams it, suitable for large files.</li><li>fs.readFile reads the entire file into memory, which can cause performance issues for large files.</li></ul><p><strong>4. What is the purpose of the </strong><strong>pipe() method?</strong></p><ul><li>pipe() connects a readable stream to a writable stream and automatically handles backpressure.</li></ul><p><strong>5. What are Transform streams? Can you provide an example?</strong></p><ul><li>Transform streams are duplex streams that can modify data while reading and writing.</li><li>Example: Compressing data using zlib (as shown above).</li></ul><p><strong>6. How do you switch a stream from flowing mode to paused mode?</strong></p><ul><li>Use stream.pause() to switch to paused mode and stream.resume() to return to flowing mode.</li></ul><p><strong>7. What are some practical applications of streams?</strong></p><ul><li>Handling large files, streaming video/audio, real-time data processing, HTTP requests and responses</li></ul><h3>Conclusion</h3><p>Node.js streams are a fundamental concept for building efficient and scalable applications. They shine in scenarios involving large data sets, real-time processing, and file handling. By mastering streams, you can build high-performance applications while optimizing resource usage.</p><p>For interviews, understanding the theoretical concepts, practical applications, and common challenges like backpressure will set you apart as a skilled developer. Experiment with streams in your projects to deepen your understanding and showcase your expertise.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36abb0a8f78e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mongodb array operation queries only few engineers adapt all of them]]></title>
            <link>https://medium.com/@vikramgyawali57/mongodb-array-operation-queries-only-few-engineers-adapt-all-of-them-ee8521b42389?source=rss-5a5de8ab70e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee8521b42389</guid>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[arrays]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[nosql]]></category>
            <dc:creator><![CDATA[Bikram Gyawali]]></dc:creator>
            <pubDate>Sat, 30 Nov 2024 16:27:06 GMT</pubDate>
            <atom:updated>2024-11-30T16:27:06.472Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/920/0*yYegBpkmZftIbVpR.jpg" /></figure><p>MongoDB’s array operations are powerful tools for managing and querying data in collections. In this blog, we’ll explore <strong>20 different use cases</strong> for working with arrays using a bank account model.</p><h3>Bank Account Model</h3><p>Each document in the accounts collection represents a bank account with transactions stored as an array:</p><pre>{<br>  &quot;_id&quot;: 1,<br>  &quot;accountNumber&quot;: &quot;123456789&quot;,<br>  &quot;accountHolder&quot;: &quot;John Doe&quot;,<br>  &quot;transactions&quot;: [<br>    { &quot;type&quot;: &quot;deposit&quot;, &quot;amount&quot;: 1000, &quot;date&quot;: &quot;2024-01-01&quot; },<br>    { &quot;type&quot;: &quot;withdrawal&quot;, &quot;amount&quot;: 200, &quot;date&quot;: &quot;2024-01-15&quot; },<br>    { &quot;type&quot;: &quot;deposit&quot;, &quot;amount&quot;: 500, &quot;date&quot;: &quot;2024-01-20&quot; }<br>  ],<br>  &quot;beneficiaries&quot;: [&quot;Alice&quot;, &quot;Bob&quot;, &quot;Charlie&quot;]<br>}</pre><h3>1. Match Documents with Specific Array Element</h3><p><strong>Query</strong>: Find accounts with a deposit transaction of exactly $1000.</p><pre>db.accounts.find({ &quot;transactions.amount&quot;: 1000 })</pre><p><strong>Result</strong>:</p><pre>[<br>  {<br>    &quot;_id&quot;: 1,<br>    &quot;accountNumber&quot;: &quot;123456789&quot;,<br>    ...<br>  }<br>]</pre><h3>2. Match Documents Where an Array Contains a Specific Value</h3><p><strong>Query</strong>: Find accounts where “Alice” is a beneficiary.</p><pre>db.accounts.find({ beneficiaries: &quot;Alice&quot; })</pre><p><strong>Result</strong>:</p><pre>[<br>  {<br>    &quot;_id&quot;: 1,<br>    ...<br>  }<br>]</pre><h3>3. Filter Documents with Multiple Array Elements Matching</h3><p><strong>Query</strong>: Find accounts with both “Alice” and “Charlie” as beneficiaries.</p><pre>db.accounts.find({ beneficiaries: { $all: [&quot;Alice&quot;, &quot;Charlie&quot;] } })</pre><p><strong>Result</strong>:</p><pre>[<br>  {<br>    &quot;_id&quot;: 1,<br>    ...<br>  }<br>]</pre><h3>4. Retrieve Only Matching Array Elements</h3><p><strong>Query</strong>: Fetch only the deposit transactions.</p><pre>db.accounts.find(<br>  { &quot;transactions.type&quot;: &quot;deposit&quot; },<br>  { &quot;transactions.$&quot;: 1 }<br>)</pre><p><strong>Result</strong>:</p><pre>[<br>  {<br>    &quot;_id&quot;: 1,<br>    &quot;transactions&quot;: [<br>      { &quot;type&quot;: &quot;deposit&quot;, &quot;amount&quot;: 1000, &quot;date&quot;: &quot;2024-01-01&quot; }<br>    ]<br>  }<br>]</pre><h3>5. Add Elements to an Array</h3><p><strong>Query</strong>: Add a new beneficiary “David” to account 1.</p><pre>db.accounts.updateOne(<br>  { _id: 1 },<br>  { $push: { beneficiaries: &quot;David&quot; } }<br>)</pre><p><strong>Result</strong>:</p><pre>{ &quot;beneficiaries&quot;: [&quot;Alice&quot;, &quot;Bob&quot;, &quot;Charlie&quot;, &quot;David&quot;] }</pre><h3>6. Remove Specific Elements from an Array</h3><p><strong>Query</strong>: Remove “Bob” from the beneficiaries.</p><pre>db.accounts.updateOne(<br>  { _id: 1 },<br>  { $pull: { beneficiaries: &quot;Bob&quot; } }<br>)</pre><p><strong>Result</strong>:</p><pre>{ &quot;beneficiaries&quot;: [&quot;Alice&quot;, &quot;Charlie&quot;, &quot;David&quot;] }</pre><h3>7. Update Specific Array Element</h3><p><strong>Query</strong>: Change the amount of the first deposit to $1200.</p><pre>db.accounts.updateOne(<br>  { _id: 1, &quot;transactions.type&quot;: &quot;deposit&quot; },<br>  { $set: { &quot;transactions.$.amount&quot;: 1200 } }<br>)</pre><h3>8. Append Multiple Elements</h3><p><strong>Query</strong>: Add multiple beneficiaries.</p><pre>db.accounts.updateOne(<br>  { _id: 1 },<br>  { $push: { beneficiaries: { $each: [&quot;Eve&quot;, &quot;Frank&quot;] } } }<br>)</pre><h3>9. Remove Multiple Elements</h3><p><strong>Query</strong>: Remove “Alice” and “Eve” from beneficiaries.</p><pre>db.accounts.updateOne(<br>  { _id: 1 },<br>  { $pull: { beneficiaries: { $in: [&quot;Alice&quot;, &quot;Eve&quot;] } } }<br>)</pre><h3>10. Query Arrays Based on Size</h3><p><strong>Query</strong>: Find accounts with exactly 3 beneficiaries.</p><pre>db.accounts.find({ beneficiaries: { $size: 3 } })</pre><h3>11. Slice Arrays in Projection</h3><p><strong>Query</strong>: Retrieve only the first 2 transactions.</p><pre>db.accounts.find(<br>  { _id: 1 },<br>  { transactions: { $slice: 2 } }<br>)</pre><h3>12. Match Nested Objects in Arrays</h3><p><strong>Query</strong>: Find accounts with a withdrawal of $200.</p><pre>db.accounts.find({<br>  transactions: { $elemMatch: { type: &quot;withdrawal&quot;, amount: 200 } }<br>})</pre><h3>13. Increment Values in Nested Arrays</h3><p><strong>Query</strong>: Increase the amount of all deposits by $100.</p><pre>db.accounts.updateMany(<br>  { &quot;transactions.type&quot;: &quot;deposit&quot; },<br>  { $inc: { &quot;transactions.$[elem].amount&quot;: 100 } },<br>  { arrayFilters: [{ &quot;elem.type&quot;: &quot;deposit&quot; }] }<br>)</pre><h3>14. Sort Array Elements</h3><p><strong>Query</strong>: Sort transactions by amount in descending order.</p><pre>db.accounts.updateOne(<br>  { _id: 1 },<br>  { $push: { transactions: { $each: [], $sort: { amount: -1 } } } }<br>)</pre><h3>15. Remove Duplicate Elements</h3><p><strong>Query</strong>: Deduplicate the beneficiaries array.</p><pre>db.accounts.updateOne(<br>  { _id: 1 },<br>  { $addToSet: { beneficiaries: { $each: [&quot;Alice&quot;, &quot;Charlie&quot;] } } }<br>)</pre><h3>16. Project Specific Fields from Array Elements</h3><p><strong>Query</strong>: Retrieve only the transaction type and amount.</p><pre>db.accounts.find(<br>  { _id: 1 },<br>  { &quot;transactions.type&quot;: 1, &quot;transactions.amount&quot;: 1 }<br>)</pre><h3>17. Find Elements Not Matching a Condition</h3><p><strong>Query</strong>: Find accounts where no transactions are of type “loan.”</p><pre>db.accounts.find({<br>  transactions: { $not: { $elemMatch: { type: &quot;loan&quot; } } }<br>})</pre><h3>18. Push Only If Not Present</h3><p><strong>Query</strong>: Add “Alice” to beneficiaries if not already present.</p><pre>db.accounts.updateOne(<br>  { _id: 1 },<br>  { $addToSet: { beneficiaries: &quot;Alice&quot; } }<br>)</pre><h3>19. Remove Last Element</h3><p><strong>Query</strong>: Remove the last beneficiary.</p><pre>db.accounts.updateOne(<br>  { _id: 1 },<br>  { $pop: { beneficiaries: 1 } }<br>)</pre><h3>20. Aggregate Array Elements</h3><p><strong>Query</strong>: Calculate the total deposit amount.</p><pre>db.accounts.aggregate([<br>  { $match: { _id: 1 } },<br>  { $unwind: &quot;$transactions&quot; },<br>  { $match: { &quot;transactions.type&quot;: &quot;deposit&quot; } },<br>  { $group: { _id: null, total: { $sum: &quot;$transactions.amount&quot; } } }<br>])</pre><p><strong>Result</strong>:</p><pre>{ &quot;total&quot;: 1700 }</pre><h3>Conclusion</h3><p>MongoDB’s array operations enable efficient and dynamic handling of complex data. By mastering these operations, you can unlock MongoDB’s full potential for your applications, such as managing bank account transactions, customer data, or other real-world use cases.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee8521b42389" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 MongoDB Indexing Strategies for Scalable System Design]]></title>
            <link>https://towardsdev.com/10-mongodb-indexing-strategies-for-scalable-system-design-dac6d6623104?source=rss-5a5de8ab70e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/dac6d6623104</guid>
            <category><![CDATA[indexing]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[scalable]]></category>
            <category><![CDATA[mongodb]]></category>
            <dc:creator><![CDATA[Bikram Gyawali]]></dc:creator>
            <pubDate>Tue, 26 Nov 2024 11:49:26 GMT</pubDate>
            <atom:updated>2024-11-28T12:56:01.064Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lecG3QUWCszdvQ-w0ZbwgQ.png" /><figcaption>indexing with mongodb</figcaption></figure><p>MongoDB indexes are crucial for optimizing query performance and enabling efficient data retrieval. When designing a scalable system, leveraging the right indexing strategies can make all the difference in performance and resource utilization. In this blog, we will explore <strong>10 different indexing strategies</strong>, their use cases, and how to apply them in real-world systems.</p><h3>1. Single-Field Index</h3><h3>Description</h3><p>A single-field index is created on a single field in a document. It is the most basic type of index and is commonly used for fields that are frequently queried.</p><h3>Use Case</h3><p>E-commerce platforms often query products by productId or category.</p><h3>Example</h3><pre>db.products.createIndex({ productId: 1 });</pre><h3>Real-World Application</h3><ul><li><strong>Online stores</strong>: Speed up searches for products by ID.</li><li><strong>User management systems</strong>: Quickly retrieve user profiles by userId.</li></ul><h3>2. Compound Index</h3><h3>Description</h3><p>A compound index is created on multiple fields in a specified order, allowing efficient querying on combinations of fields.</p><h3>Use Case</h3><p>Social media platforms querying posts by userId and createdAt for a user’s activity feed.</p><h3>Example</h3><pre>db.posts.createIndex({ userId: 1, createdAt: -1 });</pre><h3>Real-World Application</h3><ul><li><strong>Social media feeds</strong>: Retrieve the most recent posts of a user.</li><li><strong>Analytics systems</strong>: Query transactions by customerId and date.</li></ul><h3>3. Text Index</h3><h3>Description</h3><p>Text indexes enable text search on string fields and support queries like $text.</p><h3>Use Case</h3><p>Search functionality for blog platforms, allowing users to search articles by keywords.</p><h3>Example</h3><pre>db.articles.createIndex({ content: &quot;text&quot; });</pre><h3>Real-World Application</h3><ul><li><strong>Content management systems</strong>: Enable keyword searches for articles or documents.</li><li><strong>E-commerce</strong>: Implement search functionality for product descriptions.</li></ul><h3>4. Geospatial Index</h3><h3>Description</h3><p>Geospatial indexes support queries related to location data, such as finding points near a specific coordinate.</p><h3>Use Case</h3><p>Ride-hailing apps locating nearby drivers for a customer.</p><h3>Example</h3><pre>db.locations.createIndex({ location: &quot;2dsphere&quot; });</pre><h3>Real-World Application</h3><ul><li><strong>Delivery systems</strong>: Find restaurants or warehouses near a delivery address.</li><li><strong>Travel apps</strong>: Locate hotels or attractions near a user.</li></ul><h3>5. Unique Index</h3><h3>Description</h3><p>Unique indexes ensure that the indexed field(s) contain unique values.</p><h3>Use Case</h3><p>Prevent duplicate entries for user registration by enforcing unique emails.</p><h3>Example</h3><pre>db.users.createIndex({ email: 1 }, { unique: true });</pre><h3>Real-World Application</h3><ul><li><strong>Authentication systems</strong>: Ensure unique usernames or emails.</li><li><strong>Inventory systems</strong>: Prevent duplicate product SKUs.</li></ul><h3>6. TTL (Time-to-Live) Index</h3><h3>Description</h3><p>TTL indexes automatically delete documents after a specified time.</p><h3>Use Case</h3><p>Temporary sessions or logs that expire after a set period.</p><h3>Example</h3><pre>db.sessions.createIndex({ createdAt: 1 }, { expireAfterSeconds: 3600 });</pre><h3>Real-World Application</h3><ul><li><strong>Authentication</strong>: Expire user sessions after 1 hour.</li><li><strong>IoT systems</strong>: Remove outdated sensor data periodically.</li></ul><h3>7. Partial Index</h3><h3>Description</h3><p>Partial indexes index only the documents that meet a specified condition, reducing index size.</p><h3>Use Case</h3><p>Index active users only, excluding inactive accounts.</p><h3>Example</h3><pre>db.users.createIndex({ lastLogin: 1 }, { partialFilterExpression: { status: &quot;active&quot; } });</pre><h3>Real-World Application</h3><ul><li><strong>CRM systems</strong>: Optimize queries for active customers.</li><li><strong>Task management apps</strong>: Index tasks that are incomplete.</li></ul><h3>8. Sparse Index</h3><h3>Description</h3><p>Sparse indexes include only documents that have the indexed field, saving space.</p><h3>Use Case</h3><p>Index a field (optionalField) that is not present in all documents.</p><h3>Example</h3><pre>db.collection.createIndex({ optionalField: 1 }, { sparse: true });</pre><h3>Real-World Application</h3><ul><li><strong>Flexible schemas</strong>: Query optional fields without indexing every document.</li><li><strong>Metadata systems</strong>: Index fields present in only certain document types.</li></ul><h3>9. Wildcard Index</h3><h3>Description</h3><p>Wildcard indexes automatically index all fields in a document or fields that match a pattern.</p><h3>Use Case</h3><p>Dynamic schemas where fields vary widely between documents.</p><h3>Example</h3><pre>db.collection.createIndex({ &quot;$**&quot;: 1 });</pre><h3>Real-World Application</h3><ul><li><strong>NoSQL systems</strong>: Support complex queries in unstructured or semi-structured datasets.</li><li><strong>Data lakes</strong>: Quickly query fields in JSON documents with varied structures.</li></ul><h3>10. Hidden Index</h3><h3>Description</h3><p>Hidden indexes exist but are not considered by the query planner unless explicitly hinted. They are useful for testing index performance.</p><h3>Use Case</h3><p>Experiment with new indexes without affecting existing queries.</p><h3>Example</h3><pre>db.orders.createIndex({ customerId: 1 }, { hidden: true });</pre><h3>Real-World Application</h3><ul><li><strong>Performance optimization</strong>: Test new index configurations before deployment.</li><li><strong>Legacy systems</strong>: Safely experiment without disrupting current query performance.</li></ul><h3>Designing Scalable Systems with MongoDB Indexes</h3><p>Here’s how to apply these strategies:</p><ol><li><strong>Analyze Query Patterns</strong>: Use MongoDB’s query profiler to identify which fields are frequently queried.</li><li><strong>Prioritize High-Cardinality Fields</strong>: Index fields with a wide range of unique values (e.g., userId).</li><li><strong>Combine Indexes</strong>: Use compound indexes for queries involving multiple fields.</li><li><strong>Monitor Performance</strong>: Regularly evaluate index usage with db.collection.getIndexes() and explain() methods.</li><li><strong>Leverage Automation</strong>: Use TTL indexes for automatic cleanup of stale data.</li></ol><h3>References</h3><ol><li><a href="https://www.mongodb.com/docs/manual/indexes/">MongoDB Indexing Documentation</a></li><li><a href="https://www.mongodb.com/docs/manual/administration/optimization/">MongoDB Performance Best Practices</a></li><li><a href="https://developer.mongodb.com/">Building Scalable Systems</a></li></ol><p>By carefully selecting and implementing these indexing strategies, you can design scalable systems that handle growing datasets and complex queries with ease.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dac6d6623104" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/10-mongodb-indexing-strategies-for-scalable-system-design-dac6d6623104">10 MongoDB Indexing Strategies for Scalable System Design</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>