Enterprises Don’t Have a Data Problem — They Have an Access Problem

Why Secure, Agentic Access Is the New Enterprise Interface

The Missing Link: Accessing Enterprise Knowledge

Not lack of information, but access to information, is the biggest productivity problem in modern enterprises.

Here’s a scenario that plays out in offices everywhere.

• A project manager is looking for information on the technology stack for a client’s project.
• A new hire wants to understand the company’s leave policy.
• A sales lead is searching for last quarter’s performance metrics.

The information exists. Somewhere. Buried in a PDF sitting in someone’s Google Drive. Locked away in a PostgreSQL database that only the IT team can query, or fragmented across Confluence pages, SharePoint folders, and legacy systems that haven’t been updated in months.

Every enterprise sits on a goldmine of internal data, such as HR policies, project documentation, technical specifications, and financial reports. But that data lives in silos: PDFs, DOCX files, HTML pages, relational databases like MySQL and SQL Server, and NoSQL systems like MongoDB.

When someone needs an answer, they usually face two poor options.

Option one: Spend hours hunting through folders and documents, hoping the version is current.

Option two: Raise a ticket with IT and wait for someone to write and validate a SQL query.

Neither of these scales.

For non-technical teams, this friction is constant. Why should a marketing manager understand database schemas? Why should an HR specialist navigate complex folder hierarchies to answer routine questions?

This isn’t just inefficiency; it’s a fundamental disconnect between the data organizations already have and their ability to actually use it.

This is where the Enterprise Agent comes in.

From Search to Conversation and Then to Action

Traditional enterprise search tools stop at retrieval. They help you find documents, but do not understand or use the information inside them.

Large Language Models changed expectations. Employees now expect to ask questions in natural language and get direct, contextual answers. But deploying LLMs naively inside an enterprise quickly exposes real limitations:

• Hallucinated or incomplete answers
• Outdated context
• No grounding in internal systems
• Serious security and compliance risks

Retrieval-Augmented Generation (RAG) addresses part of this problem by grounding responses in enterprise data. But RAG alone is not enough. Enterprises need orchestration.

• They need governance.
• They need agents that can reason across systems — not just retrieve text.

How Quantrium Is Solving the Access Problem

At Quantrium, we’re addressing this challenge in a phased, pragmatic way, starting where enterprises feel the pain most and scaling systematically.

At the core of our approach is a simple idea: enterprise knowledge should be unified semantically, not physically.

The Enterprise Agent platform is built around three foundational layers:

1. Data Ingestion and Integration
2. Retrieval and Reasoning with RAG
3. Continuous Feedback and Improvement

Structured Data Without the Bottlenecks

Phase One delivers production-grade access to structured data stored in SQL databases, specifically MySQL, PostgreSQL, and Microsoft SQL Server. This is where critical business data often lives and where access is typically restricted to a small technical group.

Our goal is straightforward:

“Make structured data accessible to non-technical users without compromising security, control, or accountability.”

The Enterprise Agent platform securely connects to existing enterprise data sources and creates a unified semantic representation of enterprise knowledge. Raw data never leaves the enterprise boundary. Only schemas, metadata, and contextual signals are embedded for retrieval.

Phase One emphasizes secure access, schema-aware query generation, full observability, and simple containerized deployment within the enterprise’s environment.

How It Works Under the Hood

The system uses a Retrieval-Augmented Generation (RAG) pipeline designed for schema-aware reasoning. During ingestion:

• Database schemas (tables, columns, relationships, metadata) are read
• This information is chunked and embedded
• Embeddings are stored in Qdrant

No raw data is exposed externally; only contextual representations required for retrieval are accessed.

When a user submits a natural-language question:

1. The query is embedded
2. Relevant schema context is retrieved
3. The LLM generates a SQL query grounded in enterprise-specific schemas and constraints

Users can either:

• View and download results directly through the application, or
• Receive the generated SQL query itself, for teams that prefer manual execution

A note on accuracy

Like any AI-assisted system, generated queries require review. Complex joins or nuanced business logic may need iteration. The interface is intentionally designed to support verification rather than blind execution.

Either way, the dependency on IT tickets is dramatically reduced, without sacrificing oversight.

Continuous Feedback and Improvement

The Enterprise Agent maintains conversational context and supports user feedback on generated results.

This feedback loop helps to:

• Improve query disambiguation
• Increase first-attempt answer accuracy
• Refine retrieval quality over time

The system is designed to improve through real enterprise usage rather than static prompts.

Expanding the Scope: From Data Access to Agentic Workflows

From structured data, the system expands incrementally:

• Phase Two: Question answering from PDFs and unstructured documents
• Phase Three: Native support for NoSQL systems such as MongoDB
• Phase Four: A fully agentic approach where the Enterprise Agent reasons across multiple systems and MCP servers

And this is where the real enterprise challenge emerges — because answering real questions rarely involves a single system.

The Integration Problem: Why Connectivity Matters

Enterprise knowledge doesn’t live in one place. A question like “What’s the status of the Johnson project?” might require accessing:

• Project management tools
• Recent documents
• Repository / Release notes
• Internal communications

Traditionally, connecting these systems is expensive and brittle. Each integration requires custom development, ongoing maintenance, and careful handling of authentication and permissions.

As the number of systems grows, integration complexity explodes into an N×M problem.

This is where the Model Context Protocol (MCP) changes the equation.

MCP: A Standard for AI to System Connectivity

The Model Context Protocol (MCP) is an open standard introduced in late 2024 that provides a consistent way for AI systems to interact with external tools and data sources.

Instead of building bespoke integrations for every combination of AI application and system, MCP allows integrations to be built once and reused across compatible clients.

At a high level:

• MCP Servers expose data and functionality from external systems
• MCP Clients (like the Enterprise Agent) consume those capabilities via a standard protocol

The protocol has seen rapid adoption and is now managed by the Linux Foundation’s Agentic AI Foundation, which is a strong signal of its trajectory as shared industry infrastructure.

Quantrium’s MCP Servers: Enterprise-Ready by Design

Quantrium has developed a suite of enterprise-grade MCP servers that integrate seamlessly with the Enterprise Agent.

Each MCP server:

• Handles authentication and permissions
• Translates queries into system-specific operations
• Enforces access controls and auditing
• Exposes a consistent, governed interface to the agent

When an employee asks a question, the Enterprise Agent can route requests to one or more MCP servers, synthesize the retrieved information, and present a coherent response without the user needing to know which systems were consulted.

Deployment Model

All MCP servers are designed to run inside the enterprise’s own infrastructure:

• Sensitive data stays within organizational boundaries
• Identity integrates with existing access management
• Audit logs provide visibility into data usage

This is not a SaaS black box. It’s a deployable infrastructure that enterprises control.

What This Changes for Enterprise Teams

The combination of the Enterprise Agent and MCP servers creates a practical “single interface” for everyday enterprise questions.

What it enables:

• Natural-language access to multiple internal systems
• Faster answers to routine questions
• Reduced context switching and tool hopping

What it doesn’t replace:

• Human judgment
• Review for high-stakes decisions

In early deployments, the value is most visible in the long tail of routine requests: policy lookups, status checks, historical data retrieval, and cross-referencing information across systems.

This is where productivity gains compound quietly but meaningfully.

Where We Are Today

Quantrium is validating this approach through pilot deployments with enterprise clients.

The roadmap continues:

• Structured data access is live and evolving
• Unstructured document support is being developed
• NoSQL connectivity is planned
• MCP server coverage is expanding across enterprise systems

Each phase builds on the same core idea: enterprise data should be secure and usable.

When access becomes intuitive and controlled, information stops being a bottleneck and starts becoming an accelerator.

Technical Notes

For readers interested in implementation details:

• LLM: OpenAI GPT-4o
• Vector Database: Qdrant
• Orchestration: Dapr Agents
• Backend: Python, FastAPI
• Frontend: Vue.js
• Observability: OpenTelemetry with SigNoz

The MCP servers are built using the official Python SDK and follow the protocol specification for tools, resources, and prompts to ensure long-term ecosystem compatibility.

Stay tuned for more…

Enterprises Don’t Have a Data Problem — They Have an Access Problem was originally published in Quantrium.ai on Medium, where people are continuing the conversation by highlighting and responding to this story.

Optimizing Microservices with Proven Architectural Patterns

In today’s modern world, smart applications are required to be flexible, scalable, and fault tolerant. Microservices architecture enables you to achieve this by decomposing monolithic systems into independent services. However, by using design patterns you can manage microservices effectively and address various challenges like — service communication, data consistency, and resilience.

This guide introduces 37 essential microservices patterns, grouped into seven categories. In each pattern, you will see decision-making insights which will help you to choose and apply the right patterns based on your system needs.

First, let’s list down the seven categories of Microservices Design Patterns:

1. Decomposition Patterns
2. Communication Patterns
3. Data Management Patterns
4. Resilience Patterns
5. Observability Patterns
6. Security Patterns
7. Deployment Patterns

1. Decomposition Patterns

Decomposition patterns helps you in the transition from monolithic to microservices-based architectures. They define clear service boundaries between the services based on domains (scope of business), subdomains (business functions), or transactional needs (atomic units of work). Following are the four patterns that are classified as decomposition patterns.

1.1. Domain-Driven Design (DDD)

Domain-Driven Design (DDD) is a methodology for structuring software based on business domains. It breaks applications into smaller, self-contained microservices, each representing a bounded context (subdomain) — a clearly defined responsibility within a domain. This approach ensures high cohesion within services and reduces tight coupling between them.

E-commerce platforms like Amazon use DDD by separating services such as Order Management, Inventory, and Payment Processing, allowing teams to develop, deploy, and scale these independently.

1.2. Strangler Fig Pattern

The Strangler Fig Pattern allows you to carry out a gradual migration from a monolithic system to microservices. Instead of rebuilding the entire system at once, new features are implemented as microservices while progressively replacing legacy components. This reduces migration risk and avoids downtime.

A banking system modernizing its customer onboarding process could start by implementing a new microservice for digital KYC (Know Your Customer) while keeping legacy systems for existing users. Over time, all onboarding processes are transitioned to microservices.

1.3. Smart Endpoints, Dumb Pipes

This pattern ensures that business logic resides within microservices, while communication channels remain lightweight. “Smart Endpoints” refers to each microservice responsible for its own logic, data management and interactions. “Dumb Pipes” refers to infrastructure (such as message queues or network) which simply facilitates communication between services. It is an anti-pattern which discourages the use of complex middleware (e.g., Enterprise Service Buses) that centralize logic, reducing bottlenecks and improving system scalability.

Ride-sharing apps like Uber follow this approach, where the services like Rider Matching, Pricing, and Payments communicate via simple REST or gRPC calls rather than relying on a heavyweight central broker.

1.4. Self-Contained Systems

Self-Contained Systems (SCS) advocate for fully independent microservices that handle their own UI, logic, and database like an autonomous unit. This reduces dependencies and allows teams to build, deploy, and scale services independently while enhancing the overall system resilience.

Media streaming platform like Netflix implement the separate microservices for User Profiles, Content Recommendations, and Video Playback, each running independently to ensure reliability and scalability.

Decision-making Insights

Selecting the right decomposition pattern is crucial for maintainability and scalability. The table below provides insights into when each pattern is most effective.

2. Communication Patterns

Communication patterns define how microservices interact, ensuring efficiency, reliability, and fault tolerance. Communication Patterns can be broadly categorised as Synchronous or Asynchronous. The right communication strategy prevents bottlenecks and ensures seamless data flow between services.

2.1. API Gateway

An API Gateway acts as a centralized entry point for client requests, handling authentication, logging, rate limiting, request transformation, and routing to appropriate microservices. It can facilitates both synchronous communication, where requests are processed in real-time with immediate responses, or asynchronous communication, where requests are queued and processed later using event-driven mechanisms. Hence, the API Gateway pattern simplifies client interactions while enhancing security by shielding internal services from direct exposure.

E-commerce platforms like Amazon use API Gateways to manage requests from web, mobile, and third-party applications, ensuring secure and optimized access to services like product listings, cart management, and payments.

2.2. Service Mesh

A Service Mesh is a dedicated infrastructure layer that manages service-to-service communication. While it facilitates both synchronous and asynchronous communication, it was primarily designed for synchronous communication. It provides load balancing, traffic control, encryption, and observability, reducing the complexity of managing inter-service interactions. Common implementations include Istio and Linkerd.

Large-scale applications like Netflix use a Service Mesh to handle thousands of microservices (using tools developed internally like Ribbon, Eureka, and Zuul) for traffic management ensuring secure and efficient traffic management without adding logic to individual services.

2.3. Remote Procedure Invocation (RPI)

Remote Procedure Invocation (RPI) is an synchronous communication pattern which enables direct communication between services using protocols like HTTP/REST or gRPC. In RPI pattern, a service calls an API exposed by another service and waits for a response before proceeding. It ensures real-time responses but can introduce failure risks if the dependent service is slow or unavailable.

A payment gateways like Stripe relies on RPI to verify transactions in real time. When a user initiates a payment, Stripe’s microservices immediately validate the request before processing it.

2.4. Messaging Pattern

Messaging pattern enables asynchronous communication using message brokers like RabbitMQ, Apache Kafka, or AWS SQS. It allows services to communicate without waiting for an immediate response. Services send messages to a queue, allowing decoupling and resilience even if a receiving service is down.

Ride-sharing apps like Uber use messaging to handle trip status updates. When a driver updates their location, the event is published to Kafka, allowing passenger apps, maps, and billing systems to process the update asynchronously.

2.5. Service Registry

A Service Registry pattern is an asynchronous communication pattern. It maintains a dynamic directory of active microservices, enabling automatic discovery and load balancing. This ensures that services can find and communicate with each other without hardcoded configurations.

Microservices platforms like Netflix use a Service Registry (e.g., Eureka) to dynamically register and deregister services based on availability, ensuring efficient service discovery across its architecture.

2.6. Backends for Frontends (BFF)

The Backends for Frontends (BFF) pattern can be implemented in both synchronous and asynchronous ways. It creates custom backends for different client types (e.g., web, mobile, IoT) to optimize performance. This ensures that each frontend receives data in a format suited to its needs, reducing unnecessary processing on the client side.

Streaming services like Spotify use BFFs to tailor API responses for different devices — providing a lighter response for mobile apps while delivering more detailed data to the desktop web interface.

2.7. API Composition

The API Composition pattern can implement either synchronous or asynchronous communication. It aggregates the data from multiple microservices into a single response, reducing the number of direct calls clients need to make to those services separately. This pattern is useful when building dashboards or reports requiring data from multiple services.

A finance app like Mint aggregates data from bank accounts, credit cards, and investment platforms into a single user-friendly view, using API Composition to fetch and combine information.

2.8. Consumer-Driven Contracts

Consumer-Driven Contracts (CDC) pattern is primarily suited for asynchronous communication, however, it can be used with synchronous communication as well. It ensures backward compatibility by allowing consumers (clients) to define expectations for an API’s response format. This prevents breaking changes when APIs evolve.

Retail companies using third-party payment APIs (e.g., PayPal, Stripe) rely on CDCs to ensure that API changes do not disrupt their checkout systems, helping to maintain seamless payment processing.

2.9. Event-Driven Architecture

In an Event-Driven Architecture, services communicate by emitting and responding to events instead of making direct API calls. It is primarily a asynchronous communication pattern. It improves scalability and responsiveness by allowing services to act independently.

E-commerce sites like Amazon use event-driven architectures for order processing. When a customer places an order, an “Order Placed” event is triggered to services for inventory updates, payment processing, and shipment scheduling without requiring direct communication.

Decision-making Insights

Choosing the right communication pattern depends on factors like performance needs, system reliability, and scalability. While synchronous methods (RPI, API Gateway) provide immediate responses, asynchronous methods (Messaging, Event-Driven Architecture) enhance resilience and decoupling. The table below provides insights into when each pattern is most effective.

3. Data Management Patterns

Data management patterns ensure data consistency, integrity, and scalability in microservices architectures. Since each microservice often manages its own data, these patterns help maintain efficient access, synchronization, and resilience.

3.1. Database Per Service

In Database Per Service pattern, each microservice owns its own database, preventing direct data sharing between services. This approach reduces tight coupling, allows independent scaling, and enables teams to choose the most suitable database for their needs. However, it requires careful handling of data consistency across services.

E-commerce platforms like Amazon use this pattern where order management, inventory, and payment services each have separate databases, allowing them to scale independently without performance bottlenecks.

3.2. Saga Pattern

The Saga Pattern manages distributed transactions across multiple microservices by breaking them into smaller, independent local transactions within each microservice. It is useful for long-running transactions and if one step fails, compensating transactions ensure data consistency instead of rolling back everything.

There are two common approaches:

• Choreography — Each service listens for events and reacts accordingly. It is used when no single microservice controls all steps of the Saga.
• Orchestration — A central controller coordinates the flow of transactions i.e., all steps of the Saga.

In a travel booking system, separate services handle flight reservations, hotel bookings, and payments. If payment fails, previously confirmed bookings are automatically canceled using a compensating transaction.

3.3. Event Sourcing

The Event Sourcing pattern records changes to a system’s state as a series of immutable events, instead of storing only the latest state. The system can be reconstructed by replaying events and easy rollbacks as well as it provides a detailed audit trail that can help compliance and regulatory requirements.

Banking and financial applications (e.g., Revolut) use Event Sourcing to track all transactions and balance changes, ensuring transparency and enabling fraud detection by analyzing past events.

3.4. Command Query Responsibility Segregation (CQRS)

CQRS separates write operations (commands) from read operations (queries), allowing each to be optimized independently. This is useful in the systems which has high read-to-write ratios, thus, improving performance and scalability.

A social media platform like Twitter uses CQRS to handle millions of read requests efficiently while processing relatively fewer write operations (tweets, likes, comments).

3.5. Polyglot Persistence

In Polyglot Persistence pattern, microservices are not restricted to a single type of database. In this pattern, each service and even different parts of the same microservice, you can choose the most suitable database technology (SQL, NoSQL, graph, time-series, etc.) based on its needs.

A leading e-commerce platform like eBay uses MongoDB for product catalogs (flexible schema), PostgreSQL for order processing (relational consistency), Redis for caching recent searches (fast key-value lookups) and Elasticsearch for full-text search (high-speed indexing).

3.6. Data Sharding

Data Sharding pattern splits large datasets into smaller, manageable partitions (shards) and distributing them across multiple database instances. This sharding improves performance, reduce load, and enables parallel processing. Sharding strategies are generally based on a predefined logic (e.g., geographic region, user ID range).

Facebook shards its massive MySQL user database by User ID ranges across multiple servers based on geographic location, ensuring faster response times and efficient horizontal scaling. Additionally, it developed TAO (The Associations and Objects), which is a distributed data store and is optimized for handling billions of real-time social graph queries.

3.7. API Composition for Data

Instead of directly querying multiple databases, in API Composition for Data pattern the data is aggregated from multiple microservices into a single response by a central orchestrator (or composer) and present it as a single unified API. This simplifies client interactions and improves efficiency in data access. It works best when the data retrieval process is light and does not uses complex joins. If complex joins are needed, CQRS or Database Per Service patterns might be better alternatives.

Goldman Sachs’ Marcus Bank uses API Composition to consolidate financial data from multiple sources to their customer’s banking dashboard and ensures that their customers can view balances, transactions, and loans in real-time.

Decision-making Insights

Choosing the right data management pattern depends upon scalability needs, performance requirements, and consistency constraints. While Database Per Service enables independence. Techniques like Event Sourcing, Saga, and CQRS ensures data consistency and efficient processing in distributed environments. The table below provides insights into when each pattern is most effective.

4. Resilience Patterns

Resilience patterns help microservices handle failures gracefully, ensuring that even when parts of the system fail, the overall application remains functional. These patterns minimize downtime, prevent cascading failures, and optimize system recovery.

4.1. Circuit Breaker

The Circuit Breaker pattern prevents a failing service from being overloaded with repeated requests. When the failures crosses a defined threshold, the circuit opens which temporarily blocks further calls. This gives the failing service enough time to recover before resuming operations.

Netflix implemented the Circuit Breaker pattern in their recommendation engine through their Open-source library Hystrix, which is designed to handle latency and fault tolerance in distributed systems. At Netflix, if the recommendation engine becomes unresponsive, Hystrix opens a circuit breaker and stop requests, ensuring the rest of the system (like video playback) continues to work smoothly. However, now Netflix has moved from Hystrix to more advanced resilience strategies such as adaptive concurrency limits and server-side load balancing.

4.2. Bulkhead Pattern

Inspired by compartmentalized ship design, this pattern isolates services or resources so that a failure in one part does not bring down the entire system. By limiting resource usage per service, it ensures critical functions remain operational.

Microsoft Azure Service — Fabric uses the Bulkhead Pattern to isolate different microservices running on a distributed cloud infrastructure. For example, in a financial application, separate activities are handled by separate microservices for user authentication, payment processing, and fraud detection. So, even if the fraud detection service experiences a failure, authentication and payment processing remain fully operational and keep providing uninterrupted service for customers.

4.3. Retry Pattern

The Retry pattern improves the system resilience by automatically retrying failed requests after a short delay, often using exponential backoff-where the wait time keeps on increasing after each failure. This pattern helps to avoid immediate failures and provides efficient way to handle temporary issues like network congestion or temporary service downtime.

E-commerce checkout systems (like Amazon) or payment processors like Stripe uses Retry pattern to handle payment failures. If a payment request fails due to network issues, the system retries a few times before notifying the user.

4.4. Timeout Pattern

The Timeout pattern sets a maximum wait time for each request, otherwise the service will keep waiting indefinitely for a response. If the response doesn’t arrive within this maximum wait time, the request is terminated, preventing resource exhaustion.

In e-commerce platforms like Amazon, the payment gateway service has a specific timeout limit while processing the transactions. If a request to the payment provider takes too long due to server issues or network delays — the system automatically cancels the transaction and notify the user to retry with a different payment method.

4.5. Temporal Coupling

The Temporal Coupling pattern avoids any bottlenecks and failures by making interactions between services asynchronous whenever possible. When the services are tightly linked in real-time, requiring immediate responses, unresponsiveness or slowness in any service may lead to bottlenecks and failures for other services which are waiting for a response. Hence, Temporal Coupling pattern ensures continued system’s functioning and improving reliability.

Domino’s Pizza reduces temporal coupling by processing orders asynchronously. When a customer places an order, it is queued instead of waiting for real-time confirmation. The restaurant processes it independently while the system updates the user. Even if kitchen prep or driver assignment is delayed, the app remains responsive, ensuring seamless tracking and a smooth user experience.

4.6. Cross-Cutting Concerns

Cross-cutting concerns, such as logging, authentication, security, and monitoring, are functionalities that span multiple components within a microservices architecture. By implementing Cross-Cutting Concerns pattern, these concerns are managed centrally, preventing code duplication and ensuring consistency across services.

Netflix, implements centralized logging to handle cross-cutting concerns effectively. By aggregating logs from various services into a unified system, Netflix ensures consistent monitoring and troubleshooting across its platform. This approach enhances operational efficiency and maintains a seamless user experience.

4.7. Fail-Fast Principle

The Fail-Fast Principle ensures that errors or invalid conditions in the system are detected early-on and stops processing immediately. This avoids unnecessary processing and wastage of resource. Furthermore, this allows faster recovery of the failed service.

High-Frequency Trading (HFT) firms, such as Citadel Securities and Jump Trading, the decisions are made in microseconds. These firms implement fail-fast mechanisms that immediately halt processing upon detecting anomalies or invalid conditions, preventing potential cascading failures and conserving computational resources.

Decision-making Insight

Resilience patterns ensure high availability and reliability in microservices-based systems. Circuit Breakers and Bulkheads prevent cascading failures, Retries and Timeouts improve responsiveness, and Fail-Fast and Asynchronous Design optimize resource usage. The table below provides insights into when each pattern is most effective.

5. Observability Patterns

Observability patterns help developers to monitor, troubleshoot, debug, and enhance microservices-based systems. These patterns make sure that any issues such as failures or performance slowdowns are traced quickly and dealt efficiently. Hence, leading to a reliable and smooth experience for users.

5.1. Distributed Tracing

Distributed Tracing pattern helps by following the user requests across the services by assigning a unique identifier (Trace ID) to it. When the request flows through multiple microservices, distributed tracing helps in identifying slow components and performance bottlenecks enabling efficient debugging and performance analysis.

In food delivery apps like Uber Eats, an order request triggers multiple services — restaurant, payment, delivery, and notifications. If is a delay in an order update, distributed tracing tools like Jaeger or Zipkin help engineers find specific service causing the lag. Jaeger developed by Uber is now a Cloud Native Computing Foundation (CNCF) graduated project. Similarly, Zipkin was developed by Twitter.

5.2. Log Aggregation

Log Aggregation pattern centralizes logs into a single, searchable system, making it easier to track errors, detect patterns, and debug issues across microservices. Storing, managing and analyzing logs generated by each microservice separately is inefficient and hampers effectiveness.

Netflix employes the ELK stack — Elasticsearch, Logstash and Kibana to collect and aggregate the logs from different microservices (authentication, content recommendation, streaming, etc.) in one place. This centralized aggregated logging approach helps engineers to quickly search logs to identify what went wrong in case of failures.

5.3. Health Check

The Health Check pattern exposes an endpoint (e.g., /health) that reports if a service is operational or failing. Monitoring tools utilize this endpoint to proactively restart, alert, or redirect traffic when an issue is detected.

Google Cloud Platform (GCP) utilizes health checks in its Managed Instance Groups (MIGs). When a virtual machine (VM) instance fails a health check, the MIG automatically recreates the instance to ensure continuous availability and reliability of applications.

Observability patterns turn microservices into transparent, manageable systems. Distributed tracing finds bottlenecks, log aggregation simplifies debugging, and health checks ensure automatic recovery from failures. The table below provides insights into when each pattern is most effective.

6. Security Patterns

Security patterns protect microservices from vulnerabilities, unauthorized access, and data breaches. Since microservices communicate over networks, they need robust security mechanisms to ensure only legitimate users and services can access data and functionalities. This can be ensured by implementing strong encryption, API gateways, identity management, and service-to-service authentication, reducing the risk of cyber threats.

6.1. Token-Based Authentication

Instead of handling usernames and passwords repeatedly, token-based authentication uses a secure, time-limited token to verify users and services. Common formats include JWT (JSON Web Tokens) and OAuth tokens.

A mobile banking app issues a JWT token when a user logs in. This token is then attached to every request (e.g., checking balance, transferring money), allowing seamless authentication without requiring repeated logins.

6.2. API Security Gateway

An API security gateway acts as a protective layer between clients and microservices. It handles authentication, authorization, rate limiting, and encryption, ensuring that only valid requests reach backend services.

In e-commerce platforms like Flipkart or Amazon, an API gateway ensures that the users are authenticated before accessing their order history. The payment APIs are encrypted to protect credit card details and rate limits prevent bots from overwhelming the system with fake requests. Tools like Kong, Apigee, AWS API Gateway, etc. can be configured and utilised for this.

6.3. Zero Trust Security

Traditional security models trust anything inside the network — but Zero Trust assumes no one is trustworthy by default. It requires continuous verification of every request, even if it comes from an internal service.

Key Principles of Zero Trust are:

• Least Privilege — Each service gets only the access it absolutely needs.
• Multi-Factor Authentication (MFA) — Users must verify identity with passwords and biometrics.
• End-to-End Encryption — Data is always protected, even within internal networks.

A healthcare system following Zero Trust ensures that the doctor accessing patient records must authenticate every time, even within the hospital network. Communication from all services is encrypted and validated, preventing unauthorized data access.

Security in microservices requires layered protection where Token-based authentication secures users and services, API gateways act as centralized security enforcers, whereas Zero Trust ensures continuous verification, eliminating blind trust. The table below provides insights into when each pattern is most effective.

7. Deployment Patterns

Deployment patterns reduce risk and improve flexibility when releasing microservices. They help teams test, monitor, and roll out updates safely without disrupting users.

7.1. Sidecar Pattern

Instead of cluttering a microservice with extra functionalities (like logging, security, or monitoring), the sidecar pattern deploys these functionalities as separate but closely attached services. This keeps the core microservice clean and focused.

In Netflix’s microservices architecture, a sidecar service collects logs and metrics from each microservice without modifying its code. Tools like Envoy and Istio commonly implement this pattern.

7.2. Shadow Deployment

A shadow deployment introduces a new version of a service alongside the existing one, without affecting users. It mirrors real traffic to the new version to monitor performance before fully switching over.

Pizza (small) teams at Amazon first deploys a new feature/experiment in shadow mode. This allows engineers to compare real-time results with the old feature without affecting customers.

7.3. Canary Deployment

Instead of releasing updates all at once, canary deployment rolls them out gradually to a small subset of users before expanding. If issues arise, the rollout can be paused or rolled back.

Facebook tests new features on a small percentage of users before making them available to everyone. If feedback is positive, they expand the rollout.

7.4. Blue-Green Deployment

This method maintains two identical environments:

• Blue (current version) serves live traffic.
• Green (new version) is tested in parallel.

Once the new version is verified, traffic is switched from blue to green, ensuring zero downtime. If issues occur, switching back to blue is instant.

A banking app deploying a critical security update uses blue-green deployment to ensure a seamless transition with no downtime for customers.

While, Sidecar pattern keeps microservices lightweight by handling cross-cutting concerns separately, the Shadow deployment tests new versions with real traffic before rollout, whereas, Canary deployment gradually releases updates, reducing risk and Blue-green deployment ensures instant rollback, preventing downtime. The table below provides insights into when each pattern is most effective.

Conclusion

Successfully adopting microservices isn’t just about breaking an application into smaller services — it’s about using the right patterns to ensure scalability, resilience, and maintainability. Each pattern plays a crucial role in solving common challenges, from efficient communication and data management to fault tolerance and security.

By strategically combining these patterns, teams can build robust, high-performing systems that thrive in modern cloud environments. The key is to choose patterns that align with your specific needs, ensuring a balance between flexibility, reliability, and performance.

An Overview of Microservices Design Patterns was originally published in Quantrium.ai on Medium, where people are continuing the conversation by highlighting and responding to this story.

Building Scalable and Adaptive AI Agents with Modern Frameworks

In our previous article titled “What Is Agentic AI and Why All the Hype?”, we explored the foundational concepts of Agentic AI, its core components, and how it stands apart from other AI paradigms like Generative AI and Traditional AI. We discussed how Agentic AI systems excel in autonomous decision-making, proactive behavior, and adaptability.

If you haven’t read that article yet, we highly recommend giving it a look before diving into this one. While it’s not a prerequisite, it will provide valuable context to better understand the technical frameworks discussed here.

As AI technology advances, selecting the right type of AI for a given problem is more crucial than ever. Two key paradigms, Agentic AI and Generative AI, offer distinct capabilities. Agentic AI excels in autonomous decision-making and goal-oriented tasks, while Generative AI is best suited for content creation, pattern recognition, and synthesis of new ideas. But how do you determine which one is best for your specific needs?

In this article, we’ll explore technical frameworks that empower developers to build robust Agentic AI systems, with a focus on PhiData, CrewAI, and LangGraph. These frameworks provide powerful tools for constructing autonomous agents capable of handling complex workflows and dynamic environments.

How to Choose the Right AI?

Selecting the right AI system depends on the nature of your problem. Here’s a checklist to help determine whether your use case requires an Agentic AI system.

Checklist

• Do you need autonomous decision-making?
• Do you require actions to be taken without human intervention?
• Does your system need to adapt to changing tasks and environments?
• Do you need a system capable of handling a dynamic workflow?
• Do you want the AI to proactively perceive and respond to inputs from multiple sources?
• Does your system require real-time adaptation based on incoming data?
• Do you need AI to collaborate with other AI agents or systems to achieve complex goals?
• Are you looking for AI to execute multi-step tasks with minimal human supervision?

Now, let’s apply this checklist to different scenarios and determine which type of AI is best suited for each case.

Case 1: Fully Autonomous IT Incident Resolution System

A large enterprise faces frequent IT incidents, such as server failures and network issues. They need a system that can detect, diagnose, and resolve these issues autonomously, without human intervention.

Checklist

• Do you need autonomous decision-making? ✔ Yes
• Do you require actions to be taken without human intervention? ✔ Yes
• Does your system need to adapt to changing tasks and environments? ✔ Yes
• Do you need a system capable of handling a dynamic workflow? ✔ Yes
• Do you want the AI to proactively perceive and respond to inputs from multiple sources? ✔ Yes
• Does your system require real-time adaptation based on incoming data? ✔ Yes
• Do you need AI to collaborate with other AI agents or systems to achieve complex goals? ✔ Yes
• Are you looking for AI to execute multi-step tasks with minimal human supervision? ✔ Yes

This system requires Agentic AI, as it must autonomously detect, diagnose, and resolve IT issues, adapt to new problems, and operate dynamically without human intervention.

Case 2: AI-Powered Marketing Content Generator

A marketing team wants to automate content creation for social media, blogs, and product descriptions using AI. The AI must generate engaging, high-quality content, but human reviewers will decide which content to use.

Checklist

• Do you need autonomous decision-making? ❌ No (Humans make the final decision)
• Do you require actions to be taken without human intervention? ❌ No (AI assists but doesn’t act independently)
• Does your system need to adapt to changing tasks and environments? ❌ No (The task — content creation — remains the same)
• Do you need a system capable of handling a dynamic workflow? ❌ No (Workflow is structured and requires human oversight)
• Do you want the AI to proactively perceive and respond to inputs from multiple sources? ✔ Yes (AI may use different data sources for content)
• Does your system require real-time adaptation based on incoming data? ❌ No (Static content generation doesn’t need real-time changes)
• Do you need AI to collaborate with other AI agents or systems to achieve complex goals? ❌ No (AI works independently)
• Are you looking for AI to execute multi-step tasks with minimal human supervision? ❌ No (Humans oversee the final output)

Generative AI is the best fit for this case. The AI assists with content creation but doesn’t act autonomously or adapt dynamically — it supports human creativity rather than replacing decision-making.

Case 3: AI-Driven Customer Support Assistant

A company wants an AI-powered customer support assistant that can handle FAQs with static responses while also escalating urgent issues autonomously, ensuring a conversational and natural flow.

Checklist

• Do you need autonomous decision-making? ✔ Yes (For issue escalation and handling FAQs)
• Do you require actions to be taken without human intervention? ✔ Yes (AI responds automatically and escalates when needed)
• Does your system need to adapt to changing tasks and environments? ❌ No (Most responses are predefined, except escalations)
• Do you need a system capable of handling a dynamic workflow? ✔ Yes (Customer queries vary, requiring adaptable conversations)
• Do you want the AI to proactively perceive and respond to inputs from multiple sources? ✔ Yes (AI needs to process customer queries from multiple channels)
• Does your system require real-time adaptation based on incoming data? ✔ Yes (AI must adjust responses based on customer input)
• Do you need AI to collaborate with other AI agents or systems to achieve complex goals? ❌ No (Primarily a single-agent task)
• Are you looking for AI to execute multi-step tasks with minimal human supervision? ✔ Yes (Handles conversations and escalations without constant oversight)

This use case is partially agentic. The AI autonomously manages conversations and escalations but still relies on predefined rules for many responses. A hybrid approach combining Agentic AI + Generative AI is ideal, where Generative AI handles conversational interactions, while Agentic AI manages decision-making for escalations.

Final Takeaways

• Agentic AI is best for systems requiring autonomous decision-making, real-time adaptation, and dynamic workflows.
• Generative AI excels at content creation and structured tasks where human oversight is required.
• Hybrid AI works well when both adaptability and creativity are needed, such as in AI-driven customer support.

By applying this checklist to different cases, you can determine the right AI approach for your needs.

Technical Frameworks for Agentic AI

Agentic AI frameworks are gaining traction due to their ability to automate complex tasks with adaptability and decision-making capabilities. This guide covers three prominent frameworks: PhiData, CrewAI, and LangGraph, providing practical insights for implementing AI agents using these tools.

Phidata (now Agno): Phidata, recently rebranded as Agno, is an open-source framework for building multimodal AI agents and workflows. It allows developers to create agents equipped with memory, knowledge, tools, and reasoning capabilities, supporting various data types such as text, images, audio, and video.

CrewAI: CrewAI is a lean, Python-based framework designed to orchestrate role-playing AI agents that collaborate to tackle complex tasks. Independent of other agent frameworks, it offers both high-level simplicity and precise low-level control, enabling the creation of autonomous AI agents tailored to diverse scenarios.

LangGraph: Developed by LangChain, LangGraph is a low-level orchestration framework designed for building controllable AI agents. It enables the creation of stateful, multi-actor applications with large language models (LLMs), facilitating complex task automation and custom LLM-backed experiences.

Hands-on Implementation Guide

Regardless of the Agentic AI framework you choose, the core implementation steps remain the same. Follow this structured approach to build and deploy your agentic system effectively.

Step 1: Define the Problem Statement

Clearly outline the objectives of your AI agent. (Eg: Coordinating multiple agents for data scraping and analysis)

Step 2: Create an Agent

Develop the AI agent to execute tasks based on the defined objectives.

Step 3: Integrate the Necessary Tools

Equip your agent with the appropriate tools to enhance its functionality and task execution.

Step 4: Add an LLM (if required)

If natural language understanding or generation is needed, integrate a Large Language Model (LLM) to improve the agent’s capabilities.

Step 5: Monitor the Agentic System

Evaluate the initial performance of the system to ensure it functions as expected.

Step 6: Optimize with Feedback or Human-in-the-Loop

If improvements are needed, incorporate feedback loops or human oversight to enhance accuracy and efficiency.

Creating a simple finance agent using Phidata

Lets create a simple Finance agent supported by YFinance Tool in phidata. This is an episodic task (that have a terminal state). The results can directly printed on terminal.

Install Phidata

pip install -U phidata==2.7.9

Requirements

1. LLM — Add llm keys in env and configure the keys (the below code uses OpenAI)
2. Tools — YFinanceTools for Finance agent

Install Tools

Install yfinance library for YFinanceTools

pip install yfinance==0.2.52

Create a Finance Agent

Create a file finance_agent.py

from phi.agent import Agent
from phi.model.openai import OpenAIChat
from phi.tools.yfinance import YFinanceTools

finance_agent = Agent( 
    name="Finance Agent", # name for the agent 
    model=OpenAIChat(id="gpt-4o"), # llm model
    tools=[YFinanceTools(stock_price=True, analyst_recommendations=True, company_info=True, company_news=True)],  # tools for the agent
    instructions=["Use tables to display data"],  # list of instructions
    show_tool_calls=True, # logs the AI tool calls
    markdown=True, # formats in Markdown for structured text
)
finance_agent.print_response("Summarize analyst recommendations for NVDA", stream=True)

Run the agent

python finance_agent.py

This is an episodic task which prints the info on the terminal.

Advantages

• Ease of Building — Designed for quick and simple development.
• Wide Range of Tools — Offers a variety of built-in tools and supports custom tool creation.
• Playground — Provides an interactive playground for testing and building agents.
• Structured Agent Creation — PhiData agents have well-defined fields, making them easy to set up.
• Memory Support — Includes a vector database for storing and retrieving information.
• Embeddings & Chunking — Supports efficient data processing and retrieval.
• Multi-Agent Collaboration — Allows multiple agents to work together.

Drawbacks

• Performance Limitations — The PhiData Playground crashes during heavy, continuous tasks due to load and complexity.
• Limited Visualization — No built-in visualization tools for better workflow monitoring.
• Working Transparency — Debugging tools are available, but understanding dynamic workflows can be difficult due to limited interpretability.

2. CrewAI — Building a Finance Agent

Lets create a Agentic AI system with defined tasks for analyzing and reporting the stock market. Since its not a simple task we are going for multi-agent system for better performance and task delegation.

Install CrewAI

pip install crewai==0.108.0

Setup API Keys

CrewAI requires API keys for integrating with:

• LLMs (e.g., OpenAI, Anthropic, etc.)
• Databases (for data storage and retrieval)

Components of a CrewAI Agent

• Role: Defines the agent’s primary function (e.g., Financial Analyst, Reporting Specialist).
• Goal: Outlines the agent’s intended outcome or objective (e.g., Analyze NVDA stock performance).
• Backstory: Provides context about the agent’s expertise, experience, or specialization to guide its behavior.
• Tools: Specifies the resources, APIs, or services the agent will use to achieve its goal (e.g., YFinance, Google Docs).

Build a Finance Agent

from crewai import Crew, Agent, Task
from openai import OpenAI

finance_agent = Agent(
    role="Financial Analyst",
    goal="Analyze NVDA stock performance and provide insights",
    backstory="An experienced financial analyst with expertise in stock market trends, known for delivering precise insights to guide investment strategies.",
    tools=["YFinance", "AlphaVantage"],
    model=OpenAI("gpt-4o"),
)

reporting_agent = Agent(
    role="Reporting Specialist",
    goal="Generate a detailed report on NVDA performance",
    backstory="A skilled technical writer with extensive experience in financial reporting, adept at transforming data into clear, actionable insights.",
    tools=["Google Docs", "Markdown"],
    model=OpenAI("gpt-4o"),
)

analysis_task = Task(
    description="Summarize analyst recommendations for NVDA",
    agent=finance_agent
)

report_task = Task(
    description="Create a structured report with insights",
    agent=reporting_agent
)

crew = Crew(agents=[finance_agent, reporting_agent], tasks=[analysis_task, report_task])
crew.kickoff()

Run the CrewAI Agent

python crewai_agent.py

Observations

• CrewAI adopts a collaborative agent model where multiple agents can coordinate tasks.
• Ideal for multi-step workflows that require agents to share information and refine decisions.

Advantages

• Robust Collaboration: CrewAI excels in multi-agent orchestration, enabling teamwork across specialized agents.
• Flexible Workflows: Supports complex, sequential decision-making processes.
• Human-in-the-Loop Support: Enables seamless human oversight during crucial decision points.
• Dynamic Role Assignment: Agents can adapt their roles and switch tasks as objectives evolve.

Drawbacks

• Setup Complexity: Requires API keys, model configurations, and additional dependencies.
• Learning Curve: Building multi-agent workflows may require more effort compared to PhiData.
• Execution Overhead: Managing multiple agents may introduce performance overhead for smaller tasks.

3. Creating a chatbot using LangGraph

Lets create a chatbot using LangGraph with Anthropic

Install LangGraph

pip install langgraph==0.3.7

Requirements

LLM — requires API keys (used Anthropic for the code below)

Initialize an LLM

Create a file langgraph_workflow.py and add the following

from langchain_anthropic import ChatAnthropic

llm = ChatAnthropic(model="claude-3-5-sonnet-20240620")

Create a LangGraph Workflow

LangGraph is designed for workflow automation by defining nodes (steps) and edges (transitions) in the process.

from typing import Annotated
from typing_extensions import TypedDict
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages

class State(TypedDict):
    # Messages have the type "list". The `add_messages` function
    # in the annotation defines how this state key should be updated
    # (in this case, it appends messages to the list, rather than overwriting them)
    messages: Annotated[list, add_messages]

def chatbot(state: State):
    return {"messages": [llm.invoke(state["messages"])]}

graph_builder = StateGraph(State)

# first arg is the unique node name, second arg is the function or object that 
# will be called whenever the node is used.
graph_builder.add_node("chatbot", chatbot)
graph_builder.add_edge(START, "chatbot")
graph_builder.add_edge("chatbot", END)

graph = graph_builder.compile()

Visualize the workflow

with open("data/chat_bot_workflow.png", "wb") as f:
    f.write(graph.get_graph().draw_mermaid_png())

Stream the updates

def stream_graph_updates(user_input: str):
    for event in graph.stream({"messages": [{"role": "user", "content": user_input}]}):
        for value in event.values():
            print("Assistant:", value["messages"][-1].content)

while True:
    try:
        user_input = input("User: ")
        if user_input.lower() in ["quit", "exit", "q"]:
            print("Goodbye!")
            break

        stream_graph_updates(user_input)
    except Exception:
        # fallback if input() is not available
        user_input = "What do you know about LangGraph?"
        print("User: " + user_input)
        stream_graph_updates(user_input)
        break

Run the LangGraph Workflow

python langgraph_workflow.py

Observations

• LangGraph is designed for structured workflow automation.
• Each node represents a task or decision point, and edge transitions to control the workflow and also making it easy to visualize complex processes.

Advantages

• Clear Workflow Design: Nodes and edges simplify creating structured automation pipelines.
• Flexible Execution: Can dynamically adapt to different data inputs or logic paths.
• Modular Design: Encourages reusable code through modular task definitions.

Drawbacks

• Limited Tooling Integration: Compared to PhiData or CrewAI, LangGraph may require more custom code for specific data sources or APIs.

Comparative Summary

Conclusion

This technical deep dive is not aimed at identifying the “best” framework but rather exploring the strengths, limitations, and ideal use cases for each. Each framework excels in different scenarios:

• PhiData is best suited for simple tasks requiring quick implementation and minimal overhead. Its lightweight nature makes it ideal for straightforward, episodic workflows.
• CrewAI is the optimal choice for multi-agent collaboration, especially when tasks demand dynamic coordination, role management, and adaptable workflows.
• LangGraph stands out for its visual workflow design and clear process control, making it ideal for structured automation with defined steps and conditions.

In essence, there’s no universal “best” framework — the right choice hinges on your project’s complexity, data flow, and adaptability needs. Combining these frameworks strategically can unlock powerful Agentic AI solutions for enterprise-scale applications.

These observations are based on our research. However, as AI frameworks continue to evolve, future upgrades may impact their capabilities. This technical article may not remain valid over time, and it is recommended to stay updated with the latest advancements.

This article is co‑authored by Yash Goel and me, combining our expertise in Agentic AI systems. Yash Goel contributed significantly by analyzing technical frameworks and providing insightful comparisons among them.

References:

1. https://www.langchain.com/langgraph
2. https://docs.phidata.com/introduction
3. https://docs.crewai.com

Technical Frameworks for building Agentic AI Systems was originally published in Quantrium.ai on Medium, where people are continuing the conversation by highlighting and responding to this story.

Understanding the Rise of Autonomous AI Systems

The recent surge in interest around Agentic AI stems from its potential to revolutionise various industries by automating complex processes and enhancing efficiency. For instance, companies like ServiceNow and Salesforce are developing AI agents to autonomously handle tasks in customer service, HR, IT, and marketing, thereby increasing productivity and reducing operational costs.

Similarly, in healthcare, AI agents are being utilised to manage administrative tasks, assist in clinical trials, and support physicians in patient care, addressing challenges such as physician burnout and staffing shortages.

Since the start of 2025, there’s been a surge in hype around Agentic AI, along with plenty of speculation about whether AI could eventually replace humans. But what exactly is Agentic AI, and how does it differ from Generative AI? Let’s explore.

What’s Agentic AI?

Agentic AI refers to autonomous systems capable of dynamic learning, interacting within complex environments, and making decisions independently. Unlike Generative AI, which primarily creates content based on input, Agentic AI can act proactively, take initiative, and adapt to evolving situations without constant human guidance.

Agentic AI vs. Other AI Paradigms

Here we have compared the three generations of AI depicting evolution of AI over time:

1. Agentic AI (3rd Gen),
2. Generative AI (2nd Gen)
3. Traditional AI (1st Gen)

Agentic AI vs Generative AI vs Traditional AI

In simple terms:

• Agentic AI → Proactive, autonomous, decision-making.
• Generative AI → Content creation, responsive, dependent on prompts.
• Traditional AI → To solve tasks based on specific rules and pre-defined logic.

This ability to independently navigate and respond to complex environments is what sets Agentic AI apart and why it’s generating so much excitement.

How is Agentic AI different from Generative AI?

While Agentic AI promises autonomous decision-making, LLM + API setups (alongside RAG and similar architectures) already deliver immense value as highly capable Generative AI systems. Here is a detailed comparison.

Example: Research Assistant Task

Agentic AI:

• One agent autonomously finds relevant sources, another summarizes the content, and a third fact-checks for accuracy.
• The agents work together, refining outputs and continuously improving the final result — all without constant human intervention.

LLM + API:

• The user manually requests information, summarizes the findings, and verifies accuracy step by step.
• Each stage requires explicit API calls and human oversight, making the process more hands-on and time-consuming.

When to Use Each Approach

Use Agentic AI when:

• You need autonomous, multi-step decision-making.
• Tasks require collaboration among AI agents.
• Adaptability and self-improvement are important.

Use LLM + API when:

• You need simple, transactional AI responses.
• The task is well-defined and does not require adaptability.
• You can manually orchestrate API calls efficiently.

Components of Agentic AI

Agentic AI systems have several interconnected components that work together to enable intelligent behavior. Each component plays a crucial role in the overall functioning of the AI system, and they must interact seamlessly to achieve desired outcomes.

Perception Module — Captures and interprets real-world data using AI techniques like vision and NLP.

Cognition Module — Analyzes information, makes decisions, and plans actions based on learned patterns.

Action Module — Executes decisions by performing digital or physical tasks autonomously

Why Is There So Much Hype Around Agentic AI?

The buzz around Agentic AI isn’t just noise — it’s driven by the tangible benefits these systems bring to organizations. Let’s break down the key factors fueling its rising popularity:

• Automation Needs: Companies are eager to streamline operations and minimize manual effort, making autonomous systems a powerful asset for boosting efficiency.
• Adaptability: Agentic AI can adjust to evolving environments and shifting data, ensuring it stays relevant and effective in a variety of scenarios.
• Enhanced Decision-Making: These systems can analyze complex data and make informed decisions without human intervention, helping businesses act faster and more strategically.
• Scalability: Agentic AI handles growing workloads with ease, scaling operations without a linear increase in costs or resource demands.
• Cost-Effectiveness: By automating repetitive tasks and reducing reliance on human oversight, organizations can cut operational expenses while maintaining high performance.

With these capabilities, it’s easy to see why businesses across industries are eager to explore Agentic AI. But what about its impact on our day-to-day lives?

How Can Agentic AI Make Our Lives Easier?

Agentic AI isn’t just a futuristic concept — it’s a game-changer for simplifying tasks, boosting productivity, and freeing us up to focus on what truly matters. Here’s how it can transform both work and personal life:

• Automate Workflows: Seamlessly handle repetitive tasks, from scheduling meetings to managing customer inquiries, freeing up valuable time for more meaningful work.
• Focus on High-Impact Work: By handling time-consuming, low-value tasks, Agentic AI allows people to concentrate on strategic, creative, and high-priority initiatives.
• Generate & Refine Content: Create, edit, and optimize content effortlessly — whether it’s drafting reports, crafting emails, or refining presentations.
• Intelligent Data Gathering & Analysis: Collect and process data from various sources (like voice commands, text, or even receipts) to make faster, more accurate decisions — sometimes even outperforming humans in complex scenarios.

From personal assistants to workplace optimization, Agentic AI has the potential to revolutionize how we manage time and resources. However, as with any powerful technology, it comes with its own set of challenges.

Can We Trust Agentic AI Completely?

While Agentic AI holds incredible promise, it’s not without its limitations. Understanding these potential pitfalls is essential for organizations to adopt the technology responsibly.

• The Illusion of Control: Enterprises may feel in control when setting up AI agents, but many of these systems operate as black boxes. Without proper oversight, AI-driven decisions can become unpredictable — and in highly regulated sectors like finance or healthcare, even small missteps can have serious consequences.
  😇 Implement robust monitoring and feedback loops to ensure continuous human oversight and mitigate unexpected outcomes.
• The Maintenance Challenge: Although Agentic AI systems promise rapid deployment and adaptive learning, they often lack robust maintenance frameworks. Most automation work happens after deployment, and small process tweaks can cause unintended side effects, turning continuous improvement into a complex challenge.
  😇 Establish clear update and retraining protocols to ensure the AI evolves safely and aligns with changing business needs.
• The Reliability Mirage: Even with error rates as low as 1–5%, inaccuracies in high-stakes environments like banking or legal compliance are unacceptable. A single misstep can lead to financial losses, reputational damage, or legal complications, making complete trust in AI a risky bet.
  😇 Use AI as an assistive tool for decision support rather than a fully autonomous solution in high-risk areas.
• The Governance Gap: The rise of “shadow AI” — unauthorized or fragmented AI use across organizations — mirrors past issues with shadow IT. Without centralized governance, businesses face compliance breaches, inconsistent processes, and a fragmented tech landscape that’s difficult to manage.
  😇 Create a centralized AI governance team to set policies, track AI usage, and ensure compliance across the enterprise.
• The Learning Dilemma: Agentic AI lacks structured lifecycle management, which can cause processes to become outdated over time. Without a clear change management strategy, organizations risk AI systems making decisions based on obsolete information, diminishing long-term effectiveness.
  😇 Implement lifecycle management practices, including regular audits and periodic retraining, to keep AI systems aligned with evolving goals and data.

Final Thoughts

Agentic AI is undeniably powerful technology that promises to reshape industries, streamline workflows, and even make our daily lives more convenient. But with great power comes great responsibility. For businesses and individuals alike, balancing excitement with caution is key.

By understanding both the potential and the limitations of Agentic AI, we can harness its strengths while mitigating its risks. With the right mix of innovation, governance, and continuous oversight, Agentic AI can be an extraordinary tool for progress.

So, can we trust Agentic AI completely? Maybe not yet. But can we leverage it to build a smarter, more efficient future? Absolutely.

This article is co‑authored by Yash Goel and me, bringing together our shared expertise in Agentic AI research. In particular, Yash Goel played a key role in identifying real-world use cases, and providing thoughtful comparisons between different AI paradigms which were essential to shaping the article’s core insights.

References

1. https://www.youtube.com/watch?v=xOS0BhhdUbo
2. https://www.uipath.com/ai/agentic-ai
3. https://aisera.com/blog/agentic-ai/
4. https://www.automationanywhere.com/rpa/agentic-ai#:~:text=Despite%20their%20autonomy%2C%20agentic%20AI,judgment%20or%20deep%20contextual%20understanding.
5. https://scet.berkeley.edu/the-next-next-big-thing-agentic-ais-opportunities-and-risks/
6. https://markovate.com/blog/agentic-ai-architecture/
7. https://nexustrade.io/blog/you-are-an-absolute-moron-for-believing-in-the-hype-of-ai-agents-20250111

What Is Agentic AI and Why All the Hype? was originally published in Quantrium.ai on Medium, where people are continuing the conversation by highlighting and responding to this story.

Structuring, automating, and improving documentation of your Python projects

In one of my previous articles, I highlighted the key skills to build and excel in various roles in the data domain:

Exploring the opportunities in the data market

Programming and software development are essential skills for building a successful career in engineering roles (related to data). As a data, machine learning (ML) or AI engineer, a strong foundation in Object-Oriented Programming, classes, and user-defined functions is crucial for writing efficient code and developing robust data pipelines.

Additionally, developing reusable functions and modular components is paramount as they play an important role in maintainability and scalability for such data-driven projects. Docstrings act as structured function descriptions, enhancing code readability and clarity. They make it easier to understand the code, extend and re-purpose functions across different projects, and improve both collaboration and efficiency.

Building on this foundation, this article introduces the most popular Sphinx docstring format, demonstrating how it enhances code documentation and clear function descriptions. Through practical examples, you’ll see how to make use of Sphinx-style docstrings. I have used VS Code editor to create my classes and functions along with the docstrings, demonstrating how they help you to make your functions more intuitive and easier to understand.

The Sphinx docstrings format follows a structured approach to document functions, ensuring clarity in their purpose, inputs, outputs, and potential errors.

Standard Sphinx Docstrings Format

"""
<Function summary>
    
:param [parameter_name]: [parameter description]
:type [parameter_name]: [parameter datatype](, optional)
.
.
.

:raises [exception type]: [exception description]
.
.
.

:returns: [description]
:rtype: [datatype of output returned]
"""

This format provides a consistent and professional way to document code while making it easier for developers to understand function’s behavior in one glance. Additionally, it enables automatic generation of API documentation, which is widely used in Python and data-driven projects to faciliate collaboration and improve maintainability.

Docstrings in Action — Python

Before diving into examples, ensure that the python docstring extension is installed in VS Code (as shown in Figure 1). This extension streamlines the process of adding structured docstrings to your functions and classes.

Setting up Sphinx Docstrings Format in VS Code

To enable the Sphinx-style docstrings by default, follow these steps (illustrated in Figure 2).

• Go to File > Preferences > Settings
• Search for autoDocstring.docstringFormat
• Change the format to sphinx

Once configured, you can automatically generate well-structured docstrings.

Applying Docstrings to an OCR Project

To see docstrings in action, let’s revisit an Optical Character Recognition (OCR) project I previously worked on using docTR OCR.

Extracting text from an image using docTR OCR

We will rewrite the same functionality using Object-Oriented Programming (OOP) principles and structured Sphinx Docstrings.

First, let’s import the necessary modules.

from doctr.models import ocr_predictor
from doctr.io import DocumentFile
import PIL
from PIL import ImageDraw
import math
import json

Now, let’s define the DocTROCR class, which leverages the docTR OCR model to extract text from documents. The class follows an object-oriented approach, with well-structured Sphinx docstrings for auto-generated documentation.

DocTROCR Class

class DocTROCR:
    """
    Class to extract text from a document using docTR OCR model.

    :param file_path: path to the image file
    :type file_path: `str`
    """
    def __init__(self, file_path):
        """Constructor method
        """
        self.file_path = file_path
        self.model = ocr_predictor(det_arch = 'db_resnet50',    
            reco_arch = 'crnn_vgg16_bn', 
            pretrained = True
        ) 

    def convert_coordinates(self, geometry, page_dim):
        """
        Function to convert geometric coordinates to graphical coordinates.

        :param geometry: list of coordinates in geometric form ([(x_min, y_min), (x_max, y_max)])
        :type geometry: `list`
        :param page_dim: dimensions of the page ([dim_y, dim_x])
        :type page_dim: `list`

        :raises Exception: If an error occurs while converting coordinates

        :return: list of coordinates in graphical format ([x_min, x_max, y_min, y_max]) 
        :rtype: `list`
        """
        try:
            dim_x = page_dim[1]
            dim_y = page_dim[0]
            (x_min, y_min) = geometry[0]
            (x_max, y_max) = geometry[1]
            x_min = math.floor(x_min * dim_x)
            x_max = math.ceil(x_max * dim_x)
            y_min = math.floor(y_min * dim_y)
            y_max = math.ceil(y_max * dim_y)
            return [x_min, x_max, y_min, y_max]
        except Exception as ex:
            raise Exception(
                f"Error in converting coordinates: {ex}"
            )

    def get_graphical_coordinates(self, output):
        """
        Function to get graphical coordinates of the text.

        :param output: OCR output obtained from the model
        :type output: `dict`

        :raises Exception: If an error occurs while getting graphical coordinates

        :return: list of graphical coordinates of the text
        :rtype: `list`
        """
        try:
            page_dim = output['pages'][0]["dimensions"]
            text_coordinates = []
            for obj1 in output['pages'][0]["blocks"]:
                for obj2 in obj1["lines"]:
                    for obj3 in obj2["words"]:                
                        converted_coordinates = self.convert_coordinates(
                                                obj3["geometry"],page_dim
                                                )
                        print("{}: {}".format(converted_coordinates,
                                            obj3["value"]
                                            )
                            )
                        text_coordinates.append(converted_coordinates)
            return text_coordinates

        except Exception as ex: 
            raise Exception(
                f"Error in getting graphical coordinates: {ex}"
            )

    def get_ocr_output(self):
        """
        Function to get the output of the OCR model.

        :raises Exception: If an error occurs while getting OCR output

        :return: OCR output
        :rtype: `dict`
        """
        try:
            img = DocumentFile.from_images(self.file_path)
            result = self.model(img)
            self.output = result.export()
            return self.output
        except Exception as ex:
            raise Exception(
                f"Error in getting OCR output: {ex}"
            )
    
    def show_ocr_output(self):
        """
        Function to display the output of the OCR model.

        :raises Exception: If an error occurs while displaying OCR output
        """
        try:
            for obj1 in self.output['pages'][0]["blocks"]:
                for obj2 in obj1["lines"]:
                    for obj3 in obj2["words"]:
                        print("{}: {}".format(obj3["geometry"],obj3["value"]))
        except Exception as ex:
            raise Exception(
                f"Error in displaying OCR output: {ex}"
            )

    def write_ocr_output(self, output):
        """
        Function to write the output to a JSON file.

        :param output: OCR output obtained from the model
        :type output: `dict`

        :raises Exception: If an error occurs while writing output to a JSON file
        """
        try:
            with open("path/docTROCR_sample_output.json", "w") as f:
                f.write(json.dumps(output, indent=1))
            f.close()
        except Exception as ex:
            raise Exception(
                f"Error in writing output: {ex}"
            )
        
    def draw_bounding_boxes(self, image, bound):
        """
        Function to draw bounding boxes on the image."

        :param image: image on which bounding boxes are to be drawn
        :type image: `PIL.Image`
        :param bound: list of bounding boxes
        :type bound: `list`

        :raises Exception: If an error occurs while drawing bounding boxes

        :return: image with bounding boxes drawn
        :rtype: `PIL.Image`
        """
        try:
            draw = ImageDraw.Draw(image)
            for b in bound:
                p0, p1, p2, p3 = [b[0],b[2]], [b[1],b[2]], \
                                [b[1],b[3]], [b[0],b[3]]
                draw.line([*p0,*p1,*p2,*p3,*p0], fill='blue', width=2)
            return image
        except Exception as ex:
            raise Exception(
                f"Error in drawing bounding boxes: {ex}"
            )

By integrating Sphinx docstrings, we have transformed the OCR extraction class into a well-documented, readable, and maintainable Python module. Hence, the structured docstrings provide:

• clearly defined function purpose, parameters, return types, and exceptions.
• automated API documentation for improved projected scalability.
• improved collaboration between the developers and code reusability.

It is a structured and efficient way to document Python code and bridges the gap between code and documentation.

Final Thoughts

Adopting docstrings best practices also ensures that documentation stays updated with the codebase, benefiting individual developers and larger teams associated with your project. Try implementing Sphinx docstrings in your existing or future projects, and share your thoughts or queries in the comments. Stay tuned for similar content. Happy Coding!

Writing cleaner Python code with Sphinx docstrings was originally published in Quantrium.ai on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Hidden Costs of Ignoring Algorithmic Complexity in Sorting

If you’ve ever looked into sorting algorithms, you’ve likely seen this classic comparison:

• O(nlogn) sorts: Merge Sort, Heap Sort, and Quick Sort (on average)
• O(n²) sorts: Bubble Sort, Selection Sort, and Insertion Sort

And then, the cliche conclusion:
“For large datasets, always use O(nlogn)!”

But why does this actually matter? Is O(nlog⁡n) that much better? Let’s take a closer look.

A Curious Look: How Fast Sorting Algorithm Grow

Let’s take a concrete comparison to see how the complexities scale with large data sizes. Consider sorting 1 million elements.

If we use an O(n²) algorithm, the worst case gives:
n² = (10⁶)² = 10¹²

If we use an O(nlogn) algorithm, assuming base-2 logarithm:
nlog₂⁡n = 10⁶ * log₂(10⁶)

Since log₂(10⁶) ≈ 20, this simplifies to : 10⁶×20=2×10⁷

Now, let’s compare these numbers : 10¹²(quadratic) vs 2×10⁷(log-linear)
That’s a 50,000x faster!

Imagine an algorithm that runs in 1 second for O(nlog⁡n) sort.

• If you use an O(n²) sort instead, you’d wait about 14 hours for the same data size!
• When n becomes 10⁹, O(n²) would take 9,250 hours to solve whereas O(nlogn) would take just 8.3 hours!

The Computational Paradox

Even if Moore’s Law held indefinitely, pushing computational power forward, algorithms would still hit a wall where they become unusable. The growth rate isn’t just about current speed — it’s about the fundamental physics of computation.

This is why algorithmic improvements are often more valuable than raw hardware advancements.

Why Some Sorting Algorithms Avoid O(nlogn)

So, why don’t we always use O(nlog⁡n) sorting algorithms?

One big reason is space complexity. Many of the fastest sorting algorithms, like Merge Sort, require extra memory proportional to the input size O(n), making them problematic for large datasets.

Most programming languages provide built-in sorting functions, but not all of them are truly in-place.

Why Merge Sort Might Be Bad in Some Cases
While Merge Sort is often seen as a safe choice due to its worst-case guarantee of O(nlog⁡n), it can be inefficient in memory-limited environments due to the extra O(n) space requirement. This is why Python’s sort() (Timsort) and C++'s std::sort() (Introsort) combine Quick Sort + Heap Sort + Insertion Sort for in-place efficiency.

Surprising Takeaway

In many practical applications, especially those involving large-scale datasets, even an O(nlog⁡n) algorithm can start to show inefficiencies once the dataset reaches a certain size. That is because, the real-world systems are never perfect, and the overhead of managing large datasets in memory, caching, and hardware limitations can shift the balance.

The true magic is in understanding the nuances of both time and space complexities and choosing the right algorithm not just based on theoretical performance but also based on how your application will scale and interact with the system as a whole.

Final Thought

The difference between O(nlogn) and O(n²) isn’t just a technical detail — it defines what problems can be solved within a human lifetime. Understanding these differences allows better decisions in designing algorithms, choosing data structures, and performance optimization.

So next time you encounter a sorting algorithm, ask yourself: Is it O(nlogn) or O(n²)? The answer could mean the difference between completing a task in seconds or dragging-on for centuries.

Why Bother About O(nlog⁡n) vs O(n²)? was originally published in Quantrium.ai on Medium, where people are continuing the conversation by highlighting and responding to this story.

Deep dive into VACUUM revamp and key enhancements

On September 26, 2024, PostgreSQL 17 got released with host of new features and improvements. This latest version continues to solidify PostgreSQL’s position as a leading open-source database management system. Let’s straight dive into the significant enhancements to the VACUUM process and other key areas.

VACUUM Improvements

In PostgreSQL 17, the most notable upgrade is the overhaul of the VACUUM process — the process which is critical for maintaining database health and performance. Wait! does that mean there is a significant improvement in performance? Let’s find out!

TidStore — Dramatic Reduction in Memory Consumption

In PostgreSQL 17 a new internal memory structure is introduced for VACUUM which dramatically reduces the memory consumption. This new system called TidStore has replaced the previous array-based approach, which results in:

• Up to 20x reduction in memory usage during VACUUM operations
• Elimination of the previous 1GB memory usage limit for vacuuming tables
• Improved overall vacuuming performance

Enhanced Monitoring

In PostgreSQL 17 new columns are added to the pg_stat_progress_vacuum system view which provides better insights to the VACUUM process:

• dead_tuple_bytes: Shows the amount of space occupied by dead tuples
• num_dead_item_ids: Replaces the previous num_dead_tuples column

With these updates, the database administrators will be able to track the progress and efficiency of VACUUM operations more accurately.

Faster Index Cleanup

Index cleanup process removes dead index entries from indexes during VACUUM operations. In the new version, the speed of index cleanup during VACUUM got a significant improvement by changing the mechanism for removing dead tuples, now it uses the TidStore to collect dead tuples from the heap and clean up the index more efficiently.

Cleaning indexes can be resource-intensive task, especially for tables with many or large indexes.

Other Significant Upgrades

While the VACUUM improvements are substantial and impressive, the PostgreSQL 17 offers several other key enhancements:

JSON Capabilities

This one is my favorite, PostgreSQL 17 expands its JSON support with new features that align with the SQL/JSON standard:

• Introduction of the JSON_TABLE() function, which converts JSON data into a table representation
• New JSON constructors and query functions like JSON_EXISTS(), JSON_QUERY(), and JSON_VALUE()

In the current technology landscape, the JSON is the defacto widely used standard data format, these additions make working with JSON data in PostgreSQL more flexible and powerful, bridging the gap between relational and document-based data models.

Performance Improvements

In this release, several performance enhancements have been implemented:

• Improved I/O performance with a new streaming I/O interface, benefiting sequential scans and ANALYZE operations
• Better handling of IN clauses with B-tree indexes, leading to faster query execution
• Optimizations for IS NULL and IS NOT NULL clauses, reducing unnecessary evaluations

Logical Replication Enhancements

In PostgreSQL Logical Replication refers to a method of replicating data objects and their changes based on the replication identity (usually a primary key) of the objects. PostgreSQL 17 introduced following improvements to logical replication:

• Failover control for high availability environments
• Preservation of replication slots during major version upgrades (from version 17 onwards)
• Introduction of pg_createsubscriber, a utility for creating logical replicas from physical standbys

Incremental Backup Support

The most awaited significant addition is the support for incremental backups:

• New --incremental option in pg_basebackup for creating incremental backups
• Introduction of pg_combinebackup for manipulating base and incremental backups

This feature is particularly valuable for large databases where full backups are time-consuming and resource-intensive.

MERGE Command Enhancements

The MERGE command which was introduced in PostgreSQL 15, now receives further improvements:

• Addition of the RETURNING clause, allowing retrieval of modified rows
• Support for updating views using the MERGE command

Summary

PostgreSQL 17 brings substantial improvements across various areas, with noteworthy VACUUM enhancements. These upgrades collectively contribute to better performance, increased flexibility, and improved manageability of PostgreSQL databases. As organizations continue to deal with growing data volumes and complex workloads, these enhancements make PostgreSQL 17 an even more attractive option for enterprise database needs.

This is not an exhaustive list of enhancements, there are various other significant improvements in SQL, Security, Storage, etc. you can take a look at complete list here and the following references.

References:

• https://www.alibabacloud.com/blog/new-features-you-should-not-miss-in-apsaradb-rds-for-postgresql-17_601679
• https://www.postgresql.org/about/news/postgresql-17-released-2936/
• https://andyatkinson.com/postgresql-17-json-table-merge-returning-updatable-views
• https://www.postgresql.org/docs/current/release-17.html
• https://www.postgresql.org/about/news/postgresql-17-beta-1-released-2865/
• https://www.postgresql.fastware.com/blog/postgresql-17-and-trends-and-innovations-to-watch
• https://www.enterprisedb.com/blog/exploring-postgresql-17-new-features-enhancements

PostgreSQL 17: VACUUM Overhaul and More was originally published in Quantrium.ai on Medium, where people are continuing the conversation by highlighting and responding to this story.

A Closer Look at the Garbage Collector

Abstract

This article explores Python’s Garbage Collector (GC), a key feature for automated memory management. It discusses the GC’s design principles, including reference counting and its role in handling cyclic references. It also explores the Generational GC approach, which optimizes memory management by categorizing objects into generations based on their lifespan. Finally, it highlights the importance of GC optimization in real-world applications with frequent object generation and disposal, and shows how it can improve memory management and reduce performance overhead.

Introduction

Python, a popular programming language known for its simplicity and flexibility, offers a significant advantage: automatic memory management.

Garbage collection is a technique for automatic memory management. It tracks objects in memory and deallocates them when they are no longer needed. This frees up memory for other programs to use and prevents memory leaks.

Python’s garbage collector (GC) is important because it frees programmers from having to manually manage memory. This article will explore how the GC works and its role in memory management.

What is Python’s Garbage Collector?

In Python, the Garbage Collector is a built-in feature with it’s primary role to identify and remove objects that are no longer referenced and freeing up memory for other data. This process helps prevent memory leaks and keeps Python applications running efficiently.

Garbage Collector Design

Python uses reference counting to track how many references an object has. When the reference count reaches zero, the object is deallocated. This scheme covers most cases, but it doesn’t handle reference cycles.

Cyclic reference cycles are situations where objects reference each other, preventing reference counts from reaching zero.

For instance, consider a container object, container, holding a reference to itself. Even when you remove your reference to it, the reference count never reaches zero due to its internal reference.

Python’s garbage collector handles reference cycles by using a cyclic garbage collector. A cyclic garbage collector is a type of garbage collector that can detect and break reference cycles.

Memory Layout and Object Structure

Following is the representation of the normal C structure which supports a regular Python object:

Memory layout in Python is altered to accommodate garbage collection. Two extra fields are added to the memory layout of objects: _gc_next and _gc_prev.

These fields are used to maintain doubly linked lists of all objects tracked by the garbage collector. These lists partition objects into generations and support efficient operations for moving, adding, and removing objects during collection.

The Reference Counting Approach

The fundamental mechanism behind Python’s GC is reference counting. Every Python object has an associated reference count. When an object’s reference count drops to zero, it means there are no references to that object, making it eligible for removal.

Consider this example:

a = [1, 2, 3]
b = a

In this case, both a and b reference the same list. If we assign a new value to a or b, the reference count decreases by 1. When both a and b are reassigned to different values, the reference count drops to zero, and the list can be removed by the GC.

The Cyclic Reference Problem

While reference counting is effective for most situations, it has limitations, particularly with circular references. Circular references occur when two or more objects reference each other, preventing their reference counts from dropping to zero.

For instance:

python
class Node:
 def __init__(self):
 self.next = None

node1 = Node()
node2 = Node()

node1.next = node2
node2.next = node1

In this example, node1 and node2 reference each other, and their reference counts never reach zero. This is where Python’s Garbage Collector comes into play.

The Generational GC Approach

To address the cyclic reference problem, Python utilizes a generational GC approach. It is one of the algorithm which used to optimize memory management. In this approach, objects are divided into three generations: Young, Middle-Aged, and Old.

• Young objects are newly created and are considered short-lived.
• Middle-Aged objects have survived a few Garbage Collection cycles.
• Old objects are long-lived and have survived multiple GC cycles.

The GC gives more attention to Young objects since they are likely to become unreachable sooner. As objects survive more GC cycles, they are promoted to older generations.

Let’s take a deeper dive into Generational GC approach:

The Generational Hypothesis

The generational GC approach is based on the generational hypothesis, which assumes that most objects die young. In other words, newly created objects are more likely to become garbage soon, while objects that have survived multiple garbage collection cycles are more likely to remain in use for a longer time.

Generational Segregation

In Python, memory is divided into multiple generations, typically three: Generation 0 (young), Generation 1 (middle-aged) and Generation 2(old). Objects start in Generation 0, and if they survive a garbage collection cycle, they are promoted to Generation 1 and when the same object survive next garbage collection cycle, it is promoted to Generation 2.

Collection Frequency

The garbage collector in Python is designed to collect Generation 0 more frequently than Generation 1 or Generation 2 because of the generational hypothesis. This reduces the overhead of collecting objects that are likely to become garbage shortly.

Collection Algorithm

Generational GC in Python uses two primary algorithms for collection:

1. Young Generation Collection: The collector frequently checks Generation 0 for objects that are no longer referenced. This is often done using techniques like reference counting or more sophisticated methods like the tri-color algorithm.
2. Old Generation Collection: Generation 1 and Generation 2 are collected less frequently because older objects are assumed to have a longer lifespan. When a collection does occur in Generation 1 or Generation 2, it’s often more expensive as it involves traversing more objects.

Garbage Collector Modules

Python provides modules like gc that allow you to interact with and configure the garbage collector. You can manually trigger a collection using gc.collect(), and you can adjust the collection thresholds and other settings.

Tuning and Optimization

Python’s generational garbage collection can be tuned and optimized for specific applications. You can adjust the collection frequency and thresholds using the gc module, but you should be cautious and only do this when necessary.

To determine the appropriate timing for execution, the collector maintains a record of object allocations and deallocations since the previous collection. When the difference between allocations and deallocations surpasses the predefined threshold_0 value, the garbage collection process is triggered.

Initially, the collector focuses on inspecting objects in Generation 0. However, if Generation 0 has undergone more examinations than Generation 1 since the last check, the collector extends its inspection to include Generation 1. These configurable thresholds are accessible for review and adjustment via the gc.get_threshold() function.

>>> import gc
>>> gc.get_threshold()
(700, 10, 10)

The Generation 2 is bit more complex. Unlike the customizable thresholds for Generation 0 and Generation 1, the garbage collector (GC) will only initiate a full collection of the oldest generation when a specific condition is met:

the ratio of pending long-lived objects to the total long-lived objects exceeds a fixed threshold (hard-set at 25%).

This is because while partial collections (which target younger and mid-level generations) consistently inspect a roughly consistent number of objects based on the set thresholds, the cost of a full collection is directly related to the total count of long-lived objects, which can potentially be extremely high.

In practical terms, frequently performing full collections after every fixed number of object creations can severely degrade performance in scenarios where many long-lived objects are created and stored (e.g., constructing a substantial list of objects tracked by the GC can lead to quadratic performance issues instead of the expected linear performance).

Instead, using the mentioned ratio provides a more balanced approach:

each full garbage collection becomes more expensive as the total object count increases, but the frequency of these expensive operations decreases, resulting in an overall performance that scales linearly with the number of objects (meaning that as the number of objects grows, the impact of each full garbage collection is lessened).

Manual Garbage Collection

While Python’s GC system is automatic and efficient, developers can also manually trigger Garbage Collection using the gc module. This is useful in cases where memory needs to be managed more explicitly.

To manually run Garbage Collection, you can use the gc.collect() method. Here’s an example:

import gc

# …your code…

gc.collect() # Explicitly run Garbage Collection

A Real World Example of GC Optimization

Let’s consider a web application that experiences high traffic and continuously generates short-lived objects, such as request and response objects, user sessions, and database query results. In such a scenario, frequent garbage collection can lead to performance overhead.

To optimize performance in this case, the GC configuration can be adjusted:

Generational GC Configuration

You can tune the generational GC approach to control how often it runs in different generations of objects. For example, you can increase the collection frequency of Generation 0 to collect short-lived objects more aggressively, reducing the overhead of checking and cleaning them. This ensures that the most commonly generated objects are quickly collected, minimizing memory fragmentation.

Threshold Adjustments

You can adjust the garbage collection thresholds for both the number of allocations and the frequency of examinations between generations. For short-lived objects, you can lower these thresholds to trigger more frequent but quicker collections.

Manual Collection

If you know when certain objects are no longer needed, you can manually trigger the GC at the right time using the gc.collect() function. For example, you might want to run manual collection after processing a user's request and releasing associated resources.

Disabling or Reducing GC

In extreme cases, if your application can manage memory explicitly, you can temporarily disable or reduce the impact of the GC with the gc.disable() function. Use this with caution and only for specific, short-lived scenarios.

You can configure the GC to manage the frequent creation and disposal of short-lived objects efficiently, reducing the chances of performance problems due to garbage collection.

Demystifying Python’s Garbage Collector: Cleaning Up the Mess was originally published in Quantrium.ai on Medium, where people are continuing the conversation by highlighting and responding to this story.

Save time and boost accuracy with automated spaCy re-training

Introduction

Spacy is a popular NLP library that can be used for a variety of tasks, such as Named Entity Recognition (NER), Text Classification, and Sentiment Analysis. spaCy has exceptional NER capabilities which is a crucial component in Natural Language Processing (NLP) applications. It enables the identification and categorization of entities within text, such as names of people, organizations, locations, dates, and more.

When you train the NER models on a specific dataset, they excel. However, as time goes on, the data landscape evolves. New names, entities, and patterns emerge, making it essential to adapt your NER models to stay effective.

We delved deeper into spaCy’s capabilities and discovered an exciting breakthrough: automating the re-training process. Let’s see — how?

Why Automated?

Let’s consider that you are working on a project where you’re using NER to extract information from user reviews for a restaurant recommendation system. You will see that the names of new restaurants, local businesses, and even emerging food trends constantly change. To keep up your model up-to-date manually can be time-consuming and error-prone.

Manually training and re-training NER models can be arduous, involving data preprocessing, entity annotation, and feature engineering. Additionally, it’s not scalable and requires efforts everytime the model is retrained. Automation streamlines this process and often yields better model performance.

The spaCy provide features that can be used to retrain the model and the process can be automated. The benefits include:

• Increased efficiency: It can save a significant amount of time and efforts, especially, for large datasets or datasets that are constantly changing.
• Improved accuracy: It can improve the accuracy of spaCy models because the models are constantly being updated with new data.
• Reduced manual intervention: Automation can reduce the need for manual intervention and frees up time for the ML Engineers to focus on other tasks.

The Automated Process — A Closer Look

Automated updation of your NER model periodically with new data, enables it to adapt to the evolving language and entities in your domain. Here’s how it works:

Data Collection

Setup a pipeline which continuously gather new text data that represents the domain or context in which your NER model operates. This data should include examples of entities you want your model to recognize.

Data Preprocessing

Add a data preprocessing stage to your pipeline where the new data is prepared by cleaning and formatting it, ensuring it’s in a suitable format for NER training. This step will require manual review or annotation of the training data. Once prepared, combine your existing training data with the new data, creating a comprehensive dataset for retraining in .spacy format.

Initiating Automated Retraining

Once the training data is generated, for automatic retraining we can initiate the update functions feature in spacy as soon as the files are updated or added. It leverages pre-defined hyperparameters and training configurations to optimize the model to new patterns and changes in the input data, leading to a upgraded model with improved performance.

Performance Monitoring and Model Selection

After retraining, add an stage to your pipeline which does the constant monitoring of the model’s performance using metrics like precision, recall, and F1-score. And if the model’s performance is unsatisfactory, the then it should trigger retraining with a different set of hyperparameters or you may need to switch to a different architecture to enhance the model’s capabilities.

Use Cases for Automated Spacy Re-training

At Quantrium, we have used custom, trained from scratch spaCy model on our specific classes and data to automate various processes with document intelligence in Finance and Insurance Industry. For example, extraction and identification of various keys and values in:

• Paystubs or Salary Slips
• Financial Statements (such as Balance Sheet, Profit & Loss and Cash Flow statements)
• Tax Documents (such as Income Tax Returns, Form 26AS, W-2 and 1099 Forms)
• Bank Statements and Passbooks
• Insurance Claim Documents (Insurer, Insured Person, Amount, Charges, Penalty, X12 Codes, etc.)

Automated retraining with spaCy is a powerful strategy for maintaining the accuracy and relevance of your NER models. In the ever-evolving world of text data, staying up-to-date is not a luxury but a necessity. Embrace automation and watch your NER models shine in accuracy and adaptability, providing better insights and performance in your NLP applications.

Automated spaCy Re-training was originally published in Quantrium.ai on Medium, where people are continuing the conversation by highlighting and responding to this story.

Utilizing multiprocessing and Colab GPU for PPOCR recognition

Introduction

Over the last few months, I have been writing some articles on Optical Character Recognition (OCR) using Python. But, one of the challenges faced even today with OCR in Python is the recognition time taken by some OCR models. We can find some of the OCR models (such as easyOCR, docTROCR, etc.) performing very fast when implemented with a Graphical Processing Unit (GPU).

If you had gone through my article on integrating multiple OCR models (which can be found here), you would have noticed recognition models such as PPOCR, pytesseract, etc., which perform recognition coordinate-wise. A major drawback with these recognition models is that, since they recognize the text coordinate-wise, their recognition time is much higher than recognition models like easyOCR, which takes the list of all coordinates and images at a time to perform recognition.

Also, if you try performing recognition using PPOCR using a Google Colab GPU directly, you may observe that the GPU is not consumed at all by the recognition model and a warning pops out in some time stating that the GPU is not consumed. Also, GPU availability in such platforms is very limited. However, there is a procedure to perform PPOCR recognition by consuming the Colab GPU, which I have explained in the upcoming sections. PPOCR documentation can be found here.

I have also explained how PPOCR recognition can be done using multiprocessing, which is a basic solution with which one can start and try improving the process even further.

We will test the recognition using all the methods with the following image (in fact, I used the same image in my docTROCR article. The article can be found in this link):-

Implementing PPOCR recognition with ordinary CPU

For comparison purposes, we will first check and record the time taken by ordinary CPU for PPOCR recognition.

The following installations should be made:

!pip install "paddleocr>=2.0.1"
!pip install paddlepaddle

After the above installations, the following imports and definitions should be made:

from paddleocr import PaddleOCR
ocr = PaddleOCR(lang='en')

To obtain the detection coordinates, I have explained the complete procedure in the articles whose links I have shared above. Now using the text_coordinates, we will perform PPOCR recognition:

import cv2
import time
img_path = <> #define your image path here
start = time.time()
for coord in text_coordinates:
    x_min, y_min, x_max, y_max = coord[0], coord[2], \
                                 coord[1], coord[3]
    if x_min >= 0 and x_max >= 0 and \
       y_min >= 0 and y_max >= 0:
        im2 =cv2.imread(img_path)
        cropped = im2[y_min: y_max, x_min: x_max]
        text = ocr.ocr(cropped, det=False, cls=False)
end = time.time()
print(f"Time taken with ordinary CPU: {end-start} sec")

The following output was obtained in my case:

Time taken with ordinary CPU: 11.89705514907837 sec

PPOCR recognition with multiprocessing

The installations for this are the same as the previous one. The following additional imports need to be made:

from multiprocessing import Pool
from functools import partial
pool = Pool(processes=8)

More about multiprocessing can be understood from its documentation which can be found here.

Now, we will implement the recognition in a slightly different way:

def perform_ocr_recognition(coord, im2):
    text = ""
    x_min, y_min, x_max, y_max = coord[0], coord[2], \ 
                                 coord[1], coord[3]
    
    if x_min >= 0 and x_max >= 0 and \
       y_min >= 0 and y_max >= 0:
        cropped = im2[y_min: y_max, x_min: x_max]
        text = ocr.ocr(cropped, det=False, cls=False)
        return {str(coord): text[0][0]}
img = cv2.imread(img_path)
start = time.time()
text_output = pool.map(partial(perform_ocr_recognition, im2=img),
                  text_coordinates
              )
end = time.time()
print(f"Time taken with multiprocessing: {end-start} sec")

The partial module is used to fix an argument throughout the process (especially in a loop). In our case, the image array would remain the same for the entire recognition process. That is why I declare it as a fixed argument and loop through the text coordinates outside the partial method.

The map function is used to automatically append the output of each iteration to a list. You can obtain the coordinate and text as dictionary objects and this can be checked by printing out the text_output variable.

The following output was obtained in my case (and it will vary for different users and systems):-

Time taken with multiprocessing: 7.432933568954468 sec

PPOCR recognition with Colab GPU

The following installations have to be made:-

!pip install "paddleocr>=2.0.1"
!pip install paddlepaddle
!python -m pip install paddlepaddle-gpu #or python3

The paddlepaddle-gpu can be used to implement PPOCR by utilizing GPU (especially when you have GPU-CUDA set up in your platform). More information on installing this on your system can be found in the paddlepaddle documentation.

The imports and implementation are the same as that of ordinary CPU:-

from paddleocr import PaddleOCR
import cv2
import time
ocr = PaddleOCR(lang='en')
img_path = <> #define your image path here
start = time.time()
for coord in text_coordinates:
    x_min, y_min, x_max, y_max = coord[0], coord[2], \
                                 coord[1], coord[3]
    if x_min >= 0 and x_max >= 0 and \
       y_min >= 0 and y_max >= 0:
        im2 =cv2.imread(img_path)
        cropped = im2[y_min: y_max, x_min: x_max]
        text = ocr.ocr(cropped, det=False, cls=False)
end = time.time()
print(f"Time taken with GPU: {end-start} sec")

The following output was obtained in my case:-

Time taken with GPU: 0.5593817234039307 sec

Now let us try to understand the observations in the final section.

Observations and Conclusion

From the above experiments, the following observations are quite clear:-

• Ordinary CPU has taken more time when compared to the other methods. This is due to the iteration of each coordinate and a single process is utilized to recognize all coordinates in order;
• With multiprocessing, PPOCR recognition has executed faster than that of ordinary CPU, even though the time taken is still on the higher side. This is because multiple coordinates are recognized parallelly with the help of multiple processes utilized at the same time. But when testing with a large dataset, multiprocessing could save a lot of time than ordinary CPU execution;
• As expected, recognition with GPU was the quickest, but implementation was a bit different when compared to the other OCR tools;

Thus, I hope I have given you an idea of improving the PPOCR recognition process. In most cases, GPU availability would be low and hence multiprocessing may serve as a good starting point to increase PPOCR recognition speed. However with a GPU availability, one could install the paddlepaddle-gpu which serves specifically for the PPOCR model and optimize the time as shown in the previous sections. I would be more than willing to acknowledge questions and clarifications related to the topic which can be posted through the comments. Hope you have found this article useful. Thanks!

Improving PPOCR Recognition Time in Python was originally published in Quantrium.ai on Medium, where people are continuing the conversation by highlighting and responding to this story.