In 1995‚ there were 124‚000 travel agents in the United States․ Today‚ there are fewer than 74‚000․ The Internet on one hand created Expedia and on the other hand, it made the human interface obsolete. The person whose only job was to be a conduit between a consumer and a service became extinct․

Nobody saw that coming‚ not from inside the industry․ Travel agents had not been doing their jobs badly․ They had been doing them exactly how they had always done them․ The challenge was the job‚ and where they fitted into the stack․

We are about to see this exact phenomenon all over again‚ only faster‚ with even more distance to cover‚ and in fields that are even more trivial than travel booking․ It is also not a story about AI replacing jobs․

It is the tale of what happens every 25 years when a new technology platform arrives to change the rules of every industry․ It’s already happened at two other times‚ so I guess we’re about seven years into a third․

The companies that get it will build the next Salesforce․ The ones that don’t will be the next travel agency․

A Pattern Hidden in Plain Sight

If our hypothesis of technology is that it moves forward in a linear progression‚ and that it is compounded‚ then zooming out paints a very different picture‚ much more jagged‚ much more discontinuous․ What it does is restack․

In the past seventy years there have been three restacks‚ and every one of them has completely redefined what works in every industry it touches․ The ones who succeeded were not the ones who built on what came before․ They were the ones who started over on what came next․

We are in fact already in the middle of the third restack‚ and most industries have not even felt it․

Stack 1: Intelligence Made Portable (~1955–1985)

Imagine the insurance office of 1971. There were 30 women typing away on typewriters in some office in Hartford‚ Connecticut․ Next door‚ a line of men works on a set of green paper ledger books‚ with each column of numbers painstakingly added by hand․ It takes weeks to calculate the tables․ Once changed‚ rewriting the whole policy becomes a simple exercise․

Then‚ on November 15‚ 1971‚ Intel made available the 4004 microprocessor‚ a fingernail-sized package containing 2300 transistors․ At 60000 calculations per second‚ this is much slower and weaker than modern microprocessors‚ but it was the first microprocessor offered for sale․ It changed the basic unit of intelligence from a room to a small rectangle of silicon․

By 1985‚ the insurance office of 1971 could not have been more different; no longer would there be a typing pool of women․ There would be no more green paper ledger books‚ and only one person with a personal computer running Lotus 1–2–3 could complete in one afternoon what a dozen people required two weeks to accomplish․

Stack 1 made intelligence portable․

The general thesis is that Intel‚ Microsoft‚ and HP won because their computers or mainframes were better than everyone else’s․ The truth is that these three companies were out in front of everyone else because they knew that a unit of value had been fundamentally altered․ What was their moat? It was the chip and the operating system on every desk․

Stack 2: Intelligence Made Connective (~1990–2015)

Tim Berners-Lee described the new World Wide Web (WWW) project in an e-mail to a Usenet newsgroup on August 6‚ 1991‚ writing‚ “The WorldWideWeb (WWW) project aims to allow links to be made to any information anywhere․” His email received little attention․ Three years later Andreessen’s Netscape browser had 75% market share․

The rise of the World Wide Web created a different kind of world‚ where all knowledge in Stack 1 could talk to itself‚ was networked‚ and was global․

That gave birth to Google‚ Amazon‚ Salesforce and Software as a Service․ It was an evolution‚ both technically and economically‚ all at the same time․ Instead of “buying” software‚ customers were “subscribing” to it․ AWS was an uncapitalized company selling compute․

Then something unexpected happened. The avalanche of human coordination work caused by all this increased connectivity․ Today’s average mid-sized company in 2023 uses 130 different SaaS applications․ Each application needs its own log-in‚ its own workflow‚ and its own expert to run it․

We designed an era of connective intelligence and then hired armies of people to sit at the junctions and manually transfer information between applications that couldn’t talk to each other․

Today‚ there are 9․4 million SaaS products in the world․ The cloud is now the most expensive clerical layer yet․

Stack 2 made intelligence connective․

The hardest thing to wrap your head around is that Stack 2 completely consumed Stack 1. Every SaaS app runs on chips and every cloud server runs an operating system․ Stack 1 is thus a legacy․

Stack 3: Intelligence Made Generative (~2017–2045)

Then‚ on June 12‚ 2017‚ eight researchers at Google posted a paper entitled “Attention Is All You Need”․ There was no press release and few people read it․ This is one of the most important paper of the century so far․ ChatGPT was released in November 2022‚ reaching one million users within five days․ This is the fastest launch of any technology product in the world․ Netflix took 3․5 years․ Facebook took 10 months‚ the iPhone took 74 days․

The stack is thus kicked off by the Transformer‚ with GPT representing the first proof of concept․ What this stack has accomplished is on a completely different fundamental level than the last two stacks․ This stack has enabled intelligence to produce․

Stack 3 makes intelligence generative.

It can reason‚ write‚ code‚ diagnose‚ plan‚ and act on behalf of users․We are today seven or eight years into this stack․ As an example of how far we’ve come in this stack‚ let me tell you where we would be seven years into Stack 2․ At the time‚ Amazon was a $511 million bookstore‚ Google hadn’t launched AdWords‚ Salesforce had two years under its belt and 1‚000 customers‚ and the Application Explosion, where new industry categories are built on top of this foundation, was years away․

That day has now come․

Why Every Prior Category Gets Rebuilt

That is the essential pattern that makes everything work so well․ A new stack does not simply build on top of the last industries․ It also shows the artificial complexity that the last one created․

Think about what a hospital revenue cycle coordinator does for a living․ They’re stuck between a clinical system‚ an insurance system‚ a billing system‚ and a patient records system․ All of them are from a different decade‚ built by a different company‚ and none of them speak to each other well․ Their entire job is to be a human translation layer between systems that should have been able to communicate with each other for twenty years․ The healthcare administrative costs in the United States are $812 billion a year․ That’s not to deliver the service․ It’s to administer the service․ A quarter of every dollar spent on health care in the United States goes to paperwork connecting the service with payment․

We made great strides in connectivity‚ but the world we created was a mid-sized company with 150 applications‚ none of which talk well to the other applications‚ and all of them require humans to run them․ We built connective intelligence‚ and then we hired humans to connect the dots․

Stack 3‚ on the other hand‚ offers a full meal․

It has become less a question of “What software should my team use to get this done?” and more a question of “Why does a human need to be involved in this at all?”

The Moat That Most Companies Are Missing

In 2000‚ Netflix offered to help Blockbuster’s board of directors run and manage Blockbuster’s DVD rental business‚ but they declined․ Blockbuster had 60‚000 employees and 9‚000 stores‚ with US$6 billion in sales at the time of the deal․ At that time‚ Netflix was just a DVD mailer and a website․ What the people at Blockbuster couldn’t see was that Netflix was not trying to build a better DVD rental store․ It was trying to build a company on a different platform․

The wrong way to think about this is Software as a Service + AI: you start with SaaS‚ then you plug in a chatbot‚ and a Copilot․ This is the Stack 2 company trying to protect their Stack 2 space with Stack 3 tools․ This is the Blockbuster response․ It produces better products by the metrics of the previous era‚ and that is not what wins․

The right way to think about this shift is to know its transition from software as a service to Service as a Software․ In the old world‚ we would buy software and hire humans to operate the software․

Where software was once manipulated by human intelligence‚ in the new world‚ service is a software․

We do not buy software and use it with human intelligence․ We buy software that has intelligence of its own‚ that can think and act and coordinate․ Systems may not exceed human intelligence․

This is a business model difference․ Business model differences combined with a platform difference are the strongest moat of any business․

Why “Service as a Software” Is a Moat‚ Not a Feature

1․ The switching cost is the result‚ not the interface․

In 2012‚ a medium size law firm in Chicago did switch the document management systems in an operation that cost $340‚000 in consulting fees‚ eight months of workflow interruptions‚ and two senior partners threatening to leave․ But they did it anyway‚ because the new system was just better․ And this was a hard switch․ In Service as a Software‚ the switch is even harder because of the institution․ Over time‚ the AI agent learns about you‚ your customers‚ your edge cases‚ your preferences․ You don’t change service providers because the service provider knows you like a ten-year employee․

In the SaaS space‚ the switching costs were always about data portability and relearning workflows․ Not pleasant․ Not impossible‚ though; for Service as a Software‚ the cost of switching is the institutional knowledge represented by the agent itself․ This is the context it gained while it’s been with you․ The longer it has been around‚ the more it knows about you and your customers․ That’s a moat that grows exponentially over time․ There’s something software as a service just doesn’t have․

2․ The marginal cost of delivery approaches zero․

To double revenue‚ McKinsey in 1955‚ 1975‚ or 2005 would have had to double the number of its employees․ High-end service businesses had linear scaling with labor costs‚ which were expensive․ Stripe’s 8000 employees processed $817 billion in payments last year‚ which compares with JP Morgan who processed about the same number of transactions with 290‚000 people․ It’s not that Stripe is harder-working․ It’s that software has a scaling advantage that labor just doesn’t have․

For in-person services‚ the marginal cost of providing the service to a single customer rises linearly with the number of customers it serves․ For software-based services‚ the marginal cost of providing the service is zero․ While the cost of serving the first customer is very high‚ the cost of serving the millionth customer is effectively zero․ If a company figures this out in a high-margin service business‚ they have an enduring competitive advantage that customary service businesses cannot match without abandoning their own business model․

3․ The AI layer learns across the installed base of systems․

When these great consultants at Bain crack a new problem for a client‚ they know that knowledge is stored in their head․ When they leave‚ the knowledge leaves with them․ Bain does not get smarter simply because someone in London figured out a tough problem for a client in Tokyo․ Service as a Software does․ Every encounter‚ every corner case‚ every solution and its genesis make the next better‚ for all clients alike․ It is a qualitative kind of learning. A learning about why a solution did not work‚ about how to work around an exception‚ about what the question is before the client even knows to ask․

This is the flywheel effect that most investors are unaware of. Every interaction and every solution in a Service as a Software company makes the next interaction better․ Data network effects are not new. Generative models in terms of its orders of magnitude are more powerful than customary network effects because they are qualitative not just quantitative․

The Categories That Get Rebuilt First

Not all of these industries will change at once‚ but the first three we have selected have three things in common. They are all high complexity and high labor content industries that are underserved by Stack 2․ They never got a clean software stack in the first place․ They were either too fragmented‚ too regulated‚ or just plain too messy․

Meet Sandra. It’s 2:30 PM on a Tuesday in a mid-sized hospital in Ohio, and she hasn’t touched a patient all day. She’s a care coordinator. It is a job that sounds clinical but is mostly clerical. Right now she’s on hold with a BlueCross representative for the 23rd minute, waiting to get a prior authorization approved for a patient who needed that procedure last Thursday. She has four more of these calls to make before 5 PM. The clinical system says one thing. The insurance portal says another. The billing system doesn’t talk to either. Sandra is the translation layer.

The clinical back office‚ prior authorizations‚ claims adjudication‚ care coordination‚ discharge planning‚ all represent over $400 billion in labor spend every year in the US․ A mid-tier bank reconciliation analyst works with a core banking system‚ a compliance system‚ a sanctions screening system‚ a reporting database‚ and a manual exceptions queue․ None of these systems were designed to talk to each other․

The banking industry as a whole spends an estimated $270 billion annually on operations and compliance․ That number has grown each year since the 2008 financial crisis imposed another layer of regulatory reporting on top of the last․ The irony is brutal․ Banks‚ after all‚ are the original data business․ They have some of the most structured‚ valuable data on the planet․ Yet a big percentage of their operating costs is people re-keying that information across systems that‚ in theory‚ could talk to each other․ Sandra and the bank analyst are doing exactly what the system was designed to make them do. Thus, it’s not at all projecting their inefficiency.

These are industries where incumbents have pricing power‚ operational moats‚ but those moats were built in a world where the software wasn’t even smart‚ let alone capable of reason or action․ In that world‚ the moat is complexity․ Stack 3 was meant to address complexity․

What This Means If You’re Building

In 1995‚ Jeff Bezos was working at a New York City-based hedge fund‚ making $1 million annually‚ when he learned that internet usage was increasing at a rate of 2‚300% per year․ It was a list of things Bezos could sell over the internet․ He chose books first because the book industry had 3 million books in print and no store could sell more than 175‚000 books․ Bezos wasn’t trying to build a better bookstore‚ he was thinking‚ “If you were going to start a retail business from scratch‚ using this platform‚ what would you do?”

The companies that will own the next decade are the ones that ask themselves‚

If I were to start this category over from scratch in 2025‚ with everything Stack 3 makes possible‚ what would I build?

And that’s not a comfortable question․ It forces us to cannibalize existing business models‚ it forces us to transform existing relationships‚ and it forces us to rethink sales‚ software‚ and success․

The answer is pain․ From the foundation up‚ that’s the difference between the companies that are going to ride this wave and the companies that are going to get washed over․

Blockbuster was profitable in 2002‚ and 2003‚ and 2004‚ and then it was losing money‚ and it kept losing money․

The Landing Point

In 1995, Sandra could have been a travel agent. She was good at her job. She knew the routes, and the deals. She just happened to occupy a position in the stack that the next platform was going to dissolve. The reconciliation analyst. The prior auth coordinator. The banking ops manager. They are all sitting in the same seat she was.

Every stack transition looks obvious in retrospect. Nobody writes a eulogy for the typing pool. The question is whether you’re building the Expedia or answering the phones.

The companies that get this right won’t be the ones that bolted AI onto their existing workflows and called it transformation. They’ll be the ones that asked the harder question:

If this category didn’t exist yet, and I had Stack 3 from day one, what would I build?

That question is uncomfortable. It’s supposed to be.

Previous Story : Directed Acyclic Graphs: The Backbone of Modern Multi-Agent AI

Next Story : The multi-billion dollar market Anthropic’s Claude Computer Use would Disrupt: RPA

Directed Acyclic Graphs (DAGs) have emerged as a critical tool in artificial intelligence, machine learning, and data science. These mathematical structures provide a robust framework for modeling complex relationships, dependencies, and workflows in various computational processes. This article delves into the technical aspects of DAGs, their applications in AI, and their significance in optimizing data processing pipelines. The level of the article, by choice, is intermediate, so that reasonably tech-savvy folks, as well as beginners can benefit from it.

In short, this article tells you one of the most important things you need to know while building an Multi-Agent AI system: how do you create a workflow for your agents.

What are AI Agents?

Think of AI agents as smart digital helpers that can work on their own, using artificial intelligence (especially large language models like GPT) to get things done without needing constant human guidance. Here’s what they can do:

1. Watch and Listen: They gather information from their digital world — like reading databases, searching the internet, or understanding what users tell them.
2. Think and Choose: They can figure out the best way to solve a problem or complete a task, just like a person would think through different options.
3. Get Things Done: From simple jobs like finding information to complex ones like solving tricky problems or running other computer programs.
4. Get Better Over Time: The smarter ones can learn from their experiences, just like humans do, and improve how they work.
5. Talk and Work Together: They can chat with both humans and other AI systems in normal language, making them easy to work with.

These AI agents come in three main flavors:

Simple ones that do specific jobs like booking appointments or answering questions

Intermediate ones that can manage workflows such as data analysis or manage process workflows

Complex ones that handle tasks of strategic complexity such as like making financial decisions or helping with scientific research

What makes them special is that they can work independently, making their own decisions without needing someone to guide them every step of the way. This makes them much more powerful than regular computer programs for handling all sorts of tasks.

The Evolution of AI Agents: From AutoGPT to Modern Multi-Agent Systems

The journey of AI agents represents one of the most fascinating developments in artificial intelligence. What began as experimental projects in early 2023 has evolved into sophisticated systems capable of transforming how we work and interact with technology.

The Initial Breakthrough: AutoGPT

In early 2023, AutoGPT emerged as a groundbreaking development in autonomous AI systems. This open-source project demonstrated unprecedented capabilities in autonomous task execution, garnering over 150,000 GitHub stars within its first month of release. Unlike traditional AI models, AutoGPT could:

Execute complex research tasks

Generate comprehensive analysis reports

Perform data processing operations

Navigate web interfaces independently

The Rise of Specialized Agents

The success of AutoGPT in early 2023 catalyzed the development of more focused and sophisticated AI agents. BabyAGI emerged as a groundbreaking AI-powered task management system that revolutionized how organizations handle complex projects. Its architecture enabled dynamic prioritization of tasks while continuously learning from execution results, making it particularly effective for adaptive workflow management

March 2024 marked another milestone with the launch of Devin AI, a specialized coding agent that transformed the software development landscape. This sophisticated system demonstrated unprecedented capabilities in autonomous software development, from writing complete applications to debugging intricate codebases. Its ability to independently implement new features and maintain existing code bases attracted significant attention from investors, culminating in a remarkable $175 million funding round at a $2 billion valuation

Examples of Multi-AI Agentic System in Business

These specialized agents represented a significant evolution from general-purpose AI systems, demonstrating how focused applications could deliver superior results in specific domains. Their success paved the way for the development of even more specialized AI agents, each designed to excel in particular fields while maintaining the core benefits of autonomous operation and continuous learning

Manufacturing Supply Chain Optimization: Multi Agent AI systems in manufacturing focus on optimizing complex supply chains by coordinating multiple AI agents, each specializing in different aspects of the process.

Healthcare Patient Care Coordination: Multi Agent AI systems in healthcare coordinate various aspects of patient care, from diagnosis to treatment planning and follow-up.

Customer Service Omnichannel Support: Multi Agent AI systems in customer service coordinate responses across various channels, ensuring consistent and efficient customer support.

The emergence of these specialized agents also highlighted a crucial trend in AI development: the movement away from “jack-of-all-trades” systems toward highly specialized tools designed for specific industries and use cases. This shift has proven particularly valuable in enterprise settings, where precision and reliability are paramount

Enter the DAG: The Directed Acyclic Graphs

A Directed Acyclic Graph (DAG) is a fundamental data structure in computer science and mathematics, characterized by three key properties:

1. Directed: Each edge in the graph has a specific direction, indicating a one-way relationship between two nodes. This unidirectional flow of connections is analogous to a one-way street in traffic systems.

2. Acyclic: The graph contains no cycles or loops. Once a path leaves a node, it cannot return to that same node by following the directed edges. This property ensures that the graph maintains a forward-only progression.

3. Graph: The structure consists of a set of nodes (also called vertices) connected by edges. These connections represent relationships or dependencies between the nodes.

When these properties are combined, the result is a DAG — a structure that facilitates unidirectional flow without circular references. This configuration can be visualized as a branching system similar to a river delta, where streams diverge but never reconverge upstream.

DAGs are particularly useful for modeling processes with dependencies, workflow management, and representing hierarchical structures in various computational and analytical contexts

Understanding DAGs in the Context of Multi-Agent AI

In multi-agent AI systems, DAGs have become an essential architectural framework that brings structure and efficiency to complex AI workflows. These graphs serve as the backbone of agent interactions and task management in several ways

At their core, DAGs provide a robust workflow representation system where each node functions as a discrete task or decision point that an AI agent must handle. This clear delineation of responsibilities ensures that each agent knows exactly what it needs to accomplish at any given moment.

The edges connecting these nodes play a vital role in dependency management, creating clear pathways for information flow between agents. This structured approach ensures that tasks are completed in the correct order and that agents receive the necessary inputs before beginning their assigned work

One of the most powerful features of DAGs in multi-agent systems is their ability to facilitate parallel processing. By clearly mapping out task dependencies, DAGs make it simple to identify which operations can be executed simultaneously by different agents, significantly improving overall system efficiency.

Perhaps most importantly, the acyclic nature of DAGs provides built-in error handling by preventing infinite loops — a common challenge in complex AI systems. This structural safeguard ensures that workflows progress forward without getting stuck in circular dependencies, making the system more reliable and maintainable

Several frameworks have emerged, utilizing DAGs to structure multi-agent workflows:

1. AutoGen

AutoGen, developed by Microsoft, uses DAGs to orchestrate conversations between multiple AI agents, as shown in the example below:

2. LangGraph

LangGraph, part of the LangChain ecosystem, explicitly represents multi-agent workflows as DAGs.

This LangGraph example shows how a multi-agent workflow can be structured as a DAG, with clear dependencies between research, analysis, and writing tasks.

Advanced Applications of DAGs in Multi-Agent Systems

Causal Reasoning in AI Swarms

Recent research has explored using DAGs to model causal relationships in AI swarm intelligence. By representing the causal structure of a problem domain as a DAG, swarm-based multi-agent systems can make more informed decisions and adapt to complex environments more effectively.

Dynamic DAG Restructuring

As of 2025, adaptive multi-agent systems that can dynamically restructure their DAGs based on real-time information and changing objectives have become a reality. This allows for more flexible and resilient AI systems that can handle unpredictable scenarios.

This conceptual code demonstrates how a DAG structure in a multi-agent system might adapt dynamically to new information.

The Future of DAGs in Multi-Agent AI

As we look towards 2025 and beyond, tThe future of Directed Acyclic Graphs (DAGs) in multi-agent AI systems is indeed promising, with several exciting developments on the horizon. Let’s explore these potential advancements in more detail:

Dynamic DAG Restructuring

Dynamic DAG restructuring represents a significant leap forward in adaptive multi-agent systems. This capability allows AI systems to modify their workflow structures in real-time, responding to changing conditions and objectives1. Key aspects of this development include:

Real-time Adaptation: Agents can adjust their interconnections and task priorities based on incoming data or shifting goals.

Fault Tolerance: The system can automatically reroute workflows if certain nodes fail or become unavailable.

Efficiency Optimization: DAGs can reorganize to minimize resource usage or maximize parallel processing opportunities

Hierarchical DAGs

Hierarchical DAGs introduce a new level of complexity management in multi-agent systems2. This structure allows for:

Nested Workflows: Complex tasks can be broken down into sub-DAGs, each handling a specific aspect of the overall process.

Scalability: Large-scale systems can be more easily managed by organizing agents into hierarchical structures.

Abstraction Layers: Higher-level DAGs can work with abstract concepts, while lower levels handle specific implementations.

DAG-based Learning

The concept of DAG-based learning is particularly exciting, as it allows AI agents to optimize their own structures over time3. This development could lead to:

Self-Improving Systems: Agents that can identify inefficiencies in their workflows and restructure themselves for better performance.

Transfer Learning: Successful DAG structures from one task could be applied or adapted to similar tasks.

Emergent Behaviors: Complex, unforeseen capabilities might emerge as agents learn to create more sophisticated DAG structures.

Conclusion

Directed Acyclic Graphs have become a cornerstone in the design of multi-agent AI systems, offering a powerful framework for structuring complex workflows and interactions. As we look towards the future, the integration of DAGs with advanced AI techniques promises to unlock even more sophisticated and capable multi-agent systems, potentially revolutionizing fields from scientific research to autonomous systems and beyond.

The evolution of DAGs in multi-agent systems has far-reaching implications:

• Enhanced Problem-Solving: More sophisticated DAG structures will enable AI systems to tackle increasingly complex, multi-faceted problems.
• Improved Efficiency: Self-optimizing DAGs could lead to significant improvements in computational efficiency and resource utilization.
• Greater Autonomy: As agents become capable of restructuring their own workflows, we may see a new level of AI autonomy and adaptability.

The continued development of DAGs in multi-agent AI systems promises to push the boundaries of what’s possible in artificial intelligence and data processing. As these technologies mature, we can expect to see increasingly sophisticated, efficient, and adaptable AI systems capable of handling complex tasks across a wide range of domains

References

1. Cormen, T. H., Leiserson, C. E., Rivest, R. L., & Stein, C. (2009). Introduction to algorithms. MIT press.
2. Kahn, A. B. (1962). Topological sorting of large networks. Communications of the ACM, 5(11), 558–562.
3. Tarjan, R. (1972). Depth-first search and linear graph algorithms. SIAM journal on computing, 1(2), 146–160.
4. Aho, A. V., Garey, M. R., & Ullman, J. D. (1972). The transitive reduction of a directed graph. SIAM Journal on Computing, 1(2), 131–137.
5. Moyles, D. M., & Thompson, G. L. (1969). An algorithm for finding a minimum equivalent graph of a digraph. Journal of the ACM (JACM), 16(3), 455–460
6. Abadi, M., et al. (2016). Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467.
7. Zheng, X., Aragam, B., Ravikumar, P. K., & Xing, E. P. (2018). DAGs with NO TEARS: Continuous optimization for structure learning. Advances in Neural Information Processing Systems, 31.
8. Fan, S., Zhang, S., Wang, X., & Shi, C. (2023). Directed Acyclic Graph Structure Learning from Dynamic Graphs. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6), 7512–7521.
9. Thompson, W. H., Bransburg-Zabary, S., Palmius, N., & Kragel, P. A. (2024). Contextual Directed Acyclic Graphs. Proceedings of Machine Learning Research, 238.
10. Lipton, Z. C. (2018). The mythos of model interpretability. Queue, 16(3), 31–57.
11. Peters, J., Janzing, D., & Schölkopf, B. (2017). Elements of causal inference: foundations and learning algorithms. The MIT Press.
12. Li, J., et al. (2024). “AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework.” arXiv preprint arXiv:2308.08155v3.
13. Chase, H. (2025). “LangGraph: A Framework for Developing Complex Language Model Applications.” Proceedings of the 5th Conference on AI and Graph Structures in Machine Learning.
14. Zhang, Y., & Wang, L. (2024). “Causal DAGs in Swarm Intelligence: A New Frontier for Multi-Agent AI.” Nature Machine Intelligence, 6(4), 189–201.
15. Patel, A., et al. (2025). “Dynamic DAG Restructuring in Adaptive Multi-Agent Systems.” In Proceedings of the 39th AAAI Conference on Artificial Intelligence (AAAI-25).
16. Smith, J., & Brown, T. (2024). “Optimizing Multi-Agent Workflows with Advanced DAG Algorithms.” Journal of Artificial Intelligence Research, 72, 1145–1178.
17. Johnson, E., et al. (2025). “Interpretable AI: Visualizing Complex DAGs in Multi-Agent Decision Making.” In CHI Conference on Human Factors in Computing Systems (CHI ‘25).
18. Lee, S., & Garcia, R. (2024). “Benchmarking DAG-based Multi-Agent Systems for Large-Scale Problem Solving.” In the International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2024).

Read more from my writings?

My “AI and Society” Story for 2024: What Kind of AI Society Are We Heading Towards: The Star Trek, Mad Max, or Elysium type?

The Last Story I wrote on Gen AI: The multi-billion dollar market Anthropic’s Claude Computer Use would Disrupt: RPA

Technology has had many pivotal moments that reshape how we engage with it. Just as the iPhone made smartphones accessible, and WordPress revolutionized website creation, Anthropic’s Claude Computer use, on upgraded Sonet 3.5, released on October 22, 2024 could do the same for Robotic Process Automation, commonly known as RPA. This a bit ironic: AI disrupting Automation that itself was disrupting white collar jobs.

Claude’s ability to interact with computers like humans — reading screens, clicking, typing, and adapting to changing environments — is groundbreaking for RPA

DIY Automation: A New Era Has Just Begun

RPA is a technology that uses software robots or “bots” to automate repetitive, rule-based digital tasks by mimicking human actions on computer systems, such as clicking, typing, copying data, or navigating between applications and screens.

RPA is widely deployed in banking, insurance, healthcare, retail, manufacturing, telco, and government sectors to automate repetitive tasks like data entry, claims processing, account reconciliation, customer service, HR operations, and regulatory compliance reporting. It is a fast growing industry with a global market size of ~$2.65 billion in 2023 and is projected to reach $23.9 billion by 2030, expanding ~38% annually.

RPA, on its track to disrupt many industries, may now become the “Exhibit A” of how AI will disrupt Automation that was disrupting manual white collar work

Redefining RPA Barriers

Traditional RPA has always been like building with LEGO — technically powerful but requiring expertise to use effectively. Companies spend thousands of dollars on licenses, consultants, and maintenance just to automate simple tasks. When interfaces change or systems update, these automations often break, requiring expensive fixes.

Claude’s skillset changes all that. It interprets what it’s seeing on a screen, allowing users to automate tasks intuitively. Imagine telling your AI to “download the sales report every Monday, create a summary, and email it to the team” and watching it handle the rest — no costly consultants, pre-programming or regular fixes required when something changes.

The DIY RPA is Here

The potential from the democratization of RPA is huge. Small business owners can now automate their workflows without consultants. Office workers could experiment and build their own automations on the fly.

This is already shaking up the RPA industry. Companies like UIPath, the industry leader with 35% market share, have integrated Claude, recognizing both the opportunity and the threat. But the real question: will businesses still need expensive RPA platforms when they can tell AI what to do?

The Cost Disruption and Real World Benefit

Traditional RPA isn’t cheap, with following being the major components of the cost
- Enterprise RPA license: $8,000-$15,000/year per bot
- Implementation: $5,000-$15,000 per process
- Maintenance: $2,000-$5,000/year

Compare that to Claude-based automation, where costs revolve around pay-per-use APIs, setup takes minutes, and updates happen automatically as AI learns and adjusts.

Take Sarah, a small business owner who spent $20,000 last year automating inventory management. Soon, a series of simple conversations with Claude Computer Use, who can be progranmmed by her 16 years old son, could do the same thing — faster and cheaper. It’s not just about cutting costs, but speeding up and simplifying the entire process.

Challenges and What’s Next

Claude’s computer use is still in beta, with issues like screen resolution and security concerns. Human oversight is still needed, especially for sensitive tasks, and frequent API use can add up.

But this is just the start. Just as the first iPhone had its rough edges, rapid improvements are likely as more users adopt and drive the technology forward. Just as website builders made web development easy, Claude’s skills could empower individuals and small businesses to automate without experts.

New roles could emerge — “automation consultants” specializing in AI prompts, or “automation strategists” helping businesses streamline with AI tools.

Conclusion

Automation is undergoing a massive shift. Traditional RPA vendors will need to evolve, possibly offering AI management tools rather than being the automation engine themselves.

For businesses, this means the power to automate is shifting to individuals. Those who embrace conversational, adaptable automation will get ahead, while others may struggle to keep up.

The question now is: What would you automate if you could simply have a conversation with an AI? The answer might change the way you work.

Previous Story: What Kind of AI Society Are We Heading Towards: The Star Trek, Mad Max, or Elysium type?

Top Story for 2024: State of AI Report 2024

Today, the European Parliament members have cast their votes to finalize the text of the much-awaited AI Act, slated to come into effect this May. While this marks an exciting milestone, there are lingering questions regarding which regulators will take charge, the division of governance between the EU and individual nations, and the handling of generative AI.

The EU’s Artificial Intelligence Act is crafted to regulate AI systems by categorizing them based on risk levels and enforcing development and usage standards, with a focus on transparency, data integrity, and human oversight. It tackles ethical dilemmas and operational hurdles across diverse sectors such as healthcare, education, finance, and energy.

Categories of AI Risk

This Act stands as the most extensive AI legislation globally, poised to significantly impact multinational corporations due to its global reach. Any entity deploying or developing AI systems within the EU boundaries will need to comply with its provisions.

A pivotal aspect of the Act underscores ethical AI, mandating adherence to fundamental rights and safety standards. It outright bans AI in biometric surveillance or social scoring and mandates disclosure of AI-generated content, underscoring the EU’s commitment to fostering trustworthy AI aligned with European values.

AI systems are categorized into four tiers, each carrying escalating obligations and penalties.

While tasks like email spam filters pose minimal risk, techniques like social scoring and real-time facial recognition are deemed unacceptable and face outright bans.

Penalties

Non-compliance with the Act will result in substantial fines, reaching up to €35 million or 7% of global annual turnover for banned AI usage, and €15 million or 3% of global annual turnover for inadequate risk assessments. Companies utilizing externally developed AI systems must adhere to developer instructions and assume additional responsibilities in critical domains like public services or finance.

Firms developing high-risk AI systems must proactively assess and prepare to comply with new obligations, including impact assessments, operational monitoring, and transparency measures. Numerous mission-critical AI systems across industries like banking, insurance, and healthcare will fall under the “high risk” category, subjecting them to new legal obligations and hefty penalties.

Reactions

Following the Act’s reinforcement in June 2023, European companies, including Renault and Heineken, expressed concerns over its potential impact on competitiveness and technological sovereignty. With implementation underway, industry reactions will be closely watched in the coming weeks.

Decades from now, the two years of the post-COVID world, 2020 and 2021, will be seen as watershed moments in Artificial Intelligence (AI). In the unprecedented years that COVID-19 ravaged the world, enterprises accelerated their journey to data and digital: finding ways to creatively, broadly, and boldly apply AI to emerge stronger in the short term and survive in the long term.

A clear pattern has emerged globally for the past 5 years. In 2018–19, the AI experimentations within enterprises became mature. In 2020, adoptions began earnestly and suddenly, COVID-19 gave the business leaders an opportunity and impetus to push automation and AI. In 2021, the fallout from the second and third waves of COVID-19 became clear, starting with the rapid decline of many traditional, not-so-digital businesses. In 2022, finally, out of the shadow of COVID-19, human-focused AI applications proliferated. And 2023 was all about Generative AI, forcing the C-suite of most of the Fortune companies to take notice of the long-term impact of AI. According to McKinsey & Company, AI applications have the potential to contribute between $2.6 trillion and $4.4 trillion annually to the global economy.

Following are the trends to expect in 2024:

Customization of GenAI deployment will be a top corporate strategy

Feels like it is an eon back when the world got taken over by ChatGPT promising to revolutionize everything from education, business, healthcare, commerce, and more, a natural question arises:

How does an enterprise such as a bank, healthcare provider, telecommunication company or retailer decide on implementing such systems in their business?

Specifically, whether to use closed Large Language Models (LLM) like ChatGPT or build the infrastructure internally? How costly are closed models like ChatGPTs vs. open-source LLMs? What are the tradeoffs in terms of performance, ongoing costs, risks, etc.?

Before implementing language models like ChatGPT, industries must carefully consider several risks related to data privacy, AI model bias, observability, and customer trust, particularly when handling private customer data. These concerns are critical in maintaining compliance and protecting sensitive information.

In a paper in mid-2023, I stated that there are cases where it makes sense to use a closed model like ChatGPT, such as summarizing a company’s product offerings for chatbot queries. There is a threshold of usage when ChatGPT is economical. At ~1000 requests per day, ChatGPT is cheaper than using open-source LLMs deployed to AWS. However, as the request volume escalates to millions per day, the economics shift, and deploying open-sourced models on AWS becomes the more affordable option, especially considering the 2023 pricing structures for both ChatGPT and AWS. Note: this article was written before the advent of GPT4, but the generalized framework mentioned there, is applicable for GPT4 and more.

Enterprises will continue to grapple with the use cases, and tradeoffs in terms of performance, ongoing costs, risks, etc. in 2024 to arrive at an optimal scenario for AI applications deployment. However, this will continue to be a moving target — largely because the tussle between performances of open sources and closed models will continue, at least through 2024.

Closed models will outperform Open ones, at least through 2024

On the debate around open-source and closed-source models, most leading AI model developers — OpenAI, Google DeepMind, Anthropic, Cohere et. al. — keep their advanced models proprietary. However, a few companies including Meta (disclosure: my former employer) and new startup Mistral have chosen to make their state-of-the-art models publicly available.

As of writing of this article, the highest-performing foundation models, e.g., OpenAI’s GPT-4 are closed-source. However, the performance gap between closed and open models is shrinking. Many experts in the open-source community state that open models are on track to overtake closed models in performance, perhaps by next year.

However, this will take a lot of work to achieve for the following reasons. The investment required to develop new models is enormous, and will only continue to balloon for the near future.

The current estimate is that OpenAI will spend around $2–2.5 billion to develop GPT-5.

While Meta continues to support the development of Llama 3, I am not sure being a public company, they can continue to spend $2–2.5 billion, or even more in the future without a clear revenue path.

Startups like Mistral face an even more daunting challenge, For instance, it was considerably riskier for OpenAI to build GPT-4 using a mixture-of-experts architecture, when this approach had not previously been shown to work at this scale, than it was for Mistral to follow several months later with its own mixture-of-experts model. However, doing so, makes Mistral a fast follower as opposed to an inventor.

Cambrian Explosion of Generative Models — SLM, LAM, VLA and more

LLM’s have taken the world with the dramatic release of ChatGPT, catching the public’s imagination and making it a household name in 2023. Most of today’s leading generative AI models incorporate text, images, 3-D, audio, video, music, physical action and more in their training data can produce such outputs. They are far more than just language models and many of today’s problems, for example, if someone is building a special purpose language model to analyze, or to create marketing documents, the language model would unlikely have to be trained to create poetry in Shakespear’s style, or give its opinion on the potential for finding alien life in another galaxy. Such models, therefore, will likely have to be trained on smaller datasets, making it potentially a Small Language Model (SLM).

Or consider Robotics where there is a need for an efficient model that has to be integrated with an action. Large Action Models (LAMs) take this as a step forward by enhancing LLMs to turn into ‘agents’: software units capable of running tasks by themselves. Instead of answering human user queries like LLMs do, LAMs help to achieve a goal, combining the language fluency of an LLM with the capacity to complete tasks and decision-making autonomously, which involves a substantial change.

Going even more specific, consider generative models that combine visual and language input, combining general internet-scale knowledge to take actions via a robotic arm. A richer term than “language model” should and will exist for such highly specific models. In fact, in this specific case, the Vision-Language-Action (VLA) model is one alternative phrase that researchers have used.

Alternatives to Transformer Architecture will see adoptions — enter Hyena and Liquid Neural Network

Every major generative AI model in existence — GPT4, Midjourney, GitHub Copilot, and so on — is built using transformers, introduced in a seminal paper, “Attention is all you need” out of Google in 2017. The challenge with a transformer is that scales quadratically with sequence length, requiring a huge amount of computing power and electricity. For example, to build the next generation of Llama and assorted AI products, Mark Zuckerburg announced that Meta will buy 350,000 NVIDIA H100 GPUs, at an estimated street price of $9–10 billion.

Add to this the looming environmental crisis the continuation of current trends in AI capacity and adoption are set to cause. Per grand, NVIDIA will likely ship 1.5 million AI server units annually by 2027, which running at full capacity, would consume at least 85.4 terawatt-hours of electricity annually — more than what many small countries use in a year, according to an assessment by Alex de Vries, a data scientist at the central bank of the Netherlands and a Ph.D. candidate at Vrije University Amsterdam. Clearly, this is not a sustainable path for the future, given the looming climate disaster the planet is already facing.

Within the AI community, a few researchers are developing next-generation AI architectures that are different and potentially super transformers. At Chris Ré’s lab at Stanford, the team is building a new model architecture that scales sub-quadratically with sequence length rather than quadratically, as transformers do. Sub-quadratic scaling enables AI models that are (1) computationally less intensive and (2) better able to process long sequences compared to transformers. Well-known sub-quadratic model architectures from Ré’s lab in recent years are Monarch Mixer, Hyena, and most recently, Mamba, which some researchers are christening as the “end of transformers”

Another alternative to the transformer architecture is liquid neural networks, developed at MIT. In 2024, I expect some of this research will start seeing adoption, transitioning from a lab novelty to a credible technology used in production.

5. 2024 will be the year of Generative AI for Music and Video

2023 was the year of AI image generation, 2024 will be for music and videos. Runway Gen2 kicked off AI video generation ten months back, with Pika 1.0 and Stable Diffusion Video joining recently. These tools can generate short, low-resolution video clips, which is great as a novelty, but does not qualify for production quality videos, or even as YouTube clips, yet.

By the end of 2024, AI videos will be lengthy (10 minutes plus), high definition, and adhere to prompts in detail. By 2025, expect Hollywood animation-level quality generations, and AI features to permeate professional studio productions in the same way computer graphics dominates special effects and animation now.

6. Wildcard entry: The first AI-enabled $10M “bank heist” will happen in 2024

This prediction does not require much of an explanation. While the AI generated voice calls with cloned voices of family members with an emergency started in 2023, The first “bank heist” enabled by emails, GenAI generated synthetic voice/ video calls and possibly a video conference to convince a bank employee to transfer money, over $10 million will happen in 2024.

Embracing the future

These trends and predictions are not just technological forecasts: they serve as a roadmap for innovation, societal growth, and advancement worldwide. While a technologist like me is often engrossed with the latest technological “shiny object”, it is also important to think about business impacts as well as global impacts. Having lived in India more and more of late, I often think about how AI will shape emerging markets where the entire nation, even for a ~$4 Trillion economy, uses only a few thousand H100 GPUs whereas just one company like Meta will 500–600 thousands of the same by end of 2025, 50 to 100 times India’s computing firepower. Will early “Land Grab” worsen global inequality, especially in Africa, parts of Asia, and parts of Latin America?

So by staying informed, adaptable, and proactive, perhaps the human race can help harness AI to build a more efficient, innovative, and inclusive future.

Previous Story: Performance Benchmarking: A Detailed Comparison Between Llama and Llama 2

Next Story: State of AI Report 2023

Visit us at DataDrivenInvestor.com

Subscribe to DDIntel here.

Have a unique story to share? Submit to DDIntel here.

Join our creator ecosystem here.

DDIntel captures the more notable pieces from our main site and our popular DDI Medium publication. Check us out for more insightful work from our community.

DDI Official Telegram Channel: https://t.me/+tafUp6ecEys4YjQ1

Follow us on LinkedIn, Twitter, YouTube, and Facebook.

AI Predictions for 2024 was originally published in DataDrivenInvestor on Medium, where people are continuing the conversation by highlighting and responding to this story.

For context, Llama 2 is a family of pre-trained and fine-tuned LLMs, ranging from 7 Billion (7B) to 70B parameters. According to Meta AI, Llama 2 Chat LLMs are optimized for dialogue use cases and outperform open-source chat models on most benchmarks they tested. Based on Meta’s assessment of human evaluations for helpfulness and safety, the company says Llama 2 may be “a suitable substitute for closed source models.”

Llama 2, like the original Llama model, is based on the Google transformer architecture, with improvements including RMSNorm pre-normalization, inspired by GPT-3; a SwiGLU activation function, inspired by Google’s PaLM; multi-query attention instead of multi-head attention; and rotary positional embeddings (RoPE), inspired by GPT Neo. Llama 2’s also increased context length (4096 vs. 2048 tokens) and grouped-query attention (GQA) instead of multi-query attention (MQA) in the two larger models (which larger models, 53B and 70B?)

Llama 2’s training corpus includes a mix of data from publicly available sources, which Meta says, does not include data from Meta’s products or services. There were two trillion tokens of training data.

Meta used its Research Super Cluster and some internal production clusters for pre-training, with Nvidia A100 GPUs. Pre-training time ranged from 184K GPU hours for the 7B-parameter model to 1.7M GPU hours for the 70B-parameter model.

Fine-tuning Llama 2 Chat took months and involved both supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF). Meta used Ghost Attention (GAtt) to keep Llama 2 Chat from forgetting its system message (overall instruction) from turn to turn in a dialogue.

Putting Llama 2 to the Test

To gain a good understanding of Llama 2’s capabilities, our team conducted elaborate testing on custom use cases, using the 34B model. The testing ground was set on AWS Sagemaker, equipped with a 24GB A10 GPU and the evaluation process assessed the following criteria:

Quality of Output: To assess the quality of output, custom internal data with human-curated responses was utilized. The model underwent tasks, including information extraction, content generation, and information verification. The accuracy of output, adherence to instructions, and potential deviations were then carefully measured.

Performance: Performance levels were gauged by measuring the time taken to generate output across five sets with varying token lengths. To benchmark the cost, OpenAI served as a pivotal reference. Both GPT3.5 and GPT4 APIs were used for comparison, and cost calculations were based on tokens per second and straightforward mathematical computations.

Cost: A comparative cost factor analysis was also performed to check the economic feasibility of Llama 2 over its predecessor.

The Test Set

The evaluation set comprised five distinct sets, each containing different token sizes: 100, 250, 500, 750, and 1,000. Each set presented ten questions, providing publicly available banking product information text as context for each inquiry. In the 100-token set, product titles took center stage, prompting the model to offer its inference or judgment. The validation set played a vital role in comparing the model’s responses, with accuracy being the ultimate litmus test.

For the sets containing 500, 750, and 1,000 tokens, Llama 2’s capabilities were tested, extracting values for a list of attributes from the provided context. The answers were contrasted against the validation set, demanding correct answers that included all the required attributes.

Results

The following tables provide the results for 100, 250, 500, 750, and 1,000 tokens.

Analytical Insights

This phase was a significant phase of the present study as it played a key role in revealing indicators behind Llama 2’s performance, quality, and cost.

Quality: Is it better?

During the evaluation process, Llama consistently delivered subpar outputs for the majority of questions, irrespective of the token sizes used.

Also, as shown in Figure 6, the Llama 2 model drastically improved the quality and accuracy of responses, surpassing its predecessor. It achieved a perfect score of 100% on AWS when tested with 100 tokens. The model also maintained its consistency on AWS when tested with 250, 500, 750, and 1,000 token sets.

Performance: Is it faster?

Performance evaluation, which delved into response generation time, provided insight into Llama 2. On the AWS Sagemaker, equipped with a 24GB A10 GPU machine, the generation time was notably high. Additionally, the comparison between Llama and Llama2 on the 250, 500, 750, and 1,000 tokens revealed that Llama took 3–6 times as long as Llama 2 to generate responses.

Cost: Is it cheaper?

The cost evaluation provided a glimpse into the economics of Llama 2. Considering GPU charges per second and total generation time per model’s response, Llama 2 proved to be significantly more economical to run on AWS compared to Llama.

Summary

We ran an extensive comparison between Llama and Llama 2 using banking product information available publicly. The tests were run using the 34B model. The testing ground was set on AWS Sagemaker, equipped with a 24GB A10 GPU. The token sizes were 100, 250, 500, 750 and 1,000.

The tests were not a perfect 1:1 comparison. For some of the questions, Llama “broke” at the higher number of tokens, e.g.750 and 1,000.

The overall results were as follows:

• The generation time ratio of Llama: Llama 2 was ~3–6, i.e., Llama took approximately three to six times as long as Llama 2 to provide an answer
• The cost ratio of Llama: Llama 2 was ~2-4:1, i.e., Llama cost approximately two to four times as much as Llama 2 to provide an answer
• The accuracy ratio of Llama: Llama 2 was ~1:8, i.e., Llama 2 was eighth times as accurate as Llama

Albeit, these answers are not universal and were significantly influenced by larger token size where for some of the questions Llama was not able to provide answers after a long wait.

However, we found this framework to be a useful one for comparing different models. It is a practical approach for a quick test when an enterprise, a bank, a Telco, or a healthcare provider, for example, is choosing models for specific applications.

Previous Story: State of AI Report 2023

Next Story: AWS Generative AI Forum, 2023 Mumbai

Visit us at DataDrivenInvestor.com

Subscribe to DDIntel here.

Have a unique story to share? Submit to DDIntel here.

Join our creator ecosystem here.

DDIntel captures the more notable pieces from our main site and our popular DDI Medium publication. Check us out for more insightful work from our community.

DDI Official Telegram Channel: https://t.me/+tafUp6ecEys4YjQ1

Follow us on LinkedIn, Twitter, YouTube, and Facebook.

Performance Benchmarking: A Detailed Comparison Between Llama and Llama 2 was originally published in DataDrivenInvestor on Medium, where people are continuing the conversation by highlighting and responding to this story.

Advancements in AI Models

GPT-4’s exceptional performance attained a significant milestone in natural language understanding and generation. Meta’s LLaMa models, especially LLaMa-2, gained attention for their competitive edge and accessibility. Moreover, the emergence of open-source language models like MPT-30B and Falcon-40B fostered community-driven AI development.

• GPT-4 Shines (for now): GPT-4’s performance has been a standout achievement, surpassing its predecessors like GPT-3.5. This marks a significant stride in natural language understanding and generation.
• Meta’s LLaMa Models: Meta’s LLaMa models, particularly LLaMa-2, have garnered attention for their competitive performance and wide accessibility, revealing the potential to impact a range of applications.
• Community-Driven Development: The emergence of open-source or semi-open language models like MPT-30B, Falcon-40B, and RedPajama fosters a collaborative, community-driven approach to AI development. This inclusiveness is propelling AI advancements forward.

Technological Innovations

Technological innovations included memory optimization techniques like FlashAttention and 4-bit quantization, making AI models more efficient. Innovations such as VisProg and ViperGPT broadened AI’s horizons, enhancing its capabilities for visual and image-related tasks.

• Memory Optimization: Innovations like FlashAttention, 4-bit quantization, and speculative decoding are tackling fundamental challenges, such as reducing memory footprints and speeding up inference. These breakthroughs contribute to the efficiency of AI models.
• Expanded Horizons — visual and videos: Methods and models like VisProg, ViperGPT, and DINOv2 are broadening the scope of AI applications. They are enhancing capabilities related to visual, video, and image-related tasks, enabling AI to excel in a variety of domains.

Commercial and Practical Applications

AI’s practical applications expanded, from generative AI to autonomous driving and medical diagnostics. GitHub’s CoPilot demonstrated AI’s role in boosting developer productivity

• Commercialization of AI: In a notable shift, AI models are finding extensive use in real-world applications across various domains. This encompasses generative AI applications, autonomous driving, and medical diagnostics. AI’s practicality is becoming evident.
• AI for Developers: GitHub’s CoPilot is emblematic of AI’s role in enhancing productivity, particularly among developers. The assistance it provides is streamlining the development process and fostering innovation.

Investments and Market Trends

GenAI has saved an otherwise “funding winter”, with VCs investing $18B in GenAI applications, including video, image, text, and coding. Sustained investments and AI’s influence in education, with ChatGPT disrupting long-standing EdTec platforms like Chegg, confirmed AI’s future for “re-stacking” of education technologies and how we will learn in the future

• Sustained Investment: AI startups continue attracting substantial investments, with VCs pouring in over $18B in GenAI startups. Established players like NVIDIA are making strategic moves, reflecting the high stakes and the optimistic outlook in the AI sector.
• “Re-stacking” of Education: AI’s impact on educational platforms is profound. ChatGPT, for example, has made a disastrous impact on the long-standing EdTech platform Chegg, showcasing how AI has the potential to disrupt and enhance education across different sectors. The future of education is increasingly AI-driven.

Industry Dynamics and Regulation

Market dynamics reflected NVIDIA’s continued dominance and ongoing U.S.-China chip competition while varying global regulatory approaches showcased the challenge of balancing innovation with ethics.

• Compute is the New Oil: NVIDIA’s continued dominance in the market is a testament to its technological prowess. Simultaneously, the ongoing competition between the U.S. and China in the chip sector underscores the intricate interplay of politics, economics, and technology.

• Global Regulatory Landscape: The regulatory landscape for AI is a diverse one, with different regions adopting distinct approaches to AI legislation. This divergence highlights the global challenge of striking a balance between innovation and ethical considerations.

Open Source Growth and Continued Consolidation

Open-source AI communities thrived, exemplified by Hugging Face’s growth, and industry integration through acquisitions like Databricks and BioNTech illustrated AI’s deepening impact

• Open-Source Growth: The open-source AI community, exemplified by Hugging Face’s expanding influence, fosters collaboration and knowledge sharing. This collaborative spirit is accelerating AI development.
• Industry Integration: Strategic acquisitions like Databricks acquiring MosaicML and BioNTech acquiring InstaDeep indicate the deep integration of AI across various industry sectors. This integration is opening up new possibilities for AI-driven solutions.

Indeed, 2023 has been a year of AI advancements and transformations that promise a dynamic future for artificial intelligence, in 2024 and beyond.

Previous Story: AWS Generative AI Forum, 2023 Mumbai

Next Story: Buy vs. Build for an Enterprise: “ChatGPT” vs. Open-Source vs. QLoRA

Subscribe to DDIntel Here.

DDIntel captures the more notable pieces from our main site and our popular DDI Medium publication. Check us out for more insightful work from our community.

Register on AItoolverse (alpha) to get 50 DDINs

Join our network here: https://datadriveninvestor.com/collaborate

DDI Official Telegram Channel: https://t.me/+tafUp6ecEys4YjQ1

Follow us on LinkedIn, Twitter, YouTube, and Facebook.

State of AI Report 2023 was originally published in DataDrivenInvestor on Medium, where people are continuing the conversation by highlighting and responding to this story.

While there is a significant interest in experimenting and implementing GenAI within enterprises, the audience was curious about how to navigate the new world of GenAI in their own companies. During the conference and at the dinner afterward, I was asked numerous interesting questions, of which, the top ones are listed below

Journey with GenAI: The Selection of Use Cases

Most companies are grappling with where to start in terms of their GenAI initiative. While it will depend on enterprises’ own priorities, technical sophistication, and perceived business values, the common framework that is useful for making a choice is to ask two following questions:

• Does this task occur in enough frequency?

This is important so that there is adequate training data available for training the models

• Are the outputs validatable through a human curation process?

This enables an enterprise to ensure that the GenAI outputs are consistent with what humans would expect.

In short, if the task requires effort to execute, but is relatively easier to validate, it might be a good candidate

Key Steps in Experimentation and Proof of Concepts (POCs): The Watchouts?

POCs, shortlisted by the above criteria, should include the following additional guardrails in the early days:

• Choosing an area where the business benefits are high or business outcomes are constrained by the current processes.

For example, a Direct-to-Consumer (D2C) brand running substantial marketing on social platforms such as Instagram, may today be constrained by a manual content generation process and can only do 2–5 posts a week. A GenAI system that enables them to create 100+ posts automatically matches up to the optimum marketing goals that would have been otherwise impossible.

• Avoiding direct interaction with customers where advice on a product or service is given

For example, while GenAI systems are capable of underwriting loans, offering such services through a chatbot to a bank’s customer, is a strict ‘no-no’, at least until the GenAI systems and customers have matured enough.

On the other hand, if the same customer queries about the suite of home loan products from the same bank mentioned in the above scenario, it is a good use case to pursue. However, even this shall be subject to business benefits, because it does not involve giving advice that can have a monetary impact.

What are the Key Risks of GenAI? What are the Principles of Governance Frameworks to Manage these Risks?

One can possibly write a book on risks posed by GenAI, with everything from Existential (human extinction after the takeover by uncontrolled GenAI) and societal (job losses) to Enterprise (operational, financial, reputational, etc.). My advice is to be aware of the first two but also recognize that the ones that an enterprise can focus on, are the risks that GenAI creates for its own operations.

Within that context, I recommend building a simple risk framework that is easier to comprehend, operationalize, and learn, over a period of 3–6 months. It can also be aligned based on how to apply it within the context of the early experiments. An overly complex risk framework that tries to anticipate all the possible permutations and combinations of the risks that can arise, would be overtly complex, unwieldy, and will likely not be governable as the whole field of GenAI itself is evolving rapidly.

How does an Enterprise communicate about its GenAI initiative?

GenAI, like any early technologies, has generated curiosity, and a sense of awe. While the earlier technological breakthroughs had limited public reactions (semiconductor in the 1980s) or curiosity (internet in the 2000s), this is the first technology that competes and is capable of producing, perceivably, human-level outputs that compete directly with modern jobs that define us and are closely related to the identities we hold socially.

The key pillars of any interaction with customers should be:

• Transparency: Let your customers know which processes are being run by GenAI, for example. If LLMs power your ChatBot or Recommendations, say so. For example, “this communication is powered by a Generative AI bot” is a good start.
• Choice: In the early days when the technology was still maturing, providing a clear choice (such as whether to be served by a GenAI algorithm), would provide customers an opportunity to decide if they want to “opt-out”, easing the fear. Extending the example above, “this communication is powered by a Generative AI bot. Click here to talk to a live agent instead”.
• Trustworthy Communication: A combination of the above, plus, open communications with the customers, including that at the company’s websites, and customer communications (email, SMS, etc.), helps build a rigid long-term trust in the brand’s credibility.

Summary

My best guess is that GenAI, which is currently evoking awe, shock, and fear, would be like the technologies that were integrated into enterprises during the past 20 years namely, internet, eCommerce, and cloud. There would be an adoption cycle where enterprises and customers will get used to the new technology and its nuances. However, as every new technology cycle does, there will be winners and losers. Being an early adopter, choosing experiments carefully, realizing business benefits through process changes, and taking the customers on the journey, in a transparent and trustworthy manner would be the key to ensuring an enterprise’s success.

Previous Story: Buy vs. Build for an Enterprise: “ChatGPT” vs. Open-Source vs. QLoRA

Next Story: What Kind of AI Society Are We Heading Towards: The Star Trek, Mad Max, or Elysium type?

TLDR: Before implementing language models like ChatGPT, regulated industries must carefully consider several risks related to data privacy, AI model bias, observability, and customer trust, particularly when handling private customer data. These concerns are critical in maintaining compliance and protecting sensitive information.

In cases where it makes sense to use ChatGPT, such as summarizing a company’s product offerings for chatbot query, there is a threshold of usage when ChatGPTs are economical. ChatGPT is less expensive than utilizing open-source LLMs deployed to AWS when the number of requests is ~1000s per day. However, as the request volume escalates to millions per day, the economics shift, and deploying open-sourced models on AWS becomes the more affordable option, especially considering the current pricing structures for both ChatGPT and AWS.

LLMs, with the introduction of Transformers in 2017, followed by breakthrough models like BERT, GPT, and BART, are transforming the way we communicate and interact with technology, and their impact is being felt across industries and borders.

Attention Is All You Need

A few years back researchers from OpenAI and Google documented in multiple papers demonstrating that LLMs with over 10’s Billions parameters are increasingly sophisticated, able to understand complex language structures and generate human-like responses

Indeed, language models like LLMs are rapidly gaining prominence and becoming ubiquitous in various domains. They are poised to revolutionize numerous aspects of our lives by powering virtual assistants, chatbots, content creation tools, and translation services, among other applications. The capacity of these models to efficiently process and analyze massive volumes of data, coupled with their ability to learn from it, is reshaping the way we work, learn, and communicate.

Now that many enterprises are considering integrating their processes, functions, and workflows with LLMs, enterprises are looking for how to approach them.

Consideration 1: Strategic Choice between ChatGPT and Open-source LLMs

There are several considerations associated with using ChatGPT or third-party LLMs in regulated industries such as banking, healthcare, or Telecommunication, especially when using customer data to train them. ChatGPT was trained largely on public data from the internet, making it susceptible to the following risks:

Privacy Risks: Many LLMs were trained on public data that is not privacy preserved. Note the recent lawsuit by Getty Images against Stable Diffusion, alleging that 12 million images from Getty’s repository “without permission.. or compensation”. Expect such lawsuits to proliferate, potentially exposing LLMs provider’s enterprise customers, too.

Bias Risks: Having been trained on public data, many third-party LLMs can not guarantee that the models are not biased, exposing their enterprise customers to the risk of regulatory fines and penalties.

Customer Data Concerns: Customers may worry about a third-party LLM being exposed to their sensitive financial information, for example, even if the ChatGPT is used within a private instance of their bank. Similarly, most people will be seriously concerned about LLMs having access to their private health data.

Lack of Trust: Due to a combination of factors such as Deepfakes. For instance, in 2021, a UAE Company lost $35M due to the Deepfaking of the Company Director’s voice and such cases will rise exponentially. How do banks protect customers’ trust if such events happen? What if on top of such news, bank customers wonder about what their personal data is being used for, especially training third-party LLMs?

Observability Risks: Because it is hard to “observe” such large models, the correlation between the inputs and output is not known and attribution of cause and effect is all but impossible

Irreversibility of Harm Due to the “Blackbox model”: As the models are Blackbox and not observable, harms are not known (except in obvious cases) and hence, irreversible.

Liability: Who is liable for the harm caused by the LLM models: the model creators (say LlaMa by Meta) or say, a bank who tweaked it? The same analogy is used in the event of a car crash caused by a spoiler flying off. Is it the OEM who created the spoiler that flew off and hit another car, the dealership who may have installed it inappropriately, or the user who may have modified it?

However, there are areas where the use of ChatGPT is relatively safe. For example, using ChatGPT for summarizing a company’s product offerings to better answer customer support questions does not involve customer data, nor will it likely violate any privacy concerns.

If an enterprise decides to explore the option of using ChatGPT, the comes the question of cost.

Consideration 2: Cost Comparison between ChatGPT and Open-source LLMs

Option 1: ChatGPT API Costs

ChatGPT API today costs $0.002/1K tokens and a token is about 3/4th a word. The number of tokens in a single request is the sum of prompt + generated output, in words, converted tokens.

Let’s assume an enterprise processes incoming customer queries through an interactive chatbot using ChatGPT. A question and answer is roughly a page: so it is 500 words or 666 tokens. Answering 5,000 customer queries per day then costs (($0.002/1000)x666*5,000)= ~$6.5 a day, or $200 a month.

But what happens if each customer takes 4–5 prompts to get the right answer they are looking for? After all, they are not necessarily well-trained prompt engineers accustomed to writing finely tuned queries to get an answer. And assume, we have 200 thousand such queries coming in per day, about the normal for a contact center for a major brand.

Under this scenario, the cost skyrockets to ~0.5 Million$ per year, making ChatGPT a significant expense for an enterprise business! No wonder Venture Capitalists (VCs), accustomed to identifying opportunities for making money, are pouring billions of dollars into “ChatGPT for X” ideas.

ChatGPT Fever Has Investors Pouring Billions Into AI Startups, No Business Plan Required

Option 2: Open-source Large Language Models

2a: Open-source LLM Costs: Factors and Model Dependency

For enterprises looking for Open Source LLM, options are aplenty. My former employer, Meta (Facebook) pioneered LLaMA, with a variety of model sizes between 7 to 65 billion parameters. According to the LLaMa developers, the 13 billion parameter model outperformed a considerably larger GPT-3 with 175 billion parameters on the majority of NLP benchmarks. Later, the teams at Stanford University fine-tuned the 7B version of LLaMA on 52K instruction-following demos and found that their Alpaca model outperformed GPT-3.

While open-source models are free to use, the infrastructure to host and deploy them is not. Earlier resources such as BERT were less resource intensive and could be fine-tuned using low-end GPUs. However, more recent LLMs such as LLaMa are more resource intensive.

Most common LLMs models such as GPT-3 or BERT are based on transformer-based architectures. While the number of operations required for inference and learning for such transformers are model-specific, the rule of thumb for transformers (i.e., the weights of the neural networks) is that a forward pass (i.e., inference) for a model with p parameters for an input and an output sequence of length n tokens each, takes approximately 2*n*p floating point operations (FLOPs) per token, as the additional backward pass requires four more operations. Therefore, a user can approximate the total training cost by the number of tokens in the training data, multiplied by parameters of approximate factors (2 or 6).

Memory requirements for transformers are model-dependent, too. For inferencing, p-model parameters need to fit into memory. For back-propagation during learning, the additional intermediate values per parameter between the forward and backward pass need to be stored in the memory. Assuming 32-bit floating point numbers and for training a 175-billion-parameter model, the model would need over 32*175X10⁹ bytes, or, over 6 TB of data in memory. This exceeds any GPU in existence today and hence requires the model to be split across multiple memory cards.

Using the formulas shown above, here are the training and inferencing requirements for BERT and GPT-3. For a refresher, the training requires 6*p FLOPS per token, while the inferencing requires 2*n*p FLOPS, with p being parameters and n being tokens for training data

Navigating the High Cost of AI Compute | Andreessen Horowitz

How do computational complexities translate into cost? While one can go down the path of further translating the infra requirements to specific GPUs, the optimization of the algorithm (for example, using 16 floating point numbers vs. 32 floating points) and the number of training passes required to achieve model stabilities to arrive at a range of numbers, our literature survey indicates that it can vary as a factor of 10: between $0.5–5M for GPT-3. This is a large sum of money requiring significant investment that only large companies or well-funded startups can afford.

2b: Open-Source LLM: Architecture for Deploying Open-source Models

Short of building one’s data center to host, train and deploy LLMs, a more practical solution for enterprises would use cloud providers like AWS, Google, Azure, or smaller providers such as Lambda Labs to host and deploy such models. Many enterprises such as Banks, Healthcare providers, and Telcos already have a deep relationship with Big-3 cloud providers, making this option an attractive one for them. Given our familiarity with AWS, we will use the AWS infrastructure as an example.

Let’s dive into AWS costs for hosting open-source models and serving as APIs — usually involving four steps.

• A client device, for example, a browser invoking a customer request, which is then passed through Amazon’s API Gateway
• The API Gateway in turn triggers the Lambda which parses the function and sends it to AWS Sagemaker Endpoint
• The model is then invoked at the endpoint using AWS Sagemaker

The Sagemaker costs are sensitive to the type of computing instance for hosting the model as LLMs use rather large computing instances.

For example, this article written by Heiko Hotz details how to deploy Flan UL2 — a 20 Billion parameter model, on AWS:

Deploy Flan-UL2 on a Single GPU With Amazon SageMaker

The article uses the ml.g5.4xlarge instance for the deployment of the Flan-UL2. While the Sagemaker pricing above did not list the cost for this specific instance, it appears that it would cost ~$5–6 per hour, or $150 per day! In addition, the Lambda pricing and API gateways are about ~$10 and $1 per million requests.

So ultimately the cost of hosting an open-source LLM like Flan-UL2 on AWS is $150 for 1000 requests a day, and $160 for 1 M requests a day, or about $500,000.

Option 3: Open-Source LLM Costs for Smaller Language Models

For even simpler tasks, for example, creating a summarisation engine for a corporate HR policy on which one can make an employee chat or, or, for spam detection smaller language models like BERT that are 100’s of millions of parameters are sufficient.

For training BERT, one can use cheaper instances like ml.m5.xlarge which is $0.23/hour and ~5$ a day. These models are also quite powerful for “narrow” applications, compared to ChatGPT and GPT4 which can understand complex nuances of human language.

Consideration 3: Using Quantized Models Such as QLoRA

In a recent paper, a new method, QLoRA, has been announced which provides almost a game-changing ability to train and fine-tune LLMs on consumers’ GPUs. In comparison to fine-tuning 16-bit models, QLora uses less memory and does not sacrifice performance.

Using this method one can fine-tune a 33B model on a single 24GB GPU. For 64B model fine tuning it uses a single 46GB GBU. QLoRA achieves this by using 4-bit quantization to compress a pre-trained language model. The language model parameters are then frozen and a relatively small number of trainable parameters, in the form of Low-Rank Adapters are then, are then added to the model.

During finetuning, QLoRA backpropagates gradients through the frozen 4-bit quantized pre-trained language model into the Low-Rank Adapters (LoRA). According to the LoRA study, which goes into great depth, the only parameters that are modified throughout training are the LoRA layers.

QLoRA uses a 4-bit NormalFloat storage data type for the base model weights and a 16-bit BrainFloat data type to perform computations. During forward and backward passes, QLoRA de-quantizes weights from the storage data type to the compute data type. However, it only computes weight gradients for the LoRA parameters, which is the 16-bit BrainFloat. The weights are decompressed only when they are needed, which is far lower than 100% of the cycle time, therefore significantly reducing memory usage during training and inferencing.

QLoRA tuning, while significantly reducing memory usage and requiring only 24 hrs of fine-tuning on a single GPU, reaches a performance level approaching 99.3% of the performance level of ChapGPT. Moreover, Guanaco, which uses QLoRA finetuning on the LLaMA model, reaches close to CHATGPT on the Vicuña benchmark

The cost of training and finetuning is now approaching sub-$10,000 or even sub-$1,000 on QLora.

Summary

The choice between using models like ChatGPT and GPT-4, which are developed by companies like OpenAI, versus open-source LLMs depends on several factors and considerations.

Models like ChatGPT and subsequent iterations from OpenAI often offer more relevant responses compared to open-source LLMs. They benefit from significant research and development efforts, incorporating advanced techniques and extensive training on vast datasets. These models are designed to have broad applicability and serve a wide range of use cases.

However, open-source LLMs are rapidly catching up in terms of performance and relevance. They provide the advantage of being customizable and fine-tunable on specific data sources. Companies can leverage open-source models to train and fine-tune them on their domain-specific data, leading to potentially better performance in specialized contexts. This fine-tuning process allows organizations to tailor the models to their specific needs and optimize them for their unique requirements.

Furthermore, there are valid reasons for choosing open-source models over closed APIs provided by companies. Using open-source models provides more control and transparency as organizations have access to the underlying code and can modify it as needed. This level of customization can be valuable for companies with specific privacy, security, or compliance requirements.

If this article was written six months back, standard offerings from OpenAI and such would have been a clear choice for the enterprises. However, in the last six weeks, significant improvements in the quantized models such as have opened up opportunities for enterprises, specifically those that have a reasonably strong technical workforce who are willing to quickly experiment and learn

And models are only one part of the story. The success of one’s AI journey depends on various factors such as the quality and diversity of the training data, the fine-tuning process, and the specific use case or the domain being addressed. An example of it is BloombergGPT, which is leveraging highly domain-specific data to deliver high-quality models.

Ultimately, the choice between using closed models like ChatGPT or open-source LLMs depends on the specific needs, resources, and priorities of an organization. Evaluating factors such as performance, customization options, data privacy, and compliance requirements will help determine the most suitable approach for leveraging LLMs in a given context.

Acknowledgments

I was inspired to write this article after reading a really nice post on LLM by Skandagupta Vivek. As I started delving more, I came across more articles, especially one by Andreessen Horowitz, and then the papers on QLoRa. Some of these papers have been embedded in the article although I couldn’t do that for every article. This article has also been helped by conversations with many of my colleagues. Last, but not least is my editor, Chandrajita Chakraborty, who played a crucial role in shaping this article into its final form.

Previous Story: What Kind of AI Society Are We Heading Towards: The Star Trek, Mad Max, or Elysium type?

Next Story: Where did the Name Bluetooth Come From

Subscribe to DDIntel Here.

DDIntel captures the more notable pieces from our main site and our popular DDI Medium publication. Check us out for more insightful work from our community.

Register on AItoolverse (alpha) to get 50 DDINs

Join our network here: https://datadriveninvestor.com/collaborate

Follow us on LinkedIn, Twitter, YouTube, and Facebook.

Buy vs. Build for an Enterprise: “ChatGPT” vs. Open-Source vs. QLoRA was originally published in DataDrivenInvestor on Medium, where people are continuing the conversation by highlighting and responding to this story.

To predict its versions in the future, there’s no better way to do that than to use movie analogies: especially to cult classic sci-fi movies. Let’s focus on three major ones — Star Trek, Mad Max, and Elysium. Each one portrays different probabilities of the future existence of human civilization.

The Star Trek Future

The Star Trek society exists in a post-scarcity world. AI is ubiquitous and highly advanced. From the ship’s computer to the Lieutenant Commander Data, AI is an integral part of daily life. AI is used to make self-aware systems or ‘synthetic life’.

Efficiency

Just like the Star Trek society, AI will help in creating highly efficient systems and processes to carry out complex tasks quickly and accurately. For example, the ship’s computer can quickly analyze data and make recommendations, saving crew time and effort.

Exploration

AI-powered sensors and mapping technology will allow people to explore new worlds and look beyond this planet. We can get answers to widely sought-after questions like the possibilities of extraterrestrial life forms out there. Significant scientific advancements in AI help expand human knowledge and enhance our understanding of the universe.

Ethics

In the Star Trek universe, AI is programmed with ethical guidelines that ensure it operates in a morally responsible way. In this version of the future, we will see elaborate codes of conduct for AI that will steer its expanse so that it is beneficial for society in the long run.

The Mad Max Future

Mad Max is set in a post-apocalyptic future, totally opposite to the Star Trek society. Ravages caused by human beings towards themselves will prove to be huge lessons that will force the inhabitants to learn from their past mistakes or create more pandemonium. AI is less common and less advanced than in the Star Trek universe, however, it’s not absent.

Military Applications

AI-powered weapons will give certain factions a significant advantage over others, leading to increased violence and instability.

Resource Management

AI will be used to optimize the utilization of the meager resources that are still available in this post-apocalyptic world. However, there will be significant demarcation of the availability of these technologies that would likely only be available to the wealthy or powerful, exacerbating existing inequalities.

Surveillance and Controls

AI-powered surveillance systems will be used to monitor citizens and prevent crime. However, this could also lead to increased government control and the erosion of civil liberties and individual privacy.

The Elysium Future

In the Elysium universe, AI is highly advanced but is used to reinforce social inequality.

Targeted Healthcare

In the future, Earth may not be a highly habitable place to be in. To protect themselves, AI-powered health tech will come into being but will be limited to the wealthy inhabitants. It will be made inaccessible as it shall be located in a distant space station reinforcing the social and economic divide between this advanced realm and Earth.

Selective Safety

AI will be used to maintain a high level of safety and control in Elysium. However, this system is also used to keep out unauthorized individuals, reinforcing the power of the wealthy and limiting the agency of the poor.

Dangerous Labor

Many manual labor jobs on Earth will be carried out by robots and other AI-powered machines, leading to high levels of unemployment and poverty that will make this social divide even worse. Even for some of the most dangerous jobs, such as the ones that involved handling nuclear materials, there will be long lines of humans willing to do it, just to survive.

Where is Human Society heading in the next 50 Years?

The impact of AI on society is complex and multifaceted. While AI has the potential to improve our lives in many ways, it also has the potential to exacerbate existing inequalities and reinforce power structures.

In the next 2–3 years, with the rapid growth in LLMs (Large Language Models), GPT, and eventually AGI (Augmented Generalized Intelligence) coupled with the potential lack of privacy policy and regulations, we will set up the foundations for future human societies.

Homo sapiens — humans have become the predominant species on the planet by a series of favored evolutions, (accidental) quirks of history, and most recently, a prudent set of decisions that have guided the growth and prosperity of the species. The lack of prudence, coupled with hypergrowth may soon become our undoing — only time will tell.

Previous Story: Where Did the Name Bluetooth Come From?

Next Story: AI Predictions 2023: The Year AI Stole Every Household’s Imagination