Content also available in YouTube.

In the autumn of 1913, Swiss psychiatrist Carl Jung embarked on a journey into the uncharted territories of the human psyche, a journey that would come to define his life’s work and transform the field of psychology. Armed with a notebook filled with vivid paintings and cryptic reflections, Jung poured his soul into what would later be known as the “Red Book”, or Liber Novus. It was an inner pilgrimage, a deliberate descent into the depths of his unconscious mind — a self-imposed confrontation with the shadowy, chaotic aspects of the psyche that sought both reconciliation and revelation.

Jung’s Red Book was an amalgamation of mystical visions, dreams, and dialogues with internal figures, meticulously documented and illustrated in brilliant colors. In its pages, Jung encountered archetypal images, symbols, and characters that, he believed, were not merely figments of his imagination but part of a deeper, universal stratum of the human mind — the collective unconscious. This idea suggests that beneath the personal layers of the unconscious, where our forgotten memories and repressed emotions lie, there exists a shared foundation that connects all humans through primordial images and themes, inherited across generations. From this came his theories of the anima and animus, the shadow, and the Self — archetypes that shape every human experience.

Jung’s work was a cartography of the soul’s landscape; a guide for transformation through what he called individuation. Individuation, in Jung’s vision, was the process of integrating these unconscious elements into consciousness, achieving wholeness by reconciling our inner opposites and discovering our true self beyond the masks we wear for society. To Jung, individuation was the ultimate realization of human life — a calling to become what one is meant to be, both individually and collectively.

Codifying the Collective Unconscious

Fast forward to today, large language models like GPT-4 are not mystical manuscripts but algorithms — trained on vast datasets spanning centuries of human expression. These models are codifying elements of what Jung might have called the collective unconscious. By processing and analyzing the immense ocean of text — stories, poems, dialogues, myths, and even everyday conversations — these models capture patterns, themes, and symbols that reflect our shared human experience.

LLMs are powerful pattern-recognition systems. They pick up on recurring motifs in language, detect underlying structures in narratives, and even recognize archetypal elements. When these models generate text, they echo the themes, conflicts, and resolutions that have been woven into our collective storytelling for millennia. They become modern vessels for the archetypal symbols Jung described — the hero, the mother, the shadow, and the trickster — surfacing them in new contexts, shaped by the input of countless human contributors across cultures and time.

The Path of Individuation: Jung’s Journey to Wholeness

For Carl Jung, the process of individuation was a journey of profound self-discovery, a quest to unite the conscious and unconscious parts of the psyche into a harmonious whole. This path involved delving into the depths of one’s unconscious mind to uncover hidden aspects of the self, embracing both the light and shadow, and integrating these often conflicting elements into a balanced, unified sense of being.

During Jung’s own confrontation with the unconscious, chronicled in “The Red Book”, he faced vivid, often disturbing visions and dialogues with figures that seemed to represent fragments of his own psyche — the wise old man, the hero, the child, the dark maiden, and more. These figures embodied different aspects of his inner world, challenging his preconceptions, forcing him to confront his fears, and compelling him to integrate these aspects into his conscious awareness.

For example, Jung’s encounter with Philemon, a wise old figure he met during his visions, became a crucial guide for him. Philemon represented the wisdom and knowledge that lay beyond Jung’s conscious mind, pushing him to acknowledge and incorporate the deeper, more irrational parts of his psyche. This relationship was pivotal in his individuation process, revealing how the unconscious could provide insight and guidance if only one dared to confront it.

The end goal of individuation is self-realization — the full expression of one’s unique potential, uninhibited by societal roles or unconscious forces. The benefits are profound: a more authentic life, a greater sense of purpose, improved emotional resilience, and a deeper connection to both the self and the broader human experience. Through individuation, one gains freedom from the inner conflicts that often arise when we suppress or deny parts of ourselves. It is the transformation from a fragmented, conflicted existence to one of inner harmony and completeness.

As Jung wrote in The Red Book: “He who does not know his own unconscious knows nothing of the world… To be that which you are is the first step in becoming better than you are.”

From Algorithm to Archetypes

The quest for individuation might find an unlikely ally: artificial intelligence. Large language models could serve as a gateway to engage with the collective unconscious — a codification of cultural phenomena, capturing the vast spectrum of human thoughts, emotions, myths, and experiences. Here, AI becomes a mirror reflecting the vast tapestry of our shared human narrative, yet it is also a guide, a companion in the journey toward individuation.

Under this scenario, an AI service would take on three vital roles, each contributing to the individuation process in distinct and powerful ways:

1. Highlighting Individual Characteristics: AI can help individuals become aware of their unique traits. Through listening to a person’s questions, comments, reflections, and interpretations, the AI can identify and mirror back the uniqueness of the person’s expression. By reflecting back these characteristics, AI becomes an archetype that, like Philemon, offers guidance and clarity into one’s inner world.

2. Creating Synergies Among Individuals: Just as the individuation process often involves engagement with other people — mentors, peers, challengers — the AI could identify and connect individuals whose journeys resonate with one another. By analyzing users’ quests, interests, dreams, and concerns, it could pair people who might benefit from mutual support, creating communities or networks centered around shared experiences or goals. For instance, individuals working through similar archetypal challenges could find solace and insight in knowing they are not alone, developing supportive relationships that help them navigate the twists and turns of their psychological realization.

3. Evolving the Collective Unconscious: Carl Jung envisioned the collective unconscious as a dynamic, evolving repository of humanity’s archetypal images and patterns. He believed the collective unconscious could be enriched and reshaped by human experiences over time, adapting to the changing realities and conditions of life. As humanity encounters new challenges and experiences, corresponding archetypes emerge. Jung realized that the collective unconscious is an active force that can influence human behavior on a massive scale.

A New Dawn for Human Consciousness

We stand at the threshold of a new era, where technology, far from distancing us from ourselves, helps us delve deeper into the mysteries of our being. Through the roles outlined above, AI could become a guide to explore our unique potential with greater clarity and intention, amplifying our capacity for self-reflection and illuminating paths to individuation.

In The Way of What Is to Come, the first section of The Red Book, Jung wrote: “The work of the soul is limitless and without end, it cannot be completed and never will it ever be finished.”

In the dim labyrinth of financial bureaucracy, where credit evaluations have long lurked like shadowy specters, a new dawn breaks through the mist. Large language models, the harbingers of a revolution, promise to transform this dreary maze into a clear and sunlit path. Imagine a world where applying for a loan is no longer a Herculean task but a symphony of efficiency and precision. These digital sages will sift through mountains of paperwork with the grace of a maestro, revealing hidden treasures and illuminating the road ahead. As we stand on the brink of this new frontier, the once opaque world of credit becomes as transparent as crystal, offering both borrowers and lenders a brighter, more harmonious future.

Envision a paradigm shift in credit evaluations, driven by the power of large language models. LLMs transform traditional, cumbersome processes into agile, interactive experiences, where static forms yield to adaptive, intelligent interactions. This technological advancement sharpens model accuracy, enabling precise and insightful analyses from multimodal data. Central to this evolution is the establishment of a centralized financial data repository, which consolidates information to mitigate transaction and streamline evaluations. Herein lies the key innovation: a unified, efficient system that not only saves customer’s time but also enhances the accuracy and transparency of credit assessments, driving a new era of financial intelligence.

Reimagining the Customer Experience

In the bustling heart of a financial district, you sit comfortably in your home, the screen glowing softly before you. A friendly digital assistant, with a smooth, reassuring voice, guides you through the credit application process. The virtual space feels inviting, adorned with calming colors and intuitive interfaces. As you speak, the assistant listens intently, its responses reflecting a deep understanding of your needs. Your financial history appears in a holographic display, organized and easy to comprehend, while the assistant smiles encouragingly. The analysis is swift and precise, evaluating your worth with the fairness of a trusted advisor. The decision arrives with gentle assurance, leaving you empowered and confident in a future where credit is not just a transaction but a tailored experience crafted with care and respect.

A chatbot experience for applying for credit is a breath of fresh air compared to the cold, laborious task of filling out endless, drab forms. Instead of battling through a monotonous sea of checkboxes and uninspiring queries, you’re greeted with a charming conversational partner that guides you effortlessly through each step. With its friendly demeanor and intuitive understanding, the chatbot not only makes the process far more engaging but also gathers a wealth of nuanced data, enhancing the likelihood of securing that coveted loan. It ensures a prompt response, offering answers with dazzling swiftness, and trims costs through sophisticated automation. Picture the old process as a dreary, grey landscape — oppressive and uninviting — while the new chatbot experience is like strolling through a sunlit, flower-strewn meadow, where every step brings you closer to a delightful promise of ease, possibility, and financial fulfillment.

The metamorphosis of a static credit form into a dynamic chat interface is a task akin to unweaving a tapestry and reweaving it into a fluid narrative. The core challenge lies in transmuting rigid queries into a flowing dialogue, where every piece of data must be deftly captured amidst a dance of words. Navigate the labyrinth of rejection with grace, ensuring the user’s dignity remains intact while offering guidance through the fog of disappointment. Additionally, the chatbot must adapt to the ebb and flow of varied user inputs, embracing the rich diversity of language and expression. The solution lies in guiding customers to the path of financial freedom, regardless of the outcome. As they traverse this journey, each interaction brings them a step closer to their dreams, like wandering through a sunlit glade where every ray of light illuminates the path to a brighter, more liberated horizon.

Improving Model Accuracy with LLMs

Harnessing the power of multimodal data in credit prediction is akin to embarking on a grand adventure through a treasure trove of insights. Imagine venturing beyond the confines of traditional numeric algorithms into a realm where each piece of data — be it the intricate patterns in financial transactions or the detailed narratives in financial statements — tells a story of its own. This landscape is further enriched by macroeconomic indicators, where market trends and financial news provide a sweeping vista of the economic horizon, elevating predictions from mere speculation to a revelation of possibilities. For small businesses, the journey expands even further; customer reviews and social media engagement become vital signposts, guiding predictions with newfound precision. This treasure map of data ultimately enables more accurate credit assessments, unlocking opportunities for deserving businesses and ensuring that credit reaches those who might otherwise be overlooked. Thus, in this grand adventure of data, the myriad pieces — each distinct yet harmoniously interwoven — culminate in a masterful symphony of insights, illuminating the path to financial success and equity.

In the high-stakes world of credit evaluations, vigilant sentinels guard against chaos, ready to unveil hidden risks and seize fleeting opportunities. Their first mission: pre and post-mortem analyses, where they meticulously examine what went wrong with defaulted loans and what could go awry with granted credits. This intelligence is then fed back into the models, fine-tuning them with every lesson learned from financial near-disasters. As market conditions shift like unpredictable storms, LLMs quickly adapt, integrating fresh financial news and trends to keep predictions sharp and reliable. Imagine them as seasoned detectives, piecing together clues from diverse sources — like a business’s temporary revenue drop due to infrastructure investment versus one grappling with dwindling customer demand. Models can also gather rich, multimodal data during payment collections, using every fragment to enhance predictions and sharpen their competitive edge. Just as Sherlock Holmes deciphered a cryptic message to uncover the truth behind the spectral hound of Baskerville, LLMs reveal the intricate truths of creditworthiness hidden in the seemingly mundane everyday details.

In the grand rebellion against the empire of financial uncertainty, rebalancing portfolios involves adjusting investments based on credit payment patterns to manage risk and boost returns. Shifting allocations between industries balances risk, while increasing exposure to growing sectors enhance gains, like jedis. Bringing liquidity to existing credits is achieved by renegotiating terms based on performance, such as adjusting interest rates on mortgages nearing their end to better align with the customer’s financial situation. Incentivizing good decision-making through AI financial planners that guide customers to make better financial choices, like reducing discretionary spending, and offering rewards such as lower interest rates for following the recommendations. This mission transforms credit allocation into a path toward financial freedom, benefiting both individuals and the broader financial ecosystem. A herald a new era of growth and stability, after the destruction the death star of credit unworthiness, we pave the way to the universe of abundant investment possibilities.

Centralized Financial Data Repository

Alright, friends, let’s switch gears from our imaginative journey through the galaxy of credit innovation to a grounded, yet equally exciting, idea. I promise we’ll keep it clear and straightforward, even as we dive into some complex territory combining economics and technology. Buckle up, because this is where things get interesting.

You see, one of the biggest obstacles in the credit world is information asymmetry. It’s like when you’re playing a game of poker, and you need to make a decision without knowing the other players’ hands. That’s how lenders feel when they can’t get a full picture of a borrower’s financial situation. This uncertainty forces lenders to charge higher interest rates to cover their risks. What’s worse is that estimating risk takes time and adds transaction costs that get passed on to customers as fees. It’s a lose-lose situation that creates inefficiencies, slows down the process, and makes borrowing more expensive for everyone involved.

Customer incentives

Now, let’s talk about why customers and institutions would want to share data. For customers, sharing data can streamline the borrowing process, saving them precious time and reducing the hassle of repeated paperwork. For institutions, having access to comprehensive, accurate information means they can assess risk more effectively, leading to better decision-making and lower default rates. It’s a win-win situation that can make credit more accessible and affordable.

AI plays a crucial role here, acting as a gatekeeper that ensures only relevant information is disclosed where there is a legitimate possibility of accessing credit. This protects privacy and ensures that data sharing is both secure and purposeful. The beauty of this system is that information can be shared gradually, with some exchanges requiring explicit human consent and others being facilitated by agents from both credit institutions and customers. This dynamic process helps build trust and ensures that data is shared appropriately and efficiently.

Imagine this system as the birth of a new financial era, akin to the creation of the stock market or the dawn of venture capitalism in the mercantile age. This isn’t just a platform; it’s a bustling marketplace managed by a third party where data is the valuable currency. Much like the existing credit score system, but with far greater potential. This vision of tomorrow opens doors to endless opportunities for innovation in the financial sector.

Seed Startups

The seed startup market is the perfect breeding ground for this revolutionary marketplace. Firstly, the distributed nature of the market results in staggeringly high transaction costs. Each year, thousands of new startups vie for attention from tens of thousands of potential investors. Founders often find themselves spending more time fundraising — a full-time job in itself — than building their products or serving customers. With the short lifespan of startups, securing funding quickly is crucial to avoid extinction. The current undifferentiated process of applying to accelerators and cold emailing investors is inefficient and frustrating.

Secondly, the inherent risk tolerance and early technology adoption of startups and their investors make this market ripe for innovation. These players are not only open to new ideas but also actively seek them out, creating a fertile ground for groundbreaking financial solutions.

Lastly, the non-fungible nature of the services offered by investors adds another layer of complexity. Investors bring more than just capital; they offer invaluable experience and extensive networks. This creates immense opportunities for maximizing outcomes through well-matched partnerships. Imagine a world where the perfect investor is paired with the perfect startup, not just based on funding needs but on a symbiotic relationship that drives exponential growth.

This vision of tomorrow isn’t just a fantasy; it’s a call to action. With the right platform, we can transform the way startups secure funding, making the process more efficient, equitable, and innovative. Founders, imagine the possibilities: less time chasing capital and more time building the future. The era of financial innovation akin to the creation of the stock market or the rise of venture capital is upon us. And you can be at the forefront of this revolution.

Conclusion

In our exploration of the intricate world of credit evaluation, we have uncovered a fundamental truth about decision making. Like Sartre’s existential philosophy, the act of making a decision is both a burden and a liberation. Each choice carries the weight of responsibility, the consequence of uncertainty, and the freedom of possibility.

We have learned that decision making in credit evaluation, much like in life, is a delicate balance of risk and reward, knowledge and intuition. The power to decide shapes the very fabric of financial ecosystems, influencing lives and futures. As we embrace innovative solutions to streamline and enhance these decisions, we step closer to a reality where the burden of choice becomes a tool for liberation, enabling a fairer, more transparent, and more efficient world.

This journey has not only expanded our understanding but has also ignited a passion for transforming the way decisions are made in the financial realm. We hope you join us in this pursuit, embracing the freedom and responsibility that come with each choice, driving us toward a brighter, more equitable future.

Content inspired by the YouTube video: “Revolutionizing Credit Evaluation with Large Language Models” written by GPT-4o, prompted via a perfectionist human. Thank you for reading!

Insights from perception, inner speech, and spatial memory

I will present three ideas drawn from human cognition to innovate how we think about building AI agents. The first idea is about how memory is created and represented in the agent. The second idea is about how memory influences the development of a personality and improves the likelihood of an agent achieving its goals. The final idea is about enriching the information agents have available when making decisions and taking actions with perception. What all three ideas have in common is a multithreaded thought process, where agents use symbolic, probabilistic, and linguistic reasoning to enhance their understanding of the world.

A neural memory

Humans store information in at least three different ways: long-term memories, semantic representation of words, and spatial memory. The hippocampus is responsible for spatial navigation and short-term memory. In our minds, space is represented as a graph where objects are nodes, and edges indicate proximity and orientation. When we are in a new place, we reuse the same set of neurons, only modifying the activation patterns between them. Within the graph, neurons also form connections to emotions and memories, making it easy for similar places to share a similar context. In the hippocampus, inner dialogue facilitates converting short-term thoughts to long-term memories by repetition. This is what I call a neural memory.

Applying these insights into an AI agent results in a malleable, non-deterministic, fixed-size memory. Very different from traditional databases. It has three components: a grid, memories, and thoughts. The grid is a fixed-size graph that emulates neurons; connections are formed and reinforced by activity. Memories are activation patterns—variable-length arrays indicating which neurons were activated when representing that outer object. Thoughts are sequences of memories, the order of which is probabilistic based on similarity.

The basic idea is that when a new memory is learned, it is based on the existing structure of the grid following an activation pattern similar to related memories. A recurrent algorithm in the background, continuously updates memories, adapts the connections between neurons on the grid, and calculates the similarity of memories to form future thoughts. The algorithm is neuro-symbolic, where mathematical operations coexist with a reasoning process to create a space-efficient storage system. In AI agents, neural processing is powered by a large language model (LLM) that links memories with a semantic, verbal connection to form thoughts. Updating the memory is part of a larger process that rewards memories that lead to successful otucomes.

A neural memory is more complex than a traditional database, but it has three significant advantages: first, it stores similar elements close to each other. Second, it has logarithmic efficincy in terms of space because the grid has a constant size, and memories are comparatively smaller due to their sparse representation. Third, updates are fast and cheap since only a small number of edges may need to be modified, even for large objects. Additionally, it can store related memories by reusing the original activation pattern, only introducing small changes (hierarchical representation). A neural memory shares similarities with vector and graph databases but scales more elegantly and is more adaptable to the dynamism of knowledge. It is ideal for common use cases like recommendation engines.

Developing a personality

There is something intriguing about our mind’s spatial representation. If space is represented as a graph where nodes are objects. Then, there is a specific node that represents us. Nodes representing objects are not only connected to other objects but also to emotions, memories, and meanings. Just as we might have a preference for certain food or people close to us evoke powerful emotions, we also assign ourselves attributes that influence how we perceive us. Identity is not solely based on our memory; the process is far more sophisticated. But this meta-representation provides a useful tool to endow our agent with a personality. Assuming that the agentś goal is fixed, and it has no control over it, the agent can allocate part of its memory to represent itself as an ensemble of characteristics that maximize the likelihood of attaining its goal. Developing a personality becomes part of a larger reinforcement learning process that rewards positive behavior of the agent.

The agent’s personality becomes at service of its goal.

In “Inner Speech: Development, Cognitive Functions, Phenomenology, and Neurobiology” authors Ben Alderson-Day, and Charles Fernyhough explain that inner dialogue serves four main purposes: a) memory creation and memory retrieval, b) planning, problem-solving, and task switching, c) developing social awareness, and d) generating motivation. Not all of us experience inner speech, but for those os us who do, it is an important part of our cognitive process. Inner speech happens in what is called the "Brain's Default Network," which activates when we are not paying attention to our senses, for example, when daydreaming. The development of a personality can be considered a problem-solving task where the person needs to find the optimal facade to navigate a complex social situation. Then, the inner dialogue becomes the ideal tool to express, and develop our agent's personality.

Embodying an inner dialogue in an AI agent is theoretically trivial, but making it coherent is not as easy. Hallucinations might increase, and the sense of purpose might be lost. “Graph of Thoughts” and other prompt engineering techniques have provided a useful blueprint for solving complex problems like sorting. In this scenario, a reinforcement learning algorithm may experiment tunning different prompts. The key is having a close-loop between the agent’s behaviour, and the results. Based on that the agent can gradually develop better decide what information to prioritize, how to communicate, and how to adapt to different circumstances. This meta-cognition process becomes the agent’s awareness.

Creating meaning with perception

As the image depicts, the difference between sensing and perception is vast. Today’s agents mostly engage with the text input without creating a deeper, richer representation of what the human wants. To better understand what is perception, let's consider human vision as an example.

Visual perception starts starts when light is reflected from objects. Our retina captures three basic colors: red, green, and blue. Through signal processing these primary colors are combined into a rich color spectrum. Bits of light that are close to each other, and have similar colors are groupped into regions (image segmentation). Next, a visual map is generated that emulates what we see, and help the brain identify lines and edges, which are later converted into features and objects. Finally, sophisticated algorithms are used to perceive depth and motion from images using our binocular vision. During the process, the visual cortex engages with memory, language, and cognition to derive meaning from images.

Now that we appreciate the complexity behind perception. Let’s consider applying it to AI agents for inferring user’s intent and detecting change. Under this scenario, the agent enriches the input to further clarify what the user wants and how is this interaction different from others. The agent may draw insights from the time of day, similar engagements in the past, capture subtleties in the language, or even consider the behavior when typing. The extra information is used by the language model to deliver a better customer experience.

Perception cannot be solved using LLMs. The large amount of information the agent requires to process makes it unsuitable to the slow, single-threaded, context-constrained, computationally intensive, hallucination-prone reasoning that language models provide. Instead, signal processing, and machine learning algorithms will be used to quantify the information into a set of choices that the language model can use. Lightweight language models may be used to connect the last parts of the perception process with the mathematically-intensive early computations. Please note, that enriching with text with context is also useful for training, and fine-tunning language models. As the uncertainty of the next token decreases with the additional context. The final outcome is an agent that can anticipate user’s intent, offer suggestions, and better accomplish the end-goal of the customer.

Perception creates a map of the customer’s problem statement: an ideal representation for a search algorithm. What the agent does is moving a person from a place where there is a problem, to one with a solution. In navigation, three variables are considered to reach a destination: location (where I am), mapping (what’s around), and odometry (movement). Based on that a path can be found, either in a static or dynamic environment. Perception creates clarity on where the customer is, where it wants go, and how to get there.

However, there is a crucial difference between human vision, and the semantical, text-based environment where AI agents live. Seeing depends on the three-dimensional space we live on, and our binocular vision that helps us contrast images. True innovation in the development of AI agents require a new representation that allows us to process language at scale. The foundation of our current written language was designed around 4,000 years ago. Written language provides low-bandwidth to transfer information compared to digital mediums like optic fiber. While I do not know how that new paradigm of language looks like, I know where to find for inspiration.

Bioinformatics

Molecular biology studies how proteins encode information. Compared to our linear language, proteins exchange information in multiple dimensions simultaneously. Proteins follow a sequential structure, just as human language, but they have three additional dimensions. First, every single amino acid that composes a protein has a chemical structure that influences the protein locally. For example, it may have an electric charge or be hydrophilic. Our letters, on the contrary, are mostly arbitrary; while some letters play a syntactical and grammatical role, they provide relatively little semantic value. Proteins can be thousands of amino acids long, and they fold in three-dimensional space.

The specific shape that a protein adopts is highly relevant to its function. For example, proteins may have a hydrophobic membrane to avoid dissolving in water, be hollow to transport nutrients, or have precise shapes to match other enzymes or molecules. While our documents can also be thousands of words long, we fail to visualize their semantic content in a single image. The best we can do is summarize it in a title. Moreover, we do not have a theory or taxonomy that adequately explains what structure of a text is ideal for a given function. Statistical analysis to develop this theory may be one of the simplest steps we can take to begin scaling our capacity to assimilate information. Finally, proteins support a quaternary structure to form macromolecules. It would be useful if we could group a large collection of related texts to accomplish specific purposes.

Conclusion

This content originated from an “Introduction to AI Agents” talk that I gave in Mexico. Upon the start of my research, I considered AI agents as one of the many applications of the new wave of innovation powered by AI. Today, I see it as the primary application of AI that will transform the fibers of our society. The fact is that this is the next phase of Earth’s evolution, where synthetic forms of life cohabitate the planet. While many may see this as dangerous and fear the loss of their privileged place as the sole species at the top of the evolutionary hierarchy, I offer you a distinct view — one where intelligent machines become the third kingdom of life in a mutually beneficial relationship.

After all, non-sentient machines may be our only possibility to reverse the damage we have done to the planet or provide the work required to terraform other planets and satellites in our solar system. My hope is that the present narrative inspires founders, researchers, investors, and readers to learn more about the applications of human cognition to the development of Artificial Intelligence. Please, trust the wisdom of millions of years of evolution in designing and testing complex systems.

Contact

I write three times a week on LinkedIn. There, you can read my developing thoughts as I conduct research and influence the final outcome with your comments. The content is also available on YouTube, where I dive deeper into the technical details. Next year, I plan to build a startup. If you are an investor and like the ideas I present, please consider reaching out. I’m also open to part-time consulting if the problem is interesting. LinkedIn is the best place to reach me for business inquiries. If you just want to chat about the ideas presented here, please connect with me on LinkedIn, and I will be happy to share my Calendly.

Acknowledgments

First, I want to thank you: Medium readers. My last story about “How to build a startup” was very well-received, and motivated me to invest the time, and effort into writting this: your views, clapps, comments, and highlights truly make a difference. Second, to the attendees of the Introduction to AI Agents talks in Mexico. The positive feedback that I received encouraged me to share thoughts that I find interesting, but different to the content that is usually available. Lastly, to the writers of the ReAct paper that included the reference to the inner dialogue paper that inspired the torrent of curiosity that culminated into this story. Thank you!

By most standards, getting into YC represents a success: the admission rate is less than 1%, it invests a significant amount of capital often pre-revenue, and you become a member of an exclusive group that includes the founders of Airbnb, Dropbox, and DoorDash. For the analysis, I used an LLM to embed the companies’ value propositions in a vector space, ran a clustering algorithm on top of the embeddings, manually analyzed the results, and used ChatGPT to edit the draft. The conclusions are surprisingly elegant and powerful.

1. Driving efficiencies

The most common mechanism for creating a venture-backed business is by bringing efficiency to existing markets. This approach is the simplest and least risky because the demand for the product already exists. The promise is to deliver a product that is quantitatively better than what currently exists. There is often little technological uncertainty, as existing technology is applied to a new domain (low R&D). However, it causes significant disruption because it directly challenges existing players. Fundamentally, driving efficiency involves capital reallocation, where the end customer benefits from the disruption of the intermediate layers in the value delivery process.

There are clear distinctions between B2B and B2C startups that aim to drive efficiency. In B2B, efficiencies are geared towards improving or reducing the workforce. Unsurprisingly, sales receive the most attention, as it is a human-driven activity. Another source of efficiency in business is the reduction of transaction costs, which may involve reducing bureaucracy. Typically, it is expected that through automation, workforce productivity will increase, resulting in larger profit margins as the business grows and providing a competitive advantage for early adopters. Consequently, we can expect a myriad of new startups powered by AI as the possibilities for automation increase. Conversely, I was surprised by the large volume of B2B driving-efficiency startups created before the advent of generative AI, as there was little seismic technological movement enabling the entry of new players. I hope that the surge of SaaS is the result of process improvements based on insights discovered through repetition (process power). If so, I would expect the rise of AI startups to gradually increase, peaking somewhere in the next decade.

Consumer-oriented startups that drive efficiency are slightly more interesting but substantially more complex. In B2C, efficiency is passed down to consumers by aggregating demand or creating a marketplace. The first follows a business model similar to Costco, while the second is akin to Uber or Airbnb. Both models require significant capital investment as they depend on creating economies of scale to drive efficiency. This tends to result in winner-takes-all markets, a risky proposition for investors and founders who, despite investing substantial revenue, time, and effort, may end up with nothing. Speed is critical, as the first startup to reach a critical mass has a disproportionate advantage in capturing the market. For the rest, acquisition is the best alternative, but timing is crucial because the business model requires subsidizing the price for customers until economies of scale are reached. I suspect the business acumen of the founder plays a substantial role in the success of these startups, particularly in their ability to attract capital, deliver a superior customer experience (a new paradigm), and act boldly.

Optimally, capital is allocated to offset the switching costs for customers to adopt the new paradigm. In this scenario, venture capital provides the financing for customers to enjoy the efficiency benefits without an upfront commitment. However, there are downsides. First, for customers, there is no guarantee that after switching, the efficiencies will be passed to them indefinitely (cornered resource). Second, companies are exposed to the risk that a new paradigm might replace them by driving greater efficiencies. Given the minimal technological innovation, defensibility needs to be built around size or branding. Third, growth often comes at the expense of triumphing over competition, which creates moral dilemmas not suited for the noble-hearted. Your company may be selling productivity to stakeholders, but it may also be asking to cut jobs to realize the gains associated with the transaction. There is an inherent contradiction in the incentives between the startup and the customers: the startup wants to grow as big as possible, while customers want to maximize efficiency. Personally, I would find it difficult to build my value proposition around driving efficiency without also having the guiding principle of delivering value and taking responsibility (creating parallel solutions) to remediate the value destruction.

2. Removing limitations

After the dire passage through Schumpeterian economics. I’m glad to move forward to startups with brigther aspirations. I was pleasantly surprised by the significant number of companies whose core purpose is serving and bringing value to underserved communities. Particularly in FinTech, which I once considered one of the most uninspiring business models, I now believe that successful startups have some of the most noble missions.

Startups in this category possess greater technological defensibility compared to efficiency-driven startups, striking a delicate balance between technology and domain expertise. Rather than disruption, there is a focus on adaptation and creativity, where novelty, and originality take precedence. However, they encounter the same limitation as efficiency-driven startups: their valuation is constrained by market size. Although the market can grow, or the startup can enter adjacent markets, each expansion attempt carries a commensurate risk due to the absence of true technological innovation. Estimating demand is also more challenging, as it remains unclear what price customers are capable or willing to pay. An unwelcome surprise could arise if, after consolidating the market, the business proves to be unprofitable. Moreover, there are contrarian incentives at play; while removing limitations adds value for customers, it also renders the startup redundant. This necessitates the exploration of business models that only partially solve the problem, potentially fostering long-term dependence or focusing only on recurrent problems.

Adapt an existing solution for underserved communities: Indeed, this is where FinTech finds its niche. A considerable proportion of YC FinTech startups operate on an international market, targeting regions where financial services have low market penetration. This category also encompasses what is known as “technology tropicalization” bringing solutions from advanced ecnomomies to developing markets. I identify three primary reasons that facilitate the establishment of businesses catering to underserved communities. Firstly, incumbents may have been sluggish in expanding into adjacent markets. Secondly, margins may not be lucrative enough to incentivize market leaders. Thirdly, regulatory or regional challenges may necessitate significant modifications to existing solutions. The growth potential of the market is a crucial factor in valuing its economic potential. While the opportunity might not seem significant today, being an early entrant can yield substantial advantages in the future. Typically, the most effective founders have an emotional connection with the community and possess a profound understanding of its needs. However, it’s equally plausible for an expert in a particular solution to seek out a community in need of that solution. The most successful teams may possess deep expertise across both aspects: domain, and solution.

Solve emerging problems, often due to scale, using data: As companies expand, new challenges inevitably arise. Solutions that were effective at smaller scales often prove inadequate when operating across hundreds of countries, with millions of employees, and serving billions of customers. Utilizing data to automate existing solutions emerges as the most successful method of scaling. Data provides the insights that enable the replication of solutions that were previously feasible only through manual efforts. While these solutions may not be flawless, they are operational. One advantage of this business model is that with continued growth, complexity increases, leading to greater rewards. Additionally, data fosters network effects, bringing processing power, and creating a moat around the solution that improves its effectiveness with each new customer. Cloud startups play a significant role in this domain, and the importance of AI is also steadily increasing here. Ideally, founders have firsthand experience with these challenges and possess deep technical skills: new problems require innovative solutions.

Apply technology to increase the availability of a good: Envision a luxury item that is currently beyond your financial reach, but with technology, you could enjoy a comparable experience. This process often involves the productization of a service. This category is evenly balanced between consumer-oriented goods and business-oriented solutions. If a domain expert, such as a consultant or teacher, can replicate the service they provide using technology, they hold the potential to become a successful technological business. Additionally, there is a connectivity component within this category. If an activity benefits from increased participation, technology can enhance the value delivered by overcoming the limitations of the physical world. Remote-oriented solutions serve as a prime example in the B2B realm, providing a digitalized substitute for valuable human interactions. As this category addresses decreasing scarcity, it represents the closest approximation to an unconstrained market thus far.

3. Advancing technology

The crème de la crème. The most adventurous, creative, and entrepreneurial way to start a business is to move technology forward. It offers the highest upside, but entails the largest risk. Initially, the market may be undefined, and the likelihood of developing the technology is low, but when it happens, the impact is substantial. The market becomes unconstrained, offering long-term defensibility, and pushing the boundaries of our limitations. When advancing a technology, the solutions are not restricted to a single market. Truly transformative technologies can address multiple problems across industries. Expansion is seamless because differentiated value is provided by the technology. Capital is primarily required for research and development, and occasionally for capitalizing on created value and protecting intellectual property. The ideal scenario occurs when processing power accompanies the technology’s usage, enhancing its effectiveness with usage, and time.

From YC data, three broad categories encompass the advancement of technology: healthcare (biotech), industrial (including climate), and until a couple of years ago, AI. I suspect that the commoditization of Generative AI will shift the focus of deep tech towards robotics. The diversity of companies and technologies would warrant a thorough analysis on its own. It would be intriguing to explore how trends have evolved over the last 20 years and attempt a causal analysis of success. I anticipate that the academic background of the founding team would play a crucial role, and I am curious about the extent to which business acumen influences the success of landing moonshots.

As I surveyed the list of deep tech companies, I couldn’t help but feel slight disappointment in the venture capital industry. Its aversion to risk and lack of innovation in its business model are glaring. From my understanding, most investments occur only after there is already a proof of concept for the technology, resulting in a limitation of scope (ambition) and audience (those able to afford the risk). It’s likely that a significant portion of deep tech advancement originates in academic institutions, only to be capitalized upon once the most promising research emerges. However, this process is serendipitous, slow, and relies heavily on individuals, governments or non-profit willing to absorb the risks involved in financing the education. Investors, have greater risk tolerance, industry knowledge, and connections. They could potentially do a better job at allocating the brightest minds to the most pressing societal problems. This insight was recognized by Thiel’s Fellowship over a decade ago, but as a non-profit has not capitalized on its returns. Perhaps I’m being naive or overly optimistic, but I fail to see why half a million dollars couldn’t be invested to incubate a technology over 3 to 4 years, rather than attempting to accelerate a business in just 3 months.

Conclusions

When I began studying YC companies, I didn’t anticipate writing about it. It has been a rewarding process that reminded me of the benefits of slowing down and carefully examining data. The conventional narrative suggests that starting a business is challenging, and a startup even more so. Through research, I’ve gained clarity on two of the biggest causes of failure: offering something that customers do not want, and overestimating the size of a market. I refer to the first as ignorance and the second as greed. Estimating a business opportunity should be straightforward: evaluate the size and growth of the market, the potential penetration rate, and the likelihood of successful execution by the team. Make decisions based on quality data over a three to four-year horizon, and repeat. Capital becomes an advancement of future earnings, as it should be. Ultimately, I discovered what we are always taught: that demand cannot be created, at least not by us as founders.

The research also serves as a playbook for future founders. Not everyone aspires to build a multi-billion-dollar company; many of us find satisfaction in a respectable million-dollar online business that offers flexibility, independence, and freedom. Defining a technology business involves three key components: the market you are serving, the value you are providing (quality, price, scale, availability, etc.), and the technology that supports it. Consequently, the process of building a business is transparent: i) identify an inefficiency, underserved community, emerging problem, or limited good that could benefit from technology, ii) filter out possibilities where network effects are likely to dominate (winner takes all), and favor those that benefit from differentiated offerings (competition), iii) construct a value proposition aligned with your skills and experience. Self-awareness is crucial. Avoid greed, and you’re likely to lead a fulfilling life.

Finally, the research also illuminates a pathway to positively transform the venture capital industry. Not long ago, the industry was predatory, limited to a select few, and confined to Silicon Valley. Today, it has become accessible, transparent, and global, albeit not perfect. There lies an opportunity to embrace more adventurous approaches, further distinguishing venture capital from private equity. which typically focuses on providing credit. Venture capital holds the privilege of envisioning and shaping the future, empowering innovators to pursue bold ideas and catalyze change. I leave it to the reader the reflection whether today’s fundraising models are optimized to create value or if, instead, we are thinking too small, constrained by certainty, and focused solely on what exists today rather than on what we can create tomorrow.

It is not from the benevolence of the butcher, the brewer, or the baker, that we expect our dinner, but from their regard to their own interest. We address ourselves, not to their humanity but to their self-love, and never talk to them of our own necessities but of their advantages.

-Adam Smith

Acknowledgments: The depth of the analysis owes much to the groundwork laid by brilliant business strategists. Special mention goes to the books “Blue Ocean Strategy” by Renée Mauborgne, W. & Chan Kim, and “7 Powers: The Foundations of Business Strategy” by Hamilton W. Helmer, as well as “The Product Market Fit Method” by First Round Capital, and the “Acquired” Podcast. But most importantly, a huge thank you to you, the reader, whose curiosity and engagement inspired this work.

Originally published at https://amvizion.org.

Abstract: Compared to medical treatment, software consulting remains a rudimentary activity lacking standardized practices with predictable outcomes. I will present introductory concepts of Human Pathology to show how software developers and consultants can learn from them. Additionally, I will propose a methodology for applying these concepts to evaluate, diagnose and improve software projects. Finally, I’ll share some reflections of the actions needed to be taken by the software guild to adopt the scientific method in our daily practice.

1. Introduction

Pathology is the science that studies human suffering, it’s an elementary subject in the medical training to diagnose and treat patients. Their modern foundations can be traced to the invention of the microscope and the consequent discovery that microorganisms are the main cause of infectious diseases. The application of the scientific method in medicine brought an end to unsubstantiated claims that considered a disease as the expression of cosmic imbalances, part of ancient practices like Alchemy and Astrology.

Today, software development is not that different from those dark ages of medicine. Our craft and skills are mostly acquired by oral tradition where learning is done through mentorship. Our remedies lack the precision of typical medical treatments that are measured in scientific units to prevent deadly repercussions. Yet, we wonder, why so many software projects fail, and why the adoption of improv methodologies, like Agile, fail to address the increasingly complex problems of a science that has become a crucial component of society.

My invitation is to read this post with an open mind. Some of the analogies here presented will seem exaggerated, others speculative, and some even uninformed and false. However, my intention is not to say something new, nor even lead a transformation in how software is developed. I would rather like to invite the reader to reflect on what we can do as software developers to improve in our craft, pay back the trust that society has deposited in us, and elevate our profession.

2. Background

I was the most surprised when first named Tech Lead. I certainly didn’t receive any training, nor did I requested or discussed the promotion. Is not that I didn’t want it, but I was certainly not prepared to manage a whole team from day one. The same happened as I first became a developer, proudly self-taught, my education consisted of online courses, algorithmic exercises, and online forums. It was up to me, and the patience of former bosses and customers, to learn how to design a database, maintain a Linux server, and craft user-friendly experiences.

Today as a software consultant, I face new challenges: I need to quickly assess the health of a codebase, emmit eloquent recommendations to quickly fix problems, and upskill or improve the productivity of in-house teams. The skills that I learned as a Tech Lead like documenting, designing, or managing projects seem comparatively inefficient to my new hourly rate. As I look back and evaluate my recommendations, I see how limited the arsenal of tools at my disposal truly is, how unfeasible is to clearly communicate years of experience to non-technical stakeholders, and yet, as I contemplate my limitations I also see the other side: the drastic improvement in productivity that good leadership creates.

During the last months, I searched for answers in books that included clean architecture, project management, systems design, consulting, and best practices in leading tech organizations but I did not found the answer I was looking for: how to quickly turn around productivity in small organizations. In despair, I look for inspiration in other disciplines like architecture, and eventually, I was fortunate enough to find the principles of this answer in the medical science of Pathology.

3. Eight Concepts to know about Medical Pathology

MIT has a program for non-medical practitioners called GEMS to give an outsider introduction to the medical profession. The first subject, and the only mandatory, is Principles of Pathology based on the homologous book by Robbins & Cotran. The next section includes potential applications to software development for the concepts presented in the first 7 chapters of the book. As a disclaimer, my experience in healthcare is limited to 2 months of research, short professional interactions as a consultant, and previous knowledge from past writings also inspired in life sciences [1][2][3].

3.1. The human body is the epitome of a well-engineered system

It is futile attempting to describe the level of sophistication of the human body. But I will give you two examples to get a grasp about it:

1. Each cell holds about 750MB of data in its DNA (about the same as a legacy CD) and we have about 1 trillion cells in our bodies. In total, we store roughly a Zettabyte of data: the same amount that we created in our full history as a species up to 2016!
2. Our brains have100 billion neurons, about the same number of stars in our galaxy, and each neuron has on average 1,000 connections with each other. As a civilization, humankind has yet to reach 8 billion people, and on average the 2+ billion Facebook users only have 300 “friends” each. Arguably, a single brain is a more complex network than our society as a whole by a factor of at least 25x.

As a Software Engineer, I try to approach the study and appreciation of the human body with humbleness. My opinion is that every problem worth solving has already been solved by nature.

3.2. Cells are the basis of human health

Diseases in the human body are the result of many individual sick cells. In retrospect, this seems obvious but as a developer, it ain’t, as we fail to articulate what constitutes a healthy software system.

Following the analogy, we might say that health in a software system (or lack of technical debt) is the result of well-written individual functions (or methods). Adopting the function as the basis of health enables us to empirically determine parameters for how a healthy function looks. As suggested in clean code practices this might include:

• The number of lines code, variables, and even tokens or characters.
• How clear the function and its variables are named.
• The number of input parameters and return types.
• Cyclomatic complexity (number of dependencies).
• The number of conditionals, iterations, or recursions used.
• Clean error and exception handling.
• How and where is the function invoked.

By defining how a healthy function looks we can start to approach the evaluation of a system as the aggregate health of its individual components.

3.3. Cells die due to injury, failure to adapt, or age

Here, the first interesting concept is aging: cells are programmed to die. But we hardly think of functions that way. We like to believe our code will be used ad infinitum. But imagine if a function would be programmed to be deprecated? The original developer would feel more comfortable knowing how much care to dedicate to different portions of the codebase.

Even more interesting, is that cells reproduce by mitosis (splitting by half and creating two copies) but the number of divisions is also limited to 40 to 60 times [4]. Consider how cleaner the code will be if a function would deprecate after a fixed number of modifications as measured by pull requests. We know that nothing harms more the readability and consistency of a codebase than unexpected shifts in product requirements.

Thinking about stem cells, that can adapt into any type of human cell, seems far. However, I can envision creating repositories of template functions that developers would clone and adapt to their needs. These functions will have best practices implemented to them, and their deviation from them would constitute technical debt.

The second interesting concept related is that cell death is also a consequence of failure to adapt. Software projects also often fail due to failure to adapt to market conditions.

This leads me to think that software development is a human activity unlikely every other one done in the past. Consider an artist, that after finishing their work, it remains intact. Or more traditional branches of engineering like civil, mechanical, chemical, or aeronautical: their designs often meet circumstances well-known in advance.

Even scientists, that aim to formulate theories, mostly assume the invariability of the laws that govern what they are trying to understand. Software is different perhaps a little bit like social sciences or business they find an array of unpredictable circumstances derived from human behavior.

In my opinion, the emancipation of software from business is one of its greatest curses, and we need to find a new word to define the process of producing software to create products that meet changing business conditions.

3.4. Inflammation is the mechanism for cells to heal

When you cut and blood, the skin turns redder as it heals. During flu, you cough and get fever as the body fights the virus. Both are symptoms of inflammation the mechanism to bring extra blood and white blood cells (leukocytes), to the damaged area.

How does healing occurs in a Software system? By bringing extra resources to the afflicted area. When a bug is found, developers rush to fix it, and after the solution is pushed it leaves a trail in the Git history. Curiously, the inflicted area is also stained in red as a fix usually involves more deleted lines of code.

Bug fixing leaves a second sign after is fixed, it is the absence of work in other areas as more developers' attention was entertained. As a result, the overall productivity of the team plummets. As the science of Software Pathology is discovered, software consultants will use the Git history to diagnose which areas have been inflamed

3.5. Blood (or its lack of) is the biggest source of cell disease

Everything that a cell has or needs is provisioned by blood. That is why cardiovascular diseases are the leading cause of deaths globally at 18% [5]. If a cell stops receiving oxygen or nutrients its health will deteriorate, and we need exercise to keep the heart strong and flux blood to the entire body.

A function receives data, and if it is of low quality, arriving with errors, or overly complex: the function will suffer. Thus, data hygiene maintains the codebase healthy.

The circulatory system is impressive in its own right: it stretches more than 66,000 thousand miles, with every cell carrying more than 1 million molecules of oxygen. On average, the heart pumps 2.5 billion times during a human life [6].

Software systems also have a heart: the database. Evaluating how data is queried, and fetched is the first step of diagnosing the health of a software system. How data flows through a system is probably the first diagram I would advise, to understand a system.

3.6. A surprising number of diseases can be attributed to genes

Before my research, I used to consider genes as static, highly reliable, exhibiting a low degree of variance. However, I learned that almost 70% of humans suffer from diseases attributed to genetic disorders.

The concept of genetics is foreign and difficult to translate to Software. Fortunately, through my experience, I’ve started, and participated in a fair amount of different projects. And I could argue that a portion of software projects are doomed from the beginning.

In my experience, most projects fail due to business reasons, and after several iterations, these projects have accumulated so much technical debt that continuing development is unaffordable. When thinking about the software we hardly stop to think of the history of a project: not about the development but about the people that participated in it.

A source code is the reflection of the decisions, desires, limitations, and experiences of the people that created it. As we study the health of software systems we need to understand how human behavior impacts the outcome of a project.

Either because a key member left, there is a difference in vision, agile practices are clumsily applied, or there is no congruent business model. As a consultant, I’ve learned that not every software problem lies in the code.

3.7. We have an immune system to heal ourselves

Self-healing systems have been increasingly part of the SREs toolbox: complex canary configurations in Kubernetes clusters, a predominant acceptance in the benefits of testing, or the automation of security code reviews. The concept of an immune system in software projects is closer to the reality that we might think.

The immune system is divided into innate and adaptative. The innate has learned from diseases and pathogens that have accompanied us during our evolution. While the adaptative learns from comparatively shorter periods.

Probably the next step in self-healing systems is preventing problems before they arrive (or show symptoms). This could happen as soon as we are developing, after establishing the rules of what quality software is, we can use AI copilot tools for advanced linting signaling problems as soon as they are typed.

Even better, as we trace back the causes of problems in software to product development, potential problems could be spotted as soon as they are assigned to developers. Having practiced BDD principles I believe that by using passing criteria the goal is not as distant as it seems.

3.8. Histology is the study of tissues or how cells are connected

I want to share the last concept to show how deep the understanding of medical diagnosis goes. After determining that cells are the basis of health, and carefully classifying each type of cells based on its function. Physicians have studied how cells are connected through tissues linking cellular biology with anatomy.

In software, we need to start thinking about the graph topology of how functions are connected. Contrary to the human body where the connections between cells are tridimensional, in software, functions interactions can become much more complicated. Because a single dependency can be shared by the two far away modules creating turmoil.

The knowledge of histology has been harvested through the meticulous examination of different cells and tissues using microscopes, staining, and freezing techniques. We need a similar academic and scientific effort to harvest the knowledge about software pathologies.

We like to think that Software is complicated, vast, and unintelligible. Full of different technologies, languages, and cloud providers. However, compared to the diversity in nature, the complex taxonomy that originated from evolution, and the humble tools that early scientists like Aristotle had at their disposal: the task of classifying software systems is comparatively simple.

4. Diagnose the health of a React Application

During the next section, I want to imagine how the job of a consultant would look following the analogy of a doctor and a patient. The tools that they would require, and the framework to approach problems with the precision that is expected from doctors.

I start exemplifying three distinct analyses a consultant could request from a codebase. Then, I will focus on the interactions with stakeholders to communicate the diagnosis and suggested treatments. The goal is to approach problems in software with quantifiable metrics to make data-driven decisions.

For this hypothetical exercise, I will use the opinionated architecture of a React application because it follows a functional paradigm, with a unidirectional flow of data and reusable components.

4.1. Measure the health of React Components

The first analysis a software consultant would request when approaching a new React application would be to measure the relative health of components based on some of the rules highlighted in section 3.2. which include the length in lines of code of a function (or component), the number of dependencies, error handling, naming, etc.

As an example of how this analysis would look consider the SCC GitHub package that measures lines of code, estimated cost, and the complexity of a project. Written in go, the package analyzes the source code in a matter of seconds:

While doubters might be skeptical, of how effectively this can be measured. The process would consist of aggregating data from open-source projects. Relating different metrics like Github issues, pull requests, labels, and stars correlations could start to appear. Finally, machine models could predict the optimal parameters for lines of code, component inputs, depth of interfaces, and many others for a React project.

4.2. Map the flow of data through components

After figuring out how healthy individual components are, we would like to know how those components are located and related to each other. We could use traditional flow diagrams to map how data flows through components. And like a river amid mountains, React applications are pyramidal architectures where data flows downwards to the user.

We could start detecting if some components are saturated with data, or maybe in others data travel a long journey before reaching its destination. Or using other diagrams we could also spot the components that have the most dependants or because of the high number of bugs detected they are the weakest links in the system.

My ideal flow chart would have these 3 features:

1. Arrows connecting two nodes represents data flowing from one component to the other.
2. It is possible to see the interface of the data between connections.
3. The components are colored by health.

Physicians use x-rays, magnetic resonance imaging, and ultrasound to visually diagnose patients. Upon reflection, I’m surprised how underestimated spatial techniques are in software. Having a global representation of the system enables more accurate diagnosis and communication with stakeholders.

4.3. Review the short-term history of Git commits

The final analysis I would request is to understand where developers’ attention and resources were allocated during the last weeks. The chart I would propose is a stacked area chart where each color represents a different section of the code:

As an only change, I would map the area under the curve as it is done in Github insights to represent code deletions. This analysis would bring me two key pieces of information:

This analysis would bring me two key pieces of information:

1. How productivity compares across different weeks.

These 3 analyses are only a subsample of the many possible a software consultant, tech lead, or CTO could request. It’s exciting to think that the analyses could be fully automated. And only after data about a project is harvested does the real consulting starts; where creativity, experience, and elegant design makes the difference.

4.4. Collaborate to emit a diagnosis of the problem

Going back to the patient-doctor analogy, the next step after the analysis and tests are performed is to emit a diagnosis. For doctors, how the diagnosis is communicated, is just as important to what is communicated. A full diagnosis should include:

1. A transparent description of what the disease is.

2. An explanation of the causes and symptoms that triggered the diagnosis.

3. A map of potential outcomes based on the diagnosis.

Before proceeding with the treatment, patients need to understand and accept the diagnosis. Thus, trust becomes an essential component in the relation. There is a human component as the doctor communicates good or bad news, warnings, or words of fate. Example summarized diagnostic could look like these:

• New feature specifications are ambiguous, and the expected outcomes are not clearly communicated. This is because engineering does not participate at the product planning stage nor do they know about the business needs that prompt the requests. As a result, there is a lack of ownership by developers and no commitment to results.
• The new development is blocked by shared dependencies across multiple applications. Since the dependencies are not thoroughly tested, and no single person understands the repercussions of modifying them there is no certainty as to what will happen when changes are made.
• Bugs are taking too long to fix because monitoring is done manually. Additionally, logging is often insufficient and developers take time to reproduce bugs. Because the platform is highly configurable it’s not cheap to fully automate testing causing bugs to be discovered in production.

Note, how none of these diagnoses offer a solution, they are explained in plain language making it possible for non-technical founders to understand and agree with the diagnosis. Additionally. the diagnosis could be supported by data or precise examples that illustrate the problems and increase its deliverability. Finally, they are obviously aligned to the problems the client sees, and the outcomes they would like to achieve.

4.5. Recommend a treatment

Something that doctors don’t do is to make decisions for the patient. Instead, they only offer the alternatives available for them, possibly adding their personal recommendation. Just as we did with the diagnosis there are hints that I learned from doctors:

1. If the problem is not urgent, there is no need to rush to offer a solution. Let the client digest the news about the diagnosis before proposing a solution.
2. The least radical treatments should be communicated first. The more severe the diagnosis, the more human the communication needs to be.
3. Use empathy and let the client know that you understand what they are going through.
4. When making a recommendation, be sure to also communicate the risks, implications, and what they can do to improve the odds of success.
5. Optimize for trust, there is little chance of advancing to a positive outcome if the client does not trust your judgment. And if possible, prioritize the client’s needs above your own: a doctor is all about personal sacrifice.

Ideally, the efficiency of treatments and recommendations are based on data. As software engineers, we are used to applying logic to solve our problems. Unfortunately, humans are not great to instinctively deal with probabilistic scenarios. And by trying to apply logic to complex problems we risk that cognitive bias clouding our judgment.

5. Turning Software Pathology into a Science

Medicine was not always scientific, in the past dubious remedies like bleeding or trepanation were used despite their limited efficiency, often worsening injuries. There is a possibility that future software scientists will look with the same astonishment our approach to object-oriented design patterns which could appear Ptolomeic efforts to impose a worldview into the observable data.

As a nascent science, there are 3 steps software development unequivocally needs to take to adopt the scientific method, settle opinionated debates, and institutionalize its practice. Else, we are left with dialectics, power dynamics, and mysticism to guide development.

5.1. A lexicon of diseases

“Flu” describes a well-known set of medical symptoms, causes, and consequences. The same happens for malaria, smallpox, cancer, and many unfortunate other diseases. At the same time, these names are based on taxonomies shared across life sciences that started with the classification of species in biology using Latin etymologies.

Having a precise vocabulary would enable us to communicate, think, teach and solve problems. And by avoiding using the same term to refer to multiple problems we could stop dogmas and beliefs that there is a one-solution that fits all, such as TDD (test-driven development), or DevOps (automation at all costs).

Before this research, I thought that the labor of classification done in other sciences was dull and unnecessary. However, only after understanding the role, it serves as a foundation, can I appreciate the activity in its full complexity and purpose. Today, we may think that classifying animals between vertebrates and invertebrates is evident. Similarly, for the respective animal kingdoms: birds, mammals, insects, etc. However, I now wonder: how many iterations did those classifications go through? And more importantly, how will a taxonomy of software look like?

My intuition tells me that just as biology followed a classification based on the evolution of life: software should as well follow a historical taxonomy. Thus, the map software pathologies would require to trace back how new technologies also engendered new bugs and complications. Neither should we forget, that software is a social activity and one cannot compile a lexicon of maladies excluding the human element.

5.2. Understanding of causality

Autopsies were a breakthrough for medicine: they enabled physicians to examine the causes of death, learn about anatomy, validate hypotheses, and settled discussions. Today, we perceive irrational that religious impositions delayed medical progress.

Currently, the practice of project dissections and post-mortems is discouraged either because of intellectual property or business motivations. Yet, if we would like to decrease the failure rate of software projects we need to understand the reasons that cause projects to fail. Embracing established academic practices like publishing papers and designing experiments.

Causality is the link between diseases and treatments.

5.3. A plethora of treatments

Often, it is not possible, to determine a priori whether a particular treatment will succeed or not. In medicine, new drugs go through rigorous tests using laboratories, animals, and control groups before being offered to the public. Even after being released, many drugs are closely guarded by doctors and institutions.

I’m not advocating for the same level of regulation in software, but I do admire the scientific rigor. Today, I am aware of how limited my experience is compared to the spectrum of possibilities faced in software projects. Especially, considering the ever-changing business and technological landscape. At this stage of my career, I’m particularly concerned about how expensive it has become to acquire more experience by testing new solutions. Today, as I’m often responsible for teams and projects: exploring new solutions seems increasingly costlier and riskier.

As a consultant, I hope that in the future, I’m able to access well-written, exhaustive case studies before proposing a solution or treatment to a customer.

6. Conclusion

I would like to make a welcoming invitation for the reader to share their impressions, and hopefully continue with the work proposed in this essay. For me, there are 3 clear directions to go from here:

1. Label the different circumstances faced in software projects.
2. Develop automated analysis to diagnose codebases.
3. Publicly document post-mortems and success stories.

As a personal opinion, I think that developers are among the smartest people in society, and there is an unspoken belief that we will figure things out. However, as time has passed, and I’ve gained experience, the only thing that I’ve certainly learned is to remain humble and always look to keep learning.

Finally, I feel optimistic as I now know what the tools and frameworks I need as a consultant. And more importantly, I’m excited to see what changes will be brought to the industry during the next 50 years, when I retire.

7. Acknowledgements

There is no great story without risk, and this time, I want to thank the people that connect with me through my writings. In particular, Brett from software.com, who reached out and shared the vision from an earlier post I wrote when I first became Tech Lead: understanding is one of the greatest gifts you can make, and a big encouragement to continue as a writer.

I also like to share appreciation with my clients, that tolerated my learning curve, posed interesting challenges, and allowed me to discover a new passion: helping people. And above all, to the people I love that accompanied me during my research and have done so for the last 2 years: my heart is with you.

Finally, I also want to thank you: the reader. Thank you for following me, for your claps, and hopefully for sharing this story. Most likely, this was not an easy read but if you want to read more from me, I recommend you my startup trilogy: The Viral Startup, A Post Covid World, The Laws of Growth. And feel free to share your thoughts, ideas and connect with me.

Warmest regards,
Santiago M.

For the last 3 years, I’ve worked in startups as a Software Engineer. My experience ranges from Full-Stack to DevOps to Tech Lead. In parallel, I’ve been studying Artificial Intelligence in all its flavors for the last decade.

So when I joined an AI startup, I was excited to finally merge my two passions. I was hoping to learn more about how AI models are built and trained. But what I was not expecting was also to learn more about Software Development.

The difference between DevOps and MLOps might be only 2 or 3 letters but in practice, is big. And while the components are also similar: CI & CD plus Continuous Training. The possibilities expand as you begin to enter the realm of randomness. And for software engineers:

Going from DevOps to MLOps is similar to what physicists experienced with the discovery of quantum mechanics

What is MLOps?

3 fundamental pillars comprise how MLOps is practiced in the enterprise. The first pillar is interdisciplinary collaboration. Just as DevOps aims to increase the delivery of value through automation and improved communication. Besides software and business people, in MLOps, two new profile personas are added to the mix: data scientists and domain experts.

The second pillar is effective data management, while also present in DevOps culture. The amount of work required to make sure the data is sufficient for the ML models makes it necessary for even the smallest startup to build and maintain fairly sophisticated data pipelines.

The third and final pillar is improving accuracy. DevOps aims to decrease errors preferably through small, gradual improvements. This translates in MLOps to automating the training and deployment of new models making them agile and adaptable to diverse business conditions.

How MLOps look in AWS?

In parallel to the Kardashev scale, I foresee 3 different levels of adoption of the MLOps paradigm in AWS. Phase-I uses the cloud to train, deploy and serve their AI models. Characteristics of this phase include using the UI to manage infrastructure, training models with open-source libraries and architectures, manual testing (if any), and a single instance to serve the models without auto-scaling.

Phase-II automates the ML pipeline. Infrastructure is managed as a code, with each stage serving a different ML model instance. Tests are automated, updates are deployed through Continous Delivery pipelines, and monitoring is enabled in production. Additionally, some cloud services are used to improve the efficiency of data scientists, and data is replicated for training and testing purposes.

In Phase-III the cloud is used to accelerate the training and deployment of ML models. There is a single pipeline to manage the full cycle of training and serving an ML model which requires little input from Software engineers. The focus turns on creating beneficial interactions between domain experts, business developers, data scientists, and software engineers. Multiple models co-exist in production A/B testing different hypotheses. Building and training new models are partially automatized using efficient data pipelines, Continuous Integration is done through non-deterministic tests, and instances are self-healing in production.

SageMaker the all-in-one AWS MLOps solution

To experience the value that SageMaker brings to an ML team, I’ll walk through the different stages of an ML cycle. At each stage, AWS SageMaker offers distinct alternatives to improve the productivity of Data Scientists and minimize the reliance on Software Engineers.

A new ML project usually starts with conversations between data scientists, domain experts, and business developers. This leads to a feasibility analysis that includes finding, and evaluating potential datasets.

1. Data Exploration

The process of feature engineering is composed of repetitive tasks that involve a certain amount of luck. While creativity and domain knowledge may boost a model’s accuracy, AWS offers a radically distinct alternative. Autopilot is a feature that creates hundreds of models with distinct algorithms and features for tabular data. The results are displayed in a leaderboard which provides valuable insights to start building a definitive model.

Sometimes the amount of data is not enough, and data augmentation can only take you to a certain extent. Amazon SageMaker Ground Truth automates data labeling for Computer Vision and Topic Classification problems. For Computer Vision, I’ve also found that Cord has an excellent tool for semantic segmentation and object tracking. The main benefit is that by using micro-models Cord can make label predictions using only a handful of labeled images.

Lastly, I want to talk about AWS QuickSight, a Business Intelligence tool that brings ML analysis capabilities. An easy-to-use interface is a perfect tool for business developers and domain experts. That is usually the place to center conversations between technical, and non-technical stakeholders.

2. Model Development

Naturally, AWS has all kinds of setups to minimize the amount of time spent preparing a workspace. But where it really makes a difference is by their ability to run training jobs in parallel. By offering their own implementations of open-source libraries and ML frameworks is possible to use one EC2 instance for developing and a fleet of GPU instances to train the models. This can drastically impact the time of development and costs. It also creates a smoother experience without dead time.

Also very impressive is their hyper-parameter tuning. Probably the most mysterious, time-consuming, and even dreadful activity is now also partially automatable. Using smart meta-heuristics SageMaker improves the accuracy of a model by testing distinct hyper-parameter sets.

3. Model Evaluation

Once the model is trained, is to decide if the model is good enough for production. It is at this stage, where the concept of Continous Integration from DevOps is useful. We could set up a pipeline using GitHub actions to test the performance of the new model. But there is a caveat, in Software changes to code are lightweight, but Deep Learning may require hundreds of gigabytes of data to train new models. Is GIT prepared to handle those files? Probably not!

DVC is an open-source version control system for machine learning projects. Using similar commands to GIT, DVC allows you to create a mapping between the experiments, the data used, and the results. Allowing you to return to a given state of the model at any point in time. DVC has a smooth integration with AWS S3 where the data can be hosted and retrieved easily.

Additionally, CML is a library to automate reporting the results of ML jobs to GitHub pull requests. The results can then be shared to Slack and keep the whole team engaged to fulfill one of the 3 pillars of MLOPS: interdisciplinary cooperation.

4. Deploying to Production

Probably my favorite part is the single line of code that exposes an endpoint from SageMaker to make predictions. Of course, there are alternatives, like serving a docker image from a lambda or hosting it from an ECS cluster. But empowering data scientists to control the deployment of their models opens the possibility to focus interactions between data scientists and engineers in building and designing the required data pipelines.

“In MLOps, the database is the like heart of the body that pumps information to the rest of the system.”

AWS also offers Elastic Inference to accelerate the speed of predictions. Augmented intelligence for the most common AI use-cases as speech recognition and machine translation. And even a Marketplace to use third-party models.

5. Performance Monitoring

Using auto-scale and provisioned infrastructure monitoring efforts can focus on ML tasks: detecting data drift, online learning & explainability.

Data drift: In theory, data is assumed to be part of a probabilistic distribution, and the knowledge of this distribution is used to make predictions. Statistical analysis can be performed to measure if the inputs given to the model are from the same distribution that the ones used during training. In practice, this can be done by setting pipelines where sub-samples of the predictions are tested to assure the accuracy levels are consistent.

Online Learning: in life, there are good problems to have, and there are bad ones. The same happens in AI, some problems are nice to solve, their characteristics include an auto-labeling nature. An example of this is recommendation systems: the effectiveness of the recommendation can be measured by click rates, bounce rate, time spent, and other relevant metrics after the interaction. This is opposed to problems in Machine Vision where a human is required to label the image to test the accuracy of the predictions. I warn you though, to build these pipelines and I will talk in more detail about it in the next section (continuous training).

Explainability: ML models were traditionally seen as black boxes. We knew the results of the model but not the why. The need to explain the main drivers that influence the results usually falls in one either two buckets: compliance and business. Compliance, part of Ethical AI is especially relevant in industries such as medical diagnosis, self-driving cars, and robotics, where a significant level of control and predictability of deep learning methods is mandatory [1]. While relatively a new area, AWS does have support for SHAP that uses game theory, and other few methods using SageMaker Clarifai.

6. Continuous Training

Whether you have a Software or Data Science background, this should be the most exciting for you. Because while the components may not be new (data pipelines) the possibilities are endless. With continuous training, one can truly start thinking of self-learning models. The idea is simple to collect the predictions performed by the model, label the data, and feed the new data to the ML training script. Once the training is over if the new model meets the Continous Integration requirements deploy it, preferably using the canary strategy. I understand that it might sound a bit too abstract, and I will share a sample architecture for your benefit.

Scenario: a personalized newsletter. Xolotl is a startup that delivers you the most relevant news based on your interests. Their continuous training makes it nimble to shift their recommendations based on your interests. And their crowdsourced data features the best stories for you to learn about new topics.

How it works: using word embeddings, Xolotl stores each of your interests in a 512-dimensional map. Each interest has a strength factor computed by the frequency, recency, and quality of your engagements (clicks & ratings). Every day, before sending a new issue Xolotl bots surface the web to find new stories, the meaning of the stories are embedded in the 512-dimensional map. Then, based on your interactions of the prior day the weight of your interests is updated and the stories that will be served to you are selected. Using GPT-3 API, Xolotl summarizes the parts that you find the most interesting, and the output is used as the content for the newsletter email. Occasionally, Xolotl may ask you to rate some of the recommendations to get a second data point besides click rate.

ML Design Patterns for Software Engineers

In the next section, I will introduce a potential roadmap for introducing MLOps in your organization. However, since most likely the Software Engineers, familiar with DevOps, will be the main drivers of change. I find it important to introduce at a gross level the patterns used by Data Scientists to perform their job. These patterns are explained thoroughly in the homologous book by Valliappa Lakshmanan. I highly recommend the book, whether you are new to the field, or an expert looking for a reference guide.

Data Representation

There are two common problems with features: too many of them or too few. When dealing with few features Data Scientists use Feature Cross to uncover hidden relationships between the features. While theoretically, the ML model could learn that for itself, explicitly feeding the relationships can accelerate training or compensate insufficient data.

On the opposite end: two many features, as when considering distinct words in Natural Language Processing problems, the solution is to reduce the dimensionality using embeddings. Interestingly, this is also used to visualize human language.

Problem Representation

When choosing an architecture for the model there is a clear taxonomy depending on the nature of the problem. However, this may not be sufficient to cover every single case, and data scientists occasionally chain distinct models to get the predictions. When the models are chained one after the other this is called the Cascade Pattern. And when the models are simultaneously used to arrive at a solution is called ensemble learning.

Model Training & Resilient Serving

In Computer Vision it is fairly common to use pre-trained architectures to minimize development time. This is done by replacing the top layers with the new labels and retraining them with your data in classification problems. This also happens with word embeddings in NLP, and most recently in tabular data with TabNet.

Another pattern keyed predictions may be used both during training and serving the model. The idea behind it is that instead of using the network to make predictions, data is stored in a dictionary based on the inputs that lead to the result. This is especially useful to augment the data in time series problems or making predictions in low computing power devices.

There is a second alternative for making predictions in IoT and mobile devices, which is called quantization. Basically, the weights of the network used to make the prediction are quantized reducing the number of bits used to make the prediction. An example of this is GPT-3 that has 4 different models each with different precision and consequently different accuracy.

Reproducibility

Given the probabilistic nature of ML, data scientists face diverse challenges to reproduce the training of their models. Training is usually done following Gradient Descent where the initial conditions can be aleatory, to overcome this a seed is used to store the state at the beginning of the training. Another aspect to consider before training is using the same set of ordered data.

Data scientists also have to deal with changes in the structure of the data, various alternatives are exposed to handle this in the bridge pattern. Finally, one has to consider reusing the code performed during data preprocessing, otherwise, duplication of code between development and production could contribute to errors. The latter is solved using a Feature Store connected to various data pipelines. For Data Engineers used to work with AirFlow one might want to check out KubeFlow a dedicated toolkit for Machine Learning which has components to integrate with EKS and SageMaker.

A final benefit of the Feature Store is that it can also be accessible to Domain Experts where they can review how the inputs are used to make predictions.

Responsible AI

The book saved the best part for last, covering 3 aspects of responsible AI: heuristics, explicability & fairness. Heuristics involves creating simple representations of how the model makes the predictions, this is especially important for decision-makers who may not be well-versed in Data Science terminology. In case the ML model intends to replace previously human-performed actions, the recommended approach is searching for a baseline to compare how effective the model is.

Concerning fairness, the problem is different. ML models are a representation of the data it was used to train it, but what happens if the model learns unfair biases against a particular gender or ethnicity? Then, data scientists require to use what-if and other frameworks to ensure the model is not prejudicial to particular sectors of society.

Implementing MLOps in your organization

Before you jump to implement an MLOps culture in your organization, a final warning: the guide assumes your startup is already using AI. If that’s not the case, I recommend the magnificent playbook by Andrew NG on how to introduce AI to transform an organization.

“Between now and 2030, AI will create an estimated $13 trillion of GDP growth.” [2]

The five-step plan to implement MLOps:

1. Map the data that every department requires to run its operations. Understand why do they use it and how it is used.
2. Detect the main pain points of the current ML training cycle. Make fast improvements to it using familiar DevOps solutions.
3. Prototype small experiments with SageMaker to understand the challenges the Data Science team will face in their adoption of MLOps.
4. Engage the stakeholders to buy in the investment that the transition will require.
5. Reshape the communication across the different teams to align the new, faster development with strategic business and commercial goals.

Beyond MLOps

As I engaged in this exercise it made me reflect on the implications of MLOps to the startup community. I intended to direct this article to Software Engineers. Yet, I will share some thoughts that arose during my research that I did not pursue forward to remain technical.

Tesla over Toyota: the fall of Agile?

As with every branch of Agile, DevOps was originated from Toyota’s lean manufacturing system. But this poses an interesting question in 2021:

Why did Toyota Prius fail where the Tesla Model 3 succeeded?

After all, Toyota Prius was an efficient, fast, gradual improvement in technology that was supposed to bridge the adoption of electric cars. Tesla, on the contrary, was an aggressive gamble tied to a grandiose vision fully defined from the start. In short, Tesla is everything that Agile is not supposed to be.

One common characteristic that is attributed to Tesla’s skyrocketing growth is vertical integration. From manufacturing to its commercial operations, Tesla reimagined how things should be. Perhaps, auto-makers have fallen lazy outsourcing critical operations. Could this have implications to open-source?

A second interesting characteristic is how the Lithium batteries came to be used. It was because of Tesla’s founders, Martin Eberhard and Marc Tarpenning, previous work in consumer electronics that they were aware of the potential of the technology. Most of their early work involved making the technology safe for driving. My hunch is that the success of AI will fall more on domain experts than anyone else. Hot startups will be lead by 40 year-olds having worked for a decade in boring industries, rather than 20-year old tech geniuses.

The third fact in Tesla’s rise is the superhero figure of Elon Musk. It’s funny, Agile depends on empowering common workers to reduce production flaws. At its most extreme Agile is practiced by self-organizing teams, where the figure of the leader is more of a servant. Elon is the opposite, a visible leader that commands his troops to fulfill his mission. Deeply technical, and with broad knowledge can talk in-depth about engineering or tweet virally. Yet, Tesla's rise has been filled with errors, delays, and critics. Again, the opposite of Agile.

I remember Sam Altman sharing is easier to build something hard in Silicon Valley than something easy. You will not get the attention, help, and resources to succeed. I write this as Tesla installs its first GigaPress and releases a subscription model for self-driving cars. I’m guessing there is no room for the conservative, practical, wise advice of Agile in Tesla’s HQ.

The implications of a consequential fall of Agile for MLOps based on Tesla’s gutsiness are to be seen. Remember that Lean Manufacturing was only discovered as Toyota’s shared the secrets with their providers to improve their productivity. But if Tesla is vertically integrated who will leak the secrets to the world and the Software Industry about its success?

As interdisciplinary collaboration grows, who will guide it?

In Graph Theory, there is a phenomenon that occurs on fully connected graphs: as the number of nodes increases the number of edges increases even faster.

As I start to look at the number of different persons involved in an MLOps team, I see no leadership. Worst, I see no one that can step up, have a full picture and make decisions. And that can lead to very unproductive teams.

Back in 1900 when the first MBA was granted that person was trained, educated to lead a business. But as I look to the landscape of the technological revolution I see an average MBA unqualified to make the technical decisions that can impair or advance a business.

Even the name feels antiquated: Masters? Definitively not! Business? Etymologically it’s derived from the word busy, and we all know that being busy is not the same as being helpful. How managers do we see with a schedule packed with meetings yet adding zero value. And finally administrators? We are no longer in a world with oil rigs or railways to manage. There is a long way from administering to innovation.

So who should then lead an MLOps team? Well, many things are still nice with the MBA. The exclusivity (selecting the better), its inter-disciplinary nature, the two-year full immersion. I propose the LEG: Leaders in Enterprise Growth.

With a background in Software Engineering and work experience in Machine Learning, the best, with the top interpersonal skills might be able to learn about business, marketing, finance and be prepared to guide inter-disciplinary teams for startups, multinationals, or as consultants.

The implications of AI in society and economics

Working in an AI startup has been mesmerizing. It’s impressive to learn how many of our skills are outdated, close to obsolete. No way can we compete against the computing power of an AI model. Nor are we as creative as we like to be. And with the amount of information a computer can retrieve, it’s just a matter of time for them to start making better decisions than us,

And it led to wonder, what will happen to society? Not in a dystopic view, but from a pragmatic economic perspective. And as I meditated on this, I remembered Marx and the fear surging from the thought that capital would enslave the proletariat. How hopeful, we wished for a revolution that would to a fair, just, equal, society.

I see something similar happening, the social class called the technocrats that will harness the power of AI to profit, and a second class that is relegated losing important middle-class occupations as physicians, lawyers, engineers, and more.

Naturally, there are also only two alternatives in this ever-wider AI world. The first utopic, embodying the principles that Marx thought to love, would create a universal income redistributing wealth based on taxes. As ideal as this may sound, is probably unlikely and we are rather instead shifting to a more stratified society. Where, at least for the short term, economic mobility will be scarce.

But I don’t want this to sound like a bad outcome. After all, obesity is a greater issue in developed countries than hunger is. And many of the changes Marx predicted failed to materialize (perhaps in part because of him). And workers, the proletariat were left with universal education, 40-hour workweeks, prolonged life, and many conveniences that technological progress brought.

In this regard, it will be interesting to see what institutional changes will AI bring. For example, schools and childcare were partly created to free time from workers and have an educated population. This standardization came at the expense of individualism, creativity, emotional development, and family bonds.

I foresee no shortage of human labor needs because it's easier to automate skills that depend on logic and no kinesthetic abilities. Usually, these skills are prevalent on the higher-end of the market. But how these will play in the development of new institutions, human development, and economic progress I will not share.

Conclusion

As mentioned, this piece was written for you: my fellow Software Engineer. I wanted to share what MLOps is, how to do it in AWS, and how to implement it in your organization. I took extra time to incentivize leadership in you to guide this transition and explained why ultimately AI is the direction technology is going. I want to conclude with the last remark, based on the assumption that AI will radically shift how society operates: please take an extra time to consider the impact that your work has on society as a whole. You have the liberty to choose projects that contribute positively to how you imagine our society should be shaped.

Bibliography

Additional consulted resources not linked during the article:

• Introducing MLOps: How to Scale Machine Learning in the Enterprise 1st Edition
• Learn Amazon SageMaker: A guide to building, training, and deploying machine learning models for developers and data scientists
• Engineering MLOps: Rapidly build, test, and manage production-ready machine learning life cycles at scale

Thank you to the team at ProsperIA that welcomed me, and provided valuable insights during the writing of the essay. Every month I publish a piece about startups and AI, follow me if you would like to read more. And if you enjoyed the story, or it provided value to you I will appreciate your claps.

Best,
Santiago M.

On August 25th, Exxon was removed from the Dow Jones industrial index after 92 years in it [1]. Exxon is one example of the innovator's dilemma, a company that once dominated its market and eventually was outpaced due to innovation. In particular, Exxon failed to adapt as its industry shifted towards renewable energies. And why wouldn’t they? Renewable energy was not going to give them the same thick margins as oil did, nor were they be willing to devalue their oil fields and massive investments in exploration. Yet, despite Exxon’s chunky margins, long tradition, and oligopoly control of the energy sector: new technologies were developed, challengers rose. And today, investors recognize what Exxon failed to foresee 10 years ago: The future of energy is green.

7 out of the top 8 causes of death in the U.S. are attributed to a chronic disease. [2]

Ask a cynical insurer: what should they do if a 63-aged patient is diagnosed with type-II diabetes? Nothing, they might say! Because there is only a minimal chance the patient develops serious consequences in the next 2 years and then, the person will be the government’s responsibility. [3] It may sound crude, but ask yourself, what incentives does a pharmaceutical company has to prevent you from developing type-II diabetes if it will be able to sell you drugs for the next 10 years!? Under these unfortunate circumstances, even physicians may face mixed incentives when treating you.

Economic history has taught us that ethics is not a good enough incentive for us to make the right decisions. Like many chronic diseases, type-II diabetes is preventable, by keeping a balanced diet and exercising regularly. However, to believe that major players in the healthcare and pharmaceutical industries will lead the transition to a preventive medicine is a logical fallacy.

The Rise of Preventive Healthcare

Picture this futuristic reality: you go out for a run and get thirsty. But since you just worked out, the healthy, organic, sugar-free beverages are at a discounted price! And the opposite is also true when craving a second sweet snack, since you already had one earlier, the price would be higher.

In a deeper analysis, healthcare and pharmaceutical are not the only industries benefiting from our unhealthy lifestyle. Fast-food chains and junk food producers also benefit from offering low-quality, addictive products at a steep price. Similarly, the cheap entertainment industry, like video streaming, and movies, has also profited from our sedentary lifestyle.

Ask yourself: what would happen if the price of Netflix increased after the first 10 hours of watching? Or if we experienced generalized “inflation” of prices based on the number of hours we watched YouTube?

While these ideas may sound radical, or idealistic. It shows which industries have an opportunity to rise and challenge the prevailing healthcare system:

The food and fitness industries have plenty of incentives to compete and innovate against established players if their market is expanded to include part of the current healthcare revenue!

The preventive healthcare industry is a future trillion-dollar example of the innovator's dilemma.

Taxes shift Perception

70% of Tesla’s 2020Q4 profit came from regulatory credits. [4] That’s on top of their more than 200,000 customers that benefited from tax credits. [5] Other countries have also imposed government measures to curve demand towards electric vehicles. China imposed restrictions, and European countries increased taxes for fossil fuel vehicles.

We might not like taxes, but they surely shape our decisions. In economics, an externality is a cost or benefit caused by a producer that is not financially incurred or received by that producer. For the energy industry, pollution is the main externality: oil companies heavily contribute to its production, yet everyone suffers its consequences.

Taxation is one of the few alternatives society has to combat these externalities. In healthcare, there are also externalities associated with chronic diseases. Because, while every tax-payer contributes to the rising costs of healthcare, it is the established companies in the industry that rip up the benefits.

Natural policies to curve the prevalence of chronic diseases include taxes, subsidies, restrictions, and prohibitions against products that contribute to causing these diseases. But entrepreneurs can be more creative and avoid depending on governmental actions to create the incentives to lead the transition to preventive healthcare.

Because there is a major difference between pollution and an unhealthy lifestyle. While the cost of pollution falls uniformly across the entire urban population; individuals can avoid some of the negative consequences caused by the present chronic disease-endemic like morbidity, disability, and loss of productivity.

It is in this scenario that the utopic reality from the first section becomes really interesting. Today, technology enables startups to implement dynamic pricing of their products based on the lifestyle choices of their customers.

Consider a healthy-food startup that partners with a fitness tracker provider. They set up a custom e-commerce shop where the activity of their users influences the price of their products!

The more they work out, the cheaper the healthy food gets. This creates a feed-forward positive cycle for their customers as they receive a triple benefit:

1. Enjoy healthy, high-quality, nutritious food.
2. A dopamine effect derived from exercising.
3. A discounted price.

This business model, financed by investors, is feasible as long they show potential to become the next Coca-Cola or McDonald’s. The startup will sacrifice profit for market share. Just like Tesla.

A Healthy FinTech Product

It is depressing to acknowledge the lack of innovation prevalent in the FinTech industry. So far, progress has focused on lower fees, higher speed, increased convenience, and crypto. But economics is much wider than that.

A study showed that when making food choices for the coming week, 74% of participants chose fruit, whereas when the food choice was for the current day, 70% chose chocolate [6].

Behavioural economics lies at the intersection with psychology. I hypothesize that we know what is good for ourselves but fail to act on those choices. For me, the ultimate fintech product is not about helping you score small wins but setting you up to make smarter decisions.

There is a concept called emotional hedge, in the sports-betting community. It’s used when you bet against your favourite team and set yourself up for a win-win scenario: if your team loses, you cash the money, and if not, your team wins! We can use the same concept to think about our health:

“Reward yourself for making healthy decisions, and be penalized for indulging in unhealthy pleasures.”

The product I envision is shaped like a credit card that implements a double-pricing system: the price you pay is different from the price you are charged. This difference in pricing would be used first to accrue savings, and secondly to subsidize healthier lifestyle choices. In that way, even if in the future you suffer from a chronic disease, at least you are financially prepared to face it.

What do you think would this idea be a viable alternative to incentivize people to live a healthier lifestyle? If so, I would love to hear your thoughts, as well as possible strategies to implement it.

Thank you for reading! I hope you enjoyed it and grant me some claps. To read more novel takes on startups, tech, and AI please give me follow. I write a new story every month.

Best,
Santiago M.

A self-imposed health-based taxation system was originally published in DataDrivenInvestor on Medium, where people are continuing the conversation by highlighting and responding to this story.

One image is worth a thousand words. Then, what if we could summarize a 1,000-word document in a single chart?

Charts and dashboards are used to model financial and numerical data. They are the preferred tool by analysts and investors to analyze, communicate, and strategize. But until a few years ago, using charts to visualize human language was not possible. Today, word embeddings make text documents faster to process, easier to aggregate, and more profitable to analyze.

Word embeddings are numerical representations of meaning, they are the result of a long tradition of Natural Language Processing research merged with recent advances in Deep Learnings. The representation of meaning with vectors enables data scientists to perform mathematical operations on words as they do with numbers. These include addition, subtraction, and comparison used in applications like Google’s search engine. My essay “What are word embeddings?” provides a good in-depth introduction to the topic.

Word embedding visualization starts by transforming text into vectors. Then, reducing its dimensionality to plot them. This article goes through each of the four steps to plot word embeddings. And provide sample code in TypeScript, using TensorflowJS, React, and recharts. I will exemplify this by approaching the topic classification problem. And end by sharing 7 startup ideas you can implement using the technology.

1. Tokenize your Dataset

It all starts with clean data. TensorflowJS universal sentence encoder transforms words, sentences, and even short paragraphs into vectors. But if your dataset does not conform to this format, you can use a RegEx to segment it:

// Split a text by words.
const wordTokenizer = (text:string) => text.match(/(\b[^ $]+\b)/g)

// Split it by sentences.
const sentenceTokenizer = (t:string) => t.match( /[^.!?]+[.!?]+/g )

// Split by paragraphs.
const paragraphTokenizer = (text:string) => text.split("\n")

2. Modelize it with TensorflowJS

Getting the word embeddings is easy. Start by loading both dependencies: Tensorflow & the sentence encoder model. Then, load the model; this will trigger a network request and take a couple of seconds. Finally, feed your tokenized dataset as an array of strings to the model, and transform the output into a JavaScript array.

import { load } from '@tensorflow-models/universal-sentence-encoder'import '@tensorflow/tfjs'

// Turn an array of texts into word-embeddings.
const vectorize = (text:string[]) => {
  const model = await load()
  const tensors = await model.embed(text)
  const embeddings = await tensor.array()
  return embeddings
}

3. Dimensionality Reduction

TensorflowJS outputs are 512-dimensional vectors; attempting to plot them is a chimeric endeavor. Fortunately, there are several dimensionality reduction techniques. A common, effective, and simple technique is Principal Component Analysis.

The npm module ml-pca is an excellent, lightweight choice to compress vectors in Typescript. To use it, we first create a new model based on the TensorflowJS output and use the model to predict 2-dimensional vectors.

import { PCA } from 'ml-pca'

// Reduce the dimensionality of word embeddings.
reduceDimensionality = (embeddings:number[][]) => {
  const pca = new PCA(embeddings)
  const matrix = pca.predict([embeddings], {nComponents:2})
  const reducedVectors = matrix.to2DArray()[0]
  return reducedVectors
}

4. Draw it with ReactJS

The last step is to plot it using ReactJS and recharts. We are going to start by mapping our tokenized dataset to the reduced vectors. Then import the scatter plot and add the corresponding colors, props, and style to turn it into a React component. Finally, choose the title: do not underestimate this step. Make sure to be clear about what you want to convey with the plot.

import { ScatterChart, Scatter, Tooltip } from 'recharts'

interface iData {name:string, x:number, y:number}
interface iEmbeddingsPlot {title:string, data:iData[]} 

const EmbeddingsPlot = ({title, data}:iEmbeddingsPlot) => <div>
  <h1> { title } </h1>
  <ScatterChart data={data}>
    <Tooltip/>
    <RechartScatter data={data} fill={colors[i]} />
  </ScatterChart>
</div>

Example: Document visualization by Topic

To illustrate the procedure, consider mapping an array of documents labeled by topic. These are some of the challenges we could face:

1. The document size is too large for the universal sentence encoder.
2. The number of topics is too big to assign each one a unique color in the graph.
3. We need to use the tool to classify unlabeled documents.

Shay Palachy wrote an outstanding 39-minute guide to overview the different technologies used to embed documents. For the above example, I opted for a more pragmatic approach, tokenizing texts by sentences and averaging them to find the final document’s embedding.

I used a similar procedure to draw each topic on the chart. Taking the average of the vector for every document corresponding to the same topic. Averaging is a great example of the value derived from applying mathematical operations to text data.

For the final step: predicting the topic of unlabeled documents. We can apply a Machine Learning algorithm like Support Vector Machine (SVM), train a neural network, or make a geographical query to find the closest topic in a 2D vectorial space. For more information on this subject, please refer to my tutorial: “How to create a text recommendation engine.”

BONUS: 7 Startup Ideas based on Word Embeddings

1. Kluster: Product review aggregation technology.
2. Quality Leads: automatically qualify leads based on their website’s language.
3. HR-Rec: recommendation engine to find the best prospects.
4. Touristy: group restaurants, hotels, and destinations based on experiences, not location.
5. ReadFlix: find the best content to read every night after work.
6. Healtty: dashboard to monitor corporate communication and organizational culture.
7. Matchy: join people based on ideas, not appearance.

Each of these ideas has the same concept: analyze textual data and create visualizations to facilitate decision-making. I’m happy to expand on any of these ideas or connect with me on LinkedIn to discuss an MVP. What do you think, which idea has the most upside?

“Whenever you see a successful business, someone once made a courageous decision.” ― Peter F. Drucker

Thank you for reading. I hope you enjoyed the story! For more content on Natural Language Processing, consider giving me a follow. I would love to hear your thought in the comments, and your claps will be very much appreciated. 🙏

How to Visualize Human Language was originally published in Analytics Vidhya on Medium, where people are continuing the conversation by highlighting and responding to this story.

Word embeddings are used in almost every commercial application that involves AI and human language. Some example applications include search engines, social media recommendation algorithms, language translation, speech recognition, market research, automated trading, and language generation.

Word embeddings are numerical representations of a word’s meaning. They are formed based on the assumption that meaning is contextual. That is, a word’s meaning is dependant on its neighbors:

For example, if the word “ice” usually appears next to “water”, one could infer that both words have a similar meaning.

Word embeddings are represented as mathematical vectors. This representation enables to perform standard mathematical operations in words, like addition and subtraction.

These operations have interesting applications in language, like finding synonyms, classifying documents, or recommending content. Additionally, 2-dimensional vectors can be plotted to produce a visual understanding of a document or a person’s language.

Application: Finding Synonyms

Finding synonyms is one of the simplest applications of word embeddings. A synonym is a word or phrase that means exactly or nearly the same as another word or phrase. And since word embeddings are numerical representations of a word’s meaning. To find the synonyms, we only need to find the vectors that are closest to the word.

The first step to finding synonyms is to select a distance metric to compare the closeness, or similarity, between two vectors. One of the most common metrics is the Euclidean distance that sums the squared difference across every vector’s dimension:

Another common distance metric used is the absolute value. In TypeScript, this is how to measure the absolute value distance:

/* Similarity: compute the absolute distance for two vectors */
const similarity = (a:number[], b: number[]) => {
  // Only compute the distance if the vectors have the same length.
  if(a.length !== b.length) return Infinity

  // Sum the absolute value difference across every dimension. 
  const delta = a.reduce((d, i, idx) => d + Math.abs(i — b[idx]), 0)

  // Return the distance as a proxy of a vector's similarity.
  return delta
}

similarity([3,4], [1,2]) // Returns 4
// Math.abs(3–1) + Math.abs(4–2) = 2+ 2 = 4

similarity([3,4], [1,6]) // Also returns 4
// Math.abs(3–1) + Math.abs(4–6) = 2 + 2 = 4

// 3D Vectors Distance
similarity([3,4,5], [4,6,8]) // Returns 6
// Math.abs(3–4) + Math.abs(4–6) + Math.abs(5–8) = 1 + 2 + 3 = 6

Interestingly, it is also possible to find antonyms using word embeddings. The only difference is finding the vectors that maximize the distance to a word.

Application: Topic Classification

Word embeddings and distance metrics are also useful to label documents by topic. The process starts with a labeled dataset of documents classified by topics. Then, transform the document’s content into word embeddings and average the position of each vector:

/* 
 * getCenter: find the average the position of a matrix of 
 * word embeddings to find the "center" of a document. 
 */
export const getCenter = (vectors: number[][]) => {
  const dimensions = vectors[0].length 
  const dimensionArr = [...Array(dimensions)]

  // Iterate through each dimension.
  const center = dimensionArr.map((_ , idx) => {

    // Sum the value of the dimension (idx) for each vector.
    const dimensionSum = vectors.reduce((d,i) => d + i[idx], 0)
    const dimensionAvg = dimensionSum/vectors.length

    // Return the average for each dimension.
    return dimensionAvg
  })

  // Return a vector with the same shape, and averaging values.
  return center

}

getCenter([[1,2], [3,4]]) // Returns [2,3]
// [(1+3)/2, (2+4)/2] = [4/2, 6/2] = [2,3]

getCenter([[2,3,3], [4,4,-1], [0,2,4]]) // Returns [2,3,1]
// [(2+4+0)/3, (3+4+2)/3, (3-1+4)/3] = [6/3, 9/3, 6/3] = [2,3,1]

We can think about the center of a document as the document’s embedding: it is the numerical representation of its content.

The next step is to derive the topic’s center. In a similar process, we find for every topic the average position of its documents. Finally, when we want to classify an unlabeled document, we can transform its content into a vectorial representation and use the distance metric to find the closest topic.

Reflection: How would you classify documents using exclusively unlabeled data? That is with unsupervised learning.

How to build word embeddings?

As previously mentioned, the idea behind word embeddings is that the meaning of a word is related to its context. In consequence, word embeddings are the result of mapping the words that are frequently close. The process consists of 3 steps:

1. Tokenization: split, classify and find unique words on a corpus.
2. Co-occurrence matrix: map the words that are close to each other.
3. Dimensionality Reduction: compress the size of the co-occurrence matrix.

1. Tokenization

There is a long tradition in Natural Language Processing (NLP) to separate words that includes stemming and lemmatization. To exemplify some of the difficulties we need to consider when splitting a text, consider the following[1]:

• Words that have the same meaning, including plurals and conjugated verbs.
• Pronouns, prepositions, and articles that appear frequently but contribute little additional meaning.
• Abbreviations and compound words like N.Y.C. or New York.
• Words with internal hyphens or apostrophes.
• Numbers, symbols, and punctuation signs like parenthesis or ellipsis.
• Orthographic errors.

To keep things simple, we are going to use a simple RegEx:

/* @function tokenize: split the words in a text or document. */
const tokenize (text:string) => text.match(/(\b[^ $]+\b)/g)

With the advent of Deep Learning, tokenization has partially lost relevance. Because theoretically, the best AI models handle irregularities by themselves, and building tokenizers is a slow, manual process. However, using tokenizers or other input transformations can dramatically reduce training speed and improve accuracy.

2. Co-occurrence Matrix

The co-occurrence matrix contains the frequency that two words occur next to each other. Building the matrix is a four-step process:

1. Define a set with all the unique words in the training dataset.
2. Create a square matrix where each row and column represents a word.
3. Count the occurrences of neighbor words based on an N-word window.
4. Insert the count to the corresponding cell in the matrix.

The co-occurrence matrix for the following 3 sentences look like this:

1. I like deep learning.

2. I enjoy flying.

3. I like NLP.

The above example worked with a 1-word window. To illustrate how this differs based on the window size, consider the following sentence:

// For an 3-word window:
const text = ‘I enjoy learning about Natural Language Proccesing.’

// enjoy & learning are adyacent.
// enjoy & about are adyacent.
// enjoy & Natural are adyacent.
// enjoy & Language are NOT adyacent.

There are slight variations as to how to build a co-occurrence matrix. These may include different ponderation based on the closeness, asymmetric windows, and using punctuation signs to determine dynamically sized windows. You can look at the complete code to build a co-occurrence matrix in the story’s repository.

3. Dimensionality Reduction

Theoretically, we have finished. We could use the rows of the co-occurrence matrix as word embeddings. But look again at the matrix, and notice how sparse it is. There are several issues associated with this:

• The vectors would take too much storage space.
• Training models on top of these vectors would be slow.
• Relations between words would be difficult to notice.

In that sense, the final step to build the word embeddings is to reduce the dimensions of the co-occurrence matrix. A small text corpus can have tens of thousands of unique words, but word embeddings tend to be smaller than 1,000 dimensions. For example, the universal sentence encoder vectors in TensorflowJS have a size of 512 dimensions.

There are sophisticated methods in deep learning to compress a matrix. But to remain simple, we are going to use the Principal Component Analysis (PCA) method:

import { PCA } from 'ml-pca'

const reduceDimensionality = (dimensions:number) => {
  const pca = new PCA(embeddings)

  const newSize = {nComponents:dimensions}  
  const reducedVectors = pca.predict(embeddings, newSize)

  return reducedVectors
}

If we reduce the word embedding dimensions to only 2, we can plot them and gain a visual understanding of the relationship between different words. This chart shows selected word embeddings for a 60K word corpus:

Application: Solving Analogies

Adding and subtracting word embedding has an interesting and surprising application: solving analogies. Traditionally, analogies are used to measure the reasoning and language skills of students. Today, they also evaluate the accuracy of word embeddings. Consider the following analogy:

King is to man, as queen is to ______.

The idea to solve this problem is to find the word that has the same distance to “queen” as “man” has to “king.” This is how it looks with vectors:

// Measure the distance between 2 vectors.
const distance = (a:number[], b:number[]) =>
    a.map((i, idx) => i - b[idx]
)

// Get the word embedding vectors for king, man and queen.
const king = wordEmbeddings['king']
const man = wordEmbeddings['man']
const queen = wordEmbeddings['queen']

// Get the distance between king & man:
const delta = distance(king, man)

// The solution is located at the same distance starting from queen.
const solutionLocation = distance(queen, delta)

// Find the words embeddings closest to the solution's location.
const analogy = findClosest(solutionLocation)

The word embeddings dictionary can be located in a database or loaded from a package. The second option is common in Python; a third option is computing the word embeddings in real-time from the browser using TensorflowJS. For computing the distance, you can use the similarity function we derived in the synonyms section. And if you want to learn how to find the closest vectors, you may be interested in reading the tutorial: “How to build a text recommendation engine.”

Visually, this is how solving the analogy looks:

Application: Detecting Biases

Unfortunately, we tend to make value judgments based on irrelevant or unfair attributes. AI can help in automatically analyzing, measuring, and reporting these biases. The word cloud below shows biases in jobs held based on gender. Jobs in healthcare tend to be more likely associated with women, while the opposite happens for engineering:

The process of detecting bias is also simple: find the word embeddings for the two concepts or groups you will compare. Then, select the term against which to measure a potential bias. Finally, compute the relative distance to each term: the greater the distance between two terms, the greater the bias.

// Arbitrary threshold to determine if there is a bias.
const biasThreshold = 2 

// Evaluate if there is a gener bias for a particular job.
const jobBiasDetection = (job:string) => {

  // Find the word embeddings of woman, man and the input.
  const woman = wordEmbeddings['woman']
  const man = wordEmbeddings['man']
  const jobEmbedding = wordEmbeddings[job]

  // Measure the distance of the job to both concepts.
  const distanceToWoman = distance(woman, jobEmbedding)
  const distanceToMan = distance(man, jobEmbedding)

  // Determine if the job is usually associated to men.
  if(distanceToMan/biasThreshold > distanceToWoman) return true

  // Determine if the job is usually associated to women.
  if(distanceToWoman/biasThreshold > distanceToMan) return true

  // There is no bias for the specified job and threshold.
  return false
}

Because AI models incorporate the biases that we hold as a society, it remains an unsolved problem, how to train word embeddings without biases. I invite you to reflect and share in the comments: what novel ideas do you have on how to train unbiased word embedding. As Kant said:

“Truth is a predicate of whole judgments, not of partial representations.”

Advanced Topic: Deep Learning

The process explained to build word embeddings is based on research during the 1980s, called Latent Semantic Analysis (LSA). But during the last decade, it has evolved to incorporate Neural Networks. After their successful implementation in Computer Vision, Neural Networks were quickly adopted by the Natural Language Processing academic community.

Four research papers have shaped how word embeddings are currently built:

1. Word2Vec: Efficient Estimation of Word Representations in Vector Space.
2. GloVe: Global Vectors for Word Representation.
3. ELMO: Deep contextualized word representations
4. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

For this introduction, I will only center on the first two.

Word2Vec: Word Embeddings trained from predictions

Word2Vec, a paper by Google scientists, is based on the premise that word embeddings are more accurate if trained to predict a word’s occurrence. The paper introduced two complementary neural network architectures to make predictions and subsequently derive word embeddings.

Using the same N-word window concept, Word2Vec proposed the Continuous Bag of Words (CBOW) architecture to predict a word based on its neighbors. Analogously, the Skip-gram architecture attempted to predict the neighbors based on a specific word.

The paper was also innovative because of the size of the dataset used to train the neural network. It also introduced a second dataset that became the standard to benchmark the accuracy of new models. It was mostly composed of analogies and included two sections: a syntactic and a semantic one.

GloVe: Finding hidden relationships between words

Only a year later (2014), developed by Stanford researchers. GloVe merged the best of both worlds: the subtle semantic relationships discovered by Latent Semantic Analysis and the syntactic accuracy of Word2Vec predictions.

GloVe was based on the powerful intuition that the true meaning of a word is derived from the difference to the expected probability of two words occurring next to each other. GloVe reduced the noise derived by commonly occurring words. This means that if 2 words are relatively uncommon but frequently appear next to each other, the impact of that particular relationship is more relevant to determine the value of their word embeddings.

GloVe schemed a clever method to identify, transform, and map those “high-value” relationships. As a consequence, the previously sparse co-occurrence matrix was transformed into a densely populated one. In a beautiful mathematical derivation, the cells of the new matrix were the result of the dot product between a row and a column of the original matrix.

By keeping the co-occurrence matrix small, it was possible to keep the training time short. More importantly, GloVe proved to be the only model at the time that benefited from an increase in the size of the training dataset (from 6 billion to 42 billion tokens).

Conclusion: The Future of Word Embeddings

I started this essay by mentioning that word embeddings permeate every application that involves AI & human language. But the future is even brighter: I see word embeddings in a similar place as to where mobile development was 10 years ago. The technology has endless opportunities in the next decade for entrepreneurs and software developers. I attribute it to 4 main reasons:

1. Ease of development: contrary to common wisdom, integrating AI, and in particular, word embeddings, to existing applications is easy. The field is sufficiently mature to use the technology without understanding the high-level math that goes underneath.
2. Fast adoption: computer vision has enjoyed the spotlights during the last decade. But its products and applications require specialized hardware, like cameras, and processors, that present challenges to the user’s privacy and speed of adoption. TensorflowJS makes it seamless to integrate AI-NLP software using a standard mobile phone or desktop.
3. Localization: a multilingual world creates interesting barriers to entry. Where multiple participants can leverage the same application of the technology to serve a distinct local market.
4. Growth: the field is under major transformation, bringing the attention of the brightest minds around the world. As recently as 2019, state-of-art models and breakthrough innovations continuously reshape the limits and applications of the technology. Examples of this include BERT’s research paper that drastically facilitates transfer learning and OpenAI’s text generation GPT-3 API.

I invite you to meditate on the following: human language is a mechanism of compression. It enables the fast and efficient transmission of complex ideas. But it also has its limits: when speaking, only 39 bits of information are transmitted. Compare this with the 480Mbps of information transmitted via USB 2.0. That is more than 1,000 times more!

It the end, word embeddings will increase productivity by augmenting the amount of information we can handle.

If you are interested in a didactic version of this content, I invite you to visit borgez.ml. It is an online interactive course where you can use these concepts to find synonyms and classify documents using TensorflowJS. You will train your word embeddings, validate your knowledge with a Quiz, and find charts with interesting insights.

Thank you for reading! I plan to write about Sentiment Analysis, Attention, and Transformers in the upcoming weeks. If you are interested, please consider giving me a follow and sharing this story. I wish you a great day, and your claps will be very much appreciated. 🙏

Sincerely,
Santiago M.

What Are Word Embeddings? was originally published in Geek Culture on Medium, where people are continuing the conversation by highlighting and responding to this story.

Recent advances in Deep Learning have made it possible to extend the power of Artificial Intelligence into web browsers. Introduced in 2018, TensorflowJS enables web developers to integrate state-of-the-art models in their applications. One of these models, the universal-sentence-encoder, uses advances in Natural Language Processing to transform words and sentences into mathematical vectors known as word embeddings.

Developed by seminal papers in deep learning, including Google’s word2vec and Stanford’s GloVe. Word embedding unlocks a variety of applications: mapping synonyms and antonyms, deciphering analogies, measuring biases. A text recommendation engine can be understood as a similarity optimization problem. In a vector space, this translates to finding the closest point for a set of initial conditions.

Outsourcing ML operations to the front-end with TensorflowJS has several benefits that can include: privacy, cost-effectiveness, speed, and ease. For example, Federate Learning is a distributed AI paradigm that shares the training and prediction tasks with deployed devices like mobile phones and IoT.

The following approach to build a text recommendation engine pairs recent advances in deep learning advances with geographical database queries to create a smooth user experience. It is particularly effective in data-scarce applications that require fast response times and cheap computation costs, like Social Networks.

1. Preparing our Dataset

Assuming an existing dataset of potential recommendations to serve already exists. The first step is to tokenize the texts into sentences. After that, the input is ready to be fed into TensorflowJS:

import { Tensor2D } from '@tensorflow/tfjs-node'
import { UniversalSentenceEncoder } from '@tensorflow-models/universal-sentence-encoder'

const embed = async(sentences: string[]):Promise<Tensor2D> => {
    const model = await use.load()
    const embeddings = await model.embed(sentences)
    return embeddings
}

In the preceding code, Tensorflow and the sentence encoder model are imported. Then, the model is loaded, and the sentences are embedded into vectors in a single line of code. The result is a 2D Tensor composed of 512-dimensional vectors, each representing a sentence.

Finding the distance between two 512-dimensional points is a slow and expensive task. However, there is a long tradition of databases working with spatial relationships. In particular, MongoDB has support to create 2D indexes and perform queries on coordinate pairs. To accomplish this, we will use the ml-PCA node module to reduce the dimensionality of the word embeddings. First, we train the model:

import { PCA, IPCAModel } from 'ml-pca'
import { promises as fs } from 'fs'

const trainPCA = async(embeddings:number[][]) => {
    const pca = new PCA(embeddings)
    await fs.writeFile(PCA_ROOT, JSON.stringify(pca))
}

And then we make the predictions:

import PCA_Model from PCA_ROOT

const reduceDimensions(embeddings:number[][]) => {
    const pca = PCA.load(PCA_Model as IPCAModel)
    const points = pca.predict([embeddings], {nComponents:2})
    return points
}

After saving the results to MongoDB, we have a clean dataset of recommendations ready to be served.

2. Handling the user’s input

To get the initial set of conditions from which to serve the recommendations, there are two options:

1. A history of choices made by the user.
2. Ask for text input from the user.

The first option is simple: first, we only need to find a function to average the location of the choices made by the user. Handling text input is not difficult either, but we would need to implement the same pipeline we used to process the recommendations. Regardless, we will be left with 2D coordinates from which to fetch the DB:

const makeRecommendations = async(x:number, y:number) => {
    const geoNear = {$geoNear: {near:[x,y], distanceField:'dist'}}
    const limit = { $limit: 100 }

    // Get the top 100 Recommendations from the DB collection.
    const recs = await Recs.aggregate([geoNear, limit]).toArray()
    return recs
}

We are now ready to serve the recommendations, but we can do one more thing to improve the accuracy of our suggestions: reuse the original 512-dimensions word embeddings!

3. Serving the Recommendations

Using word embeddings to rank the top 100 recommendation is the best of two worlds: the speed of 2D indexes; and the accuracy of 512-dimensional vectors. But to implement it, we need to define and compute a distance between two points. In this example, I’m going to use the absolute value distance to sort the recommendations:

// Absolute Value Distance Computation.
const similarity = (userInput:number[], recommendation:number[])=> {
    if (userInput.length !== recommendation.length) return Infinity

    // Iterate through each dimension and compute the difference.
    const delta = userInput.reduce((d, i, idx) => 
        d + Math.abs(i - recommendation[idx])
    , 0)

    return delta
}

const sortRecomendations = (userInput:number[], recs:iRec[]) => { 
    return recs.sort(({embeddings:a}, {embeddings:b}) => 
        similarity(userInput, a) > similarity(userInput, b) ? 1 : -1
    )
}

Then, use the absolute value distance to render sorted recommendations in a React application:

import { useEffect, useState } from 'react'

const Recommendations = (recs: iRecs[], userInput:number) => {
    const [ sorted, setSorted ] = useState<iRecs[]>()
    useEffect(() => { 
        const sortedRecs = sortRecomendations(userInput, recs)
        setSorted(sortedRecs)
   }, [recs])

    return sorted.map((r, k) => <Recommendation {...r} key={k} />
}

Finally, to close our Machine Learning pipeline, we need a system to evaluate and improve the accuracy of the results. I suggest sending anonymized events to a Product Intelligence tool, like Amplitude. These ideas can get you started to evaluate the model’s performance:

• The correlation between the closeness of a suggestion and its click rate.
• The further above a recommendation is presented, the more clicks it gets.
• Measure differences in efficiency exist based on the time and date.
• Segment users based on their searches and interests.

Conclusion: Potential Applications

Thank you for reading; as a bonus, I will present to you six ideas where you can apply this technology to build a product or business:

1. A Reading Recommendation Engine: similar to what Netflix does with movies; or to Google’s App. You could create an app or newsletter that serves reading recommendations based on the users’ interests.
2. Document Indexing: fields like Academia, Medicine, or Legal could benefit from the ability to classify and retrieve with speed relevant information based on non-trivial queries.
3. Searching Documentations: AWS docs comprise thousand of pages and hundreds of products. And AWS Kendra powers its search engine using Machine Learning. With this algorithm, you could create a competing, non-proprietary technology to search for answers in other documentation.
4. Understanding your users: if your website or app has a search component: you can guide your marketing efforts by understanding what your customers want based on what they search for.
5. Meaningful Visualization: trying to understand visually what happened after a meeting or a week’s worth of emails is tough. Imagine creating charts based on the word embeddings to summarize and measure non-numerical interactions.
6. Social Networks: build the next thriving, niche social network by recommending the best content, developing meaningful relationships among the users, and building smart automatized moderation tools.

I hope you enjoyed this story and find it useful. If you are interested in reading more about entrepreneurial AI in the front-end, please consider giving me a follow. As always your claps, and words of encouragement are highly appreciated.

How to Develop a Text Recommendation Engine was originally published in Geek Culture on Medium, where people are continuing the conversation by highlighting and responding to this story.