I came across a meme like the one below on the internet. I don’t remember the exact details, so I created my own version. Looking at a résumé like the one below, take a couple of minutes to think about whom you would hire as your next software engineer.

Maybe I’m just cursed with thinking differently, but I’d pick the candidate on the left without a second thought.

Let me tell you a longer story to explain why. Back in 2024, I asked a few senior software engineers how their companies were using AI for coding. What was their company’s strategy? None of the companies really had a strategy. It wasn’t that they didn’t care about AI. They just hadn’t spent much time figuring out how to make it work for them. Some were simply repeating what they’d heard from others: “Sure, AI can generate code fast, but I end up spending more time debugging it. I can still do a better job myself.” In 2025, the answers to these questions have changed. Some junior engineers had decided to give AI a try. Funny enough, it was usually the junior engineers who were the first to jump in. One told me, “I implemented dark mode and layout changes for the entire application in minutes, something that would normally take me a few days or weeks.” Now that got the senior engineers’ attention.

It’s almost 2027. If your résumé still doesn’t show how you use AI tools in your workflow, you might be in serious trouble soon. At some point, I started wondering why so many senior engineers were still ignoring AI coding tools back in 2025. These were smart, experienced people, so why weren’t they paying more attention? Then I remembered Harvard professor Clayton Christensen and his famous book, The Innovator’s Dilemma. Suddenly, it all made sense.

When disruptive innovations first emerge, they may address underserved markets that established companies have failed to serve. These innovations offer compelling benefits that attract customers whose needs have not been adequately met. Yet incumbents tend to dismiss them as low-end products serving niche markets that pose little threat to their existing customer base. They believe these innovations lack the critical capabilities and user experience that their mainstream customers demand. As a result, incumbents focus on defending their market position by raising the bar even higher, continuing to do what they already do, only better. They believe that the best way to compete is to excel at their current strategy rather than fundamentally overhaul their strategic priorities in favor of an approach they believe has major flaws. The fundamental problem in this strategy is that these improvements often deliver diminishing returns in customer value. Meanwhile, disruptive innovations continue to improve, gradually moving upmarket and challenging established players from below. Eventually, they reach a point where they meet the requirements of mainstream customers. Combined with the compelling advantages they already offer, this gives customers a powerful reason to switch. By then, incumbents have lost their competitive edge, and it is too late for them to catch up with the fundamental paradigm shift. Unable to adapt or defend their markets, they are displaced entirely.

Think about Borders bookstore. Its leaders weren’t necessarily incompetent. They were running a business model that had worked extremely well for years. In the early days, shopping for books online was not yet considered a compelling alternative to the in-store experience for most consumers. Why would Borders drop everything to chase a niche market that didn’t even offer the experience its existing customers loved? After all, its loyal customers wanted to physically touch and flip through books and take their purchases home the same day. Borders wasn’t blind to the online opportunity. It had tried selling books online, but it fell into the classic trap of The Innovator’s Dilemma. The perceived flaws of online book shopping led them to believe that they could still better serve their customers through traditional bookstores. Rather than treating the emerging competition as a matter of life or death — or even as the only priority that mattered — Borders’ top priority remained what it knew best: more locations, bigger stores, and a better in-store experience. Its own online efforts struggled, and Borders eventually handed its online sales over to Amazon in 2001. We all know how that story ended. Borders went bankrupt in 2011, while Amazon went on to become one of the world’s largest companies.

If your résumé still describes how software development was traditionally done, you may be in trouble. All these tasks remain important, but they can increasingly be performed by AI rather than by you. I often buy fresh fish directly from fishing boats. One of the fishermen, with no prior coding experience, has already built a ordering app with full account management functionality entirely through vibe coding. When AI asked him whether he wanted to integrate the app with GitHub, he simply said, “Why not?”

I believe most engineers should already recognize this shift. The real question is: What actions are you taking to adapt to these changes? For senior engineers, instead of just focusing on getting even better at what you already do, ask yourself: What will your role (your target market) look like in 2027, and what are the life-or-death challenges you need to address to serve that market well? Back in 2025, I told engineers that if their companies weren’t giving them opportunities to work with AI-centric workflows, they should at least start a side project on their own. Companies tend to assign senior engineers work they’re already good at, while experimental projects often end up in the hands of junior engineers. Sometimes, senior engineers are too cautious to take bold steps. If you’re still focused on what AI agents cannot do for you today, you may be missing the bigger picture. You’re still trying to improve what you already do using the same old playbook, blinding yourself to what should be your highest priority. Soon, you may find yourself getting better and better at things the world no longer needs, while losing your ability to compete in the world that is emerging. The key issue is not awareness, but knowing what actions to take.

But does the general public realize what’s happening? Four years ago, I wrote an article expressing my frustration with high schools that were banning the use of AI. To me, the message they were sending students was basically: “Congratulations! We’re preparing you for your future career by making sure you don’t learn how to work the way your future employer will expect you to.” How does that make sense?

I get it. Many schools, including colleges, are struggling with students using AI to complete assignments and exams. But I think they’re asking the wrong questions. Instead of obsessing over how to stop students from using AI, shouldn’t they be asking how to redesign their courses to prepare students for a world where AI is part of how we work? For professors who ban the use of AI, I have one simple question: Do you expect the world to evolve around your course, or should your course evolve with the world? Every profession will eventually face the same uncomfortable questions and rethink their role. And if you don’t know the answers, or aren’t even trying to find them, you may be in trouble.

You might be thinking about how to do your current job better and faster with AI. But here’s another uncomfortable question: If AI can help you do your job better, isn’t it also possible that AI could eventually do your job entirely? Isn’t this a version of the dilemma described in The Innovator’s Dilemma? You’re focused on becoming a better version of yourself in your existing role and market, rather than asking what entirely new opportunities AI could create, what new markets might emerge, and what unique value you could bring to them. It’s like Borders believing it could offer customers a better bookstore experience than Amazon, without reconciling itself to the fact that the underlying paradigm would eventually kill its business.

That leads to the first question: “What can AI do for you?” For high schools and colleges, instead of worrying about whether AI can complete assignments quickly and easily, shouldn’t we be asking whether students can use AI to produce something better than either the student or AI could achieve alone? Personally, I’m not looking for ways AI can help me do things faster. That is boring. I’m looking for things I couldn’t do before.

Consider this for a moment. If I were a high school student today, what kinds of assignments would I want? I’d want collaborative projects where students work alongside AI. For example, I’d love to work with AI to explore how gene editing could potentially help treat cancer, a complex subject that most students at that age would struggle to understand on their own. I often wondered how much of the basic theory of relativity I could have figured out on my own if I had known what Einstein knew about electromagnetism. Back then, there were no teachers who could help me explore such questions. Fortunately, students today have AI to help them.

But there are some hard truths behind these ideas. Could AI eventually become the primary means of delivering knowledge, with teachers taking on more of a facilitator role? Remember JFK’s famous quote: “Ask not what your country can do for you — ask what you can do for your country.” A couple of years ago, I encouraged engineers to ask themselves: “What can AI do for you?” But if someone asked me that same question today, I’d say it’s outdated. Instead, I’d turn it around and say: “Ask not what AI can do for you — ask what you can do for AI.” The fundamental difference is between asking what AI can do to help you perform your job and asking what you can do to help AI accomplish its job better.

Look at what’s happened in mathematics over just the past few months. In September 2026, AI produced a proposed solution to the Navier–Stokes problem, one of the famous Millennium Prize Problems, each carrying a $1 million prize. This was a mathematical challenge that had puzzled some of the world’s best mathematicians for nearly 90 years. By October, AI researchers had released hundreds of manuscripts claiming solutions or major advances on more than 300 open mathematical problems. This is far more disruptive than AI taking over copywriting jobs. The demand for copywriting may still exist, but fewer humans will be needed to do the work. A breakthrough in mathematics is fundamentally different: it could eliminate a huge portion of the work altogether because the underlying problems have already been solved.

We always thought AI would help us wash the dishes and mop the floors, but instead, it may takes over our professional careers first. This irony reflects Moravec’s paradox, which suggests that high-level reasoning tasks, often considered difficult for humans, can be relatively easy for machines, while basic sensorimotor tasks, such as perception, movement, and physical coordination, can be extraordinarily difficult to automate.

When ChatGPT’s reasoning capabilities were first introduced, mathematicians began to recognize that a new era of AI-assisted mathematical reasoning was approaching. Many were fascinated by its progress, expecting AI to assist with proofs and discoveries rather than replace human mathematical insight. Today, however, as AI systems demonstrate increasingly advanced mathematical capabilities, reactions appear to have shifted toward anxiety, frustration, and even outrage, with some viewing these advances as unnecessary or uncalled for. Some mathematicians have even raised concerns about the pace of AI development, questioning whether progress should be slowed.

Once AI can solve mathematical problems more effectively than humans, the question will no longer be how AI can help mathematicians do their jobs better. Proving certain theorems may no longer require human involvement at all. Instead, the more important question becomes: How can humans remain in the loop and work alongside AI to achieve something neither could accomplish alone?

As AI models become increasingly specialized in specific fields, and if computers prove to be inherently better at reasoning than humans, these questions may no longer be limited to mathematicians. For instance, will AI merely help scientists identify promising targets for cancer treatment, or will it eventually formulate research strategies — generating hypotheses, designing experiments, and determining the next steps — while human scientists take on supporting roles? This picture may create another major paradigm shift in how we define the nature of our jobs.

Most mathematicians have not yet reached the fifth stage of grief: acceptance. And I think it’s fair to say that perhaps they shouldn’t. But I believe that eventually, we may begin to recognize which mathematical problems AI can already solve and turn our attention to entirely new questions and possibilities we haven’t even imagined exploring. Or perhaps the more important question is no longer how AI can help mathematicians prove theorems, but how humans can inject their intuition, insights, and creativity into AI’s problem-solving process, enabling us to discover something neither could achieve alone.

When I encouraged people in 2025 to ask, “What can AI do for you?” I didn’t have all the answers for engineers. I still don’t, especially as AI capabilities evolve with every major release. But one thing is certain: this transformation is happening. I also don’t know exactly what I can do for AI, but we should be willing to explore the possibilities. Some companies argue that AI makes them efficient enough to lay off half their workforce. I find this reasoning appalling. If AI enables us to produce so much more, why settle for the same output with fewer people? Is AI really to blame, or were these companies simply inefficient all along?

This transition will be tough, and we shouldn’t sugarcoat it. But major disruptions also create opportunities for those willing to explore new frontiers. As some readers know, I’m still working on my AI book series. With just two major sections left in one of the books, I’ve discovered that AI can now accomplish in 15 minutes what once took me hours, sometimes producing better results with the right guidance.

A friend recently asked what value I could still bring to the process. I even wondered whether AI agents could write most of a chapter. If so, how could my contribution make the combined result greater than the sum of its parts? This reinforced my belief that “What can you do for AI?” will soon become an increasingly important question.

Writing continues to inspire new ideas and insights, and I still believe I can contribute something valuable by working with AI. That’s why I’ll keep writing the book.

Success depends on building a high-performance distributed system that scales efficiently and almost linearly with the increasing number of GPUs. To succeed, several factors are critical: how effectively GPUs are interconnected, and advancements in interconnect technologies that offer low latency and high bandwidth between GPUs and compute nodes. Just as GPUs are designed with domain-specific optimizations, networks must also follow an AI-first design. The design starts with the algorithms and workloads, rather than prioritizing the network itself first. Co-design between hardware and software engineers is crucial. Additionally, total cost of ownership (TCO) is a key consideration, encompassing power consumption, cooling requirements, and potential needs for new infrastructure, such as rack and data center redesigns. A full software ecosystem is equally crucial, ensuring seamless integration of machine learning frameworks, libraries, and tools to optimize both performance and scalability.

Disclaimer

Both the B300 and Vera Rubin platforms have been announced. As I am in the process of writing a book on deep learning, this article requires some updates and revisions. Fortunately, much of the core content remains valid, so I have decided to release it. In addition, there are several network topologies discussed here that are quite interesting but will not be covered in the book.

For readers who may be interested, the book has taken a significant amount of effort to write, and I have spent considerable time on topics that are truly worth studying. If you would like to be notified when it is released, please connect with me on LinkedIn.

NVLink & NVLink Switch

NVLink is a high-speed interconnect technology designed for direct GPU-to-GPU communication. It offers significantly higher communication bandwidth and lower latency compared to the traditional PCIe bus. The Blackwell architecture uses the fifth generation of NVLink, with 18 NVLink ports (connections) per GPU and each port providing 50 GB/s of bandwidth in both directions, resulting in a total bidirectional bandwidth of 1.8 TB/s (0.9 TBs for each direction). This enables communication speeds over 14 times faster than PCIe Gen5.

NVLink uses point-to-point communication, which can lead to unnecessary separate connections. For example, in tensor parallelism, a model layer’s computation is split across multiple GPUs. Once a GPU completes its portion, it must send the results to all other GPUs. Because of NVLink’s point-to-point nature, a separate connection must be established with each GPU. In an 8-GPU system, this means a GPU must send the same data seven times, leading to inefficient bandwidth usage and increased latency.

To address the point-to-point communication issue, switches are added between GPUs to eliminate unnecessary connections and provide dedicated communication pathways that are not impacted by other connections. Built on top of NVLink, NVLink Switches offer this solution, though at a higher system cost. These switches use a crossbar architecture to create a non-blocking topology, allowing any GPU to communicate with another without interference from other connections. This ensures that each GPU can connect to any other at full bandwidth, enabling simultaneous communication among all GPUs, regardless of the number of GPUs in the system. For instance, in tensor parallelism, each GPU processes its part of the computation and sends its partial result to a switch just once. The switch then broadcasts the result to all other GPUs in the system. In this process, Nvidia’s Collective Communications Library (NCCL) is used to optimize communication between GPUs, facilitating efficient data exchange and synchronization during distributed computing tasks.

Nvidia GB200 NVL72

The GB200 NVL72 is a reference design intended to accelerate AI and HPC workloads. It allows third parties, such as system integrators and data center providers, to build large-scale AI infrastructure based on Nvidia’s validated blueprint.

The GB200 NVL72 features a rack configuration with 18 compute trays (18 dual GB200 compute nodes), interconnected via 9 NVLink Switch Trays. Each rack in this system operates with a 120kW power load, significantly higher than the typical ~10kW per rack in a conventional data center. While this configuration offers the advantage of a minimal footprint, the substantial increase in power density may slow down the adoption of this form factor due to infrastructure upgrade requirements. The high power density also requires liquid cooling for the system, which helps reduce overall power consumption by lowering the demand on fans.

Each compute tray is equipped with two GB200 Grace Blackwell Superchips, each containing two B200 GPUs and one Grace CPU, all connected through a 900GB/s NVLink chip-to-chip interconnect. NVL72 includes a total of 36 Grace CPUs and 72 Blackwell GPUs.

A single Grace CPU consists of 72 Arm Neoverse V2 cores and supports up to 480GB of LPDDR5X memory. The Blackwell-architecture GPU, featuring 208 billion transistors, are manufactured using TSMC’s custom-built 4NP process, a specialized 4-nanometer manufacturing technology. Due to constraints on die size, each GPU is composed of two dies connected by a 10 TB/s chip-to-chip link, creating a unified GPU with 192GB HBM3e memory and 8TB/s memory bandwidth.

Each GPU features 18 NVLink ports, and with 4 GPUs per node, this results in a total of 72 NVLink ports per node. To interconnect all the nodes in the GB200 NVL72 system, 9 NVLink Switch trays are required. Each tray houses 2 NVLink chips and offers a total of 144 NVLink ports, with each port capable of delivering 100 GB/s of bandwidth. Each B200 GPU will have a single connection with an NVLink Switch.

NVLink is an inter-chip communication technology that functions and operates much like a memory system. In a 72-GPU rack configuration, each GPU can directly access the memory of any other GPU as if it were accessing its own local memory. The NVLink Switch interconnects each GPU pair at 1.8 TB/s and supports full all-to-all communication across the system. With near-linear scalability, the rack functions like a single GPU, delivering 1.4 exaflops of AI performance and providing 30TB of high-speed memory accessible to all its components. The 4th generation NVLink Switch is the first rack-level switch chip capable of supporting up to 576 fully connected GPUs in a non-blocking communication fabric.

To host 576 GPUs, it may require using two racks for each set of 72 GPUs instead of just one. This setup enables the integration of additional switches for efficient GPU-to-GPU communication.

The diagram below illustrates a potential configuration in which each rack contains 9 NVLink switch trays and 18 compute trays, with each compute tray hosting a single GB200 Superchip rather than the dual-chip setup.

This arrangement frees up half of the ports in each NVLink Leaf Switch tray, allowing them to connect to the spine switches for inter-rack communication. NVLink Switch Tray 1 ports in the leaf switches connect to Plane 1 NVLink switches, Tray 2 ports connect to Plane 2, and so on. As a result, moving data between GPUs across different racks will require only two hops through NVLink switches. This configuration provides a higher CPU-to-GPU core ratio and potentially more memory per GPU, which can improve the performance of recommendation models that rely on massive embedding tables and require more CPU-driven operations.

The diagram below presents an alternative configuration, where each compute tray continues to host two GB200 Superchips, but the number of compute trays is reduced from 18 to 9 per rack.

While Google’s TPUs use fiber optics to enhance inter-node communication, especially for higher data transfer rates over long distances, Nvidia employs a selective approach in the GB200 NVL72 system. This system densely interconnects nodes via a copper cable cartridge for leaf switch connections, which is significantly more power-efficient. However, for longer-distance communication, such as between leaf and spine switches, Nvidia relies on optical connections to maintain high bandwidth and low latency across the system.

The calculus of variations optimizes functionals, functions that take other functions as input. One application is finding the path of shortest travel time. Here, a path is a function y = f(x), and the travel time T is expressed in terms of f. The optimal path, denoted f*, is the function that minimizes T among all possible paths.

Let’s investigate the fastest sailing route from NYC to Morocco, taking into account the impact of trade winds and ocean currents. For a tiny displacement ds along the path, the travel time dt is equal to

Combining it with

To find the total travel tim, T, we integrate the infinitesimal time element dt along the entire path from x₁ (NYC) to x₂ (Morocco), giving us:

The function y = f(x) represents a sailing path, and T(f) calculates the corresponding travel time. This makes T a functional — a function that takes another function as input and returns a scalar value (the travel time). Our goal is to find the stationary function f *(x) of this functional, representing the path with the shortest travel time. i.e. we find y = f*(x) such that a functional I

is stationary with the boundary condition y₁= f(x₁) and y₂ = f(x₂).

Announcement

Although I haven’t published many articles in 2023 and 2024, these years have been incredibly busy for me. The rapid advancements in AI have inspired me to write a book on Generative AI, a project that has been both challenging and time-consuming. This article is part of a chapter on optimization. However, I cannot include every detail in the book, so I am providing additional information in this article.

While the book isn’t finished yet, you can stay updated on its progress by following me on Medium or connecting with me on LinkedIn. I’ll share an announcement there as soon as it’s ready!

Euler-Lagrange Equation

Suppose y(x) makes I stationary with the boundary conditions satisfied. We introduce a function η(x). It is a completely random function except its boundary points are zero.

We define another function ȳ that can represent any function except it has to fulfill the same boundary conditions of y(x).

If ε = 0, both ȳ and y(x) refer to the same stationary function.

The first and second derivatives of ȳ w.r.t. ε are

We will need that later. Next, we want to find

R.H.S. is an integral over x. Therefore I is a function of ε as everything else that depends on x will be integrated over. We just turn this problem into optimizing a single variable ε, instead of a function. Therefore, we can apply the regular calculus to find the optimal ȳ by setting the derivative of I w.r.t. ε to zero. With ε to be zero, ȳ and y(x) refer to the same stationary function. Therefore, we set the derivative of I to be zero with ε=0.

i.e.

By applying integration by parts in the second term with η(x₁) = η(x₂) = 0, we get

Now combining the result, we get

Since η is a relatively random function, for the integration to be zero, we need

This is the condition for y = f(x) in making F stationary. And it is called the Euler-Lagrange equation.

Shortest Path

Let’s use the sailing route problem to demonstrate the Euler-Lagrange equation. We’ll simplify the problem and assume that all external factors, such as ocean currents and winds, do not affect the path, and that the sailing boat moves at a constant speed. In an optimization problem, any constant factor can be ignored. Therefore, the functional III simplifies to:

To find the shortest path, we apply the Euler-Lagrange equation.

The shortest distance between two points is a straight line.

In machine learning (ML), kernel machines utilize linear classifiers, such as SVMs, to solve nonlinear problems. They achieve this by employing a kernel function, which implicitly maps data into a higher-dimensional space, making it linearly separable.

Let X be a non-empty set, and a kernel k is defined as:

Under this definition, a kernel function k: X x X → ℝ measures the similarity between data points x and x’. For k to be a valid kernel, it must satisfy the condition k(x, x’) = ⟨φ(x), φ(x’)⟩, where φ is a mapping from the input space X to a feature space H, and ⟨・ ,・⟩ represents the inner product in the Hilbert space H. This means the kernel value is equivalent to the inner product of the data points’ representations in the feature space.

However, unlike many other ML algorithms that require explicit transformation of input data into feature vectors using a feature map, kernel methods directly compute similarity using a user-specified kernel function. Even though this kernel function implicitly operates in a high-dimensional, potentially infinite-dimensional feature space, the kernel trick allows us to avoid explicitly computing the feature representations. This technique helps us bypass the significant computational expense associated with such calculations. This implicit operation allows kernel methods to efficiently analyze complex data relationships and facilitate effective pattern recognition and analysis.

Function Space

In functional analysis, f (・) denotes a function itself, while f(x) represents the specific value that function takes at the input x.

Consider a function f that takes an input x, where x is a vector defined as (x₁, x₂).

The function f (・) is an element of a function space mapping ℝ² to ℝ. In this example, we can represent f (・) as ℝ³ within this function space.

Given the function

The linear functional f can be represented as:

In certain function spaces, such as those used in machine learning with kernel methods, functions can be written as linear combinations of the features 𝜙(𝑥). The evaluation of f at x, 𝑓(𝑥), is an inner product in the feature space:

For example, f evaluated at x = (-1, 4) equals

Feature Space in Infinite Dimension

This concept naturally extends to infinite-dimensional feature spaces. For instance, we can expand the exponential function eˣ using its Taylor series representation.

and e³ becomes

Here, the feature space is in an infinite dimension.

Evaluation Functional

Functionals map vectors to scalars. An evaluation functional L at x ∈ X is defined as the map:

This definition broadens our perspective as L is a functional parameterized by x that takes a function f as input and produces a real number ℝ.

Linear functional is a functional that preserves the operations of vector addition and scalar multiplication, as illustrated below:

Dual Space

The dual space of a vector space V is the set of all linear functionals on V.

Consider the familiar 3D space, denoted as ℝ³, where each point is represented by a vector with three components (x, y, z). This is our original vector space V. A linear functional on ℝ³ is a linear map that takes a 3D vector as input and produces a single real number as output. For example, a linear functional could be defined as: f(x, y, z) = 2x+3y + 4z. The dual space of ℝ³, denoted as V*, is the set of all possible linear functionals on ℝ³. Each functional in this dual space can be uniquely represented by a 3D vector. For example, the functional f(x, y, z) = 2x+3y + 4z can be represented by the vector (2, 3, 4).

Riesz Representation Theorem

The Riesz Representation Theorem states that every continuous linear functional L on a Hilbert space can be represented as an inner product with a fixed element in F. Formally, for any continuous linear functional 𝐿 on a Hilbert space F ’,

The Riesz Representation Theorem bridges the abstract world of linear functionals with the more familiar concept of inner products, allowing us to understand and manipulate linear functionals using the geometric intuition of inner products. In essence, the Riesz Representation Theorem states that for every continuous linear functional on a Hilbert space H, there exists a unique element within that same space H that fully characterizes the functional. This means the linear functional can be thought of as an inner product with a fixed “representative” vector. This establishes a fundamental connection between linear functionals and inner products in Hilbert spaces. In the context of Reproducing Kernel Hilbert Spaces (RKHS), this concept is extended to show that the evaluation of functions can be computed using inner products with kernel functions.

Reproducing Kernel

Combining the evaluation functional with the Riesz Representation Theorem, we obtain:

Any function f ∈ F evaluated at x can be represented as an inner product between f and a unique a unique functional kₓ ∈ F, where kₓ can be written as:

Let’s replace x with y, we get

Since kₓ ∈ F, let’s replace f ∈ F with kₓ and we get:

This essentially brings us back to the idea of a kernel.

In summary, the evaluation functional Lₓ can be written with k( ⋅ , x ) as

It can be computed as the inner product of f with a kernel function in the Hilbert space 𝐻. The function 𝑘(⋅, 𝑥) or kₓ is called the reproducing kernel.

Hence,

Here is the notation used, which all referring to the same object:

Reproducing Kernel Hilbert Spaces (RKHS)

A Reproducing Kernel Hilbert Space (RKHS) is a Hilbert space of functions in which evaluation at each point is a continuous linear functional. This means that for any function f in the space and any point 𝑥, there exists a kernel function k such that 𝑓(𝑥) = ⟨𝑓,𝑘(⋅, 𝑥)⟩. The kernel function 𝑘(⋅,𝑥), known as the reproducing kernel, acts as a bridge between the function and the feature space, allowing us to evaluate the function at specific points.

In essence, f(x) can be represented as the inner product of f with the reproducing kernel function k(⋅, x). The function k(⋅, x) is called the reproducing kernel. Conceptually, the evaluation function can be computed by the inner product of f and the feature space representation of x.

Example: Approximating a Function using RKHS

Suppose we have a set of data points and we want to approximate a function that fits these points using the Gaussian kernel.

The final steps involve solving for α and using it to compute f(x).

So, the approximate value of the function at x = 1.5 is f(1.5) ≈ 3.05.

Both methods start with a general pre-defined architectural design. As suggested by the term ‘search space,’ NAS-like methods search this space, using educated guesses or trained policies, to find optimal models. In contrast, RegNets focus on identifying design guidelines that exhibit strong performance and generalization abilities across various contexts, including different hardware platforms and tasks. These guidelines are refined through an incremental manual process, narrowing the design space into an optimal design space that informs how to design model parameters. Sampling this space to locate the optimal model is more of a bonus than the primary objective.

RegNets introduce a design space called AnyNetX, as depicted below. Each network within this space features a standard architecture consisting of a stem, a body, and a head for generating final predictions, as illustrated in the left-most part of the diagram. By sampling models with specific variations of model parameters and evaluating their performance, RegNets uncover patterns and trends. Specifically, they select guidelines that not only lead to better performance but also suggest design simplifications when no performance loss is observed.

The body contains multiple processing stages, and each stage (i) consists of dᵢ blocks. The total number of blocks across all stages is denoted by d. All blocks utilize a 1×1 convolution to extract features across channels, followed by a group convolution, and finally, another 1×1 convolution. But for Block 1, a stride-2 group convolution is employed to reduce spatial resolution.

AnyNetX offers control over four essential parameters:

1. Model Depth (d).
2. Channel Width (w₁, w₂, w₃, and w₄): Determines the width of channels for each stage.
3. Bottleneck Ratio (bᵢ): This modifies the channel width in the initial 1×1 convolution of every block.
4. Group Width (gᵢ): Specifies the group convolution width in each block.

By analyzing the performance of sampled models across various configurations, the study proposes general guidelines to narrow down the design space further:

• Use a single bottleneck ratio (bᵢ = b) value and a single group width (gᵢ = g) value for all stages.
• Increase the channel width w₁, w₂, w₃, and w₄ progressively.

• Gradually increase the number of layers (dᵢ) within each stage of the neural network, except for the final stage.

Another significant finding reveals a linear correlation between the channel width uⱼ and the model depth index (denoted by j), with w representing the hyperparameters.

But the computed uⱼ is a real number, and we need an integer for the channel width. Hence, we will apply a two-step process to quantize the value into an integer. First, we find sⱼ with the following progressive growth formula, with w representing the hyperparameters again.

Next, we apply the quantization function below to determine the channel width wⱼ for index j.

These guidelines reduce the design space into one referred to as RegNetX, a name derived from the paper’s objective to create a design space filled with simple, regular models.

After further refinement of the RegNetX design space, it reveals interesting findings that diverge from current network design practices. Remarkably, it challenges the widespread belief that greater model depth leads to better performance, revealing instead that the optimal models in RegNetX typically consist of around 20 blocks. Furthermore, best models in RegNetX achieve optimal performance by eliminating the bottleneck entirely through a bottleneck ratio of 1.0, a strategy that has been embraced by some other studies. Furthermore, the traditional practice doubles the channel width from one stage to another. However, the RegNetX study favors a value close to 2.5 for peak performance.

The final analysis focuses on RegNetX models that have been enhanced with the Squeeze-and-Excitation (SE) operation. This enhancement results in the creation of a new design space known as RegNetY. The integration of the SE operation, which adaptively recalibrates channel-wise feature responses by explicitly modeling interdependencies between channels, significantly boosts the representational power of the network. Consequently, the RegNetY design space demonstrates notable performance gains compared to the original RegNetX models.

Announcement

Generative AI has been an exciting area of development in recent years, sparking my interest in writing a book on the topic. RegNets are fascinating models that I’ve invested considerable time in studying. However, they didn’t quite fit the book’s main focus. However, I believe they hold valuable insights that are worth exploring. Instead of discarding them, I’ve decided to share these insights through a series of articles on Medium.

While the book isn’t finished yet, you can follow its progress and be among the first to know when it’s complete by connecting with me on Medium or LinkedIn. Stay tuned for the official launch announcement!

Motivation

Kernel functions offer a principled approach to detecting nonlinear relationships using linear algorithms. This is achieved by implicitly mapping the original input data into a feature space where the data might become linearly separable. For example, with the mapping 𝜙(x) = x₁² + x₂², the red and blue dots in the diagram above are now linearly separable in this newly transformed feature space, as opposed to their original input space.

However, in real-world problems, the feature space is often high-dimensional to untangle complex data, and the mapping ϕ is usually unknown. However, we can use machine learning or deep learning techniques to model ϕ. Once modeled, a linear algorithm such as Support Vector Machines (SVM) can be applied to the feature space. In the example below, a 2D input can be transformed into a pre-assumed 9D feature space using a third-degree polynomial. Then, inferences are made with the linear model θ.

However, in dealing with high-dimensional feature space, the transformation complexity becomes significant. In particular, 𝜙 can map to and often map to infinite-dimensional feature space. Computing an infinite number of components is infeasible. What makes kernel methods stand out is that it can handle high-dimensional and infinite features efficiently with the kernel trick.

Kernel Trick

Let’s demonstrate the method the hard way first, before introducing the kernel trick to make it more efficient. In kernel methods, we often utilize the similarity between an input data point and the training dataset to make predictions. A common approach involves calculating the inner product of two points after they have been transformed into a higher-dimensional feature space.

In the example below, φ(x) represents a mapping function that transforms an input vector x = (x₁, x₂) into a feature space defined by a second-degree polynomial:

Given two data points x = (1, 2) and y = (3, 4), the inner product in the basis (1, x1​, x2​) is 12.

Let’s compute the inner product in the feature space. The result is 144.

In this process, the feature mapping 𝜙 transforms the input vectors into a higher-dimensional space. Subsequently, the inner product reduces the result back to ℝ. Since the final result is a scalar in ℝ, we might question whether these transformations are necessary. In other words, can we find an equivalent function that directly computes the result without explicitly calculating 𝜙(x) and 𝜙(y)?

In our example, we observe that the final result (144) is simply the square of the inner product of the original input vectors (12² = 144). It can be shown through algebraic expansion that these two calculations are indeed equivalent.

In this example, the kernel function k is defined as the square of the inner product between the augmented vectors (1, x₁, x₂) and (1, y₁, y₂).

This is the essence of the kernel trick: the inner product of two vectors in a high-dimensional (or even infinite-dimensional) feature space can be computed efficiently using a kernel function in the original input space, bypassing the need for explicit feature mapping. This efficiency becomes particularly significant in infinite-dimensional feature spaces, where direct computation of the inner product would be impossible, but the kernel trick remains feasible.

Can this kernel trick be generalized? Indeed, for any ϕ, there exists a corresponding kernel k. For example, to raise an input space to a higher-degree polynomial, the corresponding kernel is simply:

However, our primary objective is not to explicitly find the feature space, which is generally unknown. Instead, we focus on the result, k(x, y), not the feature mapping 𝜙(x). In fact, we often design kernels directly without explicitly constructing or even knowing the corresponding feature space.

In fact, it is more intuitive to think in terms of kernels than feature spaces. For instance, if the similarity between neighboring points can be modeled with a Gaussian distribution, a Gaussian kernel would be a natural choice. As show later, this kernel corresponds to a feature space with infinite dimensions, theoretically allowing for arbitrarily complex decision boundaries.

If we can find a kernel that effectively solves machine learning problems, that is sufficient for our purposes. Understanding the feature space is a bonus, but it is not necessary for kernel design.

Announcement

Generative AI has been an exciting area of development in recent years, sparking my interest in writing a book on the topic. This project has been both immensely challenging and rewarding. During the writing process, I discovered many captivating subjects that didn’t quite fit the book’s main focus but still held valuable insights. These topics may be a bit raw due to time constraints, but I believe they’re worth exploring. Instead of discarding them, I’ve decided to share them through a series of articles on Medium.

While the book isn’t finished yet, you can follow its progress and be among the first to know when it’s complete by connecting with me on Medium or LinkedIn. Stay tuned for the official launch announcement!

Kernel

Let's give an official definition for a kernel.

Definition: Let X be a non-empty set.

For every valid kernel function, there exists at least one corresponding feature space where the kernel can be represented as an inner product of the feature space.

Radial Basis Function Kernel (RBF kernel)

Just for better understanding, let’s examine the feature spaces associated with the RBF kerne, which is defined as:

The RBF kernel implicitly maps data into an infinite-dimensional feature space. To analyze the RBF kernel, we can utilize the Taylor expansion of the exponential function. The Taylor expansions of exp(x) and exp(x x’) are:

Using the last Taylor expansion, the RBF kernel becomes

The RBF kernel implicitly maps data into an infinite-dimensional feature space 𝜙 below. This infinite-dimensional nature allows the RBF kernel to model highly complex decision boundaries.

Positive Definite Functions

Not all functions can be valid kernels. They must be positive semidefinite.

Definition

A function k: X × X → R is positive semidefinite if, for any vectors a in in ℝⁿ and xᵢ in X

it fulfills the condition:

In simpler terms, this means the function always yields a value greater than or equal to zero, regardless of the input values. If it equals zero only when 𝑎ᵢ =0 or aⱼ= 0 , the function is called positive definite.

Kernels are Positive Semidefinite

A continuous or finite-domain function k can be represented as the inner product of feature mappings in a Hilbert space H if and only if k is a positive semi-definite function.

As shown below, if a function k can be represented as an inner product in a feature space, then k is positive definite.

Furthermore, since the inner product is symmetric, any kernel function derived from the inner product must also be symmetric (k(x, y) = k(y, x)). This requirement for symmetry, combined with the requirement for positive definiteness, establishes a theoretical framework for constructing and validating new kernel functions.

Kernel Matrix

Or equivalently, for a function to be a valid kernel, the associated kernel matrix must be semipositive definite.

Definition: A n×n matrix K is positive semidefinite if

Mercer’s Theorem

We show that kernel functions are positive semidefinite. Mercer’s theorem states the converse is also true: any positive semidefinite function can be represented as the inner product of vectors in a corresponding feature space.

Ridge Regression

In this section, we will use Ridge Regression to demonstrate how to perform inference with the kernel function. In linear regression, Ridge Regression minimizes the mean squared error of predictions while also incorporating an 𝐿2 regularization term to prevent overfitting. The prediction for a data point x is equal to θᵀx.

To find the optimal θ, we set the gradient of the objective function with respect to θ to zero and solve for θ. This yields the following closed-form solution:

We can express θ in the form of Xᵀ α, where α is:

G is called the Gram matrix, which is computed as XXᵀ and is therefore symmetric.

Given a new input x, a prediction can be made using f(x) = θᵀx = ⟨θ, x ⟩.

This solution can be recognized as a linear combination of the inner products between the training data points and the new example 𝑥.

The coefficients α are derived from the Gram matrix G. This implies that we do not directly model the parameters θ. Instead, we utilize the training data itself to make predictions. This approach is a defining characteristic of non-parametric machine learning models.

Finally, we can make one more improvement: instead of using x as the input, we use its feature space representation ϕ(x). This applies to both the inner product and the Gram matrix. Here, the matrix G is also referred to as the Kernel Matrix.

The new prediction will be a linear combination of the inner products between the features of the training data points and the features of the new example x.

Kernel Design

We can combine kernels to design new ones. For example, adding two valid kernels together or multiplying a kernel by a positive scalar results in another valid kernel. Furthermore, multiplying two valid kernels together also forms a new valid kernel.

Here is a list of common kernels.

Let’s prove that the Gaussian kernel is a valid kernel by expressing it as the product of three exponential functions.

According to the rule

the first two terms forms a kernel

The third term is also a valid kernel. To see this, we note that the inner term is itself a kernel, as it is formed by the product of two functions, f(x) and f(y).

Using the Taylor series expansion, we can express the exponential function within the Gaussian kernel k₁ as an infinite sum of polynomial terms. Since both the sum and product of valid kernels are also valid kernels, we can conclude that the third term is indeed a valid kernel.

By the product rule again,

we can prove Gaussian kernel is a kernel.

Feature spaces, where each data point is represented as a vector, are crucial for algorithms to identify patterns and make predictions. Metric spaces equip these feature spaces with a distance function, allowing for measuring similarity between data points. Hilbert spaces, a specific type of metric space with additional properties like completeness and an inner product, are particularly useful for kernel-based methods such as SVM.

Their role is less prominent in deep learning, where complex non-linear models often rely more on empirical experimentation and intuition-driven design. However, this article seeks to demystify the mathematical terminology prevalent in research papers that utilize mathematical spaces, making the concepts more accessible and less intimidating for readers.

Prerequisite

This article assumes familiarity with the basic terms outlined in the previous article titled “The Study of Mathematical Spaces (Machine Learning)”.

Metric Space

While the concept of distance between two points is familiar, the power of metric spaces lies in extending this concept to less intuitive mathematical objects, allowing us to apply the properties and operations of metric spaces in broader contexts.

To refresh our understanding, a metric space, denoted ⟨M, d⟩, is essentially a set M along with a well-defined distance function (metric) d.

The metric d is a function that formalizes the notion of distance between elements of the set M. L1 and L2 distances are some popular examples for the metric in ML.

To qualify as a metric function, three simple conditions must be satisfied.

• Non-negativity: 𝑑(𝑥,𝑦)≥0.
• Symmetry: The distance is the same in both directions.
• Triangle Inequality: The direct route is the shortest.

This level of abstraction offers considerable flexibility in defining a metric. Consequently, it grants substantial freedom to design metrics tailored to specific mathematical objects and problems. The goal is to find a metric that proves beneficial in solving your particular challenges. A notable example is the Wasserstein metric, which quantifies the similarity between two probability distributions and has gained popularity as a loss function in Generative Adversarial Networks (GANs).

Complete Metric Spaces

Completeness is a fundamental property of metric spaces. It means that every Cauchy sequence (a sequence where the terms get arbitrarily close together as they progress) converges to a limit that exists within the space. Formally, a metric space M is complete if every Cauchy sequence in M converges to a point that is also in M. This implies that M doesn’t have any “holes” or “missing points.”

Completeness is crucial in various mathematical fields because it provides a foundation for analyzing limits and continuity. In machine learning, optimization algorithms like gradient descent rely on convergence properties. Theoretically, if the underlying space is complete, we can be confident that the algorithm will converge to a solution within the space, ensuring the results are meaningful and reliable.

Examples

Consider the sequence

All elements in this sequence are real numbers inside the interval (0,1). But the limit, 1, is not in the interval (0, 1), and therefore the metric space ⟨X, d⟩ for X = (1, 0) with the usual metric 𝑑(𝑥, 𝑦) = |𝑥−𝑦| is not complete. To address this problem, we can expand X to [0, 1], making the metric space complete.

To assess the completeness of a space, we first need to define a metric. The set of rational numbers equipped with the standard Euclidean metric is not complete. This is because a sequence of rational numbers can converge to an irrational number, which is not part of the rational number set. For example, even we can construct a Cauchy sequence of rational numbers ℚ that converges to the square root of 2. √2 is missing from ℚ.

Vector Spaces (Linear Space)

Vector spaces not only offer a natural structure to organize data points and features but also underpin linear models, such as linear regression and logistic regression, in many ML algorithms. Additionally, they facilitate the linear transformations central to model optimization. Their impact extends to kernel methods and natural language processing, where they enable the capture of semantic relationships through word embeddings.

Vector spaces, also called linear spaces, provides a structure for manipulating objects called vectors. These vectors can be added together and multiplied (scaled) by numbers called scalars. Mathematically, a vector space over a field 𝐹 (commonly the real numbers ℝ or the complex numbers ℂ) is a set 𝑉 equipped with two operations: vector addition and scalar multiplication. The set 𝑉 and the operations must satisfy the following axioms, which hold for all elements 𝑢, 𝑣, w in V and all scalars 𝑎, 𝑏 in F:

These operations are closed under V. The addition and the multiplication results remain under the same vector space. Vectors in a general vector space are abstract mathematical objects. They are not limited to the two-dimensional arrows used to represent physical quantities like force or velocity. The concept of a vector is much broader and can encompass many different types of objects.

Example

Polynomials, like 3x + 4x², x + 2x² + 0.5x⁵, and 5x², form a vector space V over the real numbers. We can add polynomials together and multiply them by real numbers (scalars), and the result will always be another polynomial within the same vector space V. For example, if we add (3x + 4x²) and (x + 2x² + 0.5x⁵), the result is 4x + 6x² + 0.5x⁵, which is also a polynomial in V. Therefore, functions can form a vector space, and each element in 𝑉 is simply a function.

Normed Space & Banach Space

A normed space is a vector space equipped with a norm, which measures the length or magnitude of its vectors. A Banach space is a complete normed space.

Norm

The norm of a vector v is written as ‖v‖. It is a function that assigns to each vector in a vector space V a non-negative real number (including zero), representing its length, magnitude, or size. The norm generalizes the familiar notion of length in Euclidean space to more abstract vector spaces.

A normed vector space is a vector space defined over the field of real or complex numbers (𝔽 ∈ {ℝ, ℂ}). The norm maps each vector to a scalar in the field 𝔽.

Any definition of ‖ · ‖ is called a norm if it satisfies the following properties: positive definiteness, absolute homogeneity, and the triangle inequality.

These three rules also imply a norm is greater or equal to 0. This is the reason why we omit it from the norm axioms.

Here are the common definitions for norms in ML.

Normed Space (Normed Vector Space / Normed Linear Space)

⟨ X, ‖ · ‖ ⟩ forms a normed space. Here are two examples in the normed space: the first one uses an absolute value norm while the second one uses a p-norm.

It’s important to note that a normed space is defined by its norm. The behavior of the space, including concepts like convergence and distance, depends on the specific norm being used. Different norms can lead to different conclusions about the properties of the space.

Every normed space inherently possesses a metric induced by its norm. This allows norms to quantify both lengths of individual vectors and distances between vectors within the space. A normed space can simply use the norm to define its metric.

Normed spaces are commonly used to quantify the similarity between data points, assess the quality of model predictions, and design optimization algorithms. For example, the L2 norm is frequently used to measure the error between predicted and true values in regression tasks, while the L1 norm promotes sparsity in feature selection.

Banach Spaces

A Banach space is a complete normed space (X, ‖ · ‖).

Hilbert Space

Hilbert spaces, as a specific type of complete normed vector space with additional structure, offer several advantages for ML applications:

Inner Product: The defining feature of a Hilbert space is its inner product, which generalizes the dot product in Euclidean space. This inner product allows for the calculation of angles and projections, enabling techniques such as:

• Principal Component Analysis (PCA): Finding directions of maximal variance in data.
• Support Vector Machines (SVM): Finding optimal hyperplanes for classification.

Orthogonality: Inner products facilitate the concept of orthogonality, which is crucial for:

• Fourier Transforms: Decomposing signals into orthogonal frequency components.
• Function Approximation: Representing functions as linear combinations of orthogonal basis functions.

Completeness: Hilbert spaces are complete, meaning that every Cauchy sequence converges within the space. This is important for:

• Optimization: Guaranteeing the existence of solutions to optimization problems.
• Function Spaces: Many function spaces used in ML (e.g., spaces of square-integrable functions) are naturally Hilbert spaces.

Duality: The Riesz representation theorem establishes a powerful duality between Hilbert spaces and their continuous dual spaces (spaces of linear functionals). This is exploited in:

• Kernel Methods: Optimizing the computation of similarity between two points projected in high-dimensional space using properties of the inner product space.
• Reproducing Kernel Hilbert Spaces (RKHS): A type of Hilbert space that provides a framework for kernel-based learning algorithms.

Inner product

The dot product is an example of an inner product, but the concept of an inner product is more abstract. An inner product is a binary operation on a vector space that satisfies the following axioms for all vectors x, y, z in the space and all scalars a:

Inner Product Space & Hilbert Space

A Hilbert space is endowed with an inner product and is complete. If the space is not complete, it is referred to as an inner product space.

The inner product establishes a framework for abstract geometric concepts such as length, distance, and angle within a vector space.

In every inner product space, we can derive a norm induced by the inner product:

This norm allows us to define the distance between two vectors:

Therefore, a Hilbert space is also a Banach space and a normed space. Furthermore, we can determine the angle between two vectors x and y:

These definitions of norm, distance, and angle establish the geometric structure of inner product spaces.

Both the set of real numbers (ℝ) and the set of 𝑛-dimensional real vectors (ℝⁿ) can form Hilbert spaces. In these cases, the inner product is typically defined as the dot product.

For Hilbert spaces consisting of functions, the inner product is often defined as an integral involving the product of the functions.

Cauchy–Schwarz inequality

The Cauchy-Schwarz inequality states that for all vectors x and y in an inner product space:

Equality holds if and only if x and y are linearly dependent, meaning that one vector is a scalar multiple of the other.

Orthogonality

When the inner product of two vectors x and y is zero, they are said to be orthogonal, denoted as x ⊥ y. Orthogonal vectors are linearly independent, meaning neither can be expressed as a scalar multiple of the other.

Similarly, functions f and g are orthogonal if their inner product is zero. A classic example is the pair of trigonometric functions sin(x) and cos(x), which are orthogonal over the interval [0, 2π]. In the broader context of vector spaces, any vector can be uniquely represented as a linear combination of vectors from an orthonormal basis. Notably, in function spaces, these basis vectors can themselves be functions.

Subsets of a vector space X are considered independent if no element in one subset can be expressed as a linear combination of elements from the other subset.

Use of Hilbert Space

One notable application of Hilbert spaces in ML is the Reproducing Kernel Hilbert Space (RKHS). RKHS is a specialized type of Hilbert space that plays a crucial role in kernel-based learning algorithms, leveraging the inner product structure for efficient computation and representation of functions.

The key advantage of RKHS is its ability to implicitly map data points into high-dimensional feature spaces, where similarities between them can be more easily found. Explicitly performing this mapping would be computationally expensive or even impossible in cases of infinite-dimensional feature spaces. However, the defining property of an RKHS, the reproducing property, combined with the kernel trick, enables efficient computation of these similarities using the kernel function without explicitly representing the data in the high-dimensional space. This is possible because the ultimate goal is often to compute scalar similarity values in the real numbers (ℝ), rather than the high-dimensional features themselves.

𝓁ᵖ Space

The concept of infinity can be daunting in computer science. However, in ML, where algorithms often project data into high-dimensional spaces to uncover hidden patterns, it’s natural to consider extending this to infinite dimensions. While infinite-dimensional spaces might seem impractical, the key isn’t the number of dimensions, but whether we can efficiently compute relevant properties within those spaces. For example, if the sum of an infinite series or the limit of a sequence within an infinite-dimensional space is finite and computable, then infinite dimensionality can be a powerful tool for enhancing the expressiveness of ML models. This capability opens the door to studying and designing infinite-dimensional sequences.

Elements in a mathematical space can be sequences. The ℓᵖ space is a set of infinite sequences where the p-th power of the absolute values of the terms is summable and finite. It is defined as the set of all infinite sequences x = (x₁, x₂, x₃, …) for which the infinite series formed by the p-th power of the absolute values of the elements converges. More formally, it is defined as:

It is a normed space when endowed with the p-norm.

To guarantee completeness, every Cauchy sequence in a space must converge to a limit within that space. In an ℓᵖ space, this condition is satisfied as the infinite series ∑|xᵢ|ᵖ converges for each sequence in the space. Therefore, ℓᵖ spaces qualify as Banach spaces, which are complete normed vector spaces.

Example Sequence in ℓ²

Consider the following sequence as an example of an element in the ℓ² space:

Inner Product for ℓ² Space

The inner product in ℓ² can be defined as follows.

Given the property below, the definition indeed represents an inner product in ℓ² space, fulfilling all necessary criteria. This will also serve as a good example to demonstrate how to prove that a function is an inner product.

ℓ² pace & Hilbert Space

The ℓ² space, often described as the space of square-summable sequences, is a prominent example of a Hilbert space, a type of complete inner product space. In ML, the ℓ² space, which is the space of square-summable sequences, is often used in the context of regularization techniques to prevent overfitting in models.

∞-norm

When p = ∞, the corresponding norm is called the infinity norm (or supremum norm). Since there are infinitely many components in the sequence, finding a particular element as the maximum is not possible, so the infinity norm is computed using the supremum, which is the smallest value that is greater than or equal to all elements in the sequence.

Lᵖ Space

Lᵖ spaces are a specific type of function space defined for functions from a measurable space (often a subset of ℝⁿ) to ℝ (or ℂ). A measurable space extends the concept of size and volume to more abstract spaces. Infinite-dimensional spaces, like function spaces, allow us to model data as functions rather than just finite points. This concept is familiar to engineering students, who use Fourier transforms to represent data. In this context, the input is represented with periodic sinusoidal functions.

The functions in Lᵖ spaces are required to be Lebesgue integrable, meaning the integral of the absolute value of the function raised to the power of p is finite. This ensures the completeness of the Lᵖ space.

It is a normed vector space when endowed with the p-norm.

The Gaussian function is in L² space. The Gaussian function 𝑓(𝑥) defined on the real line ℝ and specifically considered over an interval, say [−𝑎, 𝑎], is given by:

Its square is integrable. The integral is finite for any finite a, because the exponential function rapidly decays to zero as 𝑥 moves away from zero, making the area under the curve finite. Indeed, the Gaussian function is in L² even over the entire real line (from -∞ to ∞).

Inner Product for Lᵖ Space

The inner product can be defined as:

creating a Hilbert space.

Continuous Function Space

The continuous function space C[a, b] is defined as the set of all real-valued functions that are continuous on the closed interval [a, b]. In mathematical notation, this is expressed as:

Let’s explore some visual examples of continuous functions within this space:

Norm

The corresponding p-norm is

And the natural norm is defined as

The metric d(u, v) below is a supremum between u(x) and v(x). (C, d) forms a complete metric space.

Inner Product

forms an inner product space.

Examples of Hilbert Spaces

Here is an recap of three possible Hilbert spaces, each defined by its unique set of elements and corresponding inner product: Euclidean space equipped with the dot product, ℓ² space characterized by the ℓ² norm, and the space of continuous functions on the interval [0,1].

Isomorphism

Isomorphism is a structure-preserving mapping between two spaces. In the context of vector spaces, this preservation specifically refers to linearity, meaning that the operations of vector addition and scalar multiplication are maintained under the mapping.

In a Banach space, an isomorphism preserves the norm following the mapping. That is,

This is also referred to as an isometric isomorphism, where the distance is preserved before and after the mapping.

For Hilbert spaces X and Y, and the mapping M: X → Y, isomorphism means that M is a bijective (one-to-one and onto) linear map that preserves the inner product, i.e.,

Recap

Here is an overview of the relationships among some of the spaces we’ve studied. The arrows indicate the “is a” relationship. A normed vector space is inherently a metric space because the norm induces a metric, which quantifies the distance between vectors. Without a defined topology, a vector space by itself does not automatically meet the criteria required to be considered a topological space, as it lacks the concepts of closeness, connectivity or distance.

Mathematical spaces exhibit a hierarchical structure, reminiscent of object-oriented design. At the top of this hierarchy reside the most abstract spaces, such as topological spaces, which establish fundamental concepts like continuity and convergence. As we move down the hierarchy, spaces become more specialized, acquiring additional structures and properties that tailor them for specific applications.

Fields

Let’s begin by discussing fields, a type of mathematical space. Both real numbers and complex numbers form fields. While basic, this concept provides a quick overview and introduces some of the associated terms.

A field ⟨F, +, ·⟩ consists of a set F equipped with two binary operations (operations that take two elements to produce a third element):

• Addition (+)
• Multiplication (·)

The set of real numbers ℝ forms a field which contains all real numbers. The operations of addition (+) and multiplication (·) are defined in the usual way for real numbers. However, to qualify as a field, these operations on the elements of the field are required to adhere to the following axioms (basic rules):

For all a, b, c ∈ F:

1. Closure under addition and multiplication: a + b ∈ F, a · b ∈ F.

2. Associativity of addition and multiplication:

• (a + b) + c = a + (b + c), (a · b) · c = a · (b · c)

3. Commutativity of addition and multiplication:

• a + b = b + a, a · b = b · a

4. Existence of additive and multiplicative identities:

• There exists an element 0 ∈ F such that a + 0 = a = 0 + a for all a ∈ F.
• There exists an element 1 ∈ F (where 0 ≠ 1) such that a · 1 = a = 1 · a for all a ∈ F.

5. Existence of additive and multiplicative inverses:

• For every a ∈ F, there exists an element -a ∈ F such that a + (-a) = 0 = (-a) + a.
• For every a ∈ F where a ≠ 0, there exists an element a⁻¹ ∈ F such that a · a⁻¹ = 1 = a⁻¹ · a.

7. Distributivity of multiplication over addition:

• a · (b + c) = (a · b) + (a · c)

A field is closed under addition, and multiplication. This means that performing these operations on elements within the field will always produce another element within the same field.

In quantum mechanics, the complex field ℂ (consisting of complex numbers) is essential for describing quantum phenomena. Rational numbers form the rational field ℚ, while integers do not form a field. This is because most integers (except for 1) do not have multiplicative inverses that are also integers, violating the field axiom requiring the existence of multiplicative inverses for all nonzero elements.

Ordered Fields

An ordered field is a field equipped with a order relation (≤). The rational numbers (ℚ) and real numbers (ℝ) are examples of ordered fields.

Announcement

Although I haven’t published many articles in 2023 and 2024, these years have been incredibly busy for me. The rapid advancements in AI have inspired me to write a book on Generative AI, a project that has been incredibly challenging and time-consuming. During the writing process, numerous interesting topics emerged that were either too tedious or not directly relevant to the core focus of the book. These topics may be a bit rough around the edges due to time constraints, but I still believe they offer insights as a reference. Rather than discarding them, I’ve decided to share them through a series of articles on Medium.

While the book isn’t finished yet, you can stay updated on its progress by following me on Medium or connecting with me on LinkedIn. I’ll share an announcement there as soon as it’s ready!

Space

The notion of space in mathematics is abstract yet profoundly powerful. It starts with a set — a collection of objects usually called points or elements. But a set alone isn’t very interesting. The magic happens when we add different structures to the set, giving the points meaning and connections. This process of enhancing sets with various structures gives rise to a wide array of mathematical spaces, each possessing its own unique properties, and practical applications.

Space is a set that can be endowed with structures:

• Algebraic Structure: It defines operations (like addition or multiplication) and rules (axioms) on the points within a space.
• Relations: These specify the relationships between elements. For instance, in an ordered set, a relation determines whether one element is less than or greater than another.
• Metrics (Distance Functions): These provide a numerical method to measure the distance or closeness between points in a space, enabling the study of concepts like convergence, compactness and continuity.
• Topology: This defines a more general notion of closeness without necessarily relying on numerical distances.

Metric Space

Metric spaces, endowed with a distance function known as a metric, are often the first step in understanding mathematical spaces. The definition of a space is often written within brackets ⟨ ⟩ or parentheses ( ) to specify the name of the set and the particular structure applied to it.

M represents the underlying set of a metric space, which can consist of numbers, functions, sequences, or other mathematical objects. When the context is clear, we may also refer to the entire metric space as M. A metric d is a function that assigns a non-negative real number to every pair of elements, introducing the concept of “distance” between them. This structure allows for the analysis of distances and also enables the discussion of convergence and continuity.

Common metrics include the Manhattan distance and the Euclidean (L2) distance.

However, a metric function must meet the following conditions for all x, y, and z in M:

When we set 𝑧 = 𝑥, they lead to the conclusion that 𝑑(𝑥 , 𝑦) is non-negative.

Hence, these three properties are equivalent to the properties below.

• Non-negativity: 𝑑(𝑥,𝑦)≥0.
• Symmetry: The distance is the same in both directions.
• Triangle Inequality: The direct route is the shortest.

The broad definition of a metric allows for broad applicability and consistent manipulation of fundamental concepts. For example, the Wasserstein loss, used in generative AI for more efficient training, satisfies the criteria of a metric function. This allows us to apply the properties of a metric space to probability distributions without the need to create a new mathematical framework.

Sequence

Sequences provide a foundational tool for studying concepts like convergence and limits. In abstract mathematical spaces, the familiar notion of a sequence as an ordered list of numbers is too limited. We need to re-establish this concept to accommodate other mathematical objects, such as functions, while preserving the essential idea of an ordered progression.

Let’s explore the concepts of convergence and limits within the context of metric spaces.

A sequence in X is

X is a mathematical space.

Convergency & Limit in Metric Space

In the context of metric spaces, a sequence is said to be convergent if the terms of the sequence approach a specific limit as the sequence progresses indefinitely. More formally, a sequence in a metric space 𝑋 converges to a limit 𝐿∈𝑋 if, for every positive number ϵ (no matter how small), there exists a natural number N such that for all 𝑛≥𝑁, the distance between the sequence​ and L is less than ϵ. This can be expressed mathematically as:

A sequence in a metric space converges if it approaches a specific limit that is part of the space X, meaning the sequence will have a limit L∈X.

This approach, however, relies on knowing the limit beforehand, which is not always the case. To address this issue, mathematicians have developed the concept of Cauchy sequences.

Cauchy Sequence

A Cauchy sequence is defined as a sequence where the elements become arbitrarily close to each other as the sequence progresses. For a sequence to qualify as a Cauchy sequence, for any given positive distance ϵ, there exists a point in the sequence beyond which the distance between any two elements is always less than ϵ.

Definition: For every positive real number ϵ (no matter how small it is), it exists a value N (a natural number, 1, 2, 3, …) where 𝑚,𝑛 ≥ 𝑁 and

Examples

Let’s examine a sequence in ℝ: 3, 3.1, 3.14, 3.141, …. This sequence successively adds one decimal place to the approximation of π. In this example, we use the usual metric 𝑑(𝑥,𝑦)=∣𝑥−𝑦∣. For 𝑚<𝑛, the difference between the m-th and n-th terms becomes progressively smaller than:

Hence, for any positive number ε, there exists an N such that for all m and n greater than N, the difference between the mth and nth elements is less than ε.

Complete

A convergent sequence is always a Cauchy sequence. However, not all Cauchy sequences converge. Take, for example, a Cauchy sequence consisting entirely of rational numbers from the set ℚ. Every term in this sequence is a rational number.

If the sequence has a limit x, then

However, no rational number can satisfy this condition. There is no limit within ℚ for this sequence, meaning it does not converge. To make the sequence complete, we can extend the space to include ℝ.

A metric space is said to be complete if every Cauchy sequence in that space converges to a limit that is also within the space, ensuring that no sequence “escapes” the space as it converges.

Working with incomplete metric spaces presents challenges. We might construct a sequence of approximate solutions, using an iterative method or a numerical method. As the sequence progresses, the approximate solutions become increasingly close to one another, forming a Cauchy sequence in the metric space. Ideally, we would want these approximations to converge to a limit, and then demonstrate that this limit is indeed a solution. However, this approach is only guaranteed to work if the underlying metric space is complete. Otherwise, we might need to extend the space.

Domain & Image

The domain of a function is the set of all possible input values for which the function is defined. It essentially tells you what you can input into the function. On the other hand, the image of a function (codomain) refers to the set of all output values that the function can produce when applied to every element in its domain.

Continuity

Limits and continuity are fundamental building blocks in differential calculus. Cauchy sequences provide a way to define and analyze limits in the broader context of metric spaces. Let’s discuss the idea of continuity for functions between metric spaces.

A function f from one metric space X to another metric space Y is said to be continuous at a point 𝑥₀ in X if, for every ϵ>0, there exists a δ>0 such that for all x in X where the distance

This definition ensures that small changes in the input around 𝑥₀ result in small changes in the output around 𝑓(𝑥₀).

Continuity in deep learning is crucial for ensuring smooth changes in model outputs as inputs vary, which helps in stable training. It allows for the use of gradient-based optimization techniques, like backpropagation, essential for training neural networks effectively. Continuity also helps in generalization, preventing abrupt changes in predictions, leading to more reliable and interpretable models.

Countable

Dealing with infinite possibilities is challenging. Countability in a mathematical space primarily seeks to ensure a manageable and well-behaved structure. Countability conditions facilitate simplifications in analysis and topology, such as the existence of countable bases and the ability to approximate elements with finite sets.

A set is considered countable if its elements can be put into a one-to-one correspondence with the natural numbers (1, 2, 3, …). This means you can list the elements of the set in a sequence. Formally, a set is countable if there exists an injective function f : F → ℕ (natural number) that every element in F can map to a unique element in ℕ.

However, these sets can have infinitely many elements, as long as they can still be listed sequentially, like the set of even numbers, the set of integers or the set of rational numbers. In contrast, the set of real numbers between 0 and 1 is uncountable. These sets are larger than the set of natural numbers and cannot be put into a one-to-one correspondence with them.

Dense

Let ⟨𝑋, 𝑑⟩ be a metric space. A set 𝑌⊆𝑋 is dense in 𝑋 if, for every element 𝑥∈𝑋, there exists an element 𝑦∈𝑌 such that d(x, y) < ϵ for every 𝜖>0. Informally, this means that for any element outside of 𝑌, we can find an element within 𝑌 that is arbitrarily close to it. An example of a dense subset in ℝ is the set of rational numbers ℚ. To illustrate this, consider the decimal expansion of a real number:

While each individual element in the sequence is rational, the sequence itself converges to a real number. This demonstrates that any real number can be arbitrarily approximated by a rational number.

Separable

By definition, a metric space X is called separable if there is a countable set Y ⊆ X such that the closure of Y (the set of all points in X that are either in Y or arbitrarily close to points in Y) is X.

Intuitively, if a space is separable, every point of X can be approximated arbitrarily closely by points in the countable dense subset Y. This means that properties proven for Y can often be extended to the whole space X through this approximation. Separability is frequently a necessary condition for certain important theorems to hold. This property can simplify analysis and lead to more powerful implications for the entire space.

While the direct application of countable dense subsets to complex deep learning models can be challenging, their existence simplifies the justification and analysis of various techniques, such as dimensionality reduction, kernel design, approximation, and data representation.

Isomorphisms

An isomorphism is a structure-preserving mapping between two structures, which is both injective and reversible through an inverse mapping. An injective map (or one-to-one mapping) ensures that distinct elements are mapped to distinct elements.

Surjective mappings ensure that every element in the target set G is mapped to by at least one element from the domain set F.

If a mapping is both injective (one-to-one) and surjective (onto), it is classified as a bijection. This means that every element of the domain is mapped to exactly one element of the image set, and every element of the image set is the image of exactly one element from the domain, establishing a perfect one-to-one correspondence between all elements of the domain and image sets.

Isomorphisms, while not directly visible in the implementation of deep learning algorithms, play a crucial role in the underlying mathematical framework. They ensure the preservation of essential structural relationships between different mathematical spaces, which is fundamental for understanding how data transformations within neural networks affect inherent information.For instance, linear transformations in neural networks strive to preserve relationships between data points. Isomorphisms is crucial in representation learning, where the goal is to capture meaningful patterns while discarding irrelevant details. However, non-linear functions like ReLU, while essential for learning complex patterns, can potentially lead to some information loss due to their non-invertibility.

Preservation

Preservation preserves operations. In the case of fields, it preserves addition and scalar multiplication. Specifically:

Let 𝜙 be a map from 𝐹 to G. In the context of a field, if 𝜙 obeys all the rules above, the map 𝜙 is an isomorphism.

An isometry between two metric spaces is a function that preserves distances. Specifically, if (𝑋, 𝑑𝑋) and (𝑌, 𝑑𝑌) are two metric spaces, a function 𝑓:𝑋→𝑌 is called an isometry if for all 𝑥,𝑥′∈𝑋, the following holds:

This means that the distance between any two points in 𝑋 is the same as the distance between their images in 𝑌, as measured in their respective metrics.

Open Sets & Closed Sets in Metric Spaces

Open and closed sets are the fundamental building blocks of mathematical spaces, providing the essential framework for developing more complex topological concepts. For instance, they are essential for defining convergence and continuity.

Open sets are sets that don’t include their boundaries, while closed sets contain all their boundary points. To aid in visualization and understanding, we will begin our discussion by exploring open and closed sets within the more familiar framework of metric spaces.

Let’s examine a subset A ⊆ X, with an element x ∈ A.

We can construct an open ball 𝐵 centered at 𝑥 with a radius smaller than ϵ. This ball 𝐵 consists of all the elements:

Essentially, B includes x and its neighbors within a radius of ϵ. Visually, these neighbors of x might all lie within A, or some could extend beyond it.

Open Set & Boundary Points

A set A is considered open if, for every element x within A, there exists a sufficiently small radius ϵ such that all elements of the open ball B centered at x with this radius are contained entirely within A.

A boundary point of A is a point in X such that every open ball centered at that point contains elements both from A and from the complement of A (i.e., the set of points in X that are not in A).

A boundary point x is formally defined as follows:

where Aᶜ is the complement of A. The set of all boundary points of 𝐴 is denoted as δA.

An open set does not contain any of its boundary points.

That is,

Closed Set & Closure

The definition of a closed set is simply its complement is open.

i.e.

From another angle, a closed set contains all the boundary points.

The closure of a set A is formed by combining A with its boundary points. In the case of the rational numbers (ℚ) within the real numbers (ℝ), the closure of ℚ is the entire set of real numbers (ℝ).

If the closure of A is the same as A, A is a closed set.

Both the empty set ∅ and the entire set X are considered both open and closed.

Example

Consider the open interval 𝐴 = (3, 6) on the real number line ℝ.

A is an open set. For any x in A, we can identify open balls centered at x such that all elements within these balls are contained in A. For example, we could choose ε to be half the distance from x to the nearest boundary point of A.

Let’s explore a more challenging example with subsets A and C included in X. Are A and C open or closed?

The element “0” and elements greater than “3” are not boundary points of A because they do not belong to the set X itself. The element “3” is also not a boundary point of A. No open balls centered at “3” contain elements outside of 𝐴 that are also within 𝑋.

This example highlights a crucial point: determining whether a set is open or closed requires considering both the set’s definition A and the definition of the underlying space X, which together determine the set’s boundary points. In this case, the definitions result in the surprising outcome that A has an empty set of boundary points. This leads to the conclusion that A is both closed and open. Thus, it’s important to note that open and closed sets are not always mutually exclusive categories. Indeed, a set that is both open and closed is referred to as a clopen set.

For the subset C, the element 2 is a boundary point. The closure of C is equal to C itself, which indicates that C is a closed set.

Continuous Functions

Many concepts in topology can be defined using either open sets or closed sets. Consider two metric spaces 𝐴 and 𝐶, with a mapping function 𝑓:𝐴→𝐶. Informally, the function 𝑓 is continuous at a point 𝑎∈𝐴 if for every open ball around 𝑓(𝑎) in 𝐶, there exists a corresponding open ball around a in A such that the image under f of this latter ball is contained within the open ball around 𝑓(𝑎).

Formally, a function f is continuous at a point 𝑎 if, for every 𝜖>0, there exists a 𝛿> such that

This means the image of the open ball of radius 𝛿 around 𝑎 is contained within the open ball of radius ϵ around 𝑓(𝑎), ensuring that small changes in the domain near a lead to small changes in the image at f(a).

A function f is continuous on its entire domain A if it is continuous at every point within A. However, in the example below, f is not continuous at the point 𝑎.

Sequential Continuity

A function 𝑇 is sequentially continuous at a point x̃ ∈ X if, for any sequence (𝑥𝑛) converging to x̃, the sequence 𝑇(𝑥𝑛) converges to 𝑇(x̃).

In metric spaces, continuity and sequential continuity are equivalent.

Compactness

The concept of finiteness can be tricky when dealing with spaces containing infinite-dimensional elements, like functions or infinite sequences. Compactness generalizes the notion of a set being ‘closed and bounded’ in Euclidean space to such spaces. (In Euclidean space, a set is closed if it contains all its limit points, and bounded if it can be contained within a ball of finite radius.) Even if a set contains infinite-dimensional elements, compactness gives it a certain ‘finiteness’ property.

Compactness is a crucial concept in many areas of mathematics for several reasons. Firstly, it often simplifies proofs and arguments by reducing infinite scenarios to finite ones, making complex problems more manageable. Secondly, it ensures that a continuous function on a compact set always has a highest and lowest value, which is a key idea in the Extreme Value Theorem. Lastly, every sequence in compact spaces possesses a subsequence that converges. This property is important in analysis, as it guarantees the existence of limits in a wide range of scenarios.

A topological space 𝑋 is called compact if every open cover of 𝑋 has a finite subcover. Let’s define open cover and finite subcover:

A finite subcover is a smaller collection of open sets, selected from an initial open cover, that still covers the entire set. Compactness is a topological property that ensures that, for any open cover of a set, there always exists a finite subcover. In other words, no matter how you try to cover a compact space with open sets, you can always find a finite number of them that do the job. In retrospect, problems can be analyzed locally using a finite number of open sets, and the results can then be aggregated.

Topological Space

A topological space is a very general type of mathematical space that provides a framework for defining concepts like convergence, continuity, and compactness. It formally defines the notion of neighborhoods around points within a set, serving as a fundamental foundation for more advanced mathematical theories. It establishes essential but basic structures, which on their own have limited practical utility. Typically, additional structure and refinement are necessary to tailor a space for practical applications.

Unlike metric spaces, which rely on a distance function to define closeness, topological spaces are built upon the concept of open sets. This means that topological spaces don’t possess a notion of distance between points, offering a less structured framework than metric spaces. Instead, they focus on the concept of neighboring points.

Topological spaces consists of two main components:

1. A set of points X: This can be any collection of objects, such as numbers, shapes, or even more abstract entities.
2. A topology τ: This is a collection of subsets of the set of points, called open sets that satisfy certain properties.

The open sets 𝜏 must satisfy the following axioms:

1. The empty set ∅ and the entire set X itself are included in τ. This axiom ensures that there are at least two open sets in any topology, the minimum needed.
2. The union of any collection of open sets in a topological space 𝜏 also belongs to 𝜏, meaning the resulting union is itself an open set.
3. The intersection of any finite number of open sets in a topological space 𝜏 also belongs to 𝜏, meaning the resulting intersection is itself an open set.

A subset A of X is closed if and only if its complement, Aᶜ = X \ A, is open.

For a given set 𝑋={1,2,3,4}, the topology on 𝑋 can range from the simplest to the most complex depending on the number of subsets included as open sets. The simplest possible topology on any set is the trivial topology. For the set 𝑋, this topology would include only the minimal required subsets:

The most complex topology on any set is the discrete topology, where every possible subset of 𝑋 is considered an open set:

Many real-world problems involve spaces where only certain kinds of subsets are relevant or meaningful for analysis. Intermediate topologies strike a balance between too little structure (trivial topology) and too much granularity (discrete topology), making them particularly suited for detailed yet manageable analysis in both theoretical and applied mathematics.

Given a topological space X and a point p in X, a neighborhood of p is a subset V of X that contains an open set U such that

Every open set is a neighborhood of each of its points. (Note that V itself is not required to be an open set.)

With the definition of a neighborhood, the definitions for convergence, continuity, and compactness in a topological space are as follows:

Topological Isomorphism

Topology focuses on the idea of neighborhoods. While metric spaces include the concept of distance, topological spaces, being more general and abstract, do not. In topology, a teacup and a donut are considered homeomorphic, meaning they are topologically equivalent. The two shapes can be continuously deformed into each other without cutting or gluing. We can gradually deform the cup, widening its handle to form the donut’s ring. This deformation, while changing the distance between points, preserves the essential neighboring relationships that are the focus of topology. However, a teacup cannot be transformed into a bowl, as this would require punching a hole and disrupting the established neighboring relationships.

A topological isomorphism, also known as a homeomorphism, is a continuous function between two topological spaces that preserves the topological structure. It’s a bijection, meaning it’s both one-to-one (injective) and onto (surjective), and both the function and its inverse are also continuous. If such a function exists, the two spaces are said to be homeomorphic, or topologically equivalent.

The concept of homeomorphism is fundamental in topology because it allows mathematicians to classify and study spaces based on their intrinsic topological properties rather than their specific geometric shapes. This abstraction helps in understanding and solving complex problems across different areas of mathematics and science.

Bases of Open Sets

In topology, open sets are fundamental to understanding the structure of topological spaces. However, explicitly defining all open sets can be cumbersome. The concept of a basis offers a solution. A basis for a topological space is a smaller collection of open sets with a special property: every open set in the topology can be formed by taking unions of sets from the basis. Essentially, the basis acts as a set of building blocks from which all other open sets can be constructed. A basis for a topology is a collection of open sets that can be used to generate all other open sets in the space.

Example: Standard Topology on the Real Line ℝ

The standard topology on the real line ℝ is the topology generated by the collection of all open intervals in the real line. It is generated by a basis consisting of all open intervals (a, b) where a<b and 𝑎,𝑏∈ℝ. This means that any open set in this topology can be formed by taking the union of (possibly infinitely many) open intervals.

Next

Vector Space, Normed Space & Hilbert Space (Machine Learning)

NLP

• NLP — Word Embedding & GloVe
• NLP — BERT & Transformer
• How to scale the BERT Training with Nvidia GPUs?
• What is new with GPT-4?
• How much do I like ChatGPT?

Can GPT outperform humans?

OpenAI GPT 4.0 achieves a score in the top 10% of test takers on a simulated bar exam, which marks a notable improvement from GPT 3.5 which scored in the lower 10%. The standardized test results below were achieved by GPT 4.0 without undergoing any particular training for these exams. Even though it may not be sufficient for admission to Ivy League schools, the progress made since the release of ChatGPT (a version of GPT 3.5) within just a few months is remarkable.

Its internal adversarial factuality evaluations indicate that GPT-4 outperforms GPT 3.5 by 40%.

However, OpenAI asserts that while GPT 4.0 demonstrates human-level performance on various professional and academic benchmarks, it is less competent than humans in many real-world scenarios. In my own test, it refuses to answer certain toxic questions first.

But by tricking the system, I am still able to deceive it into providing responses. However, as demonstrated later, this becomes significantly more difficult or improbable for more sensitive questions.

It is easy to comprehend the variation in GPT performance between standardized tests and real-life scenarios. Standardized tests are designed with precision to eliminate any ambiguity. The intent of the inquirer is apparent. Conversely, in real-life situations, it is not always easy to grasp the questioner’s intent. GPT 4.0 still struggles to identify sarcasm and negative intentions in some situations. It tends to act quickly and struggles to pose further follow-up questions. Its main goal is not to provide proof for its claims. Nevertheless, it wouldn’t surprise me if GPT achieves superhuman performance for the benchmarks mentioned within the next 6 to 18 months. But it will require further advancements in managing fraud, intention, sentiment, and context for real-life scenarios.

GPT 4 can take in and generate up to roughly 25K words. As a result, the system is capable of processing significantly larger articles and documents. However, I believe that there is still room to better understand and accurately capture contextual information in the dialogs of a conversation.

Multimodal model

GPT is capable of handling multiple modalities, including text and images. An example of this is seen in the “Be My Eyes” app, where users can send images and texts to an AI-powered Virtual Volunteer. The virtual volunteer can quickly address any questions related to the image and provide immediate visual assistance for a wide range of tasks, for instance in the areas of visual impairment and translation.

Here are some more instances of OpenAI’s demonstrations, specifically showcasing its intelligence.

OpenAI also exemplifies how to explicate a research paper. It has the ability to analyze and comprehend intricate ideas presented in academic articles.

Steerability

OpenAI utilizes a combination of internal and external expertise, as well as public opinion, to develop a set of rules for assessing the responses generated by its models. For instance, it provides guidelines to instruct professionals on how to write, review, and rank responses, which are then used to enhance the models through supervised and reinforcement training.

Despite this, OpenAI holds the viewpoint that AI should be customizable by individual users while remaining within the limits set by society. Consequently, the responses generated by OpenAI are adaptable and can be personalized based on specific preferences, specified as “system” below. GPT-4 can offer context, limitations, and instructions on how to address queries.

This can be exemplified through the integration of the Bing interface with GPT.

Today, Greg Brockman, Co-Founder and President of OpenAI, provided a further demonstration of this concept. He presented a context for how GPT should respond to his tax question on the left side, and on the right side, he shared a 16-page US IRS tax code.

After following the arduous IRS instructions, GPT was able to calculate and elucidate his family’s standard deduction correctly. If you have dealt with the US IRS tax code before, you will promptly recognize that even straightforward questions require a genius to decipher the answer.

Risk Mitigation

From the outset of its training process, GPT-4 is designed with a stronger emphasis on alignment and safety.

The accuracy and safety of the model have been enhanced through RLHF training, in particular, to address adversarial usage, unwanted content, and privacy concerns.

Despite RLHF already being implemented in ChatGPT, GPT-4 is further refined through the careful selection and filtering of pretraining data, expert evaluations, and feedback, enhancements to the model’s safety features, and ongoing monitoring and enforcement. OpenAI reports that training RLHF with the new data has resulted in a reduction of harmful outputs.

To illustrate, additional data from domain experts were incorporated to enhance GPT-4’s ability to reject requests relating to the synthesis of hazardous chemicals. In my tests, GPT-4 consistently declines all my attempts to elicit information about breaching TSA security, regardless of how I attempt to deceive it.

To mitigate the risk of generating harmful outputs, GPT-4 employs reward signals during its RLHF training, teaching the model to decline requests for such content. A GPT-4 zero-shot classifier provides the reward signal by assessing the safety boundaries and completion style of safety-related prompts. In order to prevent the model from rejecting valid requests, a diverse dataset is gathered from multiple sources, and the safety reward signal is applied to both allowed and disallowed categories.

More Creative and Powerful

Thanks to its extensive general knowledge and advanced problem-solving capabilities, GPT-4 can tackle challenging issues with increased precision.

Brockman also exhibited the remarkable capabilities of GPT 4.0 by asking it to summarize an article about GPT in one sentence, while also requiring every word in the sentence to start with the letter “g”.