How companies are pouring millions into AI solutions that cause more harm than good

In 2019, I was entrenched in volunteer work at a small free clinic in North Hollywood, California. Actually in pursuit of my California phlebotomy (blood drawing) credential, my goal was to continue work in clinical settings, and to finish my degree to become a doctor. I wanted, like many of us, to help patients — to make a positive difference in the lives of people.

It was not very long after that I discovered computer science, artificial intelligence, and biotechnology. Hooked on the idea and disillusioned with a traditional academic path, I jumped at the opportunity to join a startup-like college in San Francisco, accelerate my degree path, and get into tech. It was all so shiny and promising. Papers on top of papers were being published with titles like “Artificial Intelligence Helping Biotech Get Real”, “AI Breakthrough Could Speed Up Lung Cancer Diagnosis …”, and many, many more. Months into pursuit of my new, shiny degree, I was doing aggressive research into the field, and even had begun working in it.

Pause. The Covid-19 pandemic seemed to accelerate the AI diagnostic space, with poor results. Venture capitalists, large investment firms, academic organizations, and big pharma continued to pour millions into so-called “AI focused biotech” startups. While large conference platforms seem to always accept a single keynote speaker on fairness or ethics in AI (mostly focused on disparate outcomes between demographic groups — a crucial part of the conversation, but a part nonetheless), my LinkedIn feed, industry contacts, and tech-news outlets served crickets on the conversation of ethical standards for the artificial intelligence boom. The biotechnology companies themselves continued to post job openings for subject matter expert “Data Scientists”, leaving no space in the room for anyone without a laser-focused Ph.D. dissertation, anyone who might have less of a stake in innovation for innovation’s sake, anyone willing to stand in the corner and ask the crucial question:

“We can, but should we?”

Building Systems Designed for Failure

Physicians swear an oath to “first do no harm”, but professionals falling under the ubiquitous umbrella of bioinformaticians, data scientists, and machine learning engineers swear no such oath.

Instead, when we’re hired, we market ourselves based on metrics. Biotechnology companies may have noble missions, but are still profit-driven companies focused on what value you bring them from a hard-number standpoint. As data practitioners, we are not selling our talents based on how fuzzy and warm we made a patient or their family feel through the promise of a technology we developed, we sell ourselves based on the percentage of test-data oncology biopsy slides our algorithm correctly identified as pre-cancerous (for example). Not to mention: assuming a technology we create gets to the “wild”, that patient and their family have no idea how our tech works. It’s up to us to be honest about how much it can really be trusted — and that kind of honesty doesn’t often get us far in our careers, or get us published or recognized.

If you’re clamoring for a better, real world example of a potentially devastating and dramatic failure, take IBM’s “Watson for Oncology” project. After a $62 million dollar investment, studies found that the recommendations made by the so-called “supercomputer” were often potentially deadly. The project inevitably was rolled back, a loss.

Or, read this article I linked earlier on the hundreds of Covid diagnostic applications that were proven to do more harm than good.

“AI for healthcare” is a term with so much potential and almost palpable kinetic energy that many of us practitioners feel galvanized to become a part of it. The truth of the matter seems to be that in many cases, AI technologies are a cry for help — the world of R&D is increasing its pace, and large, slow-moving scientific companies need an accelerator to keep up. The promise and potential of automation, accelerated discovery, fast triaging, and lowering costs for radiological, pathology, or diagnostic services are just too good to pass up. The sadder part of this truth, though, is that most of our technology just isn’t ready for clinical use. How honest we are about this seems to be contingent on how dramatically we need to increase profit margins, impress investors, or keep up with the multi-billion dollar “Joneses” to maintain a name in industry.

Taking it Back to Move Forward: Building Better Systems

What good scientists know is that good science takes time. The mavericks of medical innovation maintained equilibrium with the traditional academics who understood that sometimes you can try something new, but you shouldn’t try just for the sake of trying: not when there’s a patient on the table.

Biotech company CEOs, investors, machine learning engineers, data scientists, bioinformaticians, data analysts, data engineers, you get the point: we don’t see those patients on the table. Abstraction can be a tricky thing. It allows us to take bold steps forward, but simultaneously enables us to be bolder than we would, did we have to look into the eyes of the human being who may be put on a ventilator unnecessarily because of poorly designed Covid-19 triaging software. Or given drugs that increase bleeding because of a poor recommendation from an algorithm, or testing a new drug and dealing with severe adverse reactions, and so on.

The beginnings of the so-called “biotech” industry as we know it roughly four decades ago made us — the tech people — practitioners in a sacred space, right along with the clinicians who see, in the flesh, the impact of their work, and now ours. We need oaths — we need to develop systems with ethics in mind, from the very beginning.

In every brainstorming meeting you have with your team, you need to have someone in the corner who isn’t wrapped up in the rat race of their own career, or focused on improving potential quarterly profit margins. Biotech, in a nutshell, cannot remain what it is: money hungry, innovative for the sake of innovation and publication, full of hubris and over-trusting of the programs we develop, of the buzzwords “deep learning” and “AI”.

We need industry-wide ethical standard operating procedures. We need someone in the corner who will ask us:

“We can do this, but should we?” without fear of reproach.

In the long run, the ethical conversation is the best one you can have for your company, even if it’s sacrificial of short-term press-release glory or temporary product hype. More importantly, the ethical conversation is the best one you can have for any potential future patients impacted by your product or innovation.

Let’s Talk: The Ethical AI Conversation, and How Biotech Is Failing was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the wise words of Wikipedia, Docker is:

an open-source project that automates the deployment of software applications inside containers by providing an additional layer of abstraction and automation of OS-level virtualization on Linux.

Let’s clear this up a bit.

Why Containers? 📦

Using a service like Docker to put applications and programs into containers has the primary purpose of maintaining a consistent machine environment for an application or piece of software, no matter what server it lives on — on-premise, cloud, what OS the server has, and so on. There are a few other adjacent benefits, but for now, let’s keep it simple: Docker packages our code and dependencies, so that we can maintain a consistent machine environment for our software.

You might be asking (and you’d be right to ask) — what’s the difference between Docker and a Virtual Machine, then? Why wouldn’t I just spin up a VM for my code, and call it a day?

While a VM achieves the goal of sandboxing software and its dependencies inside an isolated environment, it does so at a cost. The computational cost of virtualizing an entire machine is pretty high. The Docker curriculum expresses how Docker is different (better than I can):

Containers take a different approach: by leveraging the low-level mechanics of the host operating system, containers provide most of the isolation of virtual machines at a fraction of the computing power.

For conceptual purposes, you can think of a Docker container like a compact version of a VM — a sandbox environment that only uses what it needs of a virtual OS to run your software.

Containers make deployment easy, reproducible, and standardized. This is why we like them.

Taking a Step Back 👣

Now that we understand what containers are, what Docker is, and why we care, let’s take a step back and set some expectations. In this article, we’ll:

• Go over some important terminology you’ll encounter when working with Docker; like containers (check! ✅), images, and registries
• Walk through installing Docker on Linux
• Go through basic Docker commands to familiarize ourselves with those we’ll use most often

Wait, What’s an Image? 🖼

Let’s use the analogy of a container ship on the ocean to get the hang of our terminology. One container ship holds many different containers from a variety of companies, and each container is full of goods. Let’s zoom in on one container. This container holds a specific amount of specific kinds of goods, and they didn’t get there on accident. Someone at the company shipping this container laid out for workers what should go inside it.

An image is the person who tells the workers what to pack in the container. In other words, it’s like a blueprint for containers to be built off of. If you’re familiar with data terms, an image has a one-to-many relationship with all containers based off of it — many containers can be spun up off of a single image.

What About a Registry? 🗂

Well, I know this is a lot of terms, but registries (also called image registries, container registries, or image repositories depending on the cloud services you use) are actually pretty straightforward. If we have a lot of software, we’re likely going to need somewhere to store all of the blueprints.

So, if our images are our blueprints, our containers are our running sandboxed applications, our registries are kind of like a bookshelf or a filing cabinet that hold our images. This allows us to pull our own images, or, if your projects are open source, for others to pull your images and be able to spin up an instance of your application on their own machine without needing to worry about installing any dependencies or configuring their environment at all (aside from just having Docker installed).

Before we move on, take a look at the diagram below to get familiar with the terms we’ve just talked about.

Setting up Docker ⚓️

While Windows and Mac allow a desktop download, most Ubuntu distros don’t have a GUI. This tutorial will walk you through setting up Docker on Ubuntu, which will be useful for deployment within Linux VMs. If you have a Windows or Mac machine, check out the links below to get Docker set up, then catch back up with us in the next section.

For Windows, use this resource.

For Mac, use this resource.

Pull up terminal on your Linux machine, and run update before installing anything:

$ sudo apt update

You can also run upgrade after, to be sure you’re up to date with installed packages:

$ sudo apt upgrade

If the kernel doesn’t upgrade, you’re good to move on. If it does, reboot with:

$ sudo reboot

Afterwards, we can install Docker:

$ sudo apt install docker.io

If you have any issues installing the docker.io package, check out the below resources to troubleshoot:

1. Docker docs
2. Linux.com

You can check that Docker was successfully installed with:

$ sudo docker --version

But why do we have to use sudo?

Docker out of the box requires admin permissions. For security reasons, you don’t always want to use Docker as the root user or always have to use sudo. The following steps take you through setting up a user to use with Docker, and then rebooting.

$ sudo usermod -a -G docker $USER

Once you’ve run this, log out and back in and try running:

$ docker --version

to check that permissions have changed. If they haven’t you’ll receive an error.

Now, try running Docker’s “Hello World” image to verify your install:

$ docker run hello-world

Nice! If that was successful, we’re ready to chat about some Docker commands.

Docker Commands 📃

There are a few Docker commands you’ll use often and want to know. For a comprehensive list of all commands and links to more details on them, check out the Docker documentation.

• run
• build
• exec
• logs
• ps
• images

Let’s start with build. The syntax for build is:

$ docker build [OPTIONS] PATH | URL | -

build builds a docker image from a Dockerfile and what’s referred to as a context. A context can be your project directory, a GitHub repository, and more, depending on what options are specified and if you provide a URL.

Docker run runs a container based off of a built image (there’s a little more to it than that, but you don’t have to worry about that for now. Check the docs if you’re interested). It’s syntax is as follows:

$ docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

Some options commonly provided to run are port mappings, volumes, detach, and so on.

Docker ps allows you to list the containers on your machine. Providing the -a flag allows you to see both running and stopped containers.

$ docker ps [OPTIONS]

Similarly, Docker images allows you to list the images on your machine:

$ docker images

Docker logs provides you with logging output for running containers, which is valuable for debugging and troubleshooting. The syntax is as follows:

$ docker logs [OPTIONS] CONTAINER

Here, CONTAINER can either specify a container name or ID, both provided in the output of docker ps.

A common use for docker logs is:

$ docker logs --follow my-container

The --follow option tells the logs to stay up and auto-update with new data as the container application is running.

Finally, Docker exec runs a new command in a running container.

$ docker exec [OPTIONS] CONTAINER COMMAND [ARG...]

Wrapping Up

Hopefully now you feel like you have a better understanding of what Docker is, why we use it, and how to get things set up and running on your own machine. We’ve covered:

• Important terminology you’ll encounter when working with Docker; like containers, images, and registries ✅
• Installing Docker on Linux ✅
• Basic Docker commands to familiarize ourselves with those we’ll use most often ✅

This is part four in a series on graph theory and graph convolutional networks.

If you’ve been reading this whole series, you’ve been with me on this entire journey — through discussing what graph theory is and why it matters, what a graph convolutional network even is and how they work, and now we’re here, to the fun part — building our own GCN.

If you’re new to this series, that’s totally fine, too! In either case, let’s get coding!

For this tutorial, we’re going to be walking through two implementations of GCNs to classify the PROTEINS benchmark dataset. If you want to find attribution or papers on this data, or download it to look at it yourself, you can find it here under the “Bioinformatics” heading. You can also take a look at the whole notebook here. Attributions for the code can be found in the repository for this project.

Part I: GCNs with Spektral

What is Spektral? According to their homepage:

Spektral is a Python library for graph deep learning, based on the Keras API and TensorFlow 2. The main goal of this project is to provide a simple but flexible framework for creating graph neural networks (GNNs).

Getting started with Spektral is extremely easy because of the forethought put into the project — if you’ve done any modeling with Keras or Tensorflow, I think you’ll find Spektral quite intuitive.

Additionally, Spektral has many benchmark graph datasets built in, meaning that you avoid the fuss of needing to ensure your data is in the right format for modeling with GNNs, and can easily get started experimenting.

Whether you’re following this tutorial in Colab or just a plain ol’ Jupyter Notebook, our first step will be the same. Spektral is not one of the built-in libraries in Colab, so we’ll need to install it:

PROTEINS is one of the benchmark datasets for graph kernals from TU Dortmund. You can access this class of datasets from the TUDataset class, which we access by first importing and then instantiating an object of it, with the name of which TUDataset we’d like to access passed in:

When we load in our dataset, we can see how many graphs it contains in the n_graphs property. Here, we can see that this dataset has 1113 graphs. In this dataset, these are split into two distinct classes.

Spektral’s GCNConv layer is based off of the paper: “Semi-Supervised Classification with Graph Convolutional Networks” by Thomas N. Kipf and Max Welling. This is perfect, as this is the paper we’ve been referencing if you’ve been following this series so far. If you haven’t, I’d recommend checking out the paper as well as my article on how these networks work to get a better grasp on what Spektral is doing for us behind the scenes!

Since this is the layer we want to use, we’re going to have to perform some preprocessing. Spektral makes this super easy with their GCNFilter class, which performs the preprocessing steps (outlined in the paper) for us in just two lines of code.

First, import GCNFilter from spektral.transforms , and then call .apply() on our dataset, passing in the instance of GCNFilter :

At this stage, we want to be sure to perform our train/test split. I do this by shuffling the data and then taking slices (about 80/20% respectively) for this simple example, but you’re welcome to optimize this step further once you get comfortable with this implementation!

Now, let’s import the layers we’ll need for our model:

Wait a minute — aren’t these import statements from Keras? Don’t worry — this isn’t by accident. Because Spektral is built on top of Keras, we can easily use the Keras functional API to build our model, adding in Spektral specific layers as we go to handle the graph-structured data.

We import Dense and Dropout layers — Dense is your typical dense neural network layer that performs forward propagation, and Dropout randomly sets input units to 0 at a rate which we set. The intuition here is that this step can help avoid overfitting*.

Then, we import our GCNConv layer, which we introduced earlier, and our GlobalSumPool layer. Spektral defines this layer concisely for us:

A global sum pooling layer. Pools a graph by computing the sum of its node features.

And that’s all there is to it! Let’s build our model:

Here, we use model subclassing to define our model. We’re going to pass n_hidden : the number of hidden layers and n_labels : the number of labels (target classes) to our model when we instantiate it for training.

Then, within __init__ , we define all of our layers as properties. Within call , we define this method to create and return our desired output by calling our layers on input in sequence.

Let’s instantiate our model for training!

Here, we’re going to initialize 32 hidden layers and the number of labels our data have. Spektral conveniently gives us an n_labels property on our TUDataset when we read it in. The benefit of this is that you can use this same code with any other Spektral dataset without modifications if you’d like to explore other data!

Above, we’re calling .compile() on our model. If you’re familiar with Keras, you’ll be familiar with this method. We’ll pass in our optimizer, adam , and define our loss function, categorical crossentropy .

Now we reach a snag. Those of you familiar with Tensorflow and Keras might be tempted to just try to call model.fit() and call it a day. However, even though Spektral makes the process of building GNNs seamless on top of Keras, we can’t exactly work with our data the same way.

Because we’re using graph-structured data, we need to create batches to feed our Keras model. For this task, Spektral still makes our lives easier by providing Loaders.

Now that we’ve taken care of batching, we can call model.fit() . We won’t need to specify batches, just pass in our loader, since it works as a generator. We will need to provide our steps_per_epoch parameter for training.

For this simple example, we’ve only chosen 10 epochs. To validate, let’s create a loader for our test data:

And we’ll feed it to our model by calling .load() .

That concludes building GCNs with Spektral! I urge you to play around with optimizing this example, or diving into other GNNs you can build with Spektral.

Part II: GCNs with Pytorch-Geometric

Even though Spektral gives us an excellent library of graph neural network layers, loaders, datasets, and more, there are times where we might want a little more fine-tuned control, or where we might want to have another tool for the job.

Pytorch-Geometric also provides GCN layers based on the Kipf & Welling paper, as well as the benchmark TUDatasets. Implementation looks slightly different with PyTorch, but it’s still easy to use and understand.

Let’s get started! We’ll be working off of the same notebook, beginning right below the heading that says “Pytorch Geometric GCN”. Attribution for this code is provided.

Our first step, as usual, is to install our required packages:

Now, let’s grab our dataset:

And take a look into our data:

Here, it’s demonstrated that there are a variety of properties provided by Pytorch-Geometric on our TUDataset object. This provides us a lot of information that we can use to fine-tune our approach later, as well as deeply understand our data.

Now that we know what our data looks like, we’re going to perform our train/test split. For this example I also use a simple shuffling and then slicing method, but as always, I encourage you to look into optimizations for this step!

Pytorch also provides us with DataLoaders for batching:

Now that we’ve taken care of that step, we can build our model. We’ll use a similar approach, but remember that now we’re using Pytorch instead of Keras.

We’ll import our functional layer (similar to Keras’ Dense layer), our GCNConv layer, and a global_mean_pool layer. This performs a similar pooling operation as Spektral’s GlobalSumPool , but by taking the mean rather than the sum of neighboring nodes.

When we build our model, we inherit from Pytorch’s GCN model, and then initialize three convolutional layers. We’ll pass in the number of hidden channels when we instantiate our model.

Then, we build a forward() method, which is similar to the call() method we build earlier in our Spektral GCN. This tells our model how to propagate our inputs through out convolutional layers. With Pytorch, we explicitly define our activation function. In this example, we use relu .

Prior to our final classification, we perform our pooling, and then set our dropout and pass our inputs through a final linear layer.

While there are a lot of opportunities to customize and fine-tune our Spektral model, I like how explicitly we define our architecture with Pytorch. When it comes to “which approach is better”, like most things, it depends on what your team needs to prioritize (explain-ability over efficiency of quickly proving concept, for example).

Let’s take a look at the resulting architecture:

Next, we’ll need to:

1. Set our optimizer — we’ll be using adam for this implementation as well
2. Define our loss function — similarly, we’ll keep categorical crossentropy
3. Define train and test functions, and then call them for a set amount of epochs.

We’re using many more epochs for this example, and as such, we achieve better metrics. There isn’t an inherent reason for this aside from the code examples I used to help build the models and learn each respective library. Also, given that the Pytorch-Geometric implementation was my final implementation, I focused more on results than in my earlier experiments. As always, I encourage you to play with and optimize the code and make it better!

Looking at the last 20 training epochs, we see that we achieve about 71.34% training accuracy and about 62.67% test accuracy. Evaluating accuracy on the test data around epoch 186, one potential optimization would be early stopping callbacks that ensure we stop training once we achieve convergence, to avoid overfitting the model.

There are more optimizations we can make, but this has been a long article, so we’ll stop here, and I’ll leave you to your own experimentation and optimization!

Notes:

* — For more information on dropout or why it works, check out this source.

Classifying the PROTEINS Benchmark Dataset was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is the third part of a series on graphs and graph theory in machine learning.

By now, if you’ve been following this series, you may have learned a bit about graph theory, why we care about graph structured data in data science, and what the heck a “Graph Convolutional Network” is. Now, I’d like to briefly introduce you to what makes these things work.

For my friends in the field who may know a bit about this subject, I’ll be talking about the intuition behind Kipf & Welling’s 2017 paper on graph convolutions for semi-supervised learning (Thomas Kipf wrote a more digestible explanation of this concept here), although other approaches have been developed since improving on this methodology. My goal is to introduce the mathematical concepts, rather than provide an extensive overview of the topic.

Now, if you aren’t a mathematician, data scientist, or even a computer scientist, don’t run away just yet. The goal is to take the academic papers and break down the concepts so that anyone can understand. Feel free to leave a reply at the end letting me know if I’ve achieved this goal or not!

Let’s Get Going! — GCNs, What’s the Goal?

In Thomas N. Kipf’s blog post, he says the goal of a Graph Convolutional Network is to:

“…learn a function of signals/features on a graph G =(v, e)”

But what does that even mean? Let’s break it down.

There are different methods for performing some prediction or classification on graphs or nodes. But we’re focused on Graph Convolutional Networks, for classification on graphs*. This means, to provide a concrete example, that for something like a protein, which has a structure represented by an entire graph:

We might want a way to embed this graph’s features such that we could classify what protein we’re looking at.

So, let’s return to our quote above. We want to learn the function of features on a graph. In other words, we want to learn the relationships between features — in the protein example, that might be specific bond representations between certain atoms, or the atomic representations themselves — given our graph’s specific network structure. In other words: how do combinations of nodes and bonds (edges) impact what protein this is that we’re looking at (target label)?

Disclaimer: this, like many things in this post, is a slight oversimplification. The purpose of this is to share the intuition behind GCNs, and make it accessible. For full explanations and mathematical proofs, I encourage you to dive deeper into the papers cited at the bottom.

Why not a normal Convolutional Neural Network? We’ve spoken a bit in previous articles about the difficulties traditional machine learning & deep learning models face when confronted with arbitrarily structured (non-Euclidean) data. What a GCN allows us to do is embed the graph structure into a two-dimensional representation, overcoming this barrier that we would encounter if we attempted to classify this graph using a traditional CNN.

You can think of embedding like taking a three dimensional molecule model, like this:

And drawing it on a piece of paper, to carry with us and memorize.

To Understand How, Let’s Look at Forward Propagation.

I may be projecting my learning style on my readers, but to really fully grasp the formulas that Kipf proposed, I had to go back to basics and review forward propagation. Let’s take a quick second to look at the forward propagation formula:

Don’t let this formula scare you. All it’s saying is that the feature representation at the next layer is equal to the result of performing some activation function on the weights at the current layer times the feature representation at the current layer plus the bias at the current layer.

Maybe that still feels dense, and if it does, that’s okay. You can think of this this way: this formula represents is a series of computational steps we take to get to an output, passing the result through each layer in our neural network until i is our final (output) layer. (If you know the basics of programming, you can conceptualize this like a for loop: “for each layer in our neural network’s layers, perform this calculation to return a final value.”)

Our activation function (represented as sigma) might be something like ReLu, and our feature representation can be thought of in terms of our protein example, where each atom in our molecule is represented by, (for the sake of our simple example), its atomic number. We would then want our final output at our last layer to be the representation of what protein we’re looking at — our classification. Don’t worry too much about bias for now if you don’t know what it is, we won’t be discussing it further in this article.

Check-In

It’s possible that if you’re maybe a bit new to all this your brain is hurting right about now, and that’s fine. Let’s take a second to review and digest what we’ve learned before we go further:

1. One goal of a GCN is to take an arbitrarily structured graph and embed it into a two-dimensional representation of a network.

2. Additionally, we want to understand the functions of features on a graph — we want to know how stuff influences other stuff (how features in our graph influence our target).

3. We’ve reviewed forward propagation for regular neural networks, and understand what this equation tells us.

Forward Propagation — How Does This Relate to GCNs?

Similar to any neural network, Graph Convolutional Networks need a way to propagate values forward through layers to achieve their goals. However, given that our data is not Euclidean, we’re going to need to make a few slight adjustments to the regular forward propagation equation we discussed above.

First, we need a way to represent our graph computationally. If you remember all the way back to my first article on Graph Theory, or if you remember your mathematics (or, for the computer scientists in the room, your data structures courses), you’ll remember that one way to represent a graph is with an Adjacency Matrix. Let’s take a look at one example:

Our adjacency matrix is effectively a sparse matrix where we have rows and columns representing node labels and a binary representation of whether or not any two nodes are connected to each other. 0 corresponds to “no connection”, where a 1 represents a connection (an edge).

So, we have our first missing piece of the GCN puzzle: our adjacency matrix, which we’ll call A.

Now, we need to make sure we insert A into our propagation equation. Since I said not to worry about bias (and since the article I’m referencing for images also omits this value from its images, this works well), you won’t see it in the following representation.

Let’s take a look at our equation so far:

But we aren’t quite done yet. There are two problems:

1. If we solely look at A, we’ll look at all of a given node’s neighbors, but not that node itself. We’ll deal with this in a moment.
2. What is A*? A, as we initially discussed it, was not normalized. Since we’re going to have to multiply our weights and feature representations by our sparse adjacency matrix A, we can anticipate that the scale of the feature representations and weights will dramatically change with the scale of A. So, A* is normalized A. Let’s talk about what that means:

Kipf & Welling introduced a method to normalize A that takes care of this second problem. First, we need the diagonal node degree matrix, D. If that sentence feels confusing, check out the linked article! One of the ways to normalize A would be to multiply with D, but Kipf & Welling introduce us to another method called symmetric normalization. This equates to:

You’ll see this in action when we provide our final equation! Now, that we’ve addressed our normalization issue, we’re almost there. We have one more thing to add: self-loops.

We want to learn representations for all of the nodes in our graph, and this means that we need to change something about our graph. Because this is a lengthy article already, I’ll once again quote Thomas Kipf from his blog post to explain:

“… multiplication with A means that, for every node, we sum up all the feature vectors of all neighboring nodes but not the node itself (unless there are self-loops in the graph). We can "fix" this by enforcing self-loops in the graph: we simply add the identity matrix to A.”

Here, we arrive at our next step: enforcing self-loops. This means that we need our nodes to also connect back to themselves, looking something like this:

If you’re not 100% sure what Kipf was talking about in his quote regarding the Identity Matrix, remember that the identity matrix is an n x n matrix with 1’s on the main diagonal (top left to bottom right) and zeroes elsewhere:

Where n is, in our use case, going to be the dimension of our adjacency matrix A.

Now that we’ve normalized A with its degree matrix and enforced self-loops by adding our identity matrix, we arrive at our final equation for forward propagation in graph convolutional networks:

Let’s break down this new equation.

This piece says: “The function of our features for layer l given adjacency matrix A”. This is what we’re looking to solve for. Remember one of our original goals, to find the functions of features for a graph.

Let’s break this second part down. Remember, this is to the right of the equal sign — this is what we’re using to find the functions of our features.

Remember that sigma (𝝈) represents an activation function (such as ReLu). So what this clause represents is the result of the activation function on the diagonal degree matrix (D̂)* of Â, which is our adjacency matrix (A) added to the identity matrix, which we’ve normalized using “symmetric normalization”. These are then multiplied by our feature representations and our weights for our current layer, l.

Wait — let’s take a step back.

Let’s talk through that last piece piece to our puzzle. Remember how we talked about our protein example, and how our nodes might have a label, like, the name or abbreviation of an atom (i.e: carbon, or C), as well as a feature representation, which might be something like their atomic number?

These representations in this example become our feature representation vectors, and for our equation above, represent H of l.

Sum it All Up — In a Few Words

You’ve just learned how to find the functions of features for a given graph, G given its adjacency matrix, A, and its degree matrix, D.

You’ve reviewed forward propagation, and learned how it differs for graph convolutional networks. You’ve learned about some important caveats to consider when dealing with arbitrarily structured data, like the importance of having or creating self-loops and of normalizing our adjacency matrix.

Nice! You’re well prepared to take on the next step — which will be coming soon — and which should be much more fun: actually classifying stuff with GCNs.

If you want to learn even more before next time, check out all of the papers and sources that were so invaluable in the writing of this article:

1. Semi-Supervised Classification with Graph Convolutional Networks, Kipf & Welling (ICLR 2017)
2. Graph Convolutional Networks, Thomas Kipf
3. Understanding Graph Convolutional Networks for Node Classification, Inneke Mayachita

Notes:

1. *GCNs can be used for node-level classification, as well, but we don’t focus on that here, for the sake of a simplified example.
2. *this represents ‘D-hat’, Medium’s mathematical notation support is lacking a bit

What Makes Graph Convolutional Networks Work? was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

A non-convoluted look into graph theory and machine learning

In my last article on graph theory, I briefly introduced my latest topic of interest: Graph Convolutional Networks. If you’re here thinking “what do those words mean?”, you’re in the right place. In this article, we’re going to break this topic down, step by step.

Part I: What’s This Graph Thing?

If this is the first you’re hearing this ‘graph’ word, I’m sorry, but you have some homework to do. Before we can dive deeper into this topic, you should check out my last article briefly introducing graph theory (and why we care about it). I’ll wait here!

Alright, now that you’re back, let’s explain a bit further. Graph theory is a mathematical theory, which simply defines a graph as:

G = (v, e) where G is our graph, and (v, e) represents a set of vertices or nodes as computer scientists tend to call them, and edges, or connections between these nodes. Graph theory exists because we need a way computationally and mathematically to represent relationships between things.These things can be anything: users on a social media platform, physical neighbors in a community, addresses or locations (coordinates) on a map, pixels in an image, or neurons in our brain. The basis of graph theory as it relates to machine learning in particular is that much of our data can be best understood when we can represent its relationships. Therefore, we’d like a way to embed these relationships so that we can then work with the whole of the data.

But let’s not get too far ahead of ourselves — we have some more terms to define.

Part II: Convolution? That sounds… convoluted.

Let’s bring it back to the things which have relationships with other things that we want to understand. Simple enough, right? Let’s consider, for example, we have pixels in an image. Those pixels are always related to every other pixel in an image. This image has a set structure, and the pixels remain within proximity to other pixels in a fixed way. Let’s take a look:

If you can tell, this fits our definition of a graph. Implicitly, an image is ‘viewed’ as a graph by a different type of neural network: a Convolutional Neural Network. In this article, I’ll be breezing through the very basic concepts of convolutional neural networks to explain graph convolutional nets. However, if you aren’t aware of CNN’s, I highly recommend taking a look at the linked source after reading this article to gain a well-rounded understanding of all of these topics.

You may be able to intuit from their name that graph convolutional networks and convolutional neural networks share some things in common. You would be correct to think this — the intuition behind GCN’s and CNN’s is extraordinarily similar.

But what is our CNN doing with this image above? If it’s technically a graph, why do we need this other thing? Well, I’m glad you asked!

Images are implicitly graphs of pixels connected to other pixels, but they always have a fixed structure. As our convolutional neural network is sharing weights across neighboring cells, it does so based on some assumptions: for example, that we can evaluate a 3 x 3 area of pixels as a “neighborhood”. The assumptions on which our convolutional neural networks work rely on 2-dimensonal, regular data (also called Euclidean data, if you’re well-versed in domain terminology).

Our social media networks, molecular structure representations, or addresses on a map aren’t two-dimensional, though. They also don’t have a necessary size or structure. We encounter difficulty when trying to cram non-Euclidean or arbitrarily structured data into CNN’s, since that’s about where they reach their limit and stop being useful.

Part III: Networks

We’ve established that we have these arbitrarily structured networks of stuff that don’t fit into our traditional convolutional neural networks. In fact, they don’t really work with a lot of different kinds of neural networks. As such, there are graph neural networks, of which graph convolutional networks are a basic variant.

In this article, I won’t be getting into the mathematics behind graph convolutional networks (even though it’s quite fun) — I just want to discuss the intuition. (Don’t worry — I cover the mathematics in the next article of the series.)

Effectively, the primary difficulty in embedding features represented as both nodes and edges is this matter of arbitrary space usage, and a lack of Euclidean distance between neighbors. With those facets, we must base approaches off of different assumptions. Here, I’ll be primarily discussing graph convolutional networks as they’ve been discussed by Kipf & Welling, although there are various approaches.

We’ve learned about how convolution in neural networks is a method of sharing weights between neighbors. First, to determine neighbors, we’re going to need to provide some data.

Where the normal neural network forward propagation function determines the feature representation of the next hidden layer by evaluating our weights, feature representation and bias for our current layer, our graph convolutional network is going to add an adjacency matrix to the equation. There is also our non-linear activation function, which, since I’m trying to not get too mathematical, I’m ignoring in our considerations for now.

Do we remember what an adjacency matrix looks like? Here’s a refresher:

What A is is a matrix representation of the connections within our graph, 𝝘₅. Each row or column label represents the node with the same number label, and a 1 in an intersecting row/column represents an edge between those nodes.

For those of you familiar with machine learning already, this looks a bit like a sparse matrix, right? See, this isn’t all so new after all.

Effectively, representing our graph as an adjacency matrix enables us to provide it to the net in the form of a tensor, something our model can work with.

Before we can just hand this matrix over to our propagation equation, though, we need to ensure that we normalize our values. The intuition behind this is similar to normalizing any data we feed through a neural network: values of vastly different degrees of magnitude can cause our network to learn higher weights for values that it shouldn’t, simply because those values were initially much higher than other values. For our purposes here, we’re just going to mention normalization. I’ll dive more deeply into Kipf & Welling’s methodology and intuition in the next article.

Once we’ve normalized our data, we’ll perform some kind of aggregation between neighboring nodes — for example, take an average. The intuition here for semi-supervised learning is that similar nodes are likely to share the same label. We can imagine this process as the passing of a message, where each layer of our GCN takes an aggregate of a neighbor node, and passes it one “hop” away, to the next node.

So if we have a three-layer GCN, we can convolve each node’s ‘third-order’ neighborhood. In human terms, this means that we can pass a message to each node’s neighbor three “hops” away and effectively embed the community structure of our graph. If, however, we have a graph with many more “hops”, we would need more layers to effectively embed our graph structure.

If you’re feeling anything like I did when I first delved into this topic, you’re probably ready for a break. Before I let you off the hook, let’s take a look at what we’ve learned:

1. We’ve recapped the essentials of graph theory, and understood why we care about this as machine learning engineers and data scientists.
2. We’ve looked at convolutional neural networks, evaluated the term “convolution” briefly, and discussed their limitations.
3. We’ve taken a brief peek behind the scenes of the intuition behind graph convolutional networks, and we mostly understand why they work!

Alright, take a rest for now, and don’t forget to give yourself a pat on the back! You’ve learned a lot in a short span. I look forward to next time, where we’ll dive deeper into the mathematics that make all of this work, and learn how to start coding up our very own GCN.

Graph Convolutional Networks — Explained. was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Tackling the impossible, step by step.

A colleague of mine, when I told him about an ongoing project, said: “Oh, you’re really building a money printer!” To predict the stock market accurately would be a game changer —this hopefully goes without saying. But, when massive financial firms with seemingly endless resources are looking to tackle the same problem, what space is there for anyone else? Besides, can one even begin to attack this in a different way, to provide even somewhat meaningful results?

Starting from… Not-Quite-Scratch

Kaggle is full of notebooks with toy examples of sentiment analysis for stock market prediction: notebooks which evaluate, based on general news headlines, if the “market” will go up or down. While these are really excellent tools for learning more about machine learning, they’re hardly interesting for high-frequency trading or actual investors. Other approaches often look at historical data, and attempt to use various regression approaches to predict where prices will be over time, for a given company. Here, figuring out what features legitimately affect price becomes an issue, as well as figuring out which historical trends will continue to hold true in the future. HFT’s (High Frequency Traders) and massive financial firms commonly scoop up these approaches as soon as anything new and promising is found, leaving those of us with less resources in the dust.

Instead of predicting price trends, I took a different approach, asking the question: “what if we could predict public sentiment about a company in a way which tends to influence their stock price, rather than JUST the price itself?”

To start, I looked at the existing sentiment analysis approaches, and played around with some of the code that I found. In considering what problem this code was actually solving, I was able to ask a crucial question:

“Based on a headline about company X, what can we learn about public sentiment, and the change that we might see in X’s stock price?”

The problem with generic sentiment analysis approaches, is that they inevitably use data on entire markets. This means that we have a binary classification (most often) that says:

“For these 25 top headlines, DOW goes up” -> 1

“For these 25 top headlines, DOW goes down” -> 0

But what would this mean for me, as an average investor? Certainly not much. For how long will this news depress the markets? For how long will it improve them? Are there any companies within this market that *aren’t* meaningfully affected by this news? What’s the actual relationship between a headline, and stock price X?

Shifting the Approach, to Prove That I Could

Can a relationship between headline y and stock price X even be proven, or are there too many moving pieces?

The first step in answering some of my own questions came after playing around with some models and generic data. I realized that this approach would never provide me with anything meaningful, and I needed to seek out more data. Specifically, I recognized that to get a better look at this problem, we need to take into consideration:

1. What company we’re looking at -> we need a label (i.e: AAPL or Apple)
2. What news has an effect on that company -> general news (i.e: “supply chains disrupted due to COVID) or specific news (i.e: “Apple CEO Steve Jobs died Tuesdsay)
3. What effect the news has -> on the day the article was published, did that company’s stock price go up, or down?

No data existed with these labels, so I set off to get my own. A partner on the project, Yin Chang, coded up some web scrapers to get market data and news. Using this, I was able to compile enough data points to at least get started. You can find the data here, if you’d like to do some playing around of your own.

The next step was to prove the concept — can we predict whether the stock will go up, down, or hold, based on relevant headlines about a company?

The importance in approaching this in stages cannot go understated. Sure, I could have compiled the headline and class data, the ticker symbol labels for the companies and loads of historical data and jumped right into a combined approach of sentiment analysis and prediction of how much a stock increased or decreased on the day of a headline, but it would be akin to just drawing a regression line on a scatterplot without knowing if the data is actually correlated. In other words, it would likely be garbage, even if I could get my metrics to look pretty.

In order to not just predict the stock market, but to predict it meaningfully, we have to take small steps towards analyzing and proving relationships between data points. There is a network of things which, when seen as a whole, can be evaluated and understood as impacting public sentiment about a company and therefore influencing its stock price. News headlines are just a small piece of this puzzle — but we first have to prove that they’re a piece of the puzzle.

At this stage, we’re working on proving this piece. The first step towards this was defining a model architecture that could perform sentiment analysis and prediction gracefully and accurately. My experiments with Naive Bayes and Random Forest Classifiers proved to me that for this portion they didn’t quite cut it. Based on my research in this area, I knew to turn to an old friend of mine, LSTM Recurrent Neural Networks.

For those who aren’t versed in data science language (because let’s be honest, it certainly wasn’t developed for beginners), an LSTM — Long Short Term Memory — is a type of neural network that “remembers” steps in its learning. To keep this section short and sweet and not too overwhelming, I’m going to oversimplify just a bit (a lot) and say that this makes it good at sentiment analysis.

Once a model was decided on, it was possible to take my toy approach as a start, and attempt to create a model that was good at this one thing. The initial approach answers our above question, and provides a classification based on headlines and Tweets about a company:

0 -> the stock will go down

1 -> the stock will go up

2 -> the stock will hold

With this initial approach, StockStalker was formed. An app prepared to inform average investors — not HFT’s — about the impact of daily news on the public sentiment about a company, and in turn, the way that company is likely to perform.

While our alpha version has a long way to go, (and lots more analysis to do!) we’ve come a long way in proving our initial concept: can we predict how a company is likely to perform based on relevant news? From here, there are a myriad of improvements to make, but we’re off to a good start.

To solve a problem everyone wants to solve, sometimes you have to look at things a little differently. This is why we created StockStalker — to predict the market, meaningfully.

Predicting the Stock Market — Meaningfully was originally published in Star Gazers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why Graph Theory Is Cooler than You Thought

Graph theory in machine learning, and how it’s changed the game

This is the first article in a four-part series on graph theory and graph neural networks.

What are Graphs?

Talk to a scientist in just about any discipline, and ask them the question — based on their discipline — “how does that stuff work?” You’ll likely find that there are systems and networks that you have to consider before you can really understand how any given thing works: whether that’s the human body, a food chain in an ecosystem, a chemical reaction, or a society as a whole. Without understanding the relationship between two animals in an ecosystem, two atoms in a molecule, or cells and tissues in our body, you just have a bunch of data: a list of cells, a readout of animals and what they eat, etc.

Traditional machine learning models often take data this way: they take lists or tables of data and do some stuff (the details of which depend on the algorithm being used as well as a few other parameters) to make predictions about a thing. But for some problems, there’s a better way.

Graphs are data structures that represent networks of, or relationships between, the data they contain. Typically, they’re represented as “nodes” and lines, or “edges”.

Figure 1, above is an example of an “undirected graph”, or, a graph in which data has a two-way relationship with other data. This means that each node in our graph, each representing a person in a social network, is connected with each other node. This might represent a networking site like LinkedIn, where once you’re connected with someone you’re connected with each other. An example of a directed social network graph might be Twitter, where users follow each other (and not always mutually).

The images which represent data (users) are called nodes or vertices, and the lines which connect them are called edges. These lines represent relationships between the vertices, and can be represented, as an “all or nothing relationship” (i.e: you’re following someone or you aren’t) or as a “weighted” relationship (i.e: a solid line can represent higher engagement between two users, while a dotted line can represent a weaker relationship or lower engagement).

At this point, it’s possible you’re feeling how I felt when I first was introduced to graphs and graph theory in a computer science class: bored and possibly slightly confused. The good news is, since we’ve covered some of the terminology that’s necessary to understand the good stuff, we can start to get into why graphs matter, and what makes them so cool.

So, what?

Graphs are already used for some pretty neat stuff in computer science: your Maps application, for example, is using graphs behind the scenes to store data about locations and the streets that connect them, and is using shortest-distance algorithms to find you the shortest route to your destination.

But it gets even better when we start to look at using graphs for machine learning. Because graphs demonstrate comprehensive relationships between pieces of data (as compared to ordered lists of data, or tensors which tell us little about the relationships between data points or features by themselves), we can use them to perform in-depth analysis and make predictions based on these relationships.

Graph Theory & Machine Learning — But How?

Before we get to reap the benefits of combining these graphs or networks we keep talking about with machine learning, we have to somehow represent our graph in a way that a computer — and then a machine learning algorithm — can understand.

Graphs can be represented traditionally in one of three basic ways:

1. An Adjacency Matrix

Adjacency matrices do… kind of just what they sound like they’d do. They represent connections, or edges, between different nodes using a matrix. We can look at an example to illustrate what this might look like:

Here, we can find an edge between two notes by finding where their labels intersect in our matrix. We can see, for example, that node 1 is not connected to itself, but that it is connected to node 2.

2. An Edge List

An edge list is another way to represent our network — or graph — in a way that’s computationally understandable. Here, we represent pairs of connected nodes within a list. You can see an example below:

3. An Adjacency List

Adjacency lists combine the above two approaches, representing a list of nodes, connected to a list of all of the nodes they’re directly connected to. To illustrate, let’s look at an example:

With the above three approaches, we’re able to tackle the difficulty of representing graphs computationally in our code. However, there are still some challenges when feeding graphs to machine learning models. Traditionally, deep learning models are good at handling data which takes up a fixed amount of space and is unidirectional. No matter how we represent them, graphs don’t take up a fixed amount of space in memory, and aren’t continuous, but rather, each node holds a reference to nodes it’s directly connected to.

There are some really fabulous ways of tackling these challenges, which I’ll dive deeper into in a later article. For now, I’ll leave you with a few resources to research on your own should you be interested, which are providing us with new ways to expand the problems machine learning is able to solve.

• Learn about Deep Learning on Graphs with Graph Convolutional Networks
• Learn about DeepWalk — a project finding another way to represent graphs for machine learning

Graph Theory and Machine Learning — What Can we Do With It?

Nothing exists in a vacuum, and understanding the interconnected networks of data that make up many of our scientific disciplines provides the exciting potential to answer so many questions — more than I can begin to wrap into this article.

What if we could better understand the human brain? What if we could make predictions about the effect of some stimulus or change on an ecosystem? Or, predict which compound is the most likely to create an effective drug?

The best part of everything we’ve just learned is that we can — and it isn’t simply a theoretical possibility, but something we’re doing right now!

Graph theory is already being used for:

1. Diagnostic modeling (predicting to a certain degree of certainty whether or not a patient has a specific diagnosis).
2. Helping with the diagnoses and treatment of patients with cancer.
3. Developing pharmaceuticals (medications).

4. Seeking to develop a theoretical synthesis between the theories of ecology and evolution.

How Graph Theory Makes it All Happen

Let’s dive a little deeper into these applications, so we can look at the utilization of graph theory within them in more detail.

Let’s use diagnostic models as an example. Specifically, I want to look at this example of network analysis being used for the diagnosis and identification of possible schizophrenia in patients:

Graph-based network analysis in schizophrenia

Using graphs to represent network analyses of the brain, neuroscientists are able to map key findings related to the diagnosis of schizophrenia. Given that there are certain markers for the onset of schizophrenia:

• less efficiently wired networks
• less local clustering
• less hierarchical organization

We could potentially evaluate these networks with a machine learning algorithm and predict the probability a patient has or will develop schizophrenia based on these markers.

Without the knowledge of these networks, in this example, this kind of analyses becomes an entirely different neurological analysis of the patient. The promising discoveries of these findings for schizophrenia has promising implications for the diagnosis and treatment of this disorder — possible early diagnosis and intervention that goes far beyond simply evaluating symptoms.

This is just an example, but is entirely illustrative of the benefits of graph theory in machine learning as it intersects with other disciplines.

The fact of the matter is, there is often much more to our data than we can represent in lists, data frames, or tensors alone. While there are ways to explore our data and present it in such a way that we can hypothesize relationships and even enable our algorithms to predict these, when we’re able to represent the connections between our data in a different way, we’re able to do more with the data that we have.

When we understand the ways in which things relate to one another, we understand them better: we can make more comprehensive predictions, and answer harder questions, with some pretty life changing results.

Want to learn more? Check out the rest of the series:

• Graph Convolutional Networks — Explained
• What Makes Graph Convolutional Networks Work?
• Classifying the PROTEINS Benchmark Dataset

Attributions:

1. Everything You Need to Know About Graph Theory and Machine Learning
2. Graph Theory & Machine Learning in Neuroscience
3. Knowing Your Neighbours — Machine Learning on Graphs
4. Network-based machine learning and graph theory algorithms for precision oncology

Why Graph Theory is Cooler Than you Thought was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

To the others like me, who don’t belong in tech. Others, who, like me, never saw a future for themselves — let alone one like this. Others who never saw themselves as seeking a career, an office job. Others like me who never thought they could change the world or build something new. Others like me who reminisce too often about childhood friends that will never have the chance to “turn things around”, who live with the weight of the privilege of walking away from a life that tried too hard to claim theirs.

Some of us are members of groups that are historically marginalized and underrepresented in tech to begin with. Some of us may fit in better. But we all carry the sense of being unable to relate to our peers. Hiding certain tattoos, not speaking about the past, we are dedicated to moving forward, to making something of ourselves. We don’t speak about survivor’s guilt, about how “turning life around” requires walking away from everything we once knew.

To the others like me, who aren’t sure if they can make it in this world of savvy children of entrepreneurs and venture capitalists, these clean-cut kids who had robotics clubs and programming courses in their high schools, I am here to tell you you can.

To the others like me, who consider what their childhood friends would say if they were here to see them now, who may be seen as stuck up by their own families: you do belong, if you want to.

To the others like me, who feel like traitors: you have this chance to do good for your loved ones. Too often it won’t feel like you’re doing anybody any good but yourself, but you are providing yourself the opportunity to make real change. What you do with these new found powers is up to you and you alone. There is a way to stay true to yourself and your background and provide a better life for yourself and your children than the one you grew up with.

To the others like me who know that speaking about class and race in tech is more than diversity initiatives and claiming that everyone belongs, we can change things. Understanding what a big leap it is for anyone to walk away from their neighborhood, their friends, understanding that culturally this is not a choice some can make, we can provide the foundation for “turning things around” to be seen as acceptable for our children and their peers. We can make making things better an option, we can bring resources back to our communities, and we can demonstrate our own worth: to ourselves, our peers, our mentors and instructors.

It’s fair to say that I have not always been the most humorous person. Growing up, I was teased quite a bit for being a bit too studious, wearing glasses, and knowing all the answers. As time went on, I got contact lenses, stopped raising my hand in class, stopped trying to be too friendly, and exited the traditional school environment as quickly as I could. Living in some interesting environments at the time, I straddled a strange line between being very young and professionally accomplished, and simultaneously needing (for a multitude of reasons) to stay close to peers engaging in behaviors that were anything but “professional”. When attempting to go to college the first two times, and integrate into these new social environments, I found myself in a position similar to high school: unable to relate to my scholarly peers on any kind of level beyond having classes together, and eventually leaving, feeling as though I didn’t belong, or didn’t deserve to be there.

Fast forward to today, after years of working to dramatically change my circumstances. Attempting to integrate is often clumsy, and this awkwardness I feel is most noticeable in my attempts at humor. Serious, deadpan, and dry, what was once seen as witty in prior social circles now often either goes unnoticed or is seen as intensely critical rather than funny. Observant to an extreme degree, cutting or sarcastic comments about things no one else may notice about a stranger once were praised and met with laughter, but now are met with comments like, “that’s a little mean.” In getting to know this new social group, I am learning how to adopt to more accepted forms of humor, rather than defensive humor. Things that my new peers find very funny and relatable leave me feeling a bit like I’ve been living under a rock for several years, but I’m slowly growing to appreciate this slightly friendlier and sweeter form of humor, shared in Zoom chats and Slack channels.