The CAT SET on the MAT: Cross Attention for Set Matching in Bipartite Hypergraphs

Given are two sets of disjoint nodes: left nodes $\mathcal{V}$ and right nodes $\mathcal{V}^{\prime}$; a simple bipartite hypergraph is defined as $\mathcal{H}_{sim}=(\mathcal{V},\mathcal{V}^{\prime},\mathcal{B})$, where $\mathcal{B}\subseteq 2^{\mathcal{V}\cup\mathcal{V}^{\prime}}$ such that there is at least one node from each node set in the elements of $\mathcal{B}$. However, we proceed further and define a different kind of bipartite hypergraph: a per-fixed²² 2 We call it “per-fixed” and not pre-fixed, since only left and right hyperedges are fixed. one wherein basically we fix the set of left and right hyperedges beforehand. Over the node sets $\mathcal{V}$ and $\mathcal{V}^{\prime}$, the set $\mathbb{F}:=2^{\mathcal{V}}\setminus\emptyset$ of potential left hyperedges and the set $\mathbb{F}^{\prime}:=2^{\mathcal{V}^{\prime}}\setminus\emptyset$ of potential right hyperedges could be noted. Once we fix the set of actual left hyperedges to be $\mathcal{F}\subseteq\mathbb{F}$ and actual right hyperedges to be $\mathcal{F}^{\prime}\subseteq\mathbb{F}^{\prime}$, we could define the set of potential bipartite hyperedges as:

A simple bipartite hypergraph is different from a usual (non-bipartite) hypergraph in that it has two disjoint sets of nodes ($\mathcal{V},\mathcal{V}^{\prime}$) instead of one. On the other hand, a per-fixed bipartite hypergraph is different from the two, since we fix the sets of left and right hyperedges ($\mathcal{F},\mathcal{F}^{\prime}$) beforehand, and are thence worried only about connections across them (as defined by $\mathcal{B}$). An important consequence of these facts is that we need not model the existence of the left or right hyperedges individually, but focus on the cross bipartite relations instead.

In the present work, we have set out to solve the problem of bipartite hyperedge prediction (BHP), which we define as follows:

It is to be noted that Definition 2 considers a per-fixed bipartite hypergraph and one could also define the BHP problem for a simple bipartite hypergraph (let’s call it simple-BHP) as well. The only difference between BHP and simple-BHP would be that while for the former, the set of possible left- and right-hyperedges is fixed even before the problem is defined, for the latter, any possible left- or right-hyperedge could be chosen as arbitrary subset choices from the respective node sets. Nevertheless, the difference shows up only for bipartite non-hyperedges: for simple-BHP, we have $\mathcal{B}\subseteq\mathbb{B}(\mathbb{F},\mathbb{F}^{\prime})$ and $\hat{\mathcal{B}}:=\mathbb{B}(\mathbb{F},\mathbb{F}^{\prime})\setminus\mathcal{B}$, but for per-fixed BHP (as per Definition 2, the set of observed left hyperedges and right hyperedges are fixed to $\mathcal{F}\subseteq\mathbb{F}$ and $\mathcal{F}^{\prime}\subseteq\mathbb{F}^{\prime}$ respectively, and then hyperedges and non-hypergedges are defined as $\mathcal{B}\subseteq\mathbb{B}(\mathcal{F},\mathcal{F}^{\prime})$ and $\hat{\mathcal{B}}:=\mathbb{B}(\mathcal{F},\mathcal{F}^{\prime})\setminus\mathcal{B}$. We will see how existing hyperege prediction algorithms apply only to simple-BHP and fail to cater the per-fixed BHP problem.

The foregoing lemma states an important result: that we could solve the BHP problem using SMP. Since we have an equivalent set matching for a given hypergraph, and that we could solve the BHCP problem using SMP, let us define a new problem called the Bipartite Hyperedge Set Matching Prediction (BHSMP) problem as follows:

Currently, the best known model to predict hyperedges is that of Hyper-SAGNN [45] (Figure 2), which uses (1) a self-attention framework to model information flow between nodes of a hyperedge, and (2) a static-vs-dynamic comparative technique to learn hyperedge formation. But the main reason why it does not deem fit for the bipartite hyperedge scenario is that it captures information flow between all the nodes of a usual (non-bipartite) hyperedge. And for a per-fixed bipartite hypergraph, the difference is subtler: “pay heed to the cross-connections, not the self-connections.” In other words, one need not model the occurrence of individual left and right hyperedges, and instead focus on the connection between them. That is, we need not predict left and right hyperedges individually, but only predict the connections between them, since the former is already fixed to be $\mathcal{F}$ and $\mathcal{F}^{\prime}$. For the symptoms-diseases case, the reason why Hyper-SAGNN does not apply to the BHP prediction problem is that we are interested in modeling/predicting new diagnoses involving observed symptom-sets and disease-sets. Paying simultaneous attention to symptoms, diseases and their connections (diagnoses) leads to a learning process that oscillates between the positive and negative classes. More about this would be discussed in Section 8.3.

Let us see what effect applying Hyper-SAGNN has on a bipartite hyperedge. Consider a bipartite hypergraph $\mathcal{H}=(\mathcal{V},\mathcal{V}^{\prime},\mathcal{F},\mathcal{F}^{\prime},\mathcal{B})$. Now, let us take two sets $f:=\{u_{1},u_{2},\ldots,u_{k}\}\subseteq\mathcal{V}$ and $f^{\prime}:=\{u^{\prime}_{1},u^{\prime}_{2},\ldots,u^{\prime}_{k^{\prime}}\}\subseteq\mathcal{V}^{\prime}$ that are “observed” to be hyperedges. Also given are the embeddings of each node: $\mathbf{x}_{i},\mathbf{x}^{\prime}_{i^{\prime}}\in\mathbb{R}^{d}$ for nodes $u_{i}$ and $u^{\prime}_{i^{\prime}}$ respectively ($1\leq i\leq k$, $1\leq i^{\prime}\leq k^{\prime}$). If $\mathbf{W}_{Q},\mathbf{W}_{K}\in\mathbb{R}^{d\times d_{K}}$, and $\mathbf{W}_{V}\in\mathbb{R}^{d\times d_{V}}$ denote weight matrices for the self-attention framework of Hyper-SAGNN, we have for the potential $(k+k^{\prime})$-sized hyperedge $f\cup f^{\prime}=\{u_{1},u_{2},\ldots,u_{k},u^{\prime}_{1},u^{\prime}_{2},\ldots,u^{\prime}_{k^{\prime}}\}$,

where $1\leq i,j\leq k$ and $1\leq i^{\prime},j^{\prime}\leq k^{\prime}$. These values are then normalized using the softmax function as follows:

where $z_{i}:=\left(\displaystyle\sum_{t=1}^{k}{\exp(a_{it})}+\sum_{t=1}^{k^{\prime}}{\exp(b_{it})}\right)$ and $z^{\prime}_{i^{\prime}}:=\left(\displaystyle\sum_{t=1}^{k^{\prime}}{\exp(a^{\prime}_{i^{\prime}t})}+\sum_{t=1}^{k}{\exp(c_{i^{\prime}t})}\right)$.

The dynamic embeddings of the hyperedge nodes would then be:

Going by the cross-attention paradigm, we introduce three extra attention parameters, amounting to parameters $\mathbf{W}_{Q},\mathbf{W}_{K},\mathbf{W}^{\prime}_{Q},\mathbf{W}^{\prime}_{K}\in\mathbb{R}^{d\times d_{K}}$ and $\mathbf{W}_{V},\mathbf{W}^{\prime}_{V}\in\mathbb{R}^{d\times d_{V}}$. We redefine $b$, $c$, $\beta$, and $\gamma$ for $1\leq i\leq k$ and $1\leq i^{\prime}\leq k^{\prime}$ as follows:

Now, the cross-attention dynamic embedding of the left and right hyperedge nodes would then be:

Finally, we have new embeddings $\bm{\delta}_{1},\bm{\delta}_{2},\ldots,\bm{\delta}_{k}$ of left nodes and $\bm{\delta}^{\prime}_{1},\bm{\delta}^{\prime}_{2},\ldots,\bm{\delta}^{\prime}_{k^{\prime}}$ of right nodes, all $d_{V}$-dimensional vectors.

We use altogether three variants of CATSETMAT: CATSETMAT-X (only CAT), CATSETMAT-SX (a pair of SAT followed by a CAT), and CATSETMAT-SXS (CATSETMAT-SX with another pair of SAT modules following CAT). Architecture diagrams for all the three variants have been shown in Figure 3.

As discussed in Section 6, we prepare four classes of baselines: node2vec-based, bipartite-graph-based, set-embedding-based and hyperedge-prediction-based, each of which has been described below.

1. 1.

   node2vec-based:
   n2v-cross-mean, n2v-full-mean, n2v-cross-min, n2v-full-min. More details could be found in Appendix B.3.1.

2. 2.

   Bipartite-graph-based: Since our hypergraph could be interpreted as a bipartite “graph” (Observation 1), we prepare some baselines on bipartite versions of two popular link prediction algorithms [23]: Common Neighbor (CN) [26] and Adamic Adar (AA) [1]. Please refer to Appendix B.3.2 for more details on how these features are computed for a bipartite hypergraph.

3. 3.

   Set-embedding-based (FSPool): For this, we first convert the node embeddings obtained from node2vec [11] (with dimension $16$, since it showed best results) of each of the left and right hyperedges into set-embeddings using FSPool [46], and then learn a simple classifier.

4. 4.

   Hyperedge-prediction-based (Hyper-SAGNN): We use the existing best performing algorithm Hyper-SAGNN [45] for hyperedge prediction, wherein we interpret our bipartite hypergraph as a usual one as per Observation 1. We use the same experiment settings as Zhang et al. [45] and their architecture as shown in Figure 2.

Although bipartite link prediction algorithms (bipartite-AA-min–bipartite-CN-avg) perform poorly with AUCs ranging from $43$–$55\%$ and $37$–$50\%$ for the first two datasets, for mag-acm-ak, some of them perform considerably better with AUCs going up to $64.5\%$ as well. These algorithms heavily depend on the interconnectivity between the left and right hyperedges, and it seems to be the best in the mag-acm-ak dataset. The next set of algorithms – the four node2vec-based ones – perform much better than the foregoing group, but still lag behind CATSETMAT by huge margins. Surprisingly, the bipartite graph based algorithms outperform the node2vec ones for the last dataset. This only shows that performances vary dramatically from dataset to dataset.

The next two algorithms – FSPool and Hyper-SAGNN, though being cutting-edge deep learning approaches in their respective fields (viz., set and hyperedge embeddings respectively), fail to capture the desired bipartite hypergraph structure. The main reason for this, as has been explained in Section 8.3, is that they struggle to learn a consistent representation of nodes since the same set of nodes get involved in bipartite hyperedges (positive class) as well as bipartite non-hyperedges (negative class) simultaneously. Moreover, Hyper-SAGNN also shows quite a high amount of deviation ($13\%$; as per Table 2), which is undesirable of any machine learning algorithm. But this only shows the huge dependence of Hyper-SAGNN on the data preparation process (i.e., negative-sampling and train-test split) that we perform five times. More about this has been discussed in Section 8.3, where we use the learning curves (Figures 5 and 6) to compare FSPool’s and Hyper-SAGNN’s respective behaviour w.r.t. CATSETMAT.

Finally, the last three algorithms – the ones we have proposed as part of this work – are the best performing ones among the lot. For tmdb-cc, we see that our best method is $13.56\%$ better than the group-wise next best (i.e., n2v-full-mean), which has an AUC of only $77.68\%$. Another of our methods (CATSETMAT-SXS) performs the best for tmdb-ck, with a much higher improvement of $33.06\%$ as compared to n2v-cross-mean, the group-wise next best. For both these datasets, we obtain quite high AUC values of around $84$–$88\%$, but the same is not true for mag-acm-ak, for which the best performance is limited to $76.62\%$, albeit given by our algorithm CATSETMAT-SX. Although this is a much lower score as compared to the other two datasets, but it is $18.65\%$ higher than bipartite-AA-avg, which is the best performing baseline. Barring our algorithms CATSETMAT-X, CATSETMAT-SX, and CATSETMAT-SXS, no other algorithm consistently performs well on all the three datasets. While at least one dataset deemed problematic for each of the baselines, these three algorithms remained consistent for all datasets. Moreover, no baseline touched the $80\%$ AUC mark except the CATSETMAT-based algorithms. We credit this success to the cross-attention paradigm, which focuses on the cross-links more than on the left and right hyperedges, thereby avoiding the positive-negative dilemma as explained in Section 8.3, where we make use of Figure 7 to explain the better performance of our algorithm.

Although the CAT module (see Figure 3) is our main contribution in this work, adding more SAT layers is expected to model the individual left- and right-hyperedges as well, since it might capture some inner domain-specific relational structure in each of them. We can clearly see this effect being illustrated in Table 2, wherein adding only one pair of SAT modules before CAT (in CATSETMAT-SX) boosts the performances of a purely CAT-based algorithm CATSETMAT-X by upto $3.92\%$, $12.59\%$, and $13.27\%$ for the three datasets respectively. While the addition of another pair of SAT modules (CATSETMAT-SXS) shows the potential of improving the AUC performance by an additional $1.93\%$ for tmdb-cc, but same is not true for the tmdb-ck and mag-acm-ak, where the performance instead drops by $2.57\%$ and $1.31\%$ respectively. This effect could be attributed to over-fitting due to an increase in model parameters. As a result of this, we conclude that addition of more number of SAT modules would not further boost the performance, and so, it should be limited to a single pair of SAT modules before CAT, as done in CATSETMAT-SX. The difference is also clear from the learning curves in Figure 7, where the relative epoch-by-epoch performance of each variant could be observed.

It is clear that our algorithm works well, and hence the baselines fail to capture the “bipartiteness” for hyperlink prediction. Figures 5, 6, and 7 depict the AUC learning curves for FSPool, Hyper-SAGNN, and CATSETMAT respectively on the three datasets. We could see that unlike CATSETMAT, the baselines FSPool and Hyper-SAGNN do not seem to converge, and keep oscillating around an AUC of 50%.

We reason for this behavior as follows. A careful look at the definition of a per-fixed bipartite hypergraph (Definition 1) would reveal a straightforward yet important fact: that left and right sets of hyperedges are fixed in advance. What makes this factor interesting is another definition: that of bipartite non-hyperedges (again, Definition 1) or the negative class, which ensures that each negative sample would be formed from the same fixed left and right hyperedges that formed samples from the positive class. What is striking to observe is that there is nothing that stops a left hyperedge $f$ that involves in forming a positive sample from getting involved in forming a negative sample as well. The same could also be said for a right hyperedge $f^{\prime}$. As a result, we observe that the positive and the negative samples are built from mostly (if not entirely) the same left and right hyperedges.

When an algorithm that treats the entire hyperedge $f\cup f^{\prime}$ as a whole is deployed to embed it, it keeps getting confused, as to how to train the parameters to account for both classes simultaneously. Thus, it does not settle for a stable output for both $f$ as well as $f^{\prime}$, owing to $f$ and $f^{\prime}$’s simultaneous association with both the classes. This is what algorithms like Hyper-SAGNN and FSPool suffer from, when it comes to a per-fixed bipartite hypergraph.

The attention parameters $\mathbf{W}_{Q}$, $\mathbf{W}_{K}$, and $\mathbf{W}_{V}$ that the Hyper-SAGNN formulation uses keep getting modified as per the positive and negative class labels, and since in our case, the bipartite hyperedges and bipartite non-hyperedges both use the same set of left and right hyperedges (owing to them being fixed), we perform an experiment that undoes this effect. For this, we change our negative sampling method to a “sized-random” [27] technique and use the following, we essentially change the “per-fixed” hypergraph (from Definition 1) to a “simple” hypergraph.

The exact procedure is as follows: Given a hypergraph $\mathcal{H}=(\mathcal{V},\mathcal{V}^{\prime},\mathcal{F},\mathcal{F}^{\prime},\mathcal{B})$:

1. 1.

   Initialize $\hat{\mathcal{B}}=\{\}$, the set of bipartite non-hyperedges.

2. 2.

   Pick a hyperedge $b=f\cup f^{\prime}\in\mathcal{B}$

3. 3.

   Pick a random set $\hat{f}\subseteq\mathcal{V}$ of left nodes such that $|\hat{f}|=|f|$, and a random set $\hat{f}^{\prime}\subseteq\mathcal{V}$ of right nodes such that $|\hat{f}^{\prime}|=|f^{\prime}|$. Define $\hat{b}:=\hat{f}\cup\hat{f}^{\prime}$.

4. 4.

   If $\hat{b}\in\mathcal{B}\cup\hat{\mathcal{B}}$, repeat Step 3. Else, add the new bipartite non-hyperedge $b$ to $\hat{\mathcal{B}}$.

5. 5.

   Repeat steps 2–4 until $|\hat{\mathcal{B}}|=5\cdot|\mathcal{B}|$.

Performing Hyper-SAGNN on this simpler hypergraph, surprisingly enough, makes the result totally different! Hyper-SAGNN now reports a test AUC of a whopping $98\%$ (see Figure 8) on mag-acm-ak, a dataset that was the toughest to handle as per Table 2. It is also advisable to see how well the learning curve boosts-up with this version of the data. It is clear from the foregoing arguments that there exists a positive-negative dilemma in usual hyperedge embedding approaches as far as bipartite hypergraphs are concerned. The unusually higher variance of $13\%$ observed in Table 2 for Hyper-SAGNN on mag-acm-ak could be easily attributed to this dilemma.