<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yunchao (Lance) Liu 刘运超 on Medium]]></title>
        <description><![CDATA[Stories by Yunchao (Lance) Liu 刘运超 on Medium]]></description>
        <link>https://medium.com/@YunchaoLanceLiu?source=rss-1f97428bc509------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3kmH_Zl2u6l33wn8tCmDtg.png</url>
            <title>Stories by Yunchao (Lance) Liu 刘运超 on Medium</title>
            <link>https://medium.com/@YunchaoLanceLiu?source=rss-1f97428bc509------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:31:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@YunchaoLanceLiu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[MolKGNN: Extending Convolution To Molecules]]></title>
            <link>https://medium.com/data-science/molkgnn-extending-convolution-to-molecules-b94a4d51f39f?source=rss-1f97428bc509------2</link>
            <guid isPermaLink="false">https://medium.com/p/b94a4d51f39f</guid>
            <category><![CDATA[drug-discovery]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[graph-neural-networks]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Yunchao (Lance) Liu 刘运超]]></dc:creator>
            <pubDate>Mon, 29 Aug 2022 21:02:54 GMT</pubDate>
            <atom:updated>2023-06-28T22:14:10.516Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understand <strong>MolKGNN, an interpretable GNN tailored for drug discovery accepted at AAAI2023</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8tyuKyuUYSLt7O80VHOQZw.png" /><figcaption>Figure 1. An analogy of (A) image convolution and (B) proposed molecular convolution. <a href="https://www.biorxiv.org/content/10.1101/2022.08.24.505155v1">Image from the original paper.</a></figcaption></figure><p>This blog introduces our latest model <strong>Mol</strong>ecular <strong>K</strong>ernel <strong>G</strong>raph <strong>N</strong>eural <strong>N</strong>etwork (MolKGNN) from the paper<br><a href="https://ojs.aaai.org/index.php/AAAI/article/view/26679"><strong><em>Interpretable Chirality-Aware Graph Neural Network for Quantitative Structure Activity Relationship Modeling in Drug Discovery</em></strong></a></p><p>This work is accepted at <a href="https://aaai-23.aaai.org/">AAAI2023</a> (oral and poster presentation), and the <a href="https://logconference.org/">Learning on Graph Conference</a> (non-archival poster)</p><h3><strong>Introduction:</strong></h3><p>Elucidating molecular structures and their pharmacological activity has been a long-standing problem in the history of drug discovery. In 1859, the German chemist Carl Stahlschmidt demonstrated that the addition of methyl iodide to strychnine and brucine apparently destroyed their physiological action [1]. His work prompted two Scottish scientists, Alxander Crum Brown (1838–1922) and Thomas R. Fraser (1841–1920) to conduct experiments on a series of chemical compounds. Those experiments helped them confirm that there exists a structure activity relationship (SAR) [1]. At that time, there was considerable optimism that a general law describing the relationship between molecular structure and their pharmacological activity would be discovered. Several mathematical/statistical and machine learning methods have been developed since then, trying to predict this relationship. This process is known as quantitative structure activity relationship (QSAR) modeling. Examples of QSAR attempts include multiple linear regression, partial least squares, discriminant analysis, decision tree, genetic algorithms and etc [2]. High hopes are held out for the therapeutic applications of QSAR studies. However, till today, predicting the biological activity of small molecules remains a challenging task.</p><p><strong>Task:</strong></p><p>QSAR modeling, i.e., predict a binary label 0 (inactive) or 1 (active) from the molecule structure. A molecule is represented as a graph, where nodes are atoms and bonds are edges.</p><p>Highlights of the paper:</p><ul><li>This paper introduces a novel SE(3)/conformation-invariance model named <strong>MolKGNN</strong>, tailored for QSAR task in drug discovery.</li><li><strong>MolKGNN </strong>features in its novel molecular convolution, lightweight chirality calculation, and its interpretability</li><li>A realistic drug discovery experiment demonstrates the pragmatic value of the proposed <strong>MolKGNN</strong></li></ul><h3><strong>MolKGNN Model Architecture</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*pUrFIntVidykzxV_CJ15dw.png" /><figcaption>Figure 2. (A) In 2D image convolution, a higher convolution value indicates a higher visual similarity pattern. (B) The image kernels offer the benefit of interpretability. Image by the author.</figcaption></figure><p>MolKGNN draws its inspiration from the 2D image convolution (Figure 2). In 2D image, convolution operation can be regarded as calculating the similarity between the image patch and the image kernel. Larger output values indicate higher visual similarity patterns such as edges, strips, and curves [3]. However, the 2D image convolution cannot be readily extended to 3D molecular graphs due to their irregularity. Therefore, a new molecular convolution is designed to convolute between a molecular neighborhood (1-hop neighbors) and a molecular kernel (1-hop), similar to an image patch convoluted with an image kernel. The molecular convolution has the following properties:</p><ul><li>Like image convolution, the more similar a molecular neighborhood is to the molecular kernel, the higher the molecular convolution value should be.</li><li>Unlike image convolution, the molecular convolution should be rotation-invariant</li><li>The molecular kernel could provide interpretability.</li></ul><p>The next question is, how do we design the molecular convolution to have the above-mentioned properties?</p><p>The similarity between a molecular neighborhood <em>S </em>and a kernel <em>S’ </em>is quantified by a similarity score <em>ϕ(S, S’). </em>This score is a combination of three subscores <em>ϕ_</em>cs, <em>ϕ_</em>ns, and <em>ϕ_</em>es, which captures the central similarity, neighborhood node similarity, and edge similarity, respectively. The calculations of these are illustrated below (notations are fully described at the end):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*dJrqX4bxiOsFdFD9qLYmvg.png" /><figcaption>Figure 3. Illustration of the similarity score calculation. Image from the <a href="https://www.biorxiv.org/content/10.1101/2022.08.24.505155v1">original paper</a>.</figcaption></figure><p>A similarity score between the neighborhood subgraph and kernel is calculated from the combination of three subscores. The subscores quantify the similarities of different aspects between the neighborhood subgraph and the kernel. The central similarity subscore<em> ϕ</em>_cs captures the similarity between the center node attributes (v and v’). The neighborhood node similarity subscore<em> ϕ</em>_ns captures the similarities of attributes of neighboring nodes (u1 and u1&#39;, u2 and u2&#39;, u3 and u3&#39;). The Neighborhood edge similarity subscore <em>ϕ</em>_es captures the similarities of attributes of neighboring edges.</p><p>Since node/edge attributes are vectors, a similarity function sim(⋅) is used to calculate the vector similarity. We use <a href="https://en.wikipedia.org/wiki/Cosine_similarity">cosine similarity</a> in our implementation. <em>ϕ_cs </em>is calculated between attributes of center node v and v’ as shown in Figure 3(a).</p><p>Note that to calculate <em>ϕ</em>_ns, there are multiple ways of matching the neighbors, and each matching gives us a score. We enumerate all matchings and define the one that gives the highest score to be the optimal neighbor matching χ* (e.g, u1 and u2&#39;, u2 and u3&#39;, u3 and u1&#39; in the right of Figure 3(b)). This enumeration of matchings is feasible because there are at most four neighbors in drug-like molecules.</p><p>For <em>ϕ_</em>es<em>, b</em>ecause the neighboring nodes have a one-to-one correspondence to neighboring edges, we can find the optimal edge matching χ^{e,*} based on the optimal neighbor matching χ* (Figure 3(c)).</p><p>Next, we would like to integrate chirality calculation. The idea is to use the kernel as an anchor for reference. The molecular neighborhood is then compared with the kernel to see if it is of the same neighbor order as the kernel or not. We leverage the tetrahedron volume calculation in vector form to capture the neighbor ordering [4]. See the illustration below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*CZRTQdHNx9O2raGJF4Rsxg.png" /><figcaption>Figure 4. Illustration of chirality calculation. Image from the <a href="https://www.biorxiv.org/content/10.1101/2022.08.24.505155v1">original paper</a>.</figcaption></figure><p>In neighborhood 1, three vectors a1, b1, and c1 are made from arbitrarily-chosen neighbors without loss of generality. The tetrahedral volume can be calculated as 1/6* a1×b1⋅c1. Note that this volume can be of positive or negative signs, which indicate the volume direction. The same calculation can be carried out in the kernel for the corresponding neighbor nodes in the optimal matching. If the sign of the tetrahedron volume of the neighborhood1 is the same as the one in the kernel, we know that they have the same neighboring node ordering. In the case of the neighborhood2 above, its volume is of different signs and we know neighborhood2 has different neighbor nodes order. Also, note that the constant 1/6 is trivial in the sign determination and can be omitted in actual implementation.</p><p>Finally, we leverage the message passing neural network (MPNN) framework [5] to get a larger receptive field. The idea is to replace the traditional aggregation of attributes of neighboring nodes, with the aggregation of similarities between a molecular neighborhood and kernels. See the illustration below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*u3Z8PEhzG1RqSP9mA0o5gg.png" /><figcaption>Figure 5. Overview of the MolKGNN model. The key idea is to replace the traditional aggregation of attributes of neighboring nodes, with the similarities between a molecular neighborhood and a set of kernels. Image from the <a href="https://www.biorxiv.org/content/10.1101/2022.08.24.505155v1">original paper</a>.</figcaption></figure><p>The final atom embedding can be learned by repeating the process of calculating molecular convolution and propagating messages several times. The final molecular embedding can be obtained by <a href="https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#pooling-layers">various pooling techniques</a>. The downstream prediction can be made by attaching a classifier such as the Multi-layer Perception (MLP) on top of the molecular embedding.</p><h3><strong>Experiments</strong></h3><p>Realistic datasets from drug discovery are used to benchmark MolKGNN [6, 7]. These datasets are carefully curated to remove false positive signals frequently seen in drug discovery campaigns. Dataset statistics can be seen below and they are available on <a href="https://figshare.com/articles/dataset/Well-curated_QSAR_datasets_for_diverse_protein_targets/20539893">FigShare</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zGMzmDzJ-rKWz6vpC1vTgQ.png" /><figcaption>Table 1. Dataset statistics. They feature in the large size, highly-imbalanced label distribution and diverse protein targets. Image from the <a href="https://www.biorxiv.org/content/10.1101/2022.08.24.505155v1">original paper</a>.</figcaption></figure><p>Two tables below show the results. The logAUC_[0.001, 0.1] is used here to bias the compounds with high predicted scores. This corresponds to the real-world drug discovery scenario: only those predicted with high scores for activity are going to be purchased or synthesized. Hence it’s of more interest to see the model performance on those compounds instead of the general model performance. See the <a href="https://www.biorxiv.org/content/10.1101/2022.08.24.505155v1">original paper</a> for more results and the experiment details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*HPbOxy3et9cd79MY0rlJcA.png" /><figcaption>Figure 7. Result table. Image from the <a href="https://www.biorxiv.org/content/10.1101/2022.08.24.505155v1">original paper</a>.</figcaption></figure><p>Moreover, <strong>MolKGNN</strong> is able to capture patterns that align with domain knowledge. Below is an example of the patterns translated from the learned kernel. This pattern is also known as an important structure in medicinal chemistry called <a href="https://en.wikipedia.org/wiki/Trifluoromethyl_group">trifluoromethyl groups</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UzDvPIYPCym-18ZKjpdYxQ.png" /><figcaption>Figure 6. A learned kernel shows an important substructure pattern, known as the trifluoromethyl group. Image from the <a href="https://www.biorxiv.org/content/10.1101/2022.08.24.505155v1">original paper</a>.</figcaption></figure><p>In addition, the experiment on the expressiveness of <strong>MolKGNN</strong> confirms its able to distinguish chiral molecules. The CHIRAL1 dataset [8] is used. that contains 102,389 enantiomer pairs for a single 1,3-icyclohexylpropane skeletal scaffold with one chiral center. The data is labeled as R or S stereocenter and we use accuracy to evaluate the performance. For comparison, we use GCN [9] and a modified version of our model, MolKGNN-NoChi, that removes the chirality calculation module. Our experiments observed GCN and <strong>MolKGNN-NoChi </strong>achieve 50% accuracy while <strong>MolKGNN</strong> achieves nearly 100%, which empirically demonstrates our proposed method’s ability to distinguish chiral molecules.</p><h3><strong>Ablation Studies</strong></h3><p>Details of this study can be found in the <a href="https://www.biorxiv.org/content/10.1101/2022.08.24.505155v1">original paper</a>.</p><p><strong>Component of <em>ϕ</em>(S, S′)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jy7_jPLB2WjI8AMphf1Ofw.png" /><figcaption>Figure 7. the alation study results for components of <em>φ(S, S’). </em>Image from the <a href="https://www.biorxiv.org/content/10.1101/2022.08.24.505155v1">original paper</a>.</figcaption></figure><p>The removal of any of the components has a negative impact on logAUC[0.001,0.1]. In fact, the impact is bigger for logAUC[0.001,0.1] than AUC in terms of the percentage of performance change. Note that in some cases such as the removal of φes, there is an increase in performance according to AUC, but this would significantly hinder the logAUC[0.001,0.1] metric.</p><p><strong>Kernel Number</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/1*XzGfn-iTlGj-Iu4s_z2Xew.png" /><figcaption>Figure 8. Performance for different kernel numbers. Image from the <a href="https://www.biorxiv.org/content/10.1101/2022.08.24.505155v1">original paper</a>.</figcaption></figure><p>When the number of kernels is too small (&lt; 5), it greatly impacts the performance. However, once it is large enough to a certain point, a larger number of kernels has little impact on the performance.</p><h3>Discussion of Computation Complexity</h3><p>It may seem to be formidable to enumerate all possible matchings described above. However, most nodes only have one neighbor (e.g., hydrogen, fluorine, chlorine, bromine and iodine). Take AID 1798 for example, 49.03%, 6.12%, 31.08% and 13.77% nodes are with one, two, three<br>and four neighbors among all nodes, respectively. For nodes with four neighbors, only 12 out of 24 matchings need to be enumerated because of chirality [8]. Since the adjacency matrix of molecular graphs are sparse, most GNNs incur a time complexity of O(|E|). And as analyzed above, the permutation is bounded by up to four neighbors (12 matchings). Thus, finding the optimal matching has a time complexity of O(1). The calculation of molecular convolution is linear to the number of K kernels and hence has a time complexity of O(K). Overall, our method takes a computation time of O(|E|K)</p><h3><strong>Conclusion</strong></h3><p>In this work, we introduce a new GNN model named <strong>MolKGNN</strong> to address the QSAR modeling problem. MolKGNN utilizes a newly-designed molecular convolution, where a molecular neighborhood is compared with a molecular kernel to output a similarity score, which is used as the new atom embedding for the next layer. Comprehensive benchmarking is conducted to evaluate MolKGNN. Well-curated datasets that consist of experimental HTS data from diverse protein target classes are used for the evaluation. The datasets are highly imbalanced, which highlights<br>the scarcity of positive signals in this real-world problem. For evaluation, not only do we use traditional AUC, but also logAUC[0.001,0.1], to evaluate the method’s performance under a high cutoff condition. This high-cutoff condition is typical of real-world applications and demonstrates the ap-<br>plicability of MolKGNN to drug discovery. Moreover, this paper provides a theoretical justification and experimental demonstration that MolKGNN is able to distinguish chiral molecules while providing interpretability for its results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*rs2Dt77LlM8jMEfGQ6D0wg.png" /><figcaption>Oral and poster presentation of this work at AAAI2023 in Washington DC</figcaption></figure><p><strong>References:</strong></p><p>[1] Parascandola, John. “Structure-Activity Relationships — The Early Mirage.” <em>Pharmacy in History</em> 13.1 (1971): 3–10.</p><p>[2] Wermuth, Camille Georges, ed. <em>The practice of medicinal chemistry</em>. Academic Press, 2011.</p><p>[3] Lin, Zhi-Hao, Sheng-Yu Huang, and Yu-Chiang Frank Wang. “Learning of 3d graph convolution networks for point cloud analysis.” <em>IEEE Transactions on Pattern Analysis and Machine Intelligence</em> 44.8 (2021): 4212–4224.</p><p>[4] Sliwoski, Gregory, et al. “BCL:: EMAS — enantioselective molecular asymmetry descriptor for 3D-QSAR.” <em>Molecules</em> 17.8 (2012): 9971–9989.</p><p>[5] Gilmer, Justin, et al. “Neural message passing for quantum chemistry.” <em>International conference on machine learning</em>. PMLR, 2017.</p><p>[6] Butkiewicz, Mariusz, et al. “Benchmarking ligand-based virtual High-Throughput Screening with the PubChem database.” <em>Molecules</em> 18.1 (2013): 735–756.</p><p>[7] Butkiewicz, Mariusz, et al. “High-throughput screening assay datasets from the pubchem database.” <em>Chemical informatics (Wilmington, Del.)</em> 3.1 (2017).</p><p>[8] Pattanaik, Lagnajit, et al. “Message passing networks for molecules with tetrahedral chirality.” <em>arXiv preprint arXiv:2012.00094</em> (2020).</p><p>[9] Kipf, Thomas N., and Max Welling. “Semi-supervised classification with graph convolutional networks.” <em>arXiv preprint arXiv:1609.02907</em> (2016).</p><p>This work is co-authored by <a href="https://yuwvandy.github.io/">Yu Wang</a>, <a href="https://www.linkedin.com/in/oanhvu/">Oanh Vu</a>, <a href="https://www.linkedin.com/in/rocco-moretti-547364110/">Rocco Moretti</a>, <a href="http://www.vuse.vanderbilt.edu/~bobbyb/">Bobby Bodenheimer</a>, <a href="https://www.linkedin.com/in/jens-meiler-4b635339/">Jens Meiler</a>, and <a href="https://tylersnetwork.github.io/">Tyler Derr</a>.</p><p>Yunchao “Lance” Liu acknowledges support fromthe NVIDIA Academic Hardware Grant Program.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*M8ANYItGmDInSkOfOF5qwg.png" /><figcaption>Table 1. Notations. Made by the author.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b94a4d51f39f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/molkgnn-extending-convolution-to-molecules-b94a4d51f39f">MolKGNN: Extending Convolution To Molecules</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Read Paper With Me: Bootstrap Your Own Latent: A New Approach To Self-Supervised Learning]]></title>
            <link>https://medium.com/@YunchaoLanceLiu/read-paper-with-me-bootstrap-your-own-latent-a-new-approach-to-self-supervised-learning-e6580ce8dae5?source=rss-1f97428bc509------2</link>
            <guid isPermaLink="false">https://medium.com/p/e6580ce8dae5</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Yunchao (Lance) Liu 刘运超]]></dc:creator>
            <pubDate>Sat, 29 Jan 2022 02:23:15 GMT</pubDate>
            <atom:updated>2022-01-29T15:39:30.965Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<h3>Read Papers With Me: Bootstrap Your Own Latent: A New Approach To Self-Supervised Learning</h3><h4>Interpretation of the BYOL, a self-supervised learning method</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EwArOBmDMX1KpXL6Tpln1A.png" /></figure><ul><li>This blog introduces<em> the </em><a href="https://arxiv.org/pdf/2006.07733.pdf"><em>Bootstrap Your Own Latent</em> (BYOL)</a> method, a self-supervised learning method proposed by DeepMind and Imperial London</li></ul><p><strong>A glance at the paper:</strong></p><ul><li>BYOL contains two networks with the same architecture, but different parameters.</li><li>BYOL does NOT need negative pairs, which most contrastive learning methods do, such as <a href="https://arxiv.org/abs/2002.05709">SimCLR</a></li></ul><p><strong>Background:</strong></p><p><a href="https://towardsdatascience.com/understanding-contrastive-learning-d5b19fd96607#:~:text=Contrastive%20learning%20is%20a%20machine,points%20are%20similar%20or%20different%20.&amp;text=In%20essence%2C%20contrastive%20learning%20allows,to%20do%20the%20same%20thing.">Contrastive learning (CL)</a> currently achieves state-of-the-art performance in self-supervised learning. In contrastive learning, views created from the same images are named positive pairs, while those from different images are named negative pairs.</p><p>However, there can be an infinite number of negative pairs and CL requires a significant number of them to ensure its performance. This work hence proposes a new framework called BYOL to eliminate the need for negative pairs.</p><p><strong>Method:</strong></p><ol><li>Overview</li></ol><p>There are two networks in this framework. One is named the online model and the other is named the target model. The online model is parameterized by θ and the target model is parameterized by ξ. Two models have dynamic behaviors:</p><ul><li>The target model updates its parameters ξ via calculating an <a href="https://en.wikipedia.org/wiki/Moving_average">exponential moving average (EMA)</a> of θ.</li><li>The online model updates its parameters θ via learning from the target model’s parameters ξ.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*pJZkb3CzBchEjci9Ltbg6Q.png" /><figcaption>The dynamics between the online and target models. Image from the author.</figcaption></figure><p>2. Parameter Updates</p><p>I will explain more on how each one gets updated below:</p><ul><li>ξ is updated by θ: This is done by calculating the exponential moving average (EMA), which is defined below:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*I-8I9MoLo0mWClIAhIcx0Q.png" /><figcaption>The formula of Exponential Moving Average (EMA). τ is a target decay rate and τ ∈ [0,1]. Suppose Compared with getting a plain average of θ over a certain time series, EMA assigned larger weights to more recent θ. This can be seen in the fact that if you expand the ξ on the right side, τ becomes τⁿ (n is the timestep). As τ is less one, τⁿ will become smaller, meaning its importance is decaying. Image from the <a href="https://arxiv.org/pdf/2006.07733.pdf">original paper</a>.</figcaption></figure><ul><li>θ is updated by ξ: This is done by optimizing an objective function. The objective function is a mean squared error between a normalized prediction q_θ_bar and a target z’_ξ_bar:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*VZb5NkWHyGDKxmKo1DRHeg.png" /><figcaption>Image from the <a href="https://arxiv.org/pdf/2006.07733.pdf">original paper</a>.</figcaption></figure><ul><li>q_θ_bar is the <a href="https://mathworld.wolfram.com/L2-Norm.html">L2 normalization</a> of q_θ:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/1*_mBpwSPXM6bDA4300-QY6Q.png" /><figcaption>Image from the <a href="https://arxiv.org/pdf/2006.07733.pdf">original paper</a>.</figcaption></figure><ul><li>z’_ξ_bar is the L2 normalization of z’_ξ:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/186/1*p9wO4K-o4MSZGpZaz3KAcQ.png" /><figcaption>Image from the <a href="https://arxiv.org/pdf/2006.07733.pdf">original paper</a>.</figcaption></figure><p>3. Construction of target z’_ξ and prediction of q_θ</p><p>This is the core part of this method. In the online model, there are three stages: encoder f_θ, projector g_θ, and predictor q_θ.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rXKcAigRajsZnIvaIsmb7w.png" /><figcaption>The online model is made from an encoder, a projector, and a predictor. The online model predicts the projection from the target via the objective function mentioned above. sg stands for stop gradient, meaning that the gradient does NOT backpropagate in the target model because the target model is updated using EMA. After training, everything except f_θ is discarded. Image adapted from the <a href="https://arxiv.org/pdf/2006.07733.pdf">original paper</a>.</figcaption></figure><p>The whole structure is likely built upon the <a href="https://arxiv.org/pdf/2002.05709.pdf">SimCLR</a> framework so I will briefly introduce it. In SimCLR, the input image is transformed by t and t’ to generate two augmented views, and then pass through an encoder f(⋅) and a projector g(⋅) to get a projected representation. The projected representations zᵢ and zⱼ are then contrasted to maximize their agreement, which is found to lead to better performance than maximizing agreements between hᵢ and hⱼ directly. Negative pairs are constructed by using views from a different input image. More details on the <a href="https://arxiv.org/pdf/2002.05709.pdf">SimCLR</a> paper.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/282/1*pF-SHPGDWxyMeA1aWLm2LQ.png" /><figcaption>Overview of the <a href="https://arxiv.org/pdf/2002.05709.pdf">SimCLR</a> framework</figcaption></figure><p>The difference, however, lies in the factor that in BYOL, two views are generated through different encoders f_θ and f_ξ. These two are of the same architecture but different parameters. Also, in BYOL, there is a predictor and a target network. As you can see, in BYOL, there is no need to have negative pairs.</p><p><strong>Results:</strong></p><p>The BYOL model outperforms SimCLR.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*12kNCDgZA8KonND3ea_FDw.png" /><figcaption>Image from the <a href="https://arxiv.org/pdf/2006.07733.pdf">original paper</a>.</figcaption></figure><p><strong>Conclusion:</strong></p><p>The BYOL provides a new way of self-supervised learning that does not need negative pairs.</p><p>BYOL has two models, with the same architecture but different parameters. The online model updates its parameters by optimizing an objective function. The target model updates its parameters by calculating an exponential moving average.</p><p><strong>Personal Remarks:</strong></p><ol><li>Constructing enough negative pairs is an important step to ensure the performance of contrastive learning. Otherwise, a problem called <a href="https://arxiv.org/abs/2110.09348">collapsed representation</a> may happen. This work opens a new avenue for self-supervised learning without using negative pairs.</li><li>The EMA is used to take a previous θ into consideration, biasing newer ones. This technique has been used in other methods such as the <a href="https://towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c">Adam</a> optimizer. The original paper also has an ablation study on the τ of the EMA.</li></ol><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e6580ce8dae5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Meta AI Published Data2Vec. A New Milestone for Learning Across Modalities]]></title>
            <link>https://medium.com/@YunchaoLanceLiu/interpretation-of-data2vec-a-general-framework-for-self-supervised-learning-in-speech-vision-and-43299fef41d4?source=rss-1f97428bc509------2</link>
            <guid isPermaLink="false">https://medium.com/p/43299fef41d4</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Yunchao (Lance) Liu 刘运超]]></dc:creator>
            <pubDate>Sat, 22 Jan 2022 04:34:03 GMT</pubDate>
            <atom:updated>2022-02-20T03:08:01.312Z</atom:updated>
            <content:encoded><![CDATA[<h3>Read Paper with Me: Data2Vec: A General Framework for Self-supervised Learning in Speech, Vision and Language</h3><h4>Interpretation of data2vec, a model for learning across modalities</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*sLst-IqlHS01QXHvNwoh-w.png" /></figure><ul><li>This blog introduces a new paper on self-supervised learning from Meta AI: <a href="https://ai.facebook.com/research/data2vec-a-general-framework-for-self-supervised-learning-in-speech-vision-and-language">data2vec: A General Framework for Self-supervised Learning in Speech, Vision, and Language</a></li><li>If you have a hard time understanding this blog, it’s recommended to read <a href="https://medium.com/@YunchaoLanceLiu/read-paper-with-me-bootstrap-your-own-latent-a-new-approach-to-self-supervised-learning-e6580ce8dae5">my blog on BYOL</a> first. This work is built upon BYOL.</li></ul><p><strong>Background: </strong>Traditional machine learning relies on labeled data for training. However, the annotation of data is costly and laborious. In a larger sense, it’s impossible to label all data in the world. Recently, s<a href="https://viso.ai/deep-learning/self-supervised-learning-for-computer-vision/">elf-supervised learning (SSL)</a> attracts attention as a promising method to solve this problem and for <a href="https://ai.facebook.com/blog/self-supervised-learning-the-dark-matter-of-intelligence/">approximating the common sense in AI systems,</a> eventually achieving Artificial General Intelligence (AGI).</p><p>Instead of using supervised signals from labeled data, SSL exploits the relationship between data. However, different modalities typically need different models for SSL. This paper hence proposes a unified framework called data2vec (the name is a play on another famous algorithm <a href="https://en.wikipedia.org/wiki/Word2vec">word2vec</a>) for SSL in three modalities: images (in the original paper, <em>computer vision </em>refers to this), texts (sometimes referred to as <em>language</em> in the original paper), speech. Data2vec achieves state-of-the-art (SOTA) results for all three modalities.</p><p>Before we move on, it’s important to notice the difference between these three modalities: Images are 2D structured data. Texts are discrete 1D data and speech is continuous 1D data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*FazEMTTCkY7TqdisLCf-BA.png" /><figcaption>Examples of three modalities: Image, speech, language (consistently called<em> texts</em> in this blog). Image from <a href="https://ai.facebook.com/research/data2vec-a-general-framework-for-self-supervised-learning-in-speech-vision-and-language">the original paper</a>.</figcaption></figure><p>*This work is NOT for training in <a href="https://towardsdatascience.com/multimodal-deep-learning-ce7d1d994f4">multimodal learning</a> (i.e., during training, only one of the three modalities is passed as an input, not a mixture of them) but can be helpful for multimodal learning</p><p><strong>Method Overview: </strong>data2vec uses one model but has two modes: the teacher mode and the student mode. In each time step, the student mode of data2vect will try to learn from the teacher mode and update the model parameters</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*NBU0sfo0sAfTcExxRn6yEQ.png" /><figcaption>In each time step, data2vec in student mode learns from its teacher mode and updates its parameters. Image by the author.</figcaption></figure><p>Specifically, the teacher mode generates representation from a given sample (i.e. image, speech, text). A masked version of the same sample is passed to the student mode. Learning happens by minimizing the objective function between the student’s prediction of a target that is constructed by teachers’ parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*414n_QE9vgkgpRhGgbNRHg.png" /><figcaption>For a certain timestep, the input of the teacher is a full sample (i.e., unchanged image, speech, text) and that of the student is a masked one. The student learns from the teacher by predicting a target constructed using the last K layers of the teacher (shown in blue). Image from <a href="https://ai.facebook.com/research/data2vec-a-general-framework-for-self-supervised-learning-in-speech-vision-and-language">the original paper</a>.</figcaption></figure><p><strong>Method:</strong></p><ol><li>Model Architecture</li></ol><p>The samples are turned into representations by the data2vec model. A representation of the full sample comes from the teacher, and a representation of the masked sample comes from the student.</p><p>These are <em>contextualized representations, </em>meaning that they encode particular timestep as well as other information from the sample, due to the use of self-attention in the Transformer. This is the major difference between this work and previous works, which lack contextual information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oSH8HhWPxnT-FTgIsLjKbQ.png" /><figcaption>Samples turn into representations by passing through the data2vec model. First of all, samples are embedded into tokens. If in student mode, then a masking method is applied to the tokens before passing them to a <a href="https://arxiv.org/abs/1706.03762">Transformer</a>. If in teacher mode, the tokens are directly passed to the Transformer. Image by the author.</figcaption></figure><p>2. Target Construction</p><p>An exponential moving average (<a href="https://en.wikipedia.org/wiki/Moving_average#%7B%7Banchor%7Cexponential_moving_average%7CExponential%7D%7DExponential_moving_average">EMA</a> [7, 8], which places greater weights for more recent data points) is used to update the teacher’s parameters. The τ here is linearly increasing, and allows more frequent updates of the teacher at the beginning of training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/229/1*dJpyiOOkPrT0v2QpYBtFVw.png" /><figcaption>θ is the student-mode model parameters. Δ the teacher-mode model parameters and is updated using the exponential moving average (<a href="https://en.wikipedia.org/wiki/Moving_average#%7B%7Banchor%7Cexponential_moving_average%7CExponential%7D%7DExponential_moving_average">EMA</a> [7, 8]) of θ. τ linearly increases from τ₀ to a target value τₑ for the first τₙ updates and then stays constant for the remaining of the training. Image from <a href="https://ai.facebook.com/research/data2vec-a-general-framework-for-self-supervised-learning-in-speech-vision-and-language">the original paper</a>.</figcaption></figure><p>The target is then constructed by using the top K (closer to the output) blocks of the transformer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/1*v3HA16lcKmWCfnHmquDbOg.png" /><figcaption>L is the total blocks in the network. aₜˡ-circumflex (cannot type the caret symbol) is obtained by a normalization from aₜˡ. aₜˡ denotes the output of block <em>l</em> at timestep t. Image from <a href="https://ai.facebook.com/research/data2vec-a-general-framework-for-self-supervised-learning-in-speech-vision-and-language">the original paper</a>.</figcaption></figure><p>3. Objective Function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G23_truB_bBusqWa28jS8Q.png" /><figcaption>yₜ is the target. fₜ(x) is the prediction. β controls the transition from a squared loss to an L₁ loss. When the gap is large, L₁ is used to make the loss less sensitive to outliers. Image from <a href="https://ai.facebook.com/research/data2vec-a-general-framework-for-self-supervised-learning-in-speech-vision-and-language">the original paper</a>.</figcaption></figure><p>In a word, the teacher and the student have a dynamic behavior: student’s parameters are updates by optimizing the obejective function in step 3, while the teacher’s parameters are updated by calculating the the EMA in step 1. This dynamic is believed to prevent the model from collapsing into a constant representation [7]. I will write another blog to introduce [7] in the future.</p><p><strong>Results:</strong></p><p>The paper reports SOTA results on all three modalities.</p><ol><li>Image (metric: accuracy. Higher value, better performance):</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*CmMr9DPFulHsHSuB-tlrIQ.png" /><figcaption>Image from <a href="https://ai.facebook.com/research/data2vec-a-general-framework-for-self-supervised-learning-in-speech-vision-and-language">the original paper</a>.</figcaption></figure><p>2. Speech (metric: word error rate. Lower value, better performance)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N68U2S6eyT-oBX3WHmbC5A.png" /><figcaption>Image from <a href="https://ai.facebook.com/research/data2vec-a-general-framework-for-self-supervised-learning-in-speech-vision-and-language">the original paper</a>.</figcaption></figure><p>3. Texts (metric: GLUE score. Higher value, better performance)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K6l1LfcHRnqqN2S3ukfHAw.png" /><figcaption>Image from <a href="https://ai.facebook.com/research/data2vec-a-general-framework-for-self-supervised-learning-in-speech-vision-and-language">the original paper</a>.</figcaption></figure><p><strong>Ablation Study：</strong></p><ol><li>Top K blocks</li></ol><p>The paper argues that using the average of top K blocks in the teacher mode is better than using just the top one (In this section in the paper, the authors use “top K layers”, which is inconsistent with “top K blocks” in the “Targets” section in the paper. I’m assuming “layers” and “blocks” are used interchangeably and both mean the blocks in the Transformer).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LA_BR4c0F9t-_aJC7q6m5A.png" /><figcaption>Using the average of top K blocks of the teacher model is better than just using the single top block. The results shown have better performance when the value is lower(speech), higher(NLP, i.e., texts) and higher(Vision, i.e., images ) respectively. The effect is more pronounced in speech and texts than in images. Image from <a href="https://ai.facebook.com/research/data2vec-a-general-framework-for-self-supervised-learning-in-speech-vision-and-language">the original paper</a>.</figcaption></figure><p>2. Target Feature Type</p><p>Rather than just use the top K blocks, the authors also tried using different parts of the teacher mode and found that using the FFN is the best.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*Uhk3rFnEMUcFa8i9z7TLZw.png" /><figcaption>WER stands for word error rate. Image from <a href="https://ai.facebook.com/research/data2vec-a-general-framework-for-self-supervised-learning-in-speech-vision-and-language">the original paper</a>.</figcaption></figure><p><strong>Conclusion: </strong>The paper introduces a new general self-supervised learning framework and achieves SOTA performance for three modalities.</p><p>The framework involves one model with two modes: teacher and student. The teacher is given a full sample input while the student is given a masked input of the same sample. Self-supervised learning is achieved by letting the student learn from the teacher.</p><p><strong>Personal Remarks:</strong></p><ol><li>It’d be more interesting to see how this method performs for unstructured modality, e.g. graphs :P</li><li>Transformer and <a href="https://medium.com/mlearning-ai/read-paper-with-me-bootstrap-your-own-latent-a-new-approach-to-self-supervised-learning-e6580ce8dae5">BYOL</a> play an important role in the success of this method. Transformer is a flexible architecture not constraint to a specific modality, so it can be applied to different modalities. And BYOL provides the core self-supervised learning part of this method.</li><li>This work serves as a key step for unifying inputs from different modalities. As humans are likely to use a similar learning process to understand the visual world as they do for language [9, 10], this work is significant in bringing us closer to AGI.</li></ol><p><strong>References:</strong></p><p>[1] Bao, H., Dong, L., and Wei, F. <a href="https://arxiv.org/abs/2106.08254">Beit: BERT pre-training of image transformers</a> (2021). ArXiv</p><p>[2] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. <a href="https://arxiv.org/abs/2010.11929">An image is worth 16x16 words: Transformers for image recognition at scale</a> (2020). arXiv</p><p>[3] Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. <a href="https://proceedings.neurips.cc/paper/2020/file/92d1e1eb1cd6f9fba3227870bb6d7f07-Paper.pdf">wav2vec 2.0: A framework for self-supervised learning of speech representations</a> (2020). In Proc. of NeurIPS</p><p>[4] Sennrich, R., Haddow, B., and Birch, A. <a href="https://aclanthology.org/P16-1162/">Neural machine translation of rare words with subword units</a> (2016). In Proc. of ACL</p><p>[5] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. <a href="https://aclanthology.org/N19-1423.pdf">Bert: Pre-training of deep bidirectional transformers for language understanding</a> (2019). Proc. of NAACL</p><p>[6] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. <a href="https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf">Attention is all you need</a> (2017). In Proc. of NIPS</p><p>[7] Grill, J.-B., Strub, F., Altche, F., Tallec, C., Richemond, ´ P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., Piot, B., Kavukcuoglu, K., Munos, R., and Valko, M. <a href="https://arxiv.org/abs/2006.07733">Bootstrap your own latent: A new approach to self-supervised learning</a> (2020). arXiv.</p><p>[8] Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bo- ´ janowski, P., and Joulin, A. <a href="https://arxiv.org/abs/2104.14294">Emerging properties in selfsupervised vision transformers </a>(2021). arXiv</p><p>[9] Friston, K. and Kiebel, S. <a href="https://royalsocietypublishing.org/doi/10.1098/rstb.2008.0300">Predictive coding under the freeenergy principle</a> (2009). Philosophical transactions of the Royal Society: Biological sciences</p><p>[10] Friston, K. <a href="https://www.nature.com/articles/nrn2787">The free-energy principle: a unified brain theory?</a> (2010), Nature reviews neuroscience</p><p><strong>Further reading:</strong></p><p><a href="https://medium.com/@YunchaoLanceLiu/read-paper-with-me-bootstrap-your-own-latent-a-new-approach-to-self-supervised-learning-e6580ce8dae5">BYOL </a>( The method that data2vec is built upon)</p><p><a href="https://jimmy-shen.medium.com/finally-we-have-a-more-general-modality-unspecific-representation-method-data2vec-5dcba6c853ef">Jimmy Chen’s blog on data2vec</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43299fef41d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>