<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bridget Leonard on Medium]]></title>
        <description><![CDATA[Stories by Bridget Leonard on Medium]]></description>
        <link>https://medium.com/@leonardbridget01?source=rss-a678462d8691------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*4JdqOUVPMvJyAu_guV-MRA.jpeg</url>
            <title>Stories by Bridget Leonard on Medium</title>
            <link>https://medium.com/@leonardbridget01?source=rss-a678462d8691------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 18:29:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@leonardbridget01/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Can Intelligence Live in Language Alone?]]></title>
            <link>https://medium.com/@leonardbridget01/can-intelligence-live-in-language-alone-e9b22496457b?source=rss-a678462d8691------2</link>
            <guid isPermaLink="false">https://medium.com/p/e9b22496457b</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[cognitive-science]]></category>
            <category><![CDATA[neuroscience]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Bridget Leonard]]></dc:creator>
            <pubDate>Thu, 26 Mar 2026 16:22:24 GMT</pubDate>
            <atom:updated>2026-03-26T16:22:24.721Z</atom:updated>
            <content:encoded><![CDATA[<p>We know intelligence exists outside language, but can it exist entirely within it?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HRqkAFgv4WKwMYjIf-MsMg.jpeg" /></figure><p>Large language models (LLMs) can explain complex ideas, write code, and pass exams. But they still fail at tasks that even children solve with ease. So what exactly are we measuring when we call a system “intelligent”?</p><p>Humans do not just record the world; we build internal models of it, combining sensory input with prior expectations to interpret what we see. These internal representations, the organized information we use to make sense of the world, shape how intelligence works. If we want to compare biological and artificial intelligence, these representations may be one of the most important things to study.</p><p>Although language captures many aspects of the statistical structure of the world, intelligence spans beyond it. And even when language is useful, it is not always the best format for representing every kind of information, especially when the task depends on perception, space, or action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ExjB46ildXU8Q2K4X60f9Q.jpeg" /></figure><p>The success of LLMs has brought old questions from cognitive science and neuroscience back into focus. If a language model can behave intelligently, what parts of intelligence can be built from language alone? Vision-language models (VLMs) make that question even more interesting. They allow us to ask not just whether a model can encode visual information, but whether it can use that information meaningfully once it has been folded into a system organized around language.</p><p>Spatial reasoning is a particularly useful test. It asks whether a model can move beyond recognizing objects in a scene and instead reason about their relationships, positions, and viewpoints. VLMs are still notoriously weak at tasks like perspective-taking, where a model must represent a scene from someone else’s point of view rather than its own.</p><p>One reason for this may be how visual information is integrated into these systems. Many VLMs rely on vision models trained to align images with captions, which is excellent for semantic understanding but may be less suited for representing spatial structure. That raises an important question: do these models fail because they never extract the right spatial information, or because they do not use it effectively once it is there?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*B27399fNsjAYwtB3f8Zd_w.png" /><figcaption>Base LLaVA often defaults to the viewer’s perspective, while perspective tokens help the model reason from the agent’s point of view.</figcaption></figure><p>To test this idea, I worked on a perspective-taking task for a vision-language model. These models often default to their own viewpoint rather than an agent’s viewpoint in a scene, revealing a strong egocentric bias. By adding <a href="https://github.com/bridgetleonard2/EmbodimentTokens">spatially explicit tokens</a> and fine-tuning on perspective-taking examples, we substantially improved performance. That result suggests the issue may not be a total absence of spatial information, but a failure to represent or use it effectively during inference.</p><p>We also found evidence that base LLaVA already contains some sensitivity to spatial structure in a scene. A small number of internal units responded reliably to features like an agent’s orientation in the scene. After fine-tuning, more of these units appeared to track spatial information, suggesting that the model may already contain some of the right ingredients but does not use them efficiently by default.</p><p>Language may be an especially powerful space for semantic relationships and some forms of abstract reasoning. But current models still struggle when reasoning requires moving beyond immediate correlations, as in spatial reasoning, negation, or flexible generalization. Testing those limits may help us better understand both the strengths and the boundaries of language-based intelligence.</p><h4>Further reading / Project links</h4><p><strong>arXiv:</strong> <a href="https://arxiv.org/abs/2601.16378">Cognitively-Inspired Tokens Overcome Egocentric Bias in Multimodal Models</a><br><strong>Code:</strong> <a href="https://github.com/bridgetleonard2/EmbodimentTokens">Embodiment Tokens</a> <br><strong>Code: </strong><a href="https://github.com/bridgetleonard2/RotationTokens">Rotation Tokens</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9b22496457b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leveraging Multi-Modal Models to Predict Human Brain Activity]]></title>
            <link>https://medium.com/@leonardbridget01/leveraging-multi-modal-models-to-predict-human-brain-activity-37bb461b5d40?source=rss-a678462d8691------2</link>
            <guid isPermaLink="false">https://medium.com/p/37bb461b5d40</guid>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[multimodal]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[brain]]></category>
            <category><![CDATA[tower-bridge]]></category>
            <dc:creator><![CDATA[Bridget Leonard]]></dc:creator>
            <pubDate>Sun, 26 May 2024 21:20:05 GMT</pubDate>
            <atom:updated>2024-05-26T21:20:05.599Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*HlaZdmqjlECxFGDxMD4BwQ.png" /></figure><h3>Introduction</h3><p>One of my big interests in AI and machine learning is comparing neural network architecture to brain architecture. A lot of research has found similarities between the two. Specifically, activations in different layers of neural networks mimic brain responses in different regions when both are exposed to the same stimulus.</p><p>A few months ago, I read a <a href="https://arxiv.org/abs/2305.12248">paper</a> published by the Huth Lab at UT Austin investigating the multimodal <a href="https://huggingface.co/docs/transformers/en/model_doc/bridgetower">BridgeTower</a> model which encodes text and prepares images by wrapping the <a href="https://huggingface.co/docs/transformers/v4.41.0/en/model_doc/roberta#transformers.RobertaTokenizer">RobertaTokenizer</a> and <a href="https://huggingface.co/docs/transformers/v4.41.0/en/model_doc/bridgetower#transformers.BridgeTowerImageProcessor">BridgeTowerImageProcessor</a> into one processor instance. They found that encoding models based on this model and fMRI data can transfer across language and vision; models trained on one modality can successfully predict the brain response to the other modality. This reveals the shared representations between visual and language features that exist in both multimodal models and the human brain.</p><h3>Path to Replication</h3><p>I was already interested in starting some project comparing fMRI data to neural network layer activations when I discovered the Huth Lab’s work. Since all the fMRI data they used was open source, including the raw stimuli in their experiments, and BridgeTower was freely accessible on HuggingFace, I saw replication as a great way to introduce myself to this field.</p><p>After reading the paper many times to break down the analysis, I conducted my project in five steps (<a href="https://github.com/bridgetleonard2/BridgeTower-Brain?tab=readme-ov-file">available as notebook tutorials on my GitHub</a>).</p><ol><li>Introducing myself to the BridgeTower model and its layers</li><li>Extracting features from movies and stories using a specific BridgeTower layer</li><li>Creating matrices that align caption and image representations in BridgeTower to project visual features into the language space and language features into the visual space</li><li>Creating modality-specific voxelwise encoding models that map the relation of all the features extracted from BridgeTower being fed frames of a movie/pieces of story transcript to a subject’s brain activity while watching a movie/listening to a story for each voxel in the subject’s brain</li><li>Using the voxelwise encoding models on the input data of the modality it wasn’t trained on to predict brain responses — using the vision encoding model to predict brain responses to stories with BridgeTower features and using the language encoding model to predict brain responses to movies with BridgeTower features</li></ol><h3>Part 1: BridgeTower Intro</h3><p>I first began by accessing the BridgeTower model through the HuggingFace Transformers Python library.</p><pre>from transformers import BridgeTowerModel, BridgeTowerProcessor</pre><p>I first looked at the model architecture and the model’s outputs before learning how to set forward hooks in the model to extract the feature representations of specific layers:</p><pre># input (text and image)<br>processor = BridgeTowerProcessor.from_pretrained(&quot;BridgeTower/bridgetower-base&quot;)<br>image = Image.open(str(&#39;data/intro_data/elephant.png&#39;)).convert(&#39;RGB&#39;)<br>text = &quot;An elephant in grass&quot;<br>inputs = processor(image, text, return_tensors=&#39;pt&#39;)<br><br># model<br>device = torch.device(&#39;cuda&#39; if torch.cuda.is_available() else &#39;cpu&#39;)<br>model = BridgeTowerModel.from_pretrained(&quot;BridgeTower/bridgetower-base&quot;)<br>model = model.to(device)<br><br># placeholder for batch features<br>features = {}<br>def get_features(name):<br>    def hook(model, input, output):<br>        # detached_outputs = [tensor.detach() for tensor in output]<br>        last_output = output[-1].detach()<br>        features[name] = last_output  # detached_outputs<br>    return hook<br><br>hook_names = []<br>hooks_to_remove = {}<br># register forward hooks with layers of choice<br># First, convolutional layers<br>patch_embed = model.vision_model.visual.embeddings.patch_embedding.register_forward_hook(get_features(&#39;patchEmbed&#39;))<br>hook_names.append(&#39;patch_embed&#39;)<br><br>final_vis = model.vision_model.visual.ln_post.register_forward_hook(get_features(&#39;final_vis&#39;))<br>hook_names.append(&#39;final_vis&#39;)</pre><p>After loading the final feature maps you can visualize different layer activations for any image input. For example this elephant:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*RIT-o3sKUCi2IsixL9wlwA.png" /></figure><p>Is represented like this in different BridgeTower layers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*1FjhM1O5ZdKUorSm61xuCw.png" /></figure><p>This feature extraction method is the foundation of how to use BridgeTower to train the encoding models.</p><h3>Part 2: Feature Extraction</h3><p>The fMRI datasets used to train and test the encoding models consist of a movie-watching fMRI dataset (vision) and a story-listening fMRI dataset (language).</p><p>I used layer 8 throughout this tutorial since it was noted to create the best encoding model performance in the paper.</p><p><strong>Movie Feature Extraction</strong></p><p>Our movie stimuli consist of 12 training movies and one test movie where each training run consists of 9000 images total and the test run contains 8100 images total. The associated fMRI data was collected by showing the videos at 30 frames per second with a sampling rate of TR=1 (1 sample per second). I mimicked this procedure with BridgeTower by averaging the feature representation over every 30 frames so training movies will have 300 time points and the test movie will have 270 time points:</p><pre>def get_movie_features(movie_data, layer, n=30):<br>    &quot;&quot;&quot;Function to average feature vectors over every n inputs.<br><br>    Parameters<br>    ----------<br>    movie_data: Array<br>        An array of shape (n_images, 512, 512). Represents frames from<br>        a color movie.<br>    n (optional): int<br>        Number of frames to average over. Set at 30 to mimick an MRI<br>        TR = 2 with a 15 fps movie.<br><br>    Returns<br>    -------<br>    data : Dictionary<br>        Dictionary where keys are the model layer from which activations are<br>        extracted. Values are lists representing activations of 768 dimensions<br>        over the course of n_images / 30.<br>    &quot;&quot;&quot;<br>    print(&quot;loading HDF array&quot;)<br>    movie_data = load_hdf5_array(movie_data, key=&#39;stimuli&#39;)<br><br>    # Define Model<br>    device, model, processor, features, layer_selected = setup_model(layer)<br><br>    # create overall data structure for average feature vectors<br>    # a dictionary with layer names as keys and a list of vectors as it values<br>    data = {}<br><br>    # a dictionary to store vectors for n consecutive trials<br>    avg_data = {}<br><br>    print(&quot;Running movie through model&quot;)<br>    # loop through all inputs<br>    for i, image in tqdm(enumerate(movie_data)):<br><br>        model_input = processor(image, &quot;&quot;, return_tensors=&quot;pt&quot;)<br>        # Assuming model_input is a dictionary of tensors<br>        model_input = {key: value.to(device) for key,<br>                       value in model_input.items()}<br><br>        _ = model(**model_input)<br><br>        for name, tensor in features.items():<br>            if name not in avg_data:<br>                avg_data[name] = []<br>            avg_data[name].append(tensor)<br><br>        # check if average should be stored<br>        if (i + 1) % n == 0:<br>            for name, tensors in avg_data.items():<br>                first_size = tensors[0].size()<br><br>                if all(tensor.size() == first_size for tensor in tensors):<br>                    avg_feature = torch.mean(torch.stack(tensors), dim=0)<br>                    avg_feature_numpy = avg_feature.detach().cpu().numpy()<br>                    # print(len(avg_feature_numpy))<br>                else:<br>                    # Find problem dimension<br>                    for dim in range(tensors[0].dim()):<br>                        first_dim = tensors[0].size(dim)<br><br>                        if not all(tensor.size(dim) == first_dim<br>                                   for tensor in tensors):<br>                            # Specify place to pad<br>                            p_dim = (tensors[0].dim()*2) - (dim + 2)<br>                            # print(p_dim)<br>                            max_size = max(tensor.size(dim)<br>                                           for tensor in tensors)<br>                            padded_tensors = []<br><br>                            for tensor in tensors:<br>                                # Make a list with length of 2*dimensions - 1<br>                                # to insert pad later<br>                                pad_list = [0] * ((2*tensor[0].dim()) - 1)<br>                                pad_list.insert(<br>                                    p_dim, max_size - tensor.size(dim))<br>                                # print(tuple(pad_list))<br>                                padded_tensor = pad(tensor, tuple(pad_list))<br>                                padded_tensors.append(padded_tensor)<br><br>                    avg_feature = torch.mean(torch.stack(padded_tensors),<br>                                             dim=0)<br>                    avg_feature_numpy = avg_feature.detach().cpu().numpy()<br>                    # print(len(avg_feature_numpy))<br><br>                if name not in data:<br>                    data[name] = []<br>                data[name].append(avg_feature_numpy)<br><br>            avg_data = {}<br><br>    layer_selected.remove()<br><br>    # Save data<br>    data = np.array(data[f&quot;layer_{layer}&quot;])<br>    print(&quot;Got movie features&quot;)<br><br>    return data</pre><p>Layer 8 has 768 features so for each training movie, I ended with an array of shape (300, 768), and for the test movie, a shape of (270, 768).</p><p><strong>Story Feature Extraction</strong></p><p>The story stimuli used in the fMRI experiment consisted of 10 stories from the <em>Moth Radio Hour.</em> To put this modality through BridgeTower, transcripts of each story were obtained and each word within the story was fed through the model with a pad of 20 words before and after for context.</p><pre>def get_story_features(story_data, layer, n=20):<br>    &quot;&quot;&quot;Function to extract feature vectors for each word of a story.<br><br>    Parameters<br>    ----------<br>    story_data: Array<br>        An array containing each word of the story in order.<br>    n (optional): int<br>        Number of words to to pad the target word with for<br>        context (before and after).<br><br>    Returns<br>    -------<br>    data : Dictionary<br>        Dictionary where keys are the model layer from which activations are<br>        extracted. Values are lists representing activations of 768 dimensions<br>        over the course of each word in the story.<br>    &quot;&quot;&quot;<br>    print(&quot;loading textgrid&quot;)<br>    story_data = textgrid_to_array(story_data)<br><br>    # Define Model<br>    device, model, processor, features, layer_selected = setup_model(layer)<br><br>    # Create a numpy array filled with gray values (128 in this case)<br>    # THis will act as tthe zero image input***<br>    gray_value = 128<br>    image_array = np.full((512, 512, 3), gray_value, dtype=np.uint8)<br><br>    # create overall data structure for average feature vectors<br>    # a dictionary with layer names as keys and a list of vectors as it values<br>    data = {}<br><br>    print(&quot;Running story through model&quot;)<br>    # loop through all inputs<br>    for i, word in tqdm(enumerate(story_data)):<br>        # if one of first 20 words, just pad with all the words before it<br>        if i &lt; n:<br>            # collapse list of strings into a single one<br>            word_with_context = &#39; &#39;.join(story_data[:(i+n)])<br>        # if one of last 20 words, just pad with all the words after it<br>        elif i &gt; (len(story_data) - n):<br>            # collapse list of strings into a single one<br>            word_with_context = &#39; &#39;.join(story_data[(i-n):])<br>            # collapse list of strings into a single one<br>        else:<br>            word_with_context = &#39; &#39;.join(story_data[(i-n):(i+n)])<br><br>        model_input = processor(image_array, word_with_context,<br>                                return_tensors=&quot;pt&quot;)<br>        # Assuming model_input is a dictionary of tensors<br>        model_input = {key: value.to(device) for key,<br>                       value in model_input.items()}<br><br>        _ = model(**model_input)<br><br>        for name, tensor in features.items():<br>            if name not in data:<br>                data[name] = []<br>            numpy_tensor = tensor.detach().cpu().numpy()<br><br>            data[name].append(numpy_tensor)<br><br>    layer_selected.remove()<br><br>    # Save data<br>    data = np.array(data[f&#39;layer_{layer}&#39;])<br>    print(&quot;Got story features&quot;)<br><br>    return data</pre><p>Each story was a different length so I ended with arrays of shape (n_words, 768) for the story feature extraction.</p><h3>Part 3: Feature Alignment</h3><p>To account for the possibility that language and visual feature spaces extracted from each layer of BridgeTower are only related based on a linear transformation, you can estimate the matrices that align BridgeTower feature spaces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*AMrbDD2oU-oMuUKnBlbsaw.png" /><figcaption>Caption: Two young guys with shaggy hair look at their hands while hanging out in the yard. Two young, White males are outside near many bushes. Two men in green shirts are standing in a yard. A man in a blue shirt standing in a garden. Two friends enjoy time spent together.</figcaption></figure><p>To do this, I followed the Huth Lab’s method of using the Flickr30K dataset (caption and image pairs, see example above) to extract features from each caption and image.</p><pre># Stream the dataset so it doesn&#39;t download to device<br>test_dataset = load_dataset(&quot;nlphuji/flickr30k&quot;, split=&#39;test&#39;, streaming=True)<br><br># Define Model<br>device, model, processor, features, layer_selected = setup_model(layer)<br><br>data = []<br><br>for item in tqdm(test_dataset):<br>    # Access data directly from the item, no need for indexing<br>    image = item[&#39;image&#39;]<br>    image_array = np.array(image)<br>    caption = &quot; &quot;.join(item[&#39;caption&#39;])<br><br>    # Run image<br>    image_input = processor(image_array, &quot;&quot;, return_tensors=&quot;pt&quot;)<br>    image_input = {key: value.to(device)<br>                   for key, value in image_input.items()}<br><br>    _ = model(**image_input)<br>    image_vector = features[f&#39;layer_{layer}&#39;]<br><br>    # Run caption<br>    # Create a numpy array filled with gray values (128 in this case)<br>    # This will act as the zero image input<br>    gray_value = 128<br>    gray_image_array = np.full((512, 512, 3), gray_value,<br>                               dtype=np.uint8)<br><br>    caption_input = processor(gray_image_array, caption,<br>                              return_tensors=&quot;pt&quot;)<br>    caption_input = {key: value.to(device)<br>                     for key, value in caption_input.items()}<br>    _ = model(**caption_input)<br><br>    caption_vector = features[f&#39;layer_{layer}&#39;]<br><br>    data.append([image_vector.detach().cpu().numpy(),<br>                caption_vector.detach().cpu().numpy()])</pre><p>Then, using ridge regression, I estimated the image-to-caption matrix that predicts each language feature from all the visual features, and the caption-to-image matrix that predicts each visual feature from all the language features.</p><pre># Variables<br>captions = data[:, 1, :]<br>images = data[:, 0, :]<br><br>alphas = np.logspace(-6, 6, 50)  # alphas = np.logspace(1, 20, 20)<br>scaler = StandardScaler(with_mean=True, with_std=False)<br><br>ridge_cv = RidgeCV(<br>    alphas=alphas, cv=5,<br>    solver_params=dict(n_targets_batch=500, n_alphas_batch=5,<br>                       n_targets_batch_refit=100))<br><br>pipeline = make_pipeline(<br>    scaler,<br>    ridge_cv<br>)<br><br>_ = pipeline.fit(images, captions)<br>coef_images_to_captions = backend.to_numpy(pipeline[-1].coef_)<br><br>_ = pipeline.fit(captions, images)<br>coef_captions_to_images = backend.to_numpy(pipeline[-1].coef_)<br><br># Save coefficients<br>np.save(f&#39;results/alignment/layer_{layer}/coef_images_to_captions.npy&#39;,<br>        coef_images_to_captions)<br>np.save(f&#39;results/alignment/layer_{layer}/coef_captions_to_images.npy&#39;,<br>        coef_captions_to_images)</pre><h3>Part 4: Voxelwise Encoding (and GPU-acceleration)</h3><p>The goal of the voxelwise encoding model is to map the relationship between all 768 BridgeTower features to each voxel in the brain. This way, you see which areas of the brain are more related to the BridgeTower features and better predicted by them. Since the fMRI data is very large (81111 voxels), I used the Himalaya python library which allows for GPU-accelerated ridge regression. I ran my code on an HPC cluster to leverage multiple GPUs and created a container for BridgeTower, Himalaya, and other important dependencies (all this can be found on my <a href="https://github.com/bridgetleonard2/BridgeTower-Brain/tree/main/container">GitHub</a>).</p><p><strong>Vision Encoding Model</strong></p><pre>def vision_model(subject, layer):<br>    &quot;&quot;&quot;Function to build the vision encoding model. Creates a<br>    matrix mapping the linear relationship between BridgeTower features<br>    and brain voxel activity.<br><br>    Parameters<br>    ----------<br>    subject: string<br>        A reference to the subject for analysis. Used to load fmri data.<br>    layer: int<br>        A layer reference for the BridgeTower model. Set&#39;s the forward<br>        hook on the relevant layer.<br><br>    Returns<br>    -------<br>    average_coef: Array<br>        Array of shape (layer_output_size*4, num_voxels) mapping<br>        the relationship of delayed feature vectors to each voxel<br>        in the fmri data.<br>    &quot;&quot;&quot;</pre><p>I started by extracting the BridgeTower features from the raw movie data using my previously defined get_movie_features() function:</p><pre>    data_path = &#39;data/raw_stimuli/shortclips/stimuli/&#39;<br><br>    # Extract features from raw stimuli<br>    train00 = get_movie_features(data_path + &#39;train_00.hdf&#39;, layer)<br>    train01 = get_movie_features(data_path + &#39;train_01.hdf&#39;, layer)<br>    train02 = get_movie_features(data_path + &#39;train_02.hdf&#39;, layer)<br>    train03 = get_movie_features(data_path + &#39;train_03.hdf&#39;, layer)<br>    train04 = get_movie_features(data_path + &#39;train_04.hdf&#39;, layer)<br>    train05 = get_movie_features(data_path + &#39;train_05.hdf&#39;, layer)<br>    train06 = get_movie_features(data_path + &#39;train_06.hdf&#39;, layer)<br>    train07 = get_movie_features(data_path + &#39;train_07.hdf&#39;, layer)<br>    train08 = get_movie_features(data_path + &#39;train_08.hdf&#39;, layer)<br>    train09 = get_movie_features(data_path + &#39;train_09.hdf&#39;, layer)<br>    train10 = get_movie_features(data_path + &#39;train_10.hdf&#39;, layer)<br>    train11 = get_movie_features(data_path + &#39;train_11.hdf&#39;, layer)<br>    test = get_movie_features(data_path + &#39;test.hdf&#39;, layer)</pre><p>Next, I loaded all the fMRI data and cleaned it by removing NaNs:</p><pre>    # Load fMRI data<br>    fmri_train = np.load(&quot;data/moviedata/&quot; + subject + &quot;/train.npy&quot;)<br>    fmri_test = np.load(&quot;data/moviedata/&quot; + subject + &quot;/test.npy&quot;)<br><br>    # Prep data<br>    train_fmri = remove_nan(fmri_train)<br>    test_fmri = remove_nan(fmri_test)</pre><p>Then, set up the fMRI and feature data to be run through ridge regression:</p><pre>    fmri_arrays = [train_fmri, test_fmri]<br>    feature_arrays = [train00, train01, train02, train03, train04,<br>                      train05, train06, train07, train08, train09,<br>                      train10, train11, test]<br><br>    # Combine data<br>    Y_train = np.vstack(fmri_arrays)<br>    X_train = np.vstack(feature_arrays)</pre><p>To define the cross-validation such that data from one movie would not be split up, I found the run onset of each movie in feature_arrays:</p><pre>    # Define cross-validation<br>    run_onsets = []<br>    current_index = 0<br>    for arr in feature_arrays:<br>        next_index = current_index + arr.shape[0]<br>        run_onsets.append(current_index)<br>        current_index = next_index<br><br>    n_samples_train = X_train.shape[0]<br>    cv = generate_leave_one_run_out(n_samples_train, run_onsets)<br>    cv = check_cv(cv)  # cross-validation splitter into a reusable list<br><br>    # Define the model<br>    scaler = StandardScaler(with_mean=True, with_std=False)<br><br>    delayer = Delayer(delays=[1, 2, 3, 4])<br><br>    backend = set_backend(&quot;torch_cuda&quot;, on_error=&quot;warn&quot;)<br>    print(backend)<br><br>    X_train = X_train.astype(&quot;float32&quot;)<br><br>    alphas = np.logspace(1, 20, 20)<br><br>    print(&quot;Running linear model&quot;)<br>    ridge_cv = RidgeCV(<br>        alphas=alphas, cv=cv,<br>        solver_params=dict(n_targets_batch=500, n_alphas_batch=5,<br>                           n_targets_batch_refit=100))<br><br>    pipeline = make_pipeline(<br>        scaler,<br>        delayer,<br>        ridge_cv,<br>    )<br><br>    _ = pipeline.fit(X_train, Y_train)</pre><p>Lastly, I extracted the coefficients from the fit model to create my vision encoding model in matrix form:</p><pre>    coef = pipeline[-1].coef_<br>    coef = backend.to_numpy(coef)<br>    print(&quot;(n_delays * n_features, n_voxels) =&quot;, coef.shape)<br><br>    # Regularize coefficients<br>    coef /= np.linalg.norm(coef, axis=0)[None]<br><br>    # split the ridge coefficients per delays<br>    delayer = pipeline.named_steps[&#39;delayer&#39;]<br>    coef_per_delay = delayer.reshape_by_delays(coef, axis=0)<br>    print(&quot;(n_delays, n_features, n_voxels) =&quot;, coef_per_delay.shape)<br>    del coef<br><br>    # average over delays<br>    average_coef = np.mean(coef_per_delay, axis=0)<br>    print(&quot;(n_features, n_voxels) =&quot;, average_coef.shape)<br>    del coef_per_delay<br><br>    print(&quot;Finished vision encoding model&quot;)<br>    return average_coef</pre><p><strong>Language Encoding Model</strong></p><p>The language encoding model is similar to the vision encoding model but requires a little more effort to prep the data. Specifically, the size of the first dimension in feature arrays is the number of words in the transcript, not the number of fMRI data samples. So, I had to resample the features to match the time dimension of the corresponding fMRI data:</p><pre>def resample_to_acq(feature_data, fmri_data):<br>    dimensions = fmri_data.shape[0]<br>    data_transposed = feature_data.T<br>    data_resampled = np.empty((data_transposed.shape[0], dimensions))<br><br>    for i in range(data_transposed.shape[0]):<br>        data_resampled[i, :] = resample(data_transposed[i, :],<br>                                        dimensions, window=(&#39;kaiser&#39;, 14))<br><br>    print(&quot;Shape after resampling:&quot;, data_resampled.T.shape)<br>    return data_resampled.T<br><br><br>def prep_data(fmri_data, feature_data):<br>    fmri_reshaped = remove_nan(fmri_data)<br><br>    feature_resampled = resample_to_acq(feature_data, fmri_reshaped)<br><br>    return fmri_reshaped, feature_resampled</pre><p>Using these helper functions, I extracted the features, loaded the fMRI data, and prepped them for ridge regression:</p><pre>def language_model(subject, layer):<br>    &quot;&quot;&quot;Function to build the language encoding model. Creates a<br>    matrix mapping the linear relationship between BridgeTower features<br>    and brain voxel activity.<br><br>    Parameters<br>    ----------<br>    subject: string<br>        A reference to the subject for analysis. Used to load fmri data.<br>    layer: int<br>        A layer reference for the BridgeTower model. Set&#39;s the forward<br>        hook on the relevant layer.<br><br>    Returns<br>    -------<br>    average_coef: Array<br>        Array of shape (layer_output_size*4, num_voxels) mapping<br>        the relationship of delayed feature vectors to each voxel<br>        in the fmri data.<br>    &quot;&quot;&quot;<br>    data_path = &#39;data/raw_stimuli/textgrids/stimuli/&#39;<br><br>    # Extract features from raw stimuli<br>    alternateithicatom = get_story_features(data_path + &#39;alternateithicatom.TextGrid&#39;, layer)<br>    avatar = get_story_features(data_path + &#39;avatar.TextGrid&#39;, layer)<br>    howtodraw = get_story_features(data_path + &#39;howtodraw.TextGrid&#39;, layer)<br>    legacy = get_story_features(data_path + &#39;legacy.TextGrid&#39;, layer)<br>    life = get_story_features(data_path + &#39;life.TextGrid&#39;, layer)<br>    yankees = get_story_features(data_path + &#39;myfirstdaywiththeyankees.TextGrid&#39;, layer)<br>    naked = get_story_features(data_path + &#39;naked.TextGrid&#39;, layer)<br>    ode = get_story_features(data_path + &#39;odetostepfather.TextGrid&#39;, layer)<br>    souls = get_story_features(data_path + &#39;souls.TextGrid&#39;, layer)<br>    undertheinfluence = get_story_features(data_path + &#39;undertheinfluence.TextGrid&#39;, layer)<br><br>    # Load fmri data<br>    fmri_alternateithicatom = np.load(&quot;data/storydata/&quot; + subject + &quot;/alternateithicatom.npy&quot;)<br>    fmri_avatar = np.load(&quot;data/storydata/&quot; + subject + &quot;/avatar.npy&quot;)<br>    fmri_howtodraw = np.load(&quot;data/storydata/&quot; + subject + &quot;/howtodraw.npy&quot;)<br>    fmri_legacy = np.load(&quot;data/storydata/&quot; + subject + &quot;/legacy.npy&quot;)<br>    fmri_life = np.load(&quot;data/storydata/&quot; + subject + &quot;/life.npy&quot;)<br>    fmri_yankees = np.load(&quot;data/storydata/&quot; + subject + &quot;/myfirstdaywiththeyankees.npy&quot;)<br>    fmri_naked = np.load(&quot;data/storydata/&quot; + subject + &quot;/naked.npy&quot;)<br>    fmri_ode = np.load(&quot;data/storydata/&quot; + subject + &quot;/odetostepfather.npy&quot;)<br>    fmri_souls = np.load(&quot;data/storydata/&quot; + subject + &quot;/souls.npy&quot;)<br>    fmri_undertheinfluence = np.load(&quot;data/storydata/&quot; + subject + &quot;/undertheinfluence.npy&quot;)<br><br>    # Prep data<br>    fmri_alternateithicatom, ai_features = prep_data(fmri_alternateithicatom, alternateithicatom)<br>    fmri_avatar, avatar_features = prep_data(fmri_avatar, avatar)<br>    fmri_howtodraw, howtodraw_features = prep_data(fmri_howtodraw, howtodraw)<br>    fmri_legacy, legacy_features = prep_data(fmri_legacy, legacy)<br>    fmri_life, life_features = prep_data(fmri_life, life)<br>    fmri_yankees, yankees_features = prep_data(fmri_yankees, yankees)<br>    fmri_naked, naked_features = prep_data(fmri_naked, naked)<br>    fmri_ode, odetostepfather_features = prep_data(fmri_ode, ode)<br>    fmri_souls, souls_features = prep_data(fmri_souls, souls)<br>    fmri_undertheinfluence, under_features = prep_data(fmri_undertheinfluence, undertheinfluence)<br><br>    fmri_arrays = [fmri_alternateithicatom, fmri_avatar, fmri_howtodraw,<br>                   fmri_legacy, fmri_life, fmri_yankees, fmri_naked,<br>                   fmri_ode, fmri_souls, fmri_undertheinfluence]<br>    feature_arrays = [ai_features, avatar_features, howtodraw_features,<br>                      legacy_features, life_features, yankees_features,<br>                      naked_features, odetostepfather_features,<br>                      souls_features, under_features]<br>    # Combine data<br>    Y_train = np.vstack(fmri_arrays)<br>    X_train = np.vstack(feature_arrays)<br><br>    # Continue with rest of model that mirrors the vision encoding model<br>    # i.e., Set up cross val, scaler, delayer, alphas<br>    # i.e, Fit model<br>    # i.e, Extract and normalize coefficients<br><br>    return average_coef</pre><p>The rest of the model is the same as the vision model but returns the matrix that contains the language encoding model.</p><h3>Part 5: Crossmodal Prediction</h3><p>Finally, to put it all together, I used the vision model to predict brain responses to stories using story features extracted by BridgeTower and used the language model to predict brain responses to movies using movie features extracted by BridgeTower.</p><p><strong>Story Prediction</strong></p><pre>def story_prediction(subject, layer, vision_encoding_matrix):<br>    &quot;&quot;&quot;Function to run the vision encoding model. Predicts brain activity<br>    to story listening and return correlations between predictions and real<br>    brain activity.<br><br>    Parameters<br>    ----------<br>    subject: string<br>        A reference to the subject for analysis. Used to load fmri data.<br>    layer: int<br>        A layer reference for the BridgeTower model. Set&#39;s the forward<br>        hook on the relevant layer.<br>    vision_encoding_matrix: array<br>        Generated by vision_model() function. A matrix mapping the relationship<br>        between feature vectors and brain activity<br><br>    Returns<br>    -------<br>    correlations: Array<br>        Array of shape (num_voxels) representing the correlation between<br>        predictions and real brain activity for each voxel.<br>    &quot;&quot;&quot;</pre><p>To account for any linear transformations between visual and language spaces within the BridgeTower model, I first projected the story features into the opposite space using the caption to image alignment matrix:</p><pre>    _, coef_captions_to_images = alignment(layer)<br><br>    data_path = &#39;data/raw_stimuli/textgrids/stimuli/&#39;<br>    <br>    # Get story features<br>    alternateithicatom = get_story_features(data_path + &#39;alternateithicatom.TextGrid&#39;, layer)<br>    avatar = get_story_features(data_path + &#39;avatar.TextGrid&#39;, layer)<br>    howtodraw = get_story_features(data_path + &#39;howtodraw.TextGrid&#39;, layer)<br>    legacy = get_story_features(data_path + &#39;legacy.TextGrid&#39;, layer)<br>    life = get_story_features(data_path + &#39;life.TextGrid&#39;, layer)<br>    yankees = get_story_features(data_path + &#39;myfirstdaywiththeyankees.TextGrid&#39;, layer)<br>    naked = get_story_features(data_path + &#39;naked.TextGrid&#39;, layer)<br>    ode = get_story_features(data_path + &#39;odetostepfather.TextGrid&#39;, layer)<br>    souls = get_story_features(data_path + &#39;souls.TextGrid&#39;, layer)<br>    undertheinfluence = get_story_features(data_path + &#39;undertheinfluence.TextGrid&#39;, layer)<br><br>    # Project features into opposite space<br>    alternateithicatom_transformed = np.dot(alternateithicatom, coef_captions_to_images.T)<br>    avatar_transformed = np.dot(avatar, coef_captions_to_images.T)<br>    howtodraw_transformed = np.dot(howtodraw, coef_captions_to_images.T)<br>    legacy_transformed = np.dot(legacy, coef_captions_to_images.T)<br>    life_transformed = np.dot(life, coef_captions_to_images.T)<br>    yankees_transformed = np.dot(yankees, coef_captions_to_images.T)<br>    naked_transformed = np.dot(naked, coef_captions_to_images.T)<br>    ode_transformed = np.dot(ode, coef_captions_to_images.T)<br>    souls_transformed = np.dot(souls, coef_captions_to_images.T)<br>    undertheinfluence_transformed = np.dot(undertheinfluence, coef_captions_to_images.T)</pre><p>Then, I loaded and prepped (resampled, removed NaNs) the story fMRI data:</p><pre>    # Load fmri data<br>    fmri_alternateithicatom = np.load(&quot;data/storydata/&quot; + subject + &quot;/alternateithicatom.npy&quot;)<br>    fmri_avatar = np.load(&quot;data/storydata/&quot; + subject + &quot;/avatar.npy&quot;)<br>    fmri_howtodraw = np.load(&quot;data/storydata/&quot; + subject + &quot;/howtodraw.npy&quot;)<br>    fmri_legacy = np.load(&quot;data/storydata/&quot; + subject + &quot;/legacy.npy&quot;)<br>    fmri_life = np.load(&quot;data/storydata/&quot; + subject + &quot;/life.npy&quot;)<br>    fmri_yankees = np.load(&quot;data/storydata/&quot; + subject + &quot;/myfirstdaywiththeyankees.npy&quot;)<br>    fmri_naked = np.load(&quot;data/storydata/&quot; + subject + &quot;/naked.npy&quot;)<br>    fmri_ode = np.load(&quot;data/storydata/&quot; + subject + &quot;/odetostepfather.npy&quot;)<br>    fmri_souls = np.load(&quot;data/storydata/&quot; + subject + &quot;/souls.npy&quot;)<br>    fmri_undertheinfluence = np.load(&quot;data/storydata/&quot; + subject + &quot;/undertheinfluence.npy&quot;)<br><br>    # Prep data<br>    fmri_ai, ai_features = prep_data(fmri_alternateithicatom, alternateithicatom_transformed)<br>    fmri_avatar, avatar_features = prep_data(fmri_avatar, avatar_transformed)<br>    fmri_howtodraw, howtodraw_features = prep_data(fmri_howtodraw, howtodraw_transformed)<br>    fmri_legacy, legacy_features = prep_data(fmri_legacy, legacy_transformed)<br>    fmri_life, life_features = prep_data(fmri_life, life_transformed)<br>    fmri_yankees, yankees_features = prep_data(fmri_yankees, yankees_transformed)<br>    fmri_naked, naked_features = prep_data(fmri_naked, naked_transformed)<br>    fmri_ode, odetostepfather_features = prep_data(fmri_ode, ode_transformed)<br>    fmri_souls, souls_features = prep_data(fmri_souls, souls_transformed)<br>    fmri_under, under_features = prep_data(fmri_undertheinfluence, undertheinfluence_transformed)</pre><p>To predict fMRI responses for each story, I took the dot product of the vision encoding matrix and transformed story features:</p><pre>    # Make fmri predictions<br>    ai_predictions = np.dot(ai_features, vision_encoding_matrix)<br>    avatar_predictions = np.dot(avatar_features, vision_encoding_matrix)<br>    howtodraw_predictions = np.dot(howtodraw_features, vision_encoding_matrix)<br>    legacy_predictions = np.dot(legacy_features, vision_encoding_matrix)<br>    life_predictions = np.dot(life_features, vision_encoding_matrix)<br>    yankees_predictions = np.dot(yankees_features, vision_encoding_matrix)<br>    naked_predictions = np.dot(naked_features, vision_encoding_matrix)<br>    odetostepfather_predictions = np.dot(odetostepfather_features,<br>                                         vision_encoding_matrix)<br>    souls_predictions = np.dot(souls_features, vision_encoding_matrix)<br>    under_predictions = np.dot(under_features, vision_encoding_matrix)</pre><p>And calculated the correlation between the predicted brain responses and actual brain responses:</p><pre>    # Calculate correlations<br>    ai_correlations = calc_correlation(ai_predictions, fmri_ai)<br>    avatar_correlations = calc_correlation(avatar_predictions, fmri_avatar)<br>    howtodraw_correlations = calc_correlation(howtodraw_predictions,<br>                                              fmri_howtodraw)<br>    legacy_correlations = calc_correlation(legacy_predictions, fmri_legacy)<br>    life_correlations = calc_correlation(life_predictions, fmri_life)<br>    yankees_correlations = calc_correlation(yankees_predictions, fmri_yankees)<br>    naked_correlations = calc_correlation(naked_predictions, fmri_naked)<br>    ode_correlations = calc_correlation(odetostepfather_predictions, fmri_ode)<br>    souls_correlations = calc_correlation(souls_predictions, fmri_souls)<br>    under_correlations = calc_correlation(under_predictions, fmri_under)<br><br>    # Get mean correlation<br>    all_correlations = np.stack((ai_correlations, avatar_correlations,<br>                                 howtodraw_correlations, legacy_correlations,<br>                                 life_correlations, yankees_correlations,<br>                                 naked_correlations, ode_correlations,<br>                                 souls_correlations, under_correlations))<br><br>    story_correlations = np.nanmean(all_correlations, axis=0)<br>    print(&quot;Max correlation:&quot;, np.nanmax(story_correlations))<br><br>    return story_correlations</pre><p><strong>Movie Prediction</strong></p><p>The process for generating movie predictions is the same with the exception of transforming the BridgeTower features into the language space, as seen below:</p><pre>def movie_predictions(subject, layer, language_encoding_model):<br>    &quot;&quot;&quot;Function to run the language encoding model. Predicts brain activity<br>    to movie watching and return correlations between predictions and real<br>    brain activity.<br><br>    Parameters<br>    ----------<br>    subject: string<br>        A reference to the subject for analysis. Used to load fmri data.<br>    layer: int<br>        A layer reference for the BridgeTower model. Set&#39;s the forward<br>        hook on the relevant layer.<br>    language_encoding_matrix: array<br>        Generated by language_model() function. A matrix mapping the<br>        relationship between feature vectors and brain activity<br><br>    Returns<br>    -------<br>    correlations: Array<br>        Array of shape (num_voxels) representing the correlation between<br>        predictions and real brain activity for each voxel.<br>    &quot;&quot;&quot;<br>    coef_images_to_captions, _ = alignment(layer)<br><br>    data_path = &#39;data/raw_stimuli/shortclips/stimuli/&#39;<br>    # Get movie features<br>    train00 = get_movie_features(data_path + &#39;train_00.hdf&#39;, layer)<br>    train01 = get_movie_features(data_path + &#39;train_01.hdf&#39;, layer)<br>    train02 = get_movie_features(data_path + &#39;train_02.hdf&#39;, layer)<br>    train03 = get_movie_features(data_path + &#39;train_03.hdf&#39;, layer)<br>    train04 = get_movie_features(data_path + &#39;train_04.hdf&#39;, layer)<br>    train05 = get_movie_features(data_path + &#39;train_05.hdf&#39;, layer)<br>    train06 = get_movie_features(data_path + &#39;train_06.hdf&#39;, layer)<br>    train07 = get_movie_features(data_path + &#39;train_07.hdf&#39;, layer)<br>    train08 = get_movie_features(data_path + &#39;train_08.hdf&#39;, layer)<br>    train09 = get_movie_features(data_path + &#39;train_09.hdf&#39;, layer)<br>    train10 = get_movie_features(data_path + &#39;train_10.hdf&#39;, layer)<br>    train11 = get_movie_features(data_path + &#39;train_11.hdf&#39;, layer)<br>    test = get_movie_features(data_path + &#39;test.hdf&#39;, layer)<br><br>    # Project features into opposite space<br>    test_transformed = np.dot(test, coef_images_to_captions.T)<br>    train00_transformed = np.dot(train00, coef_images_to_captions.T)<br>    train01_transformed = np.dot(train01, coef_images_to_captions.T)<br>    train02_transformed = np.dot(train02, coef_images_to_captions.T)<br>    train03_transformed = np.dot(train03, coef_images_to_captions.T)<br>    train04_transformed = np.dot(train04, coef_images_to_captions.T)<br>    train05_transformed = np.dot(train05, coef_images_to_captions.T)<br>    train06_transformed = np.dot(train06, coef_images_to_captions.T)<br>    train07_transformed = np.dot(train07, coef_images_to_captions.T)<br>    train08_transformed = np.dot(train08, coef_images_to_captions.T)<br>    train09_transformed = np.dot(train09, coef_images_to_captions.T)<br>    train10_transformed = np.dot(train10, coef_images_to_captions.T)<br>    train11_transformed = np.dot(train11, coef_images_to_captions.T)<br><br>    # Load fmri data<br>    fmri_train = np.load(&quot;data/moviedata/&quot; + subject + &quot;/train.npy&quot;)<br>    fmri_test = np.load(&quot;data/moviedata/&quot; + subject + &quot;/test.npy&quot;)<br><br>    # Prep data<br>    fmri_train = remove_nan(fmri_train)<br>    fmri_test = remove_nan(fmri_test)<br><br>    # Make fmri predictions<br>    feature_arrays = [train00_transformed, train01_transformed,<br>                      train02_transformed, train03_transformed,<br>                      train04_transformed, train05_transformed,<br>                      train06_transformed, train07_transformed,<br>                      train08_transformed, train09_transformed,<br>                      train10_transformed, train11_transformed]<br><br>    features_train = np.vstack(feature_arrays)<br>    features_test = test_transformed<br><br>    predictions_train = np.dot(features_train, language_encoding_model)<br>    predictions_test = np.dot(features_test, language_encoding_model)<br><br>    # Calculate correlations<br>    correlations_train = calc_correlation(predictions_train, fmri_train)<br>    correlations_test = calc_correlation(predictions_test, fmri_test)<br><br>    # Get mean correlation<br>    all_correlations = np.stack((correlations_train, correlations_train,<br>                                 correlations_train, correlations_train,<br>                                 correlations_train, correlations_train,<br>                                 correlations_train, correlations_train,<br>                                 correlations_train, correlations_train,<br>                                 correlations_train, correlations_train,<br>                                 correlations_test))<br><br>    correlations = np.nanmean(all_correlations, axis=0)<br>    print(&#39;max correlation&#39;, np.nanmax(correlations))<br><br>    return correlations</pre><h3>Results</h3><p>The best part about voxelwise encoding models is they can be easily visualized onto the brain in flatmap or MNI-space (standard brain) form. This way, you can see what areas of the brain are best “cross-modally” predicted and whether our results line up with those of the Huth lab. Here are the results in flatmap form next to the original results from the Huth lab:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*A5zZ5Oxoiwi3zXFrsAmWjA.png" /></figure><p>As seen in the labeled images on the left, you see activation in both the auditory and visual cortex (AC and VC, respectively) for the original and replicated results. Overall, information between modalities can be represented and predicted in similar ways in both human brains and multi-modal neural networks. That is, the information contained in the neural network for one modality also holds information about what that information would look like in the opposite modality. Imagine that feeding an image of the elephant generates similar information that a story about an elephant would, similar to how this information is distributed in the human brain. This finding can help us better understand how to successfully implement intelligent and efficient multimodal models based on the architectures of our own brains.</p><h3>Next Steps</h3><p>After successfully replicating this work, I’m now working on creating a pipeline to compare fMRI data to different open-source models. I hope to first investigate LLaVA-1.6–13B since it is one of the best open-source multi-modal models.</p><p>I hope this tool will be readily available in the future so anyone with fMRI data and the raw stimulus that evoked it can compare brain architecture and output to that of neural networks to better understand our own brains and the models that are rapidly approaching our cognitive abilities.</p><h3>Acknowledgments</h3><ul><li><a href="https://www.cs.utexas.edu/~huth/">Huth Lab</a> at UT Austin for their pioneering work in this domain and providing the publicly available fMRI datasets used in this work</li><li><a href="https://gallantlab.org/">Gallant Lab</a> at UC Berkeley for providing open-source code for voxelwise encoding models and providing publicly available fMRI datasets used in this work</li><li>Open source community for providing tools and frameworks that make projects like this possible</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=37bb461b5d40" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Research to Webpage: Building a Reaction-Time Experiment within a Webpage with Shiny Apps]]></title>
            <link>https://medium.com/@leonardbridget01/from-research-to-webpage-building-a-reaction-time-experiment-within-a-webpage-with-shiny-apps-742035fce6a8?source=rss-a678462d8691------2</link>
            <guid isPermaLink="false">https://medium.com/p/742035fce6a8</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[webpage]]></category>
            <category><![CDATA[shiny]]></category>
            <category><![CDATA[experiment]]></category>
            <category><![CDATA[r]]></category>
            <dc:creator><![CDATA[Bridget Leonard]]></dc:creator>
            <pubDate>Thu, 09 May 2024 03:57:15 GMT</pubDate>
            <atom:updated>2024-05-09T03:57:15.774Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Visit the webpage here: <a href="https://bridgetleonard.shinyapps.io/ErrorLearning/">https://bridgetleonard.shinyapps.io/ErrorLearning/</a></p><p>As a student studying cognitive psychology, I was always fascinated by how we can learn from memory errors. I thought it was interesting how I would never forget the correct answer to a problem I had gotten wrong in the past. I chose to study this phenomenon as my honors thesis in undergrad by creating computational models of different cognitive mechanisms of this error learning to see if one fits human data better. After completing the entire research process from start to finish and sharing my results through papers and at conferences, I wanted to expand my project outside the scientific community.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*Zs7EUf1UndbBbn94sPylUA.png" /></figure><p>After creating many iterations of dashboards to document my findings, I realized I could take my report a step further by allowing people to not only read and see my results but to contribute to my results and learn about themselves. So, I set on a journey to incorporate my reaction-time-based experiment with computational models into an interactive webpage.</p><h3>The Experiment</h3><p>The experiment worked like this:</p><ul><li>People memorize 60 different word pairs (e.g., whale-tail)</li><li>Half the word pairs are presented in a study condition where both words are shown together</li><li>The other half are presented in an error condition where only the first word is shown, people must guess the second, and then the correct word is shown after</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*gTNy8-G70VYgm_3E_VnKaQ.png" /></figure><p>The results:</p><ul><li>People remember the error words better on a final test</li><li>People take longer to respond to the error words than the study words</li></ul><h3>Individualized Models of Cognition</h3><p>Two models of error learning were created based on two common theories:</p><ul><li><strong>Elaborative learning</strong>: related errors increase learning by activating the semantic network between cue and target, creating a more meaningful story behind the fact</li><li><strong>Mediator learning</strong>: errors are used as a secondary cue to retrieve the correct answer</li></ul><p>The models were written in Python based on the ACT-R cognitive architecture where memories decay over time and are retrieved based on an activation level which is calculated based on the recency and frequency of presentation and how relevant the memory is to the current environment. The speed at which memories are retrieved is also based on their activation and how many steps are taken to retrieve them.</p><p>In our elaborative model, errors provide more contextual activation to the memories which increases the chance of them being retrieved and also increases the speed they’ll be retrieved at. In our mediator model, errors were retrieved using an extra step that increase specificity by using both the first word and error word, increasing accuracy and also increasing the time it takes to be retrieved.</p><p>So, both models predict better accuracy for error items but faster response times for error items in elaborative learners and slower response times in mediator learners.</p><p>After optimizing parameter fitting for each person who did the task, the mediator model fits almost everyone better. We calculated this fit trial-by-trial using maximum likelihood estimation. Generally, we quantified the likelihood that the given human responses were generated by either model to see which one fit better.</p><h3>The Webpage</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*EwCyBBor2BSWXuWH4fWOHw.gif" /></figure><p>The goal of my webpage was to first introduce my project and findings and then prompt people to try the task themselves. I soon decided that incorporating a way to update my results to show the results of those who do the task would be a great way to make the page interactive and the science of it more fun. So, after building the report side of the page, I started incorporating the experiment with the main goals:</p><ul><li>Administer the word-pair task through Shiny/JavaScript to collect responses and response times</li><li>Run those responses through simple analysis code written in R, then run the responses through the model code written in Python</li><li>Dynamically update the group-level visuals at the beginning of the report so people can see how they compare to others</li></ul><p><strong>Administering the task</strong></p><p>I first built the Shiny UI with HTML using the includeHTML() and allowed my JavaScript functions to interact with the UI using shinyjs::useShinyjs():</p><pre># Code from my ui.R file<br><br>library(shiny)<br><br>fluidPage(<br>  shinyjs::useShinyjs(),  # Initialize shinyjs<br>  includeHTML(&quot;www/index.html&quot;),<br>  uiOutput(&quot;plotFrame&quot;),<br>  titlePanel(&quot;The Task&quot;),</pre><ul><li>For the study and test session, my server.R file iterates through each row of word pair data and presents it in either the error or study condition using JavaScript functions that capture responses and response times</li></ul><pre>// An example of shinyjs word presentation for the study session<br>shinyjs.updateWordPair = function(cue, target, condition) {<br>   // Clear previous content<br>   $(&quot;#cueContainer&quot;).empty();<br>   $(&quot;#targetContainer&quot;).empty();<br>    <br>   var startTime; // Variable to store the start time<br>    <br>   if (condition === 1) {<br>      $(&quot;#cueContainer&quot;).text(cue);<br>      startTime = new Date(); // Record the start time<br>    <br>      // Create and append a text input element<br>      var textBox = $(&quot;&lt;input&gt;&quot;).attr({<br>        type: &quot;text&quot;,<br>        id: &quot;textBoxInput&quot;,<br>        placeholder: &quot;Type your response...&quot;<br>      });<br>          <br>      $(&quot;#targetContainer&quot;).append(textBox);<br>      // Automatically focus the textbox<br>      textBox.focus();<br>    <br>      $(&quot;#textBoxInput&quot;).keypress(function(event) {<br>        // Check if the pressed key is Enter (keyCode 13)<br>        if (event.which === 13) {<br>          var responseValue = $(&quot;#textBoxInput&quot;).val();<br>          var responseTime = new Date() - startTime; // Calculate elapsed time<br>          console.log(responseValue);<br>              <br>          // Add the response and response time to the object<br>          responsesObject[cue] = [target, condition, responseValue, responseTime];<br>          console.log(responsesObject)<br>              <br>          // Clear the input field<br>          $(&quot;#textBoxInput&quot;).val(&quot;&quot;);<br>              <br>          // Show the study pair<br>          $(&quot;#targetContainer&quot;).empty();<br>          $(&quot;#targetContainer&quot;).text(target);<br>        }<br>      });<br>    <br>      // start study trial after 5 seconds<br>      setTimeout(function() {<br>        $(&quot;#targetContainer&quot;).empty();<br>        $(&quot;#targetContainer&quot;).text(target);<br>      }, 5000);<br>    } else if (condition === 2) {<br>      $(&quot;#cueContainer&quot;).text(cue);<br>      $(&quot;#targetContainer&quot;).text(target);<br>    }<br>  };</pre><p><strong>Conducting analysis</strong></p><ul><li>After data is collected, shinyJS sent the data (responses and response times) back to R where data analysis begins</li><li>Summary stats of accuracy and average response times for study vs error condition are run first, then Python functions are loaded with the Reticulate package to run the log-likelihood models</li></ul><pre>## PYTHON<br>reticulate::py_install(&quot;numpy&quot;)<br>reticulate::py_install(&quot;scipy&quot;)<br>reticulate::py_install(&quot;pandas&quot;)<br>    <br># Import packages<br>numpy &lt;- import(&quot;numpy&quot;)<br>scipy &lt;- import(&quot;scipy&quot;)<br>pandas &lt;- import(&quot;pandas&quot;)<br>math &lt;- import(&quot;math&quot;)<br>    <br># Run MLE to find learner type: output: &quot;you fit the ___ model x% better than the ___ model&quot;<br>source_python(&quot;www/LLerror.py&quot;)<br>    <br># Append new results<br>## remove index column and add participant<br>if (acquire_lock(lock_file)) {<br>  # full_data &lt;- read.csv(&#39;www/updatingData/formatted_data.csv&#39;) ###########<br>  reactiveData$full_data &lt;- read_sheet(&quot;1pmeYWRJTQvIRFRzK4PjgB83DypA-XXtea_pj1a13WDo&quot;) ###########<br>      <br>  full_data &lt;- reactiveData$full_data<br>  ppt_code &lt;- (max(full_data$participant, na.rm = TRUE) + 1)<br>      <br>  clean_data$index &lt;- NULL<br>  clean_data$participant &lt;- ppt_code<br>      <br>  sheet_append(&quot;google-cloud-sheet-code&quot;, clean_data)<br>  reactiveData$full_data &lt;- read_sheet(&quot;google-cloud-sheet-code&quot;) ###########<br>      <br>      <br>  reactiveData$LL_data &lt;- read_sheet(&quot;google-cloud-sheet-code&quot;) #############<br>  LL_data &lt;- reactiveData$LL_data<br>      <br>  LL_results &lt;- ll_participant(clean_data, ppt_code, LL_data)<br>  <br>  participant_ll &lt;- LL_results[[1]]<br>  column_names &lt;- c(&quot;Participant&quot;, &quot;elab.decay&quot;, &quot;elab.temp&quot;, &quot;elab.ter&quot;, &quot;elab.LL&quot;, &quot;med.decay&quot;, &quot;med.temp&quot;, &quot;med.ter&quot;, &quot;med.LL&quot;, &quot;best.model&quot;, &quot;diff.LL&quot;)<br>      <br>  ppt_ll_df &lt;- data.frame(matrix(unlist(participant_ll), nrow = 1, byrow = TRUE))<br>  colnames(ppt_ll_df) &lt;- column_names<br>      <br>  print(ppt_ll_df)<br>  sheet_append(&quot;google-cloud-sheet-code&quot;, ppt_ll_df)<br>  reactiveData$LL_data &lt;- read_sheet(&quot;google-cloud-sheet-code&quot;) #############<br>      <br>  # Release the lock<br>  release_lock(lock_file)<br>} else {<br>  print(&quot;another process occuring, longer wait times may occur&quot;)<br>}</pre><ul><li>Whenever data was accessed or appended to the Google Cloud Sheet, a manual lock was put on the functions in case multiple users finished the task at the same time</li></ul><p><strong>Updating figures</strong></p><ul><li>Lastly, the main summary plots at the start of the webpage were updated whenever new data was appended to the Google Cloud Sheets using observeEvent(reactiveData$LL_data) and observeEvent(reactiveData$full_data)</li></ul><pre># Snippet of the figure updating process<br>observeEvent(reactiveData$full_data, {<br>    full_data &lt;- reactiveData$full_data<br>    <br>    cleandata_participant &lt;- full_data %&gt;% <br>      group_by(participant, condition) %&gt;% <br>      summarize(<br>        performance = (sum(correct) / n())<br>      )<br>    <br>    cleandata_summary &lt;- cleandata_participant %&gt;% <br>      group_by(condition) %&gt;% <br>      summarize(<br>        mean_performance = mean(performance),<br>        sd_performance = sd(performance),<br>        n = n(),<br>        stError = sd_performance / sqrt(n)<br>      )<br>    <br>    # Update figures<br>    summary_plot &lt;- plot_ly(data = cleandata_participant, x = ~condition, y = ~performance) %&gt;%<br>      add_markers(alpha = 0.75, color = I(&quot;#D9D6C7&quot;), split = ~participant, marker = list(size=5),<br>                  text = ~performance,<br>                  textposition = &quot;auto&quot;,<br>                  hoverinfo = &quot;text&quot;,<br>                  hovertext = ~paste0(&quot;Participant: &quot;, participant,<br>                                      &quot;&lt;br&gt; Observed Accuracy: &quot;, round(performance*100, 2), &quot;%&quot;),<br>                  showlegend=FALSE) %&gt;% </pre><h3>Summary</h3><p>With lots of iterations and trouble-shooting — finding how to store data and update figures was so much harder than I thought it’d be, but learning Google Cloud Platform tools was so useful 🥳, I finally designed a webpage that both communicated my research findings and made the findings accessible to people on an individual-scale.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=742035fce6a8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine Tuning LLaVA on an Emotion Detection Task]]></title>
            <link>https://medium.com/@leonardbridget01/fine-tuning-llava-on-an-emotion-detection-task-2e626db842d4?source=rss-a678462d8691------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e626db842d4</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[emotion-detection]]></category>
            <category><![CDATA[llava]]></category>
            <dc:creator><![CDATA[Bridget Leonard]]></dc:creator>
            <pubDate>Sat, 13 Apr 2024 21:59:08 GMT</pubDate>
            <atom:updated>2024-05-10T00:00:07.388Z</atom:updated>
            <content:encoded><![CDATA[<h3>Fine-Tuning LLaVA on an Emotion Detection Task</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*H7gwaynlxVgOGMLBhdq-Rw.png" /></figure><h3>Introduction</h3><p>I’ve recently been fascinated by the capabilities of multi-modal models and why and how they excel in so many domains of what used to be seen as human-specific behavior. Administering assessments using AI APIs can be a quick way to quantify its abilities in a specific domain. Here we delve into the idea of emotion detection with a common cognitive assessment of Theory of Mind, or the ability to understand someone else’s feelings and how they may differ from your own, known as the Reading the Mind through the Eyes Test or RMET for short.</p><p>While taking the RMET, you must choose one word out of four options to best describe a photo of someone’s eyes like the example below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*_ytynjoW_JjBa-8pbR-1Yw.jpeg" /><figcaption>Playful Comforting Irritated Bored</figcaption></figure><p>The RMET can distinguish varying abilities of emotion detection. Populations with diagnoses related to social/emotional processing reflect differences in scores. For example, individuals with Autism Spectrum Disorder and Social Anxiety Disorder typically score lower than healthy controls on emotion recognition in the task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FC5ONuhxW5SnSpncBxjG7Q.png" /><figcaption>RMET Score Distributions for Neurotypical Humans</figcaption></figure><p>Follow along with my code here: <a href="https://github.com/bridgetleonard2/Eyes-Mind-Model">https://github.com/bridgetleonard2/Eyes-Mind-Model</a></p><h3>GPT-4 vs LLaVA</h3><p>I first adapted the task for the GPT-4 vision API due to its impressive ability to generate accurate descriptions when prompted with text and image queries. GPT-4v averaged around 24–26 items correct, demonstrating its ability to meet the higher end of human performance.</p><p>I next extended the task to the LLaVA API to see if a model trained on less data could mimic these results. However, there was a substantial lack of performance where LLaVA could only correctly identify 17–20 items.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*I2JEFpcJ5Fb-0lLqkHavxw.png" /><figcaption>Model performance overlayed on typical human performance</figcaption></figure><p>While GPT-4 captures the higher end of human performance, LLaVA struggles to reach minimal human performance.</p><p>To better understand whether GPT’s superior performance was due to its model architecture or access to more training data, I set out to fine-tune LLaVA on task-relevant data, attempting to increase its performance to GPT standards. I wanted to investigate if fine-tuning can increase the performance of smaller models and to what extent this can happen. Ideally, a small additional step can lead models originally trained on smaller amounts of data to have the same task-based capabilities as those trained on larger amounts of data. However, it may be that model architecture plays a large role even in task-specific performance.</p><h3>Fine-Tuning LLaVA — The Data</h3><p>Instead of fine-tuning LLaVA on specific task data (i.e., pictures of eyes and corresponding emotions), I wanted to train it in a more realistic approach. Specifically, I wanted task-relevant caption-image pairs related to pictures of people and the emotions they look like they’re feeling. I took each of the words contained in the RMET and generated a data point by:</p><ol><li>Querying Adobe’s Firefly to generate an image of a person who looks “happy”</li><li>Asking GPT-4V to explain why the person in the image looks like they are “happy”</li><li>Formatting our data to fit LLaVA training requirements with images generated by Firefly and captions generated by GPT-4</li></ol><p>The final data looked like this:</p><pre>[<br>    {<br>      &quot;id&quot;: &quot;01&quot;,<br>      &quot;image&quot;: &quot;playful.jpg&quot;,<br>      &quot;conversations&quot;: [<br>        {<br>          &quot;from&quot;: &quot;human&quot;,<br>          &quot;value&quot;: &quot;&lt;image&gt;\nWhat does this person look like they are thinking or feeling?&quot;<br>        },<br>        {<br>          &quot;from&quot;: &quot;gpt&quot;,<br>          &quot;value&quot;: &quot;The person in the image has a bright, wide smile that suggests a cheerful disposition and is indicative of playfulness. The sparkle in their eyes and the overall expression of enjoyment on their face contribute to a lively, playful appearance. The open mouth and visible teeth in the smile also enhance the perception of a joyful, unguarded, and friendly demeanor, which are typical characteristics associated with being playful.&quot;<br>        }<br>      ]<br>    },<br>    {<br>        &quot;id&quot;: &quot;02&quot;,<br>        &quot;image&quot;: &quot;comforting.jpg&quot;,<br>        &quot;conversations&quot;: [<br>          {<br>            &quot;from&quot;: &quot;human&quot;,<br>            &quot;value&quot;: &quot;&lt;image&gt;\nWhat does this person look like they are thinking or feeling?&quot;<br>          },<br>          {<br>            &quot;from&quot;: &quot;gpt&quot;,<br>            &quot;value&quot;: &quot;The individual pictured here appears comforting due to several visual cues: 1. Gentle facial expression: The person&#39;s face shows a soft, empathetic expression, indicated by relaxed features and a slight, warm smile. 2. Attentive gaze: The person is making what appears to be direct and meaningful eye contact with someone outside of the image frame, which is often associated with active listening and a sense of presence. 3. Body language: With hands clasped and elbows on the table, the pose suggests a calm, open, and collected demeanor, which can be reassuring to someone seeking solace or comfort. 4. Posture: Leaning slightly forward, the individual appears engaged and interested in what the other person is saying, reinforcing a sense of support and attentiveness. The combination of these elements contributes to the perception that this person is likely to be comforting and supportive in their interaction with the other individual.&quot;<br>          }<br>        ]<br>      },</pre><p>With image-caption pairs like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*1h7LAENKPErgbrge7mGGdg.jpeg" /><figcaption><strong>The person in the image has a bright, wide smile that suggests a cheerful disposition and is indicative of playfulness. The sparkle in their eyes and the overall expression of enjoyment on their face contribute to a lively, playful appearance. The open mouth and visible teeth in the smile also enhance the perception of a joyful, unguarded, and friendly demeanor, which are typical characteristics associated with being playful.</strong></figcaption></figure><h3>Fine-Tuning a Containerized LLaVA</h3><p>I ran the fine-tuning process on an HPC cluster node to gain access to GPUs and CPUs to optimize training. This cluster already contained LLaVA-v1.5-13B in a software container so I chose that model as the base for my fine-tuning. Using the pre-existing training scripts for task fine-tuning found at <a href="https://github.com/haotian-liu/LLaVA/tree/main">https://github.com/haotian-liu/LLaVA/tree/main</a>, I developed a shell script to run the training script through the LLaVA container</p><pre>apptainer run --nv \<br>    --bind llava_hyak/train:/container/training_script \<br>    --bind llava_hyak/dataset:/container/dataset \<br>    --bind llava_hyak/output:/container/output \<br>    --bind llava_hyak/scripts:/container/scripts \<br>    oras://ghcr.io/uw-psych/llava-container/llava-container-train:latest \<br>    deepspeed /container/training_script/train_mem.py \<br>    --lora_enable True \<br>    --lora_r 128 \<br>    --lora_alpha 256 \<br>    --mm_projector_lr 2e-5 \<br>    --deepspeed /container/scripts/zero3.json \<br>    --model_name_or_path liuhaotian/llava-v1.5-13b \<br>    --version v1 \<br>    --data_path /container/dataset/train/train_data.json \<br>    --validation_data_path /container/dataset/validation/val_data.json \<br>    --image_folder /container/dataset/images \<br>    --vision_tower openai/clip-vit-large-patch14-336 \<br>    --mm_projector_type mlp2x_gelu \<br>    --mm_vision_select_layer -2 \<br>    --mm_use_im_start_end False \<br>    --mm_use_im_patch_token False \<br>    --image_aspect_ratio pad \<br>    --group_by_modality_length True \<br>    --bf16 True \<br>    --output_dir /container/output/checkpoints/llava-v1.5-13b-task-lora \<br>    --num_train_epochs 5 \<br>    --per_device_train_batch_size 16 \<br>    --per_device_eval_batch_size 4 \<br>    --gradient_accumulation_steps 1 \<br>    --evaluation_strategy &quot;epoch&quot; \<br>    --save_strategy &quot;steps&quot; \<br>    --save_steps 50000 \<br>    --save_total_limit 1 \<br>    --learning_rate 2e-4 \<br>    --weight_decay 0. \<br>    --warmup_ratio 0.03 \<br>    --lr_scheduler_type &quot;cosine&quot; \<br>    --logging_steps 1 \<br>    --model_max_length 2048 \<br>    --gradient_checkpointing True \<br>    --dataloader_num_workers 4 \<br>    --lazy_preprocess True \<br>    --report_to wandb<br></pre><p>LLaVA’s training script offers an argument to report results to Wandb which conveniently visualizes training and evaluation loss and other performance metrics. I fine-tuned LLaVA five times with different numbers of train epochs to assess its performance on the RMET at different stages of training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9DWSkzDcI49alNslIOO2Vw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N9wDm-_bNGo_MNJgMJo6UQ.png" /><figcaption>radient-latern-4: 7B model trained on 1 epoch; incandescent-firecracker-5: 13B model trained on 1 epoch; virtous-cake-9: 13B model trained on 5 epochs; abundant-pig-10: 13B model trained on 10 epochs; luminous-monkey-11: 13B model trained on 7 epochs</figcaption></figure><h3>Qualitative Check of Fine-tuning Procedure</h3><p>To assess whether our fine-tuning procedure modified the LLaVA base model how I expected it to, I queried the fine-tuned models with some held-out training data and the prompt: “What does this person look like they are thinking or feeling?”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*3FzXNUFxDb7go63fALO5Ag.png" /><figcaption>Different model responses from least fine-tuned (base LLaVA) at the top to most fine-tuned (10 epochs) at the bottom</figcaption></figure><p>From these two examples, three things are apparent. First, the fine-tuning had an effect. From this effect, there’s an increase in task-related performance over training epochs and also some overfitting of the 10-epoch model. While creating the training data, when asked to describe why a photo of a person looked ‘happy’ (or any other emotion description), GPT-4v would instead argue the person looked differently but might still be interpreted as the original word due to certain characteristics.</p><p>We see that the 10 epoch fine-tuned LLaVA outputs similar behavior even to an open-ended question. It’s clear from these qualitative examples that 1 epoch does not provide enough training to significantly adjust the weights to improve emotion detection, 5–7 epochs is the sweet spot, and 10 epochs leads to overfitting —the model mimics nuances in the training data that inhibit emotion recognition.</p><h3>Performance on the RMET</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*H2kYP2mdcCmKX8wRwwdJTA.png" /></figure><p>After the qualitative check, I ran the fine-tuned models through the RMET and saw that I was able to successfully improve LLaVA’s performance on the RMET and approach the lower end of human performance. I improved LLaVA first by running it on the HPC cluster with a 7.5% increase in performance. Fine-tuning the model with 5 epochs led to the greatest increase in performance— a 12.5% increase from the base model and a 5% increase from the cluster-run base model. These results show the ability of task-relevant fine-tuning to increase model performance in a way that targets improving more general cognition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*28Ab2ojU1DTg_lXdR_vnFw.png" /></figure><p>Comparing these results to human performance and GPT-4v performance, we see that the best-fine-tuned model (7 epoch) can meet the lower threshold of human performance on a few of its runs. Fine-tuning can provide a boost in LLaVA’s performance to bridge the large gap that originally existed, but it is not quite powerful enough to overcome the differences completely.</p><h3>Doubling the data</h3><p>To further explore if GPT’s performance can be attributed to larger training datasets, we created a second set of training data using another 77 AI-generated images on the RMET words. Now, for each unique RMET word, there were two image-caption pairs. This increase in training data led the 7-epoch model to almost reach the GPT-4 preview performance and allowed it to capture the lower-average end of human performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*la_ygtY-zQyxj3B3ROQgPg.png" /></figure><h3>Conclusion</h3><p>Current multi-modal models can meet human standards on emotion detection tasks. Specifically, GPT-4v meets the higher end of human performance while smaller models like LLaVA-1.5–13B reflect the lower end of human performance. This difference in model performance can likely be attributed to 1) different model architecture, 2) varying amounts of training data, or perhaps a combination of both.</p><p>To investigate if the performance difference is due to the quantity of training data, we fine-tuned LLaVA with task-relevant data generated with Adobe Firefly (images) and GPT-4v (captions). We found that fine-tuning the LLaVA with 7 epochs significantly increases the model performance to meet the lower-average range of human performance. With this increase in performance, LLaVA closely approaches the GPT-4v preview performance. Since few changes are likely made to the GPT architecture between the preview and turbo versions, the success of fine-tuning likely demonstrates that the difference in performance on the RMET is due to the quantity of training data. As such, fine-tuning efforts with more task-relevant data can likely increase LLaVA to GPT-4v standards.</p><h3>Acknowledgments</h3><p>A big thank you to the <a href="https://github.com/haotian-liu/LLaVA">LLaVA</a> creators and open-source community for sharing code and resources to assist with task-based fine-tuning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e626db842d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>