Merge branch 'ml4code:source' into source

Author: iterater

_layouts/publication.html

@@ -4,7 +4,7 @@
 
 <div class="page">
   <h1 class="page-title">{{ page.title }}</h1>
-  <h5>{{ page.authors }}. {{ page.conference }} {{ page.year }}</h5>
+  <h5>{{ page.authors }}. {{ page.conference | default: page.journal }} {{ page.year }}</h5>
   <p>
     {% for additional_link in page.additional_links %}
       [<a href="{{ additional_link.url }}" target="_blank">{{ additional_link.name }}</a>]

_publications/add_from_arxiv.py

@@ -1,4 +1,4 @@
-#!/bin/python3
+#! /usr/bin/env python3
 
 import argparse
 import arxiv
@@ -8,7 +8,7 @@
 
 
 def _first_non_stopword(title: str) -> str:
-    for word in re.split("\W", title.lower()):
+    for word in re.split(r"\W", title.lower()):
         if word in ("a", "an", "the", "is", "are", "what", "who", "your"):
             continue
         return word
@@ -20,8 +20,9 @@ def _author_lastname(author_name: str) -> str:
 
 
 def get_info(paper_id: str, out_dir: str) -> None:
+    client = arxiv.Client()
     search = arxiv.Search(id_list=[paper_id])
-    paper = next(search.results())
+    paper = next(client.results(search))
 
     summary = (
         paper.summary.replace("\n\n", "@@--@@")
@@ -30,15 +31,15 @@ def get_info(paper_id: str, out_dir: str) -> None:
     )
 
     tmpl = textwrap.dedent(
-        f"""
+        f"""\
         ---
         layout: publication
         title: "{paper.title}"
         authors: {", ".join(a.name for a in paper.authors)}
         conference:
         year: {paper.published.year}
         additional_links:
-        - {{name: "ArXiV", url: "https://arxiv.org/abs/%3Cspan%20class="pl-s1">{paper_id}"}}
+          - {{name: "ArXiV", url: "https://arxiv.org/abs/%3Cspan%20class="pl-s1">{paper_id}"}}
         tags: ["TODO"]
         ---
         {summary}

_publications/agrawal2023monitor.markdown

@@ -0,0 +1,17 @@
+---
+layout: publication
+title: Monitor-Guided Decoding of Code LMs with Static Analysis of Repository Context
+authors: Lakshya A Agrawal, Aditya Kanade, Navin Goyal, Shuvendu K Lahiri, Sriram Rajamani
+conference: NeurIPS
+year: 2023
+additional_links:
+   - {name: "ArXiV", url: "https://arxiv.org/abs/2306.10763"}
+   - {name: "NeurIPS website", url: "https://neurips.cc/virtual/2023/poster/70362"}
+   - {name: "code", url: "https://github.com/microsoft/monitors4codegen"}
+tags: ["autocomplete", "benchmark", "code completion", "code generation", "compilation", "completion", "dataset", "evaluation", "language model", "large language models", "program analysis", "static analysis", "tool"]
+---
+Language models of code (LMs) work well when the surrounding code provides sufficient context. This is not true when it becomes necessary to use types, functionality or APIs defined elsewhere in the repository or a linked library, especially those not seen during training. LMs suffer from limited awareness of such global context and end up hallucinating.
+
+Integrated development environments (IDEs) assist developers in understanding repository context using static analysis. We extend this assistance, enjoyed by developers, to LMs. We propose monitor-guided decoding (MGD) where a monitor uses static analysis to guide the decoding. We construct a repository-level dataset PragmaticCode for method-completion in Java and evaluate MGD on it. On models of varying parameter scale, by monitoring for type-consistent object dereferences, MGD consistently improves compilation rates and agreement with ground truth. Further, LMs with fewer parameters, when augmented with MGD, can outperform larger LMs. With MGD, SantaCoder-1.1B achieves better compilation rate and next-identifier match than the much larger text-davinci-003 model.
+
+We also conduct a generalizability study to evaluate the ability of MGD to generalize to multiple programming languages (Java, C# and Rust), coding scenarios (e.g., correct number of arguments to method calls), and to enforce richer semantic constraints (e.g., stateful API protocols). Our data and implementation are available at https://github.com/microsoft/monitors4codegen.

_publications/ahmed2024studying.markdown

@@ -0,0 +1,11 @@
+---
+layout: publication
+title: "Studying LLM Performance on Closed- and Open-source Data"
+authors: Toufique Ahmed, Christian Bird, Premkumar Devanbu, Saikat Chakraborty
+conference:
+year: 2024
+additional_links:
+  - {name: "ArXiV", url: "https://arxiv.org/abs/2402.15100"}
+tags: ["Transformers"]
+---
+Large Language models (LLMs) are finding wide use in software engineering practice. These models are extremely data-hungry, and are largely trained on open-source (OSS) code distributed with permissive licenses. In terms of actual use however, a great deal of software development still occurs in the for-profit/proprietary sphere, where the code under development is not, and never has been, in the public domain; thus, many developers, do their work, and use LLMs, in settings where the models may not be as familiar with the code under development. In such settings, do LLMs work as well as they do for OSS code? If not, what are the differences? When performance differs, what are the possible causes, and are there work-arounds? In this paper, we examine this issue using proprietary, closed-source software data from Microsoft, where most proprietary code is in C# and C++. We find that performance for C# changes little from OSS --> proprietary code, but does significantly reduce for C++; we find that this difference is attributable to differences in identifiers. We also find that some performance degradation, in some cases, can be ameliorated efficiently by in-context learning.

_publications/barchi2019code.markdown

@@ -0,0 +1,12 @@
+---
+layout: publication
+title: "Code Mapping in Heterogeneous Platforms Using Deep Learning and LLVM-IR"
+authors: Francesco Barchi, Gianvito Urgese, Enrico Macii, Andrea Acquaviva
+conference: DAC
+year: 2019
+additional_links:
+   - {name: "ACM", url: "https://dl.acm.org/doi/10.1145/3316781.3317789"}
+   - {name: "code", url: "https://gitlab.com/ecs-lab/deepllvm"}
+tags: ["optimization", "program analysis", "static analysis", "natural language processing"]
+---
+Modern heterogeneous platforms require compilers capable of choosing the appropriate device for the execution of program portions. This paper presents a machine learning method designed for supporting mapping decisions through the analysis of the program source code represented in LLVM assembly language (IR) for exploiting the advantages offered by this generalised and optimised representation. To evaluate our solution, we trained an LSTM neural network on OpenCL kernels compiled in LLVM-IR and processed with our tokenizer capable of filtering less-informative tokens. We tested the network that reaches an accuracy of 85% in distinguishing the best computational unit.

_publications/barchi2021exploration.markdown

@@ -0,0 +1,12 @@
+---
+layout: publication
+title: "Exploration of Convolutional Neural Network models for source code classification"
+authors: Francesco Barchi, Emanuele Parisi, Gianvito Urgese, Elisa Ficarra, Andrea Acquaviva
+journal: Engineering Applications of Artificial Intelligence
+year: 2021
+additional_links:
+   - {name: "ScienceDirect", url: "https://www.sciencedirect.com/science/article/pii/S0952197620303353"}
+   - {name: "code", url: "https://gitlab.com/ecs-lab/deepllvm"}
+tags: ["optimization", "static analysis", "program analysis", "language model"]
+---
+The application of Artificial Intelligence is becoming common in many engineering fields. Among them, one of the newest and rapidly evolving is software generation, where AI can be used to automatically optimise the implementation of an algorithm for a given computing platform. In particular, Deep Learning technologies can be used to the decide how to allocate pieces of code to hardware platforms with multiple cores and accelerators, that are common in high performance and edge computing applications. In this work, we explore the use of Convolutional Neural Networks (CNN)s to analyse the application source code and decide the best compute unit to minimise the execution time. We demonstrate that CNN models can be successfully applied to source code classification, providing higher accuracy with consistently reduced learning time with respect to state-of-the-art methods. Moreover, we show the robustness of the method with respect to source code pre-processing, compiler options and hyper-parameters selection.

_publications/barchi2022deep.markdown

@@ -0,0 +1,11 @@
+---
+layout: publication
+title: "Deep Learning Approaches to Source Code Analysis for Optimization of Heterogeneous Systems: Recent Results, Challenges and Opportunities"
+authors: Francesco Barchi, Emanuele Parisi, Andrea Bartolini, Andrea Acquaviva
+journal: Journal of Low Power Electronics and Applications
+year: 2022
+additional_links:
+   - {name: "MDPI", url: "https://www.mdpi.com/2079-9268/12/3/37"}
+tags: ["optimization", "review"]
+---
+To cope with the increasing complexity of digital systems programming, deep learning techniques have recently been proposed to enhance software deployment by analysing source code for different purposes, ranging from performance and energy improvement to debugging and security assessment. As embedded platforms for cyber-physical systems are characterised by increasing heterogeneity and parallelism, one of the most challenging and specific problems is efficiently allocating computational kernels to available hardware resources. In this field, deep learning applied to source code can be a key enabler to face this complexity. However, due to the rapid development of such techniques, it is not easy to understand which of those are suitable and most promising for this class of systems. For this purpose, we discuss recent developments in deep learning for source code analysis, and focus on techniques for kernel mapping on heterogeneous platforms, highlighting recent results, challenges and opportunities for their applications to cyber-physical systems.

_publications/berabi2024deepcode.markdown

@@ -0,0 +1,11 @@
+---
+layout: publication
+title: "DeepCode AI Fix: Fixing Security Vulnerabilities with Large Language Models"
+authors: Berkay Berabi, Alexey Gronskiy, Veselin Raychev, Gishor Sivanrupan, Victor Chibotaru, Martin Vechev
+conference:
+year: 2024
+additional_links:
+  - {name: "ArXiV", url: "https://arxiv.org/abs/2402.13291"}
+tags: ["repair", "vulnerability"]
+---
+The automated program repair field has attracted substantial interest over the years, but despite significant research efforts, creating a system that works well for complex semantic bugs such as security vulnerabilities has proven difficult. A promising direction to solve this challenge is by leveraging large language models (LLMs), which are increasingly used to solve various programming tasks. In this paper, we investigate the effectiveness of LLMs for solving code-repair task. We show that the task is difficult as it requires the model to learn long-range code relationships, a task that inherently relies on extensive amounts of training data. At the same time, creating a large, clean dataset for complex program bugs and their corresponding fixes is non-trivial. We propose a technique to address these challenges with a new approach for querying and fine-tuning LLMs. The idea is to use program analysis to limit the LLM's attention mechanism on the portions of code needed to perform the fix, drastically reducing the amount of required training data. Concretely, for training and inference, rather than feeding the entire program to the LLM, we reduce its code to a much shorter snippet that contains the reported defect together with the necessary context - and use that instead. Our evaluation shows that this code reduction approach substantially improves available models such as GPT-4 using few-shot learning, as well as fine-tuning models. To train and evaluate our system, we created a comprehensive code fixing dataset by extensively labeling 156 bug patterns (including 40 security rules), requiring complex interprocedural dataflow to discover. Our best system with Mixtral-8x7B can remove more than 80% of the reported defects while exactly matching the human fix in between 10 and 50% of cases, outperforming baselines based on GPT-3.5 and GPT-4, or based on window-based models like TFix.

_publications/bouzenia2024repairagent.markdown

@@ -0,0 +1,11 @@
+---
+layout: publication
+title: "RepairAgent: An Autonomous, LLM-Based Agent for Program Repair"
+authors: Islem Bouzenia, Premkumar Devanbu, Michael Pradel
+conference:
+year: 2024
+additional_links:
+  - {name: "ArXiV", url: "https://arxiv.org/abs/2403.17134"}
+tags: ["repair"]
+---
+Automated program repair has emerged as a powerful technique to mitigate the impact of software bugs on system reliability and user experience. This paper introduces RepairAgent, the first work to address the program repair challenge through an autonomous agent based on a large language model (LLM). Unlike existing deep learning-based approaches, which prompt a model with a fixed prompt or in a fixed feedback loop, our work treats the LLM as an agent capable of autonomously planning and executing actions to fix bugs by invoking suitable tools. RepairAgent freely interleaves gathering information about the bug, gathering repair ingredients, and validating fixes, while deciding which tools to invoke based on the gathered information and feedback from previous fix attempts. Key contributions that enable RepairAgent include a set of tools that are useful for program repair, a dynamically updated prompt format that allows the LLM to interact with these tools, and a finite state machine that guides the agent in invoking the tools. Our evaluation on the popular Defects4J dataset demonstrates RepairAgent's effectiveness in autonomously repairing 164 bugs, including 39 bugs not fixed by prior techniques. Interacting with the LLM imposes an average cost of 270,000 tokens per bug, which, under the current pricing of OpenAI's GPT-3.5 model, translates to 14 cents of USD per bug. To the best of our knowledge, this work is the first to present an autonomous, LLM-based agent for program repair, paving the way for future agent-based techniques in software engineering.

_publications/casey2024survey.markdown

@@ -0,0 +1,11 @@
+---
+layout: publication
+title: "A Survey of Source Code Representations for Machine Learning-Based Cybersecurity Tasks"
+authors: Beatrice Casey, Joanna C. S. Santos, George Perry
+conference:
+year: 2024
+additional_links:
+  - {name: "ArXiV", url: "https://arxiv.org/abs/2403.10646"}
+tags: ["survey", "cybersecurity", "vulnerability"]
+---
+Machine learning techniques for cybersecurity-related software engineering tasks are becoming increasingly popular. The representation of source code is a key portion of the technique that can impact the way the model is able to learn the features of the source code. With an increasing number of these techniques being developed, it is valuable to see the current state of the field to better understand what exists and what's not there yet. This paper presents a study of these existing ML-based approaches and demonstrates what type of representations were used for different cybersecurity tasks and programming languages. Additionally, we study what types of models are used with different representations. We have found that graph-based representations are the most popular category of representation, and Tokenizers and Abstract Syntax Trees (ASTs) are the two most popular representations overall. We also found that the most popular cybersecurity task is vulnerability detection, and the language that is covered by the most techniques is C. Finally, we found that sequence-based models are the most popular category of models, and Support Vector Machines (SVMs) are the most popular model overall.