Reproduction Package for CAV 2025 Submission `Btor2-Select: Machine Learning Based Algorithm Selection for Hardware Model Checking'

Abstract

This artifact is a reproduction package for the manuscript “Btor2-Select: Machine Learning Based Algorithm Selection for Hardware Model Checking” (#16), submitted to CAV 2025. It is archived on Zenodo with the DOI 10.5281/zenodo.15209621.

The manuscript investigates machine-learning-based algorithm selection for hardware model checking in the Btor2 language. It proposes a framework to train an algorithm selector that predicts the best available off-the-shelf model checker for a given verification task. The selector then invokes the predicted backend model checker to solve the task.

This artifact supports the reproduction of:

1. the training of the algorithm selector using the provided performance data and
2. the evaluation of the algorithm selector (integrated with backend model checkers) against state-of-the-art hardware model checkers.

The artifact consists of source code, precompiled executables, and input data used in the training and evaluation of the manuscript, as well as the results produced from the experiments. Specifically, it includes the selector trained by our framework, the backend verifiers to be selected, a set of Btor2 verification tasks collected for training and evaluation, the experimental data generated from the evaluation, and instructions to run the tools and experiments.

Badge Claims

We claim all three badges: Available, Functional, and Reusable. The artifact is available because it is archived on Zenodo with a DOI; It is functional because (1) it includes all the required dependencies, raw data, source code, and evaluation scripts bundled (completeness); (2) it can reproduce the results in the manuscript (consistency); and (3) the evaluation is conducted using BenchExec, which ensures the reliability of performance measurement and the correctness of verification results (correctness) We also provide license and detailed documentation for the usage of our software to facilitate the reuse of (part of) the artifact in other contexts. Our open-source project includes the license, up-to-date dependencies, and detailed instructions on building, interfacing, and usage on different environments.

Artifact Requirements

This reproduction package works best with the SoSy-Lab Virtual Machine, which runs Ubuntu 24.04 LTS and has all the required dependencies installed. If you test the artifact with this VM, you do not need to install any package. Please login the VM as user vagrant via GUI or SSH. The VM has been tested with VirtualBox 7.0 on a Linux (Ubuntu 24.04) computer.

The benchmarking framework BenchExec used in our experiments relies on control groups and namespaces provided by modern Linux kernels. Our tool, Btor2-Select, was tested with Python 3.12. Additionally, one of the compared tools, super_prove, is executed in a containerized environment with Podman (tested with version 4.9.3).

To perform most of the experiments included in this artifact, a machine with 16 GB of RAM, 4 CPU cores, and 15 GB of disk space is needed. Reproducing the training of certain models with the exact settings used in our manuscript requires more than 50 GB of RAM. We also provide a lightweight setting that allows these models to be trained with only 16 GB of RAM, while maintaining similar performance to the original configuration.

A full reproduction of the training part required over 25 hours of wall-clock time on a server equipped with 2 TB of RAM and two 2.0 GHz AMD EPYC 7713 CPUs, each with 128 processing units. The evaluation phase consumed more than 346 hours of CPU time on machines with 3.4 GHz processors. For demonstration purposes, a subset of benchmark tasks can be used. Training on a subset of 450 Btor2 tasks took approximately less than a minute, while evaluation on 30 selected simple Btor2 tasks took roughly 5 minutes on a standard laptop. This artifact README includes time estimates for the various commands referenced throughout.

Contents

This artifact contains the following items:

• README.{html,md}: this documentation (we recommend viewing the HTML version with a browser)
• cav2025-paper16.pdf: the submitted manuscript
• LICENSE.txt: license information of the artifact
• btor2-select/: the machine-learning-based framework for algorithm selection and the trained selector (our open-source project, at commit b60d073d)
  • bin/: contains the executables of Btor2-Select and Btor2-Para (a parallel portfolio constructed for evaluating Btor2-Select)
  • btor2select/: contains the main production codes for Btor2-Select, including:
    • btor2_select.py: performs inference using the trained selector and executes the selected backend model checker
    • train.py: trains the proposed algorithm selector
    • cross_validation.py: conducts cross-validation analysis
    • Other supporting scripts for different ML models, e.g., PWC-SVM-BoKW, PWC-SVM-WL
  • data/demo/: a small collection of Btor2 verification tasks and their performance data, intended for use in a training demo.
  • README: for additional information
• perf-eval-hwmc: a directory for evaluating performance of backend model checkers, consisting of:
  • benchmarks/: a set of Btor2 verification tasks collected for training and evaluation
  • dataset/: the performance dataset used for training
  • verifiers/: tool archives of several backend model checkers (including both hardware and software verifiers)
  • benchexec/: a checkout of BenchExec, a reliable benchmarking framework with precise resource management, used to perform the evaluation
  • bench-defs/: benchmark definitions used by BenchExec
  • README: for further information
• data-submission/: a directory containing the raw and processed data obtained from our experiments
  • cross_val/: the cross-validation results
  • evaluation/: the evaluation results of Btor2-Select, Btor2-Para, ABC, and super_prove
    • paper-results/: the results presented in the manuscript
    • demo-results/: the results of the demo runs
• Makefile: a file that assembles commands for running experiments and processing data