CALL FOR ARTIFACT

The reproducibility of experimental results is crucial to foster an atmosphere of trustworthy, open, and reusable research. To improve and reward reproducibility, FormaliSE 2024 continues its Artifact Evaluation (AE) procedure. A main goal of the artifact evaluation is to enable future researchers to more effectively build on and compare with previous work.

An artifact is any additional material (software, data sets, machine-checkable proofs, etc.) that substantiates the claims made in the paper and ideally makes them fully reproducible. Submission of an artifact is optional but encouraged for all papers where it can support the results presented in the paper.

Artifact review is single-blind (the paper corresponding to an artifact must still follow the double-blind submissions requirements) and will be conducted concurrently with the paper reviewing process.

Artifacts will be reviewed by a separate Artifact Evaluation Committee. We will be attributing 3 badges, according to EAPLS guidelines:

1. Artifact functional: documented, consistent, complete, exercisable;
2. Artifact reusable: exceeding functional, by being carefully documented and well-structured for reuse and repurposing, see below for details;
3. Artifact available: available on a publicly accessible archival repository for permanent availability that provides a Digital Object Identifier (DOI).

Artifacts will be assessed with respect to their consistency with the results presented in the paper, their completeness, their documentation, and their ease of use.

The AE will include an initial check for technical issues. Authors of artifacts will interact with the reviewers from 22 December 2023 to 12 January 2024 by means of a shared document, to help resolve any technical problems that prevent the evaluation of an artifact, if necessary.

IMPORTANT DATES

• Artifact Registration 08 December 2023 15 December 2023
• Artifact Submission 15 December 2023 22 December 2023
• Address technical problems (light review) 22 December 2023 - 12 January 2024 05 January 2024 - 14 January 2024
• Artifact Notification: 23 January 2024

ARTIFACT SUBMISSION GUIDELINES

Submission site: https://formalise24.hotcrp.com/ (artifacts track).

When the paper is submitted, authors should indicate that they plan to submit an artifact by registering via the submission site.

A final artifact submission should consist of

• an abstract: that summarizes the artifact and explains its relation to the paper including:
  • a URL from which a .zip file containing the artifact can be downloaded – we encourage you to provide a DOI – and
  • if applicable, a description of any special requirements beyond a VM image (e.g., cloud-computing resources, certain hardware, etc.), and,
  • if you are aiming for a reusable badge, an explanation why you believe your artifact is reusable, and
  • detailed specific instructions for an early light review that allows reviewers to: (1) verify that the artifact can properly run; and (2) perform a short evaluation of the artifact before the full evaluation and detect any difficulties, and
• a .pdf: file of the submitted paper.

Packaging Guidelines

Your artifact .zip file must contain the following elements.

• The artifact, i.e., data, software, libraries, scripts, etc. required to replicate the results of your paper. Please prepare a Virtual Machine. You could use VirtualBox to save a VM image as an OVA file.
• A LICENSE file. The license needs to allow the artifact evaluation chairs to download and distribute the artifact to the artifact evaluation committee members and the artifact evaluation committee members must be allowed to evaluate the artifact, e.g., use, execute, and modify the artifact for the purpose of artifact evaluation.
• A README ext file that introduces the artifact to the user and guides the user through replication of your results. Ideally, it should consist of the following parts:
  • Any additional requirements for running the artifact, such as hardware requirements or additional proprietary software;
  • The expected total runtime to run the experiments;
  • Detailed and specific reproducibility instructions to setup and use the artifact to replicate the results in the paper; including an explanation which claims and results cannot be replicated and why.

If you are not in a position to prepare the artifact as above, please contact PC chairs for an alternative arrangement. For instance, if you cannot provide us with a VM that contains licensed software, e.g., MatLab, please contact us, so we can find a solution..

Members of the artifact evaluation committee and the program committee are asked to use submitted artifacts for the sole purpose of evaluating the contribution associated with the artifact..

Evaluation Criteria

All artifacts are evaluated by the artifact evaluation committee. Each artifact will be reviewed by at least two committee members, ideally three. Reviewers will read the paper and explore the artifact to evaluate how well the artifact supports the claims and results of the paper..

Criteria for the “functional” badge

The evaluation and the awarding of the functional badge is based on the following questions:

• Is the artifact documented, i.e., at minimum, an inventory of artifacts is included, and sufficient description to enable the artifacts to be exercised is included.
• Is the artifact consistent, i.e., relevant to the associated paper, significantly contributing to the generation of its main results?
• Is the artifact complete, i.e., rand as far as possible, are all components relevant to the associated paper included?
• Is the artifact runnable, i.e., can the software/scripts that generates the results in the associated paper be executed successfully, and can included data be accessed and appropriately manipulated?

Criteria for the “available” badge

To get the available badge, please upload your VM to a permanent repository that provides a DOI, such as Zenodo, figshare, or Dryad, and use this DOI link in your artifact submission..

Additional criteria for the “reusable” badge

Artifacts seeking the “reusable” badge need to clear a significantly higher bar than functional artifacts. First, they must be available, i.e., receive an “available” badge. Second, we expect a higher level of quality during the evaluation of the functional level. Third, in addition to the criteria from the functional level, they are evaluated against the following criteria:

• Does the artifact have a license which allows reuse, repurposing, and which is easy to use?
• Are all dependencies and used libraries well documented and up to date?
• Does the artifact README explain in sufficient detail how the artifact can be used beyond the paper?
• Does the artifact provide documented interfaces for extensions, or is the artifact open source?
• Can the artifact be used in a different environment, e.g., built on another system, used outside of the VM image, etc.?

Artifact Evaluation Chairs

For any questions please contact the two artifact evaluation chairs:

• Tom van Dijk (t.vandijk@utwente.nl)
• Raúl Pardo (raup@itu.dk)