ISSTA 2026
Sat 3 - Fri 9 October 2026 Oakland, California, United States
co-located with SPLASH/ISSTA 2026
SPLASH/ISSTA 2026 (series) / ISSTA 2026 (series) /

Artifact evaluationISSTA 2026

The Artifact Evaluation process is a service provided by the community to help authors of accepted papers provide more substantial supplements to their papers so that future researchers can more effectively build on and compare with previous work. ISSTA invites submissions for artifact evaluation. Research artifacts denote digital objects that were either created by the authors of a research article to be used as part of their study or generated by their experiments.

The artifact evaluation (AE) process aims to foster reproducibility and reusability. Reproducibility refers to researchers or practitioners being able to validate the paper’s results using the provided artifact. Reusability means that researchers can extend or use the artifact in a different context or for a different use case. Overall, the artifact evaluation process allows our field to progress by incentivizing and supporting authors to make their artifacts openly available and improve their quality. Furthermore, a formal artifact evaluation documents the outstanding nature of the published research through recognizable and recognized badges stamped directly on the published papers. Therefore, it is common to offer the authors of accepted papers at high-quality conferences, such as ISSTA, an artifact evaluation service before publication.

More details can be found here: ACM guidelines on Artifact Review and Badging Version 1.1.

Call for Artifacts

ISSTA 2026 will offer a voluntary artifact evaluation for accepted submissions to the research track. The outcome will not alter the paper acceptance decision.

Packaging

To avoid installation problems (e.g., due to software dependencies) during artifact review, we require that authors provide their artifacts as Container (Docker, Podman) or VM (OVF/OVA) images. If you think that your artifact needs to be submitted in a different format, please contact the artifact evaluation committee (AEC) chairs.

The AEC members should be able to evaluate your artifact quickly. For longer runs, please try to show progress messages (e.g., completion percentage). For long-running experiments, please provide a reduced experiment scope that can be validated in one day or less and provide execution instructions for both the full and the reduced scope in the artifact documentation. In case of special hardware requirements (e.g., GPUs, specific CPU models), please indicate them in the submission form.

Please try to avoid downloading content over the internet during experiments or tests (to ensure self-containedness), as well as closed source software libraries, frameworks, operating systems, and container formats unless these are necessary for your submitted work. If possible, widely supported file formats should be used for the artifact (e.g., .zip or .tar.gz for archives, .odt or .pdf for documents, and CSV/JSON for data).

In case your artifact relies on commercial services (e.g., closed AI models), you have to make sure that these are accessible to reviewers during the entire review period and without compromising reviewers’ anonymity.

Every artifact submission must include:

  • a README main file consisting of two parts:
    • A Getting Started guide, which should contain an artifact description, installation instructions (if any), and a method to test the installation (a “smoke test”). This could be, for instance, a command to confirm that the code is installed and working, and its expected output. Reviewers should be able to complete the Getting Started guide within 30 minutes.
    • Step-by-step instructions, which should contain detailed reproduction steps for any experiments or activities supporting the paper conclusions. You should state all paper claims supported by the artifact (and how), as well as all paper claims not supported by the artifact (and why). Depending on your artifact’s nature, the instructions may differ:
      • Data artifacts should cover aspects relevant to understanding the context, data provenance, ethical and legal statements (if relevant), and storage requirements.
      • Software artifacts should contain instructions on how to reproduce the paper claims (e.g., tables/figures), and documentation on how to use the tool.
      • Proof artifacts should cover how the individual parts of the proof relate to formalisms presented in the paper. Please consult these guidelines for submitting and reviewing proof artifacts.

      In addition to the scenarios above that reproduce the paper results, we encourage you to include further instructions on how the AEC can run the artifact on different experiments, as well as documentation on the artifact code and layout.

  • a REQUIREMENTS file covering the architecture in which your artifact was packaged (e.g., x86, ARM) and hardware/software requirements (e.g., storage or non-commodity peripherals, Docker, VM, and OS). We encourage you to also include machine-readable files describing dependencies (e.g., Dockerfile, Pipfile, dune-project), if relevant.
  • a STATUS file stating the badge(s) you are applying for, as well as a short justification why you think that the artifact deserves the respective badge(s)
  • a LICENSE file describing the terms of use and distribution rights

All files must be provided as plain text (e.g., txt, md) or PDF and packaged with the artifact.

Submission

Submit your artifact through the submission system: https://issta2026-ae.hotcrp.com/

Please host your artifact on a platform that does not track IP addresses, so as to not undermine reviewer anonymity. We will check the submitted links and remove them if they violate this property.

Evaluation Phases

Kick-the-tires

Reviewers check the artifact’s packaging and integrity, and identify any possible setup problems that may prevent the artifact from being properly evaluated. Authors are given a 4-day period to address the identified issues by submitting an updated artifact link in their response text. The author response period will resemble paper rebuttals, i.e., there will be no interactive discussions. AEC members will phrase any issues encountered as concisely as possible, and authors are expected to address these issues in a single response.

Artifact assessment

Reviewers check whether the artifacts live up to the claims made in the accompanying documentation. Based on their reports and the ensuing AEC discussion, the artifact may be awarded up to two badges (see below).

Publication

Once the artifact has been evaluated, the authors are notified about the outcome, and the publication chairs are notified of any badges to be added to the paper. As badges are awarded for the artifact version available to AEC after any kick-the-tires reviews, this is the version authors should reference in their papers and that will be referenced from the ACM DL if the authors choose to make their artifact publicly available (which we strongly encourage).

Evaluation Criteria and Outcome

An artifact may be awarded one of the following badges, in accordance with the ACM artifact badging guidelines.

FunctionalThe artifact is found to be documented, consistent, complete, exercisable, and to include appropriate evidence of verification and validation.
ReusableThe artifact is of a quality that significantly exceeds minimal functionality. That is, it has all the qualities of the Functional level, but, in addition, it is very carefully documented and well-structured to the extent that reuse and repurposing is facilitated.
AvailableAuthor-created artifacts relevant to the paper have been placed on a publicly accessible archival repository. A DOI for the object is provided. Repositories used to archive data should have a declared plan to enable permanent accessibility (e.g., Zenodo, FigShare, or Dryad). . We recommend adding the DOI link in a dedicated data availability statement at the end of your paper.

Artifacts that go beyond expectations of quality will receive a Distinguished Artifact award. The selection procedure will be based on review scores and feedback from the AEC.

FAQ

What is the purpose of the artifact evaluation?

We want to encourage authors to provide more substantial evidence to their papers, and reward authors who create research artifacts. At the same time, we want to simplify the reproduction of results presented in the paper and ease future comparison with existing approaches. To ensure that the submitted artifacts provide the best possible value to the community, our goal is to be constructive and to improve the submitted artifacts.

What qualifies as an artifact?

Artifacts include (but are not limited to) software, tools, frameworks, datasets, test suites, machine-checkable proofs, or any combination of the above. We will assess the artifacts themselves, and not the quality of the research that produced them (which has been assessed by the conference PC).

Why are ACM’s “Results Validated” badges not awarded in the artifact evaluation?

The badges indicate that a study’s results are confirmed by a subsequent study with or without artifacts of the original study. Therefore, they indicate consistency across published results in a paper rather than quality of an artifact. Hence, artifact evaluation is not a necessary prerequisite for awarding these badges and the decision to award these badges may be made without any artifact evaluation involved. If a reproducibility/replicability study is accepted for publication and it is accompanied by an artifact, that artifact can be submitted to the artifact evaluation to obtain one of the “Artifacts Evaluated” badges.

Important Dates AoE (UTC-12h)
Mon 29 Jun 2026
Submission deadline
Wed 8 - Sun 12 Jul 2026
Author response period
Wed 12 Aug 2026
Author notification
Sat 30 May 2026
Submission system opens
Submission Link
https://issta2026-ae.hotcrp.com/
Artifact Evaluation Committee