Artifact EvaluationASE 2026

Call for Artifacts

The artifact evaluation track aims to review, promote, share, and catalog the research artifacts of accepted papers to the Research, Industry showcase, NIER, and Tool demonstration tracks of the current edition of ASE (2026). Authors can submit an artifact for the Artifacts Available and Artifacts Reusable badges. Our primary goal will be to help authors make their artifacts available and reusable. Definitions for all badges can be found on ACM Artifact Review and Badging Version 1.1.

Guidelines

Authors must perform the following steps to submit an artifact:

1. Prepare the artifact

Both executable and non-executable artifacts may be submitted.

Every artifact submission must include:

• a README main file consisting of two parts:
  • A Getting Started guide, which should contain an artifact description, installation instructions (if any), and a method to test the installation (a “smoke test”). This could be, for instance, a command to confirm that the code is installed and working, and its expected output. Reviewers should be able to complete the Getting Started guide within 30 minutes.
  • Step-by-step instructions, which should contain detailed reproduction steps for any experiments or activities supporting the paper’s conclusions. You should state all paper claims supported by the artifact (and how), as well as all paper claims not supported by the artifact (and why). Depending on your artifact’s nature, the instructions may differ. Here are a few examples:
    • Data artifacts should cover aspects relevant to understanding the context, data provenance, ethical and legal statements (if relevant), and storage requirements.
    • Software artifacts should contain instructions on how to reproduce the paper claims (e.g., tables/figures), and documentation on how to use the tool.
    • Proof artifacts should cover how the individual parts of the proof relate to formalisms presented in the paper. Please consult these guidelines for submitting and reviewing proof artifacts.
  • In addition to the scenarios above that reproduce the paper results, we encourage you to include further instructions on how the AEC can run the artifact on different experiments, as well as documentation on the artifact code and layout.
• a REQUIREMENTS file covering the architecture in which your artifact was packaged (e.g., x86, ARM) and hardware/software requirements (e.g., storage or non-commodity peripherals, Docker, VM, and OS). We encourage you to also include machine-readable files describing dependencies (e.g., Dockerfile, Pipfile, dune-project), if relevant.
• a STATUS file stating the badge(s) you are applying for, as well as a short justification of why you think that the artifact deserves the respective badge(s)
• a LICENSE file describing the terms of use and distribution rights. For submissions aiming for the Available badge, the license needs to ensure public availability. In the spirit of Open Science, we recommend adopting an open source license for executable artifacts and a data license for non-executable artifacts.

All files must be provided as plain text (e.g., txt, md) or PDF and packaged with the artifact.

Executable artifacts contain a script, tool, or software system. For these artifacts, authors should consider the following:

1. To avoid installation problems (e.g., due to software dependencies) during artifact review, we require that authors provide their artifacts as Container (Docker, Podman) or VM (OVF/OVA) images. If you think that your artifact needs to be submitted in a different format, please contact the artifact evaluation committee (AEC) chairs.
2. The AEC members should be able to evaluate your artifact quickly. For longer runs, please try to show progress messages (e.g., completion percentage). For long-running experiments, please provide a reduced experiment scope that can be validated in one day or less and provide execution instructions for both the full and the reduced scope in the artifact documentation.
3. In case of specific hardware requirements (e.g., GPUs, specific CPU models), please indicate them in the submission form.
4. Please try to avoid downloading content over the internet during experiments or tests (to ensure self-containedness), as well as closed source software libraries, frameworks, operating systems, and container formats unless these are necessary for your submitted work. If possible, widely supported file formats should be used for the artifact (e.g., .zip or .tar.gz for archives, .odt or .pdf for documents, and CSV/JSON for data).
5. In case your artifact relies on commercial services (e.g., closed AI models), you have to make sure that these are accessible to reviewers during the entire review period and without compromising reviewers’ anonymity. This could be achieved by providing API keys in the submission form. This information will only be available to the reviewers of your artifact and the chairs. Please make sure these API keys are rate-limited, if possible.

Non-executable artifacts only contain data and documents that can be used with a simple text editor, a PDF viewer, or some other common tool (e.g., a spreadsheet program in its basic configuration). These artifacts can be submitted as a single, optionally compressed package file (e.g., a tar, zip, or tar.gz file).

1. Prepare instructions and list the tools that are required to open the files.
2. In case of datasets: Please document aspects related to understanding the context, data provenance, ethical and legal statements (as long as relevant), and storage requirements.
3. Clearly describe and explain the features (e.g., columns) in the dataset and the schema of the dataset.
4. List the usage scenarios.

2. Make the artifact available

The authors need to make the packaged artifact available so that the PC can access it. Artifacts must be made available via an archival repository.

• There are several permanent archival repositories that can be used. An example is Software Heritage (see their submission guide), which provides long-term availability of software source code. Other solutions include Zenodo and FigShare.
• Please note that platforms that do not guarantee long-term archival, which presently includes GitHub, do not qualify.
• Obtain the assigned DOI for the artifact.
• Temporary drives (e.g., Dropbox, Google Drive) are considered to be non-persistent, the same as individual websites of the submitting authors, as these are prone to changes.

3. Submit the artifact

Submit a short abstract (maximum two pages) that has subsections with all the following titles:

1. Paper title
2. Link to the accepted paper. The paper should be made available so the reviewers can access it.
3. Purpose: a brief description of what the artifact does.
4. Badge: A list of badge(s) the authors are applying for, as well as the reasons why the authors believe that the artifact deserves that badge(s).
5. Technology skills assumed by the reviewer evaluating the artifact and hardware requirements.
6. Provenance: where the artifact can be obtained.
7. Instructions: Provide a list of instructions on how the reviewers should access the artifact, including a list of tools required. Please also mention if running your artifact requires any specific Operating Systems or other, unusual environments, including GPUs. If the data is very large or requires a special tool to open, the instructions should be submitted with the artifact. The submission should also include the explanation and schema of the dataset and its usage scenarios. If reviewers have to spend more than 30 minutes installing the tool, it is unlikely that the artifact will be accepted.

By submitting your article to an ACM Publication, you are hereby acknowledging that you and your co-authors are subject to all ACM Publications Policies, including ACM’s new Publications Policy on Research Involving Human Participants and Subjects. Alleged violations of this policy or any ACM Publications Policy will be investigated by ACM and may result in a full retraction of your paper, in addition to other potential penalties, as per ACM Publications Policy.

Review

The ASE artifact evaluation track uses a single-anonymous review process. All artifacts will receive three reviews.

Three AEC members will review each submission. If AEC members do not find sufficient substantive evidence for the availability or reusability of the artifact according to the definitions of the badges, the abstract will be rejected.

Evaluation Criteria and Outcome

An artifact may be awarded one of the following badges, in accordance with the ACM artifact badging guidelines.

	Functional	The artifact is found to be documented, consistent, complete, exercisable, and to include appropriate evidence of verification and validation.
	Reusable	The artifact is of a quality that significantly exceeds minimal functionality. That is, it has all the qualities of the Functional level, but, in addition, it is very carefully documented and well-structured to the extent that reuse and repurposing is facilitated.
	Available	Author-created artifacts relevant to the paper have been placed on a publicly accessible archival repository. A DOI for the object is provided. Repositories used to archive data should have a declared plan to enable permanent accessibility (e.g., Zenodo, FigShare, or Dryad). . We recommend adding the DOI link in a dedicated data availability statement at the end of your paper.

Artifacts that go beyond expectations of quality will receive a Distinguished Artifact award. The selection procedure will be based on review scores and feedback from the AEC.

FAQ

What is the purpose of the artifact evaluation?

We want to encourage authors to provide more substantial evidence to their papers, and reward authors who create research artifacts. At the same time, we want to simplify the reproduction of results presented in the paper and ease future comparison with existing approaches. To ensure that the submitted artifacts provide the best possible value to the community, our goal is to be constructive and to improve the submitted artifacts.

What qualifies as an artifact?

Artifacts include (but are not limited to) software, tools, frameworks, datasets, test suites, machine-checkable proofs, or any combination of the above. We will assess the artifacts themselves, and not the quality of the research that produced them (which has been assessed by the conference PC).

Why are ACM’s “Results Validated” badges not awarded in the artifact evaluation?

The badges indicate that a study’s results are confirmed by a subsequent study with or without artifacts of the original study. Therefore, they indicate consistency across published results in a paper rather than quality of an artifact. Hence, artifact evaluation is not a necessary prerequisite for awarding these badges and the decision to award these badges may be made without any artifact evaluation involved. If a reproducibility/replicability study is accepted for publication and it is accompanied by an artifact, that artifact can be submitted to the artifact evaluation to obtain one of the “Artifacts Evaluated” badges.