Artifact Evaluation Survey

Welcome to this survey on artifact evaluation! 

Several conferences currently feature artifact evaluation tracks, to which authors can submit the data, proofs, or code of an accepted paper. In this survey, we are interested in your opinion on artifact evaluation.

Your responses will help to form a clearer picture on the expectations for artifacts in computer science research.

This survey should take approximately 20 minutes.

Thank you!

Ben Hermann (Universität Paderborn)
Janet Siegmund (Universität Passau)
Stefan Winter (Technische Universität Darmstadt)

There are 36 questions in this survey.

General

1 On which of the following artifact evaluation committees have you served or have you chaired?

2011 2012 2013 2014 2015 2016 2017 2018 2019
ICSE
FSE
ISSTA
SAS
VISSOFT
OOPSLA
ECOOP
POPL
PLDI
SLE
PPoPP
ICFP
CGO
MODELS
CAV
TACAS

2 Are you familiar with the ACM Policy on Artifact Review and Badging?

Please choose only one of the following:

  • Yes
  • No

3 How helpful do you find these criteria when reviewing artifact?

Only answer this question if the following conditions are met:
Answer was 'Yes' at question '2 [g1]' (Are you familiar with the ACM Policy on Artifact Review and Badging?)

Please choose the appropriate response for each item:

Not helpful
1		 2 3 4 5 6 7 8 9 Very helpful
10

4 Please elaborate on your previous answer:

Only answer this question if the following conditions are met:
Answer was NOT at question '3 [g2]' (How helpful do you find these criteria when reviewing artifact? ())

Please write your answer here:

5

In your words, what is the purpose of artifact evaluation?

Please write your answer here:

Artifact Evaluation

In the following, we will ask questions on your experience as an artifact evaluation committee member or chair.

Many conferences differentiate between three types of artifact: code (e.g., an implementation of a prototype), proofs (e.g., for the coq proof assistant), and data (e.g., the results of a controlled experiment). In this survey, we also make this distinction, as requirements might differ between these categories.

6 What is your minimum requirement to accept an artifact in an artifact evaluation process in general? (e.g., reproducibility of article results, easy setup, documentation)

Please write your answer here:

7

What is your minimum expectation for the code portion of an artifact?
(e.g., code quality, documentation, packaging, size)

Please write your answer here:

8 What is your minimum expectation for the proof portion of an artifact?
(e.g., completeness, understandability)

Please write your answer here:

9 What is your minimum expectation for the data portion of an artifact?
(e.g., raw data, data format, format documentation, size)

Please write your answer here:

10 Do you have any other comments on the requirements for accepting an artifact?

Please write your answer here:

11

Do you think that the effort of artifact evaluation is justified?

Please choose only one of the following:

  • Yes
  • No

12 Please elaborate on your previous answer:

Only answer this question if the following conditions are met:
Answer was 'Yes' or 'No' at question '11 [ae6]' (Do you think that the effort of artifact evaluation is justified? )

Please write your answer here:

13

What are the reasons why you have recommended to accept or reject an artifact?

Please write your answer here:

14 Which arguments of your fellow artifact evaluation committee members for the acceptance or rejection of an artifact do you recall?

Please write your answer here:

15 Have you ever submitted an artifact to an artifact evaluation track?

Please choose only one of the following:

  • Yes
  • No

16

Which reasons for the acceptance/rejection of your artifact do you recall?
Please indicate for each reason if you consider it justified or not.

Only answer this question if the following conditions are met:
Answer was 'Yes' at question '15 [ae10]' (Have you ever submitted an artifact to an artifact evaluation track?)

Please write your answer here:

Artifact Usage

In the following, we will ask questions on your experience as a researcher with artifacts.
 

17 From how many artifacts have you used code for other reasons than evaluating them? *

Please choose the appropriate response for each item:

None 1-5 5-10 10-20 20-30 > 30

18 Were your expectations met by the code from the artifacts?

Only answer this question if the following conditions are met:
Answer was '1-5' or '5-10' or '10-20' or '20-30' or '> 30' at question '17 [au1]' (From how many artifacts have you used code for other reasons than evaluating them? ())

Please choose only one of the following:

  • Yes
  • No

19

Please elaborate on the (un-)met expectations towards the code:

Only answer this question if the following conditions are met:
Answer was 'Yes' or 'No' at question '18 [au3]' (Were your expectations met by the code from the artifacts?)

Please write your answer here:

20 From how many artifacts have you used proofs for other reasons than evaluating them? *

Please choose the appropriate response for each item:

None 1-5 5-10 10-20 20-30 > 30

21 Were your expectations met by the proofs in the artifact?

Only answer this question if the following conditions are met:
Answer was '1-5' or '5-10' or '10-20' or '20-30' or '> 30' at question '20 [au4]' (From how many artifacts have you used proofs for other reasons than evaluating them? ())

Please choose only one of the following:

  • Yes
  • No

22

Please elaborate on the (un-)met expectations towards the proofs:

Only answer this question if the following conditions are met:
Answer was 'Yes' or 'No' at question '21 [au6]' (Were your expectations met by the proofs in the artifact?)

Please write your answer here:

23 From how many artifacts have you used data for other reasons than evaluating them? *

Please choose the appropriate response for each item:

None 1-5 5-10 10-20 20-30 > 30

24 Were your expectations met by data in the artifact?

Only answer this question if the following conditions are met:
Answer was '1-5' or '5-10' or '10-20' or '20-30' or '> 30' at question '23 [au7]' (From how many artifacts have you used data for other reasons than evaluating them? ())

Please choose only one of the following:

  • Yes
  • No

25

Please elaborate on the (un-)met expectations towards the data:

Only answer this question if the following conditions are met:
Answer was 'Yes' or 'No' at question '24 [au9]' (Were your expectations met by data in the artifact?)

Please write your answer here:

26 Have you noticed a difference between artifacts with and without a successful evaluation?

Please choose only one of the following:

  • Yes
  • No

27 Please elaborate on your previous answer:

Only answer this question if the following conditions are met:
Answer was 'Yes' at question '26 [au10]' (Have you noticed a difference between artifacts with and without a successful evaluation?)

Please write your answer here:

28 Do you have any other comments on artifact usage from the perspective of a researcher?

Please write your answer here:

Concluding Questions

29 Are you satisfied with the current artifact evaluation process?

Please choose only one of the following:

  • Yes
  • No

30 Please elaborate:

Only answer this question if the following conditions are met:
Answer was NOT at question '29 [f0]' (Are you satisfied with the current artifact evaluation process?)

Please write your answer here:

31

In your perception, has the quality of accepted artifacts improved compared to earlier years? Please elaborate.

Please write your answer here:

32

What factors influence the quality of an artifact (e.g., the research group, the domain, the type of artifact)? Please elaborate.

Please write your answer here:

33 If you have any comments that did not fit into our survey questions and that you wish to share, please elaborate here:

Please write your answer here:

34 Please check if you would like us to send you the results from the survey. *

Please choose only one of the following:

  • Yes
  • No

35 Please check if we may keep your email to approach you for future surveys. *

Please choose only one of the following:

  • Yes
  • No

36 Please provide your email address

Only answer this question if the following conditions are met:
-------- Scenario 1 --------
Answer was 'Yes' at question '34 [f1]' (Please check if you would like us to send you the results from the survey.)
-------- or Scenario 2 --------
Answer was 'Yes' at question '35 [f2]' (Please check if we may keep your email to approach you for future surveys.)

Please check the format of your answer.
Please write your answer here:

Thank you for completing our survey!