Artefact Evaluation platform

[acocac]

Summary Sentence

A platform to handle the artefact evaluation process.

What needs to be done?

• Conduct pre-feasibility assessments of suitable management platforms.
• Create mini-tutorials how to use the selected platform.
• Set-up the platform for the AE process.

See also:

• Artefact Evaluation process #1

---

Platforms

HotCRP

• https://hotcrp.com
• Test sites created by @acocac and @rolyp

System admin (to set before)

• Users
• Topics
• Deadline
• Review Scores
  • Overall merit
  • Reviewer familiarity
• Labels for badges

Users

• Add accounts ok, but
  • require user to sign in manually
  • invite in the SPAM folder (Alejandro's gmail account)
  • Problems with inviting existing accounts (Roly)
• Manually confirm with program committe

Submission (Authors)

• Do we need separate author and reviewer checklists? (Probably just latter.)
• Checklist
  • Add README or artifact-evaluation.md (example here)
  • Create a branch

Review

• Initial dummy review to enable interaction with authors (15 days)
  • PC chair notifies reviewers of a hard deadline to submit a draft reviews and began discussion about what badges to award
• Actual review comes towards the end of the process (after author interaction) (5 days)
• Authors submit revised artifact (ideally) by pushing a new revision to repo (or updating the existing one)
• Review Scores
  • Overall merit: available, functional, reusable
  • Reviewer expertise: some familiarity

Badges

• Not in HotCRP
• Visible in the published paper
• Artefact (version assessed) curated in Zenodo, doi

[rolyp]

@acocac Renamed to avoid any suggestion that the platforms are used to manage the artefacts themselves!