Codabench: Flexible, Easy-to-Use and Reproducible Meta-Benchmark Platform - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue Patterns Année : 2022

Codabench: Flexible, Easy-to-Use and Reproducible Meta-Benchmark Platform

Zhen Xu
  • Fonction : Auteur
  • PersonId : 1092603
Adrien Pavao
Magali Richard
Wei-Wei Tu
  • Fonction : Auteur
Huan Zhao
  • Fonction : Auteur
Isabelle Guyon
  • Fonction : Auteur
  • PersonId : 1087533

Résumé

Obtaining standardized crowdsourced benchmark of computational methods is a major issue in data science communities. Dedicated frameworks enabling fair benchmarking in a unified environment are yet to be developed. Here we introduce Codabench, an open-source, community-driven platform for benchmarking algorithms or software agents versus datasets or tasks. A public instance of Codabench (https://www.codabench.org) is open to everyone, free of charge, and allows benchmark organizers to compare fairly submissions, under the same setting (software, hardware, data, algorithms), with custom protocols and data formats. Codabench has unique features facilitating the organization of benchmarks flexibly, easily and reproducibly, such as the possibility of re-using templates of benchmarks, and supplying compute resources on-demand. Codabench has been used internally and externally on various applications, receiving more than 130 users and 2500 submissions. As illustrative use cases, we introduce 4 diverse benchmarks covering Graph Machine Learning, Cancer Heterogeneity, Clinical Diagnosis and Reinforcement Learning.
Fichier principal
Vignette du fichier
Codabench.pdf (430.21 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03374222 , version 1 (12-10-2021)
hal-03374222 , version 2 (06-01-2022)
hal-03374222 , version 3 (25-02-2022)
hal-03374222 , version 4 (27-06-2022)

Identifiants

Citer

Zhen Xu, Sergio Escalera, Adrien Pavao, Magali Richard, Wei-Wei Tu, et al.. Codabench: Flexible, Easy-to-Use and Reproducible Meta-Benchmark Platform. Patterns, 2022, 3 (7), pp.100543. ⟨10.1016/j.patter.2022.100543⟩. ⟨hal-03374222v4⟩
1151 Consultations
304 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More