Codabench: Flexible, Easy-to-Use and Reproducible Meta-Benchmark Platform - Archive ouverte HAL
Accéder directement au contenu Accéder directement à la navigation
Pré-publication, Document de travail

Codabench: Flexible, Easy-to-Use and Reproducible Meta-Benchmark Platform

Zhen Xu 1 Sergio Escalera 2 Adrien Pavao 3 Magali Richard 4 Wei-Wei Tu 1, 5 Quanming Yao 6 Huan Zhao 1 Isabelle Guyon 5, 3 
3 TAU - TAckling the Underspecified
Inria Saclay - Ile de France, LISN - Laboratoire Interdisciplinaire des Sciences du Numérique
4 TIMC-MAGe - Modèles et Algorithmes pour la Génomique
TIMC - Translational Innovation in Medicine and Complexity / Recherche Translationnelle et Innovation en Médecine et Complexité - UMR 5525
Abstract : Obtaining standardized crowdsourced benchmark of computational methods is a major issue in data science communities. Dedicated frameworks enabling fair benchmarking in a unified environment are yet to be developed. Here we introduce Codabench, an open-source, community-driven platform for benchmarking algorithms or software agents versus datasets or tasks. A public instance of Codabench (https://www.codabench.org) is open to everyone, free of charge, and allows benchmark organizers to compare fairly submissions, under the same setting (software, hardware, data, algorithms), with custom protocols and data formats. Codabench has unique features facilitating the organization of benchmarks flexibly, easily and reproducibly, such as the possibility of re-using templates of benchmarks, and supplying compute resources on-demand. Codabench has been used internally and externally on various applications, receiving more than 130 users and 2500 submissions. As illustrative use cases, we introduce 4 diverse benchmarks covering Graph Machine Learning, Cancer Heterogeneity, Clinical Diagnosis and Reinforcement Learning.
Type de document :
Pré-publication, Document de travail
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/hal-03374222
Contributeur : Zhen Xu Connectez-vous pour contacter le contributeur
Soumis le : lundi 27 juin 2022 - 09:22:07
Dernière modification le : mercredi 5 octobre 2022 - 17:23:50

Fichier

Codabench.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-03374222, version 4

Citation

Zhen Xu, Sergio Escalera, Adrien Pavao, Magali Richard, Wei-Wei Tu, et al.. Codabench: Flexible, Easy-to-Use and Reproducible Meta-Benchmark Platform. 2022. ⟨hal-03374222v4⟩

Partager

Métriques

Consultations de la notice

158

Téléchargements de fichiers

63