Reproducible and hackable software benchmarking with(out) compute clusters

Maximilian Heisinger; Martina Seidl

doi:10.1007/s10009-025-00811-3

Reproducible and hackable software benchmarking with(out) compute clusters

Maximilian Heisinger^*
, Martina Seidl

^*Corresponding author for this work

Institute for Symbolic Artificial Intelligence

Research output: Contribution to journal › Article › peer-review

Abstract

We present Simsala, an easy-to-install and easy-to-use collection of scripts supporting benchmarking on clusters of compute nodes. While designed with applications for benchmarking solving technologies like SAT and extensions, our solution can be easily transferred to many other application scenarios as well. In this work, we discuss the design objectives behind Simsala, provide some implementation details, and illustrate with case studies how Simsala has been applied in the past for extensive evaluations.

Original language	English
Pages (from-to)	403-409
Number of pages	7
Journal	International Journal on Software Tools for Technology Transfer
Volume	27
Issue number	4
DOIs	https://doi.org/10.1007/s10009-025-00811-3
Publication status	Published - 19 May 2025

Fields of science

101013 Mathematical logic
102031 Theoretical computer science
102011 Formal languages
102022 Software development
102001 Artificial intelligence
102030 Semantic technologies
102 Computer Sciences

JKU Focus areas

Digital Transformation

Access to Document

10.1007/s10009-025-00811-3

Cite this

@article{94260905dfd845f79d0b4ac2b735b4b4,

title = "Reproducible and hackable software benchmarking with(out) compute clusters",

abstract = "We present Simsala, an easy-to-install and easy-to-use collection of scripts supporting benchmarking on clusters of compute nodes. While designed with applications for benchmarking solving technologies like SAT and extensions, our solution can be easily transferred to many other application scenarios as well. In this work, we discuss the design objectives behind Simsala, provide some implementation details, and illustrate with case studies how Simsala has been applied in the past for extensive evaluations.",

keywords = "Benchmarking, Cluster, Data-aggregation, Database, Slurm, Tool",

author = "Maximilian Heisinger and Martina Seidl",

note = "Publisher Copyright: {\textcopyright} The Author(s) 2025.",

year = "2025",

month = may,

day = "19",

doi = "10.1007/s10009-025-00811-3",

language = "English",

volume = "27",

pages = "403--409",

journal = "International Journal on Software Tools for Technology Transfer",

issn = "1433-2779",

publisher = "Springer Nature",

number = "4",

}

TY - JOUR

T1 - Reproducible and hackable software benchmarking with(out) compute clusters

AU - Heisinger, Maximilian

AU - Seidl, Martina

N1 - Publisher Copyright: © The Author(s) 2025.

PY - 2025/5/19

Y1 - 2025/5/19

N2 - We present Simsala, an easy-to-install and easy-to-use collection of scripts supporting benchmarking on clusters of compute nodes. While designed with applications for benchmarking solving technologies like SAT and extensions, our solution can be easily transferred to many other application scenarios as well. In this work, we discuss the design objectives behind Simsala, provide some implementation details, and illustrate with case studies how Simsala has been applied in the past for extensive evaluations.

AB - We present Simsala, an easy-to-install and easy-to-use collection of scripts supporting benchmarking on clusters of compute nodes. While designed with applications for benchmarking solving technologies like SAT and extensions, our solution can be easily transferred to many other application scenarios as well. In this work, we discuss the design objectives behind Simsala, provide some implementation details, and illustrate with case studies how Simsala has been applied in the past for extensive evaluations.

KW - Benchmarking

KW - Cluster

KW - Data-aggregation

KW - Database

KW - Slurm

KW - Tool

UR - https://www.scopus.com/pages/publications/105005792234

U2 - 10.1007/s10009-025-00811-3

DO - 10.1007/s10009-025-00811-3

M3 - Article

AN - SCOPUS:105005792234

SN - 1433-2779

VL - 27

SP - 403

EP - 409

JO - International Journal on Software Tools for Technology Transfer

JF - International Journal on Software Tools for Technology Transfer

IS - 4

ER -

Reproducible and hackable software benchmarking with(out) compute clusters

Abstract

Fields of science

JKU Focus areas

Access to Document

Other files and links

Cite this