In Proceedings of the 1st international workshop on benchmarking cyber-physical networks and systems (CPSBench), pages 6, April, 2018. IEEE. Citation Key Alias: schuss18beyondcompetitionPaper doi abstract bibtex
Performance comparisons of low-power wireless systems are often not substantiated by accurate and realistic evaluations, which raises the need of a proper benchmark. In a ﬁrst attempt towards a rigorous comparison of protocol performance under the exact same settings, we have developed in 2016 a prototype benchmarking infrastructure called D-Cube, and used it to run the ﬁrst of a series of competitions aiming to quantitatively assess the performance of low-power wireless protocols in speciﬁc scenarios. Given the success of the competition among both academia and industry, we have signiﬁcantly extended the benchmarking infrastructure in the following two editions: D-Cube now also supports, among others, remote experimentation, multiple trafﬁc patterns and loads, a custom description of how to derive performance metrics, and is further able to control the network density as well as the harshness of the RF environment. In this paper we perform a critical analysis of the current capabilities of D-Cube and argue that its main limiting factor is that the trafﬁc patterns and node identities are manually embedded in the source code by developers and cannot be changed automatically. We show that we can overcome this limitation by utilizing a well-known data structure and by having developers describe its memory address using a conﬁguration ﬁle that is passed to the benchmarking infrastructure. Following this concept, we extend D-Cube with the ability of building and applying patches to binary ﬁles and show that this allows not only to automatically change trafﬁc patterns and node identities, but to also change user-deﬁned protocol parameters. We believe that this extension is one of the last missing stepping stones to make D-Cube a full-ﬂedged benchmarking infrastructure for low-power wireless systems.