Renaissance 0.11 Released

A new release of the Renaissance benchmark suite includes several changes to the workloads:

  • movie-lens bug fix, where part of the setup was moved out of the core benchmark iteration,
  • log-regression bug fix, where input features are cached to avoid parsing the input during each benchmark iteration,
  • neo4j-analytics improvement that adds indices and uses placeholder for queries (contributed by Michael Hunger from Neo4J), and
  • dotty fix for Windows compatibility.

The Renaissance harness now forces a garbage collection run before each iteration. This changes especially the behavior of Spark based workloads, which would otherwise retain weakly referenced objects until Spark controlled cleanup once every 30 minutes. The garbage collection forced by the harness can be disabled with the --no-forced-gc flag.

The harness also includes two plugins, jmx-timers for collecting the JIT compilation times, and ubench-agent for collecting the values of arbitrary hardware performance counters.

The JSON result format now also includes OS and system metadata, which help capture the benchmark execution conditions across experiments.

We also note that Renaissance helped tune the OpenJDK HotSpot inlining settings.

Special thanks go to:

  • Michael Hunger from Neo4J for his contribution on the neo4j-analytics benchmark,
  • Guillaumes Martres from EPFL for his input on fixing the dotty benchmark for Windows, and
  • Tom Rodriguez from Oracle Labs for spotting a bug in log-regression.

Renaissance 0.10 Released

Two months after the announcement of the Renaissance benchmark suite, we are pleased to release version 0.10.0 that includes a new benchmark, several improvements and bug fixes.

Most importantly, we added the scala-doku benchmark which is a Sudoku solver written in Scala. Other changes include making finagle-http more parallel, addressing an important bug in finagle-chirper and removing the native libraries dependencies of db-shootout. Moreover, the compatibility with different architectures has been improved and JDK11 is now also supported. All changes are listed in the GitHub release.

The numerous reports and positive feedback from the community are very encouraging and show a quick adoption of the suite across the industry. It helps companies like Oracle, IBM or SAP track performance and/or fix bugs. For instance, this OpenJ9 bug running Apache Spark has been spotted thanks to the Renaissance suite.

We address special thanks to:

  • tkrodriguez from Oracle for the in-depth analysis of the finagle-chirper bug
  • vkostyukov from Twitter, for his feedback on the Finagle benchmarks
  • MBaesken and the SAP Machine team for compatibility testing on less common infrastructures

Renaissance 0.9 Released

We are pleased to announce the release of the Renaissance benchmark suite for the Java Virtual Machine. The suite is primarily focused on parallelism and concurrency, and provides workloads that exercise the modern parallel programming abstractions and primitives provided by the JVM and the Java Class Library. Through these workloads, the suite aims to aid in understanding of how modern applications and data processing frameworks use the concurrency features of the JVM, and to foster development of new optimizations that enable more efficient execution of such workloads.

The Renaissance benchmark suite is intended to provide workloads that are not available elsewhere, and thus complement the workloads found in the existing, commonly used benchmark suites for the JVM, such as DaCapo, ScalaBench, and SPECjvm2008, which we gratefully acknowledge. We have found a great deal of inspiration in these suites, and strive to match and, where possible, exceed, the level of quality and usability established by these suites.

We realize that keeping a benchmarking suite alive (so that it evolves along with the platform and applications) requires a tremendous amount of work. We therefore aim to make the Renaissance development process as open as possible, and invite the community to participate in the process. The code is open-source and the contribution mechanism is open. The suite will be regularly updated through minor releases, while new benchmarks will be introduced or retired in major releases.

Version 0.9 marks the first public open-source release of the Renaissance suite and is considered a major-release candidate with respect to the selection of workloads included in the suite. The initial selection is based on the upcoming PLDI’19 paper, which also provides additional details on motivation behind the suite as well as detailed analysis of individual benchmarks and initial performance results. We are currently stabilizing the internal design of the harness and its public APIs, and work on a number of features that need to be included in version 1.0, the first major release of the suite which should be ready in few weeks.

We welcome any comments and contributions.

Happy benchmarking!