News

We are pleased to announce a new release of the Renaissance benchmark suite. This release adds result validation to several benchmarks, improves compatibility with current Java releases, and updates numerous underlying frameworks and libraries.

Validation was improved or added to the following 7 workloads: als, fj-kmeans, future-genetic, gauss-mix, log-regression, movie-lens, and philosophers, bringing Renaissance closer to having fully validated workloads (there are still 4 missing).

Issues related to the Hadoop Client API library (which plagued the Apache Spark benchmarks) and the Chronicle Map library (which plagued the db-shootout benchmark) were resolved. As a result, all benchmarks in this release (with the exception of neo4j-analytics which requires at least version 17 to run), run on LTS versions from 11 to 21 as well as on early access version of Java 24.

The philosophers and scala-stm-bench7 benchmarks were migrated to Scala 3, continuing the migration towards more recent Scala versions. The reactors benchmark is the last to require Scala 2.12.

Even though the source code of most benchmarks remains unchanged, the actual code executed at runtime may be affected by the dependency updates, resulting in different performance. Apart from the indirect impact of library updates, the performance of the future-genetic, gauss-mix, movie-lens, and philosophers benchmarks is known to differ from the previous release due to benchmark changes.

See the GitHub release for more details.

Any comments and contributions are welcome.

Happy benchmarking!

We are pleased to announce a new release of the Renaissance benchmark suite. This release is primarily aimed at improving compatibility with Java 21 (LTS) and Java 22 (EA), but it also marks the departure from Java 8 by requiring at least Java 11 to build the suite and to run the benchmarks. Several benchmarks now use Scala 3 and a memory leak was plugged in one of the benchmarks.

To improve compatibility with Java 21, the Apache Spark framework used by the apache-spark benchmarks was updated to version 3.5.0 and the Neo4j framework used by the neo4j-analytics benchmark was updated to version 5.12.0 (the benchmark now requires Java 17 to run). The database libraries used in the db-shootout benchmark were updated to more recent versions, which makes the benchmark compatible with Java 18 (while still far from Java 21, it is an improvement over Java 11, where the benchmark was stuck for some time).

The Scala 2.12 and 2.13 versions used by various benchmarks were updated to versions 2.12.18 and 2.13.12, respectively. The akka-uct, dotty, and scala-kmeans are now using Scala 3.3.1, and the neo4j-analytics benchmark now uses Scala 2.13. This leaves only the philosophers, reactors and stala-stm-bench7 benchmarks in the Scala 2.12 camp.

Even though the source code of most benchmarks remains unchanged (apart from bug fixes and changes due to library APIs), the actual code executed at runtime may be affected by the dependency updates.

Other changes include a fix for a memory leak in the reactors benchmark, and two fixes that improve compatibility with JMH.

See the pull requests referenced in the GitHub release for more details.

Any comments and contributions are welcome.

Happy benchmarking!

We have released a maintenance update of the Renaissance benchmark suite with two changes in the underlying libraries. One is to avoid packaging vulnerable versions of the log4j framework and the other is to improve compatibility with the LoongArch64 processor architecture.

Even though we believe that the Renaissance suite itself (and the way it is used) is not susceptible to the vulnerabilities, the presence of JAR files with vulnerable classes can trigger alarms from various security scanners in some environments. We have removed the log4j library from all (even transitive) dependencies and replaced it with the reload4j library where necessary. The logging configuration of the Apache Spark, Twitter Finagle, and Akka benchmarks was updated to restore the previous behavior.

To improve compatibility with the LoongArch64 processor architecture, we updated the JNA library to a version that includes support for the processor.

See the pull requests referenced in the GitHub release for more details.

Any comments and contributions are welcome.

Happy benchmarking!

We have released a bugfix update of the Renaissance benchmark suite, fixing two race conditions discovered in the finagle-chirper and reactors benchmarks. Apart from the bug fixes, there have been no other changes to the benchmark code or the underlying libraries. The changes in the affected benchmarks are relatively minor, and the unaffected benchmarks are identical to Renaissance 0.14. Consequently, there should be no significant changes in the workloads.

Both race conditions resulted in exceptions visible in the benchmark output, but only the exception in the reactors benchmark caused the benchmark to fail in a difficult-to-ignore deadlock. The exception in finagle-chirper caused a particular future computation to terminate prematurely, but that did not cause the entire benchmark to fail. We apologize for the inconvenience.

More details can be found in the respective pull requests (#356 and #357 in the case of finagle-chirper, and #360 in the case of reactors).

Any comments and contributions are welcome.

Happy benchmarking!

We are pleased to announce a new release of the Renaissance benchmark suite. This release contains a mix of benchmark dependency updates and some internal changes. The most visible change is an update of Apache Spark to version 3.2.0, which enabled migrating the apache-spark benchmarks from Scala 2.12 to Scala 2.13. In addition, Scala 2.13 benchmarks were updated to use Scala 2.13.8. The internal changes make it easier to run individual benchmarks in what we call standalone mode, which is desired for GraalVM Native Image benchmarking.

The source code of the benchmarks remains unchanged, but updates to some of their dependencies may affect the code executed at runtime. Some of the dependencies used by multiple benchmarks were updated and their versions unified. This allows to only bundle a single version of certain components and prevents component version clashes in standalone mode. For some dependencies, such as the JNA framework, this also improves compatibility on less common combinations of CPU architectures and JVM implementations.

See below for more details.

We welcome any comments and contributions.

Happy benchmarking!

Workload changes

Any workload changes are solely due to updates to benchmark runtime or primary and secondary (transitive) dependencies. Details can be found in the respective pull requests (#342 and #344), here we only provide a brief summary:

• akka-uct benchmark
  • Updated akka-actor dependency (2.6.12 -> 2.6.18)
• apache-spark benchmarks
  • Updated Apache Spark components (3.1.2 -> 3.2.0)
  • Updated Netty components to common version (4.1.50.Final, 4.1.68.Final -> 4.1.72.Final)
  • Updated/commonized many secondary dependencies
  • Migrated to Scala 2.13
• db-shootout benchmark
  • Updated JNA framework to common version (4.2.1 -> 5.10.0)
  • Updated/commonized a few secondary dependencies
• dotty benchmark
  • Updated scala3-compiler_3 dependency (3.0.0 -> 3.0.2)
  • Updated JNA framework to common version (5.3.1 -> 5.10.0)
• future-genetic benchmark
  • Updated jenetics dependency (4.4.0 -> 5.2.0)
• neo4j-analytics benchmark
  • Updated neo4j dependency (4.2.4 -> 4.4.2)
  • Updated JNA framework to common version (5.6.0 -> 5.10.0)
  • Updated Netty components to common version (4.1.55.Final -> 4.1.72.Final)
• rx-scrabble benchmark
  • Updated rxjava dependency (1.3.7 -> 1.3.8)
• twitter-finagle benchmarks
  • Updated Finagle components (21.10.0 -> 21.12.0)
  • Updated Netty components to common version (4.1.66.Final -> 4.1.72.Final)
  • Updated/commonized several secondary dependencies

Standalone mode

Standalone mode means executing a benchmark without the help of the launcher in the main Renaissance bundle. The benchmark harness is still used, but both the harness and benchmark code are loaded using a single class loader. This is useful in situations where using multiple class loaders makes the runtime too convoluted, e.g., when benchmarking AOT-compiled code with GraalVM Native Image.

Because the launcher is not used, the bundle needs to be manually extracted. In addition to directories containing the benchmark and dependency jars, there is also a directory called single, which contains metadata-only jars, one for each benchmark. These jars can be used to execute the corresponding benchmark in standalone mode simply by running java -jar <jar-file-path>. The correct version of the Renaissance harness (matching the Scala version used by the benchmark) will be used.

We are pleased to announce a new release of the Renaissance benchmark suite. This release is primarily a JDK compatibility update, introducing support for the newly released JDK17. The Renaissance suite now supports the complete set of LTS releases of the JDK (JDK8, JDK11, and JDK17) on Linux, MacOS X, and Windows. A few benchmarks have dependencies that limit their compatibility to a particular range of JDK versions and these are automatically excluded when running the suite on an unsupported JDK.

The changes in this release are minimal. The most important one just brings the Scala 2.12 benchmarks over to Scala 2.12.15, which now properly supports JDK17 – we merely pass on the compatibility improvements provided by the Scala community! This change affects the following benchmarks, which were previously using Scala 2.12.14:

• finagle-chirper, finagle-http
• neo4j-analytics
• philosophers, scala-stm-bench7
• reactors
• all apache-spark workloads

Here we just want to note that no benchmark code has been changed in this release, only project documentation and parts of the harness responsible for plugin loading (plugin main class can be now specified in its manifest).

We welcome any comments and contributions.

Happy benchmarking!

We are pleased to announce a new release of the Renaissance benchmark suite. Apart from a number of internal cleanups, most changes in this release focused on improving compatibility with modern platforms. Most notably, we have moved away from Scala 2.11, Scala 2.12, and Spark 2 towards Scala 2.12, Scala 2.13 and Spark 3. Some benchmarks were unaffected, some needed just dependency updates, and some needed a bit more work.

The workloads, as implemented by the benchmarks, are largely unchanged, but the code being executed as a result of the benchmark workload may differ substantially from previous version of Renaissance, mainly due to substantial dependency updates in some of the benchmarks. See the sections below for more details.

On the platform compatibility front, the suite supports JDK8 and JDK11 on Linux, MacOS X, and Windows. While most benchmarks also run on more recent JDK versions, some of them have dependencies that limit their compatibility to a particular range of JDK versions.

We welcome any comments and contributions.

Happy benchmarking!

Harness changes

The harness now automatically excludes benchmarks that are known to be incompatible with the JVM on which it executes. This can be disabled using the --no-jvm-check option on the command line.

The harness now allows using a specific base directory for scratch files. The current directory is still the default, but it can be changed using the --scratch-base option on the command line. For debugging purposes, the --keep-scratch option disables deleting the scratch directories when the harness terminates.

Finally, to make experimentation with benchmarks a bit easier, the harness allows overriding benchmark parameters using the --override command line option.

Spark benchmark changes

The Spark benchmarks moved from Spark 2.0.0 on Scala 2.11.8 to Spark 3.1.2 on Scala 2.12.14, which is a major change in the underlying libraries. In addition to various cleanups and compatibility fixes for Windows, there have been several changes that may influence the overall performance.

• The number of threads used by the Spark local executor is now limited to reflect the parallelism of a particular workload, which prevents scalability issues on huge machines with tens of cores. The limit is configured using the spark_thread_limit parameter. - All workloads use the default partitioning of input data and limit the storage level of input data to memory only.
• als and movie-lens also limit the storage level of intermediate data sets to memory only to avoid benchmarking storage IO.
• log-regression now avoids repeated conversion from an RDD to DataFrame.
• naive-bayes now uses ML classes from the org.apache.spark.ml package because classes from the org.apache.spark.mllib package only served as wrappers, introducing unnecessary conversion from RDD to DataFrame.

Benchmarks using Scala 2.13.6

The following lists Java bencharks with a simple Scala wrapper or Scala benchmarks that have been updated to use Scala 2.13.6, along with notable changes in some of them:

• akka-uct (previously Scala 2.11.8)
  • Updated akka-actor dependency (2.3.11 -> 2.6.12)
  • Very minor benchmark update for the 2.6.x API.
• db-shootout (Java)
  • Maximum supported JVM version set to 11.
• dotty (previously Scala 2.12.8)
  • Updated Dotty compiler (0.12.0 -> 3.0.0)
  • Compiles scalap sources instead of “type-system puzzles”.
  • Adds hash-based validation of generated Tasty files.
• fj-kmeans, future-genetic (Java)
• mnemonics, par-mnemonics, scrabble (Java)
• rx-scrabble (Java)
• scala-kmeans (previously Scala 2.12.8)
• scala-doku (previously Scala 2.11.7)
  • Bundled cafesat dependency updated (0.01 -> 0.01-28-gd0edeaa) and cleaned up for Scala 2.13
  • Bundled scala-smtlib dependency updated (0.1 -> 0.2.1-52-ga71d6b0) and cleaned up for Scala 2.13

Benchmarks using Scala 2.12.14

The following lists benchmarks that use Scala 2.12.14, along with notable changes in some them:

• finagle-chirper, finagle-http (previously Scala 2.11.8)
• neo4j-analytics (previously Scala 2.12.8)
  • Benchmark ported to Neo4J 4.x API
  • Updated org.neo4j.neo4j dependency (3.5.12 -> 4.2.4)
  • Updated net.liftweb.lift-json dependency (3.2.0 -> 3.4.3)
  • Requires JVM versions between 11 and 15, inclusive (Neo4J requirements).
• philosophers, scala-stm-bench7 (previously Scala 2.12.3)
• reactors (previously Scala 2.11.8)
  • io.reactors.reactors-core updated (0.7 -> 0.9-e605e145-60-g9838414) and cleaned up for Scala 2.12
• Spark workloads (previously Scala 2.11.8)
  • See Spark benchmark changes above.

A new release of the Renaissance benchmark suite includes several changes to the workloads:

• movie-lens bug fix, where part of the setup was moved out of the core benchmark iteration,
• log-regression bug fix, where input features are cached to avoid parsing the input during each benchmark iteration,
• neo4j-analytics improvement that adds indices and uses placeholder for queries (contributed by Michael Hunger from Neo4J), and
• dotty fix for Windows compatibility.

The Renaissance harness now forces a garbage collection run before each iteration. This changes especially the behavior of Spark based workloads, which would otherwise retain weakly referenced objects until Spark controlled cleanup once every 30 minutes. The garbage collection forced by the harness can be disabled with the --no-forced-gc flag.

The harness also includes two plugins, jmx-timers for collecting the JIT compilation times, and ubench-agent for collecting the values of arbitrary hardware performance counters.

The JSON result format now also includes OS and system metadata, which help capture the benchmark execution conditions across experiments.

We also note that Renaissance helped tune the OpenJDK HotSpot inlining settings.

Special thanks go to:

• Michael Hunger from Neo4J for his contribution on the neo4j-analytics benchmark,
• Guillaumes Martres from EPFL for his input on fixing the dotty benchmark for Windows, and
• Tom Rodriguez from Oracle Labs for spotting a bug in log-regression.

Two months after the announcement of the Renaissance benchmark suite, we are pleased to release version 0.10.0 that includes a new benchmark, several improvements and bug fixes.

Most importantly, we added the scala-doku benchmark which is a Sudoku solver written in Scala. Other changes include making finagle-http more parallel, addressing an important bug in finagle-chirper and removing the native libraries dependencies of db-shootout. Moreover, the compatibility with different architectures has been improved and JDK11 is now also supported. All changes are listed in the GitHub release.

The numerous reports and positive feedback from the community are very encouraging and show a quick adoption of the suite across the industry. It helps companies like Oracle, IBM or SAP track performance and/or fix bugs. For instance, this OpenJ9 bug running Apache Spark has been spotted thanks to the Renaissance suite.

We address special thanks to:

• tkrodriguez from Oracle for the in-depth analysis of the finagle-chirper bug
• vkostyukov from Twitter, for his feedback on the Finagle benchmarks
• MBaesken and the SAP Machine team for compatibility testing on less common infrastructures

We are pleased to announce the release of the Renaissance benchmark suite for the Java Virtual Machine. The suite is primarily focused on parallelism and concurrency, and provides workloads that exercise the modern parallel programming abstractions and primitives provided by the JVM and the Java Class Library. Through these workloads, the suite aims to aid in understanding of how modern applications and data processing frameworks use the concurrency features of the JVM, and to foster development of new optimizations that enable more efficient execution of such workloads.

The Renaissance benchmark suite is intended to provide workloads that are not available elsewhere, and thus complement the workloads found in the existing, commonly used benchmark suites for the JVM, such as DaCapo, ScalaBench, and SPECjvm2008, which we gratefully acknowledge. We have found a great deal of inspiration in these suites, and strive to match and, where possible, exceed, the level of quality and usability established by these suites.

We realize that keeping a benchmarking suite alive (so that it evolves along with the platform and applications) requires a tremendous amount of work. We therefore aim to make the Renaissance development process as open as possible, and invite the community to participate in the process. The code is open-source and the contribution mechanism is open. The suite will be regularly updated through minor releases, while new benchmarks will be introduced or retired in major releases.

Version 0.9 marks the first public open-source release of the Renaissance suite and is considered a major-release candidate with respect to the selection of workloads included in the suite. The initial selection is based on the upcoming PLDI’19 paper, which also provides additional details on motivation behind the suite as well as detailed analysis of individual benchmarks and initial performance results. We are currently stabilizing the internal design of the harness and its public APIs, and work on a number of features that need to be included in version 1.0, the first major release of the suite which should be ready in few weeks.

We welcome any comments and contributions.

Happy benchmarking!