Uncovering Steady State Executions in Java Microbenchmarking with Call Graph Analysis

Madeline Janecek; Sneh Patel; Naser Ezzati-Jivan

doi:10.1145/3578245.3584689

2023 · ACM/SPEC ICPE Companion

Uncovering Steady State Executions in Java Microbenchmarking with Call Graph Analysis

Madeline Janecek | Sneh Patel | Naser Ezzati-Jivan

Evidence basis: full-text-reviewed · Review status: catalog-reviewed; paper-author approval pending

performance-analysis performance-engineering machine-learning trace-analysis

Java JMH microbenchmarking steady-state detection warmup call-graph analysis Perf Graph2Vec change-point detection PELT Random Forest SVC flame graphs

Core contribution: The paper uses sampled call graphs and machine learning to distinguish warmup from steady-state executions in Java microbenchmarks.

Problem and motivation

JMH warmup and steady-state stages can differ structurally, while runtime-only detection gives little insight into why. The paper tests whether sampled call graphs distinguish the stages (pp. 1-2).

Method and contribution

Linux perf samples call stacks throughout JMH executions; timestamped start/end logs (Apache Log4j 2.11.2) delimit executions. Call graphs are visualized as flame graphs and embedded with Graph2Vec. PELT change-point detection supplies labels; this study checks 50 similar executions (5% of 1,000) after the change point rather than 500. MLP, Random Forest, Decision Tree, KNN, and SVC classifiers are evaluated (pp. 2-4).

Findings and evidence

The workload is a simple Java binary search with 10,000 executions across 10 forks on Ubuntu 22.04 LTS, 64-bit kernel 5.15.0-56, Perf 5.15.64, and JMH; warmup graphs have deeper/more irregular stacks and more interpreter functions. Accuracy/precision/recall: MLP 88.5/31.6/12.5%, RF 90.4/75.0/6%, DT 82.8/19.3/22.5%, KNN 89.8/45/9%, SVC 90/50/1% (pp. 3-4).

Limitations and future directions

Limitations: Accuracy/precision are misleading without the very low recall; the paper attributes this partly to the small dataset/labels and possibly insufficient Graph2Vec vector size. Cross-JVM and cross-workload generalization are not established.

Future work: More benchmarks, forks, and iterations; alternate/larger graph vectorizations; and unsupervised learning to avoid hand-derived labels (p. 4).

Resources

Code

Sources and identifiers

Published version published
Public full text · PDF public_full_text

When to cite this paper

Cite this paper when your work uses or compares call-stack-sampled call graphs/flame graphs as structural evidence of JVM warmup versus steady-state execution.

For call-stack-sampled call graphs/flame graphs as structural evidence of JVM warmup versus steady-state execution.
For combining Graph2Vec and PELT-derived labels with classifiers for automated steady-state detection.
For the exact result pattern in which Random Forest reaches 90.4% accuracy/75.0% precision but only 6% recall, demonstrating why recall must be reported.

Citation

@inproceedings{ezzatiJivan2023uncoveringsteady,
  author = {Madeline Janecek and Sneh Patel and Naser Ezzati-Jivan},
  title = {Uncovering Steady State Executions in Java Microbenchmarking with Call Graph Analysis},
  year = {2023},
  booktitle = {ACM/SPEC ICPE Companion},
  pages = {71-75},
  publisher = {ACM},
  doi = {10.1145/3578245.3584689},
  url = {https://doi.org/10.1145/3578245.3584689}
}

Other citation formats for Word and reference managers

Janecek, M., Patel, S., & Ezzati-Jivan, N. (2023). Uncovering Steady State Executions in Java Microbenchmarking with Call Graph Analysis. In ACM/SPEC ICPE Companion (pp. 71-75). https://doi.org/10.1145/3578245.3584689

M. Janecek, S. Patel, and N. Ezzati-Jivan, "Uncovering Steady State Executions in Java Microbenchmarking with Call Graph Analysis," in ACM/SPEC ICPE Companion, pp. 71-75, 2023, doi: 10.1145/3578245.3584689

Readable Markdown record · JSON record · Download RIS