2013 · ACM SIGOPS Operating Systems Review
A Framework to Compute Statistics of System Parameters from Very Large Trace Files
Evidence basis: full-text-reviewed · Review status: catalog-reviewed; paper-author approval pending
kernel-tracing trace-analysis performance-analysis resource-analysis performance-engineering
trace statistics LTTng Linux kernel 2.6.38.6 disk-resident interval tree history tree granularity degree linear interpolation hierarchical query large-scale tracing online analysis CPU usage I/O throughput
Core contribution: The framework computes system-parameter statistics for arbitrary intervals and resource hierarchies from very large traces using a disk-resident history structure and controlled granularity.
Problem and motivation
Re-reading large traces for every interval query is slow, while storing every metric change can consume storage comparable to the source trace. The framework targets scalable, compact, interactive metric statistics (pp. 2-4).
Method and contribution
A Java/LTTng prototype builds a disk-resident interval history in one trace pass. A granularity degree controls persistence frequency; intermediate values are linearly interpolated. Stabbing queries answer point values, endpoint subtraction answers interval statistics, and a resource/metric hierarchy supports roll-up/drill-down queries (pp. 5-11).
Findings and evidence
Tests use Linux kernel 2.6.38.6, a 2.8 GHz/6 GB machine, metrics including CPU usage, I/O throughput, HTTP/FTP/DNS connection counts, and event counts. Traces span 1-40 GB; GD=1 creates a store about 2.5-4.5x the source size, while coarser GD values reduce storage/construction cost. Twenty runs use 100 random intervals; GD=1000 is the best tested query case in the figures (pp. 11-13).
Limitations and future directions
Limitations: Interpolation accuracy depends on metric behavior and granularity degree; coarse degrees trade precision for space and construction time. Online construction is described as possible but was not investigated in this phase (pp. 7-8, 12-13).
Future work: Relate granularity to metrics and trace size; study interpolation effects; support other tracing systems; connect the framework to kernel fault/attack detection; and evaluate online construction (pp. 12-13).
Sources and identifiers
- Published version published
- Institutional full text · PDF public_full_text
When to cite this paper
Cite this paper when your work uses or compares granularity-degree-controlled, disk-resident trace-statistics computation with interpolation.
- Granularity-degree-controlled, disk-resident trace-statistics computation with interpolation.
- Endpoint/stabbing-query construction for interval statistics without re-reading the queried trace range.
- Resource-hierarchy aggregation with `O(log n + K)` reported query work.
- The 1-40 GB LTTng scalability study and the storage/precision trade-off across GD values.
Citation
@article{ezzatiJivan2013aframework,
author = {Naser Ezzati-Jivan and Michel R. Dagenais},
title = {A Framework to Compute Statistics of System Parameters from Very Large Trace Files},
year = {2013},
journal = {ACM SIGOPS Operating Systems Review},
volume = {47},
number = {1},
pages = {43-54},
publisher = {Association for Computing Machinery (ACM)},
issn = {0163-5980},
doi = {10.1145/2433140.2433151},
url = {https://doi.org/10.1145/2433140.2433151}
}Other citation formats for Word and reference managers
Ezzati-Jivan, N., & Dagenais, M. R. (2013). A Framework to Compute Statistics of System Parameters from Very Large Trace Files. ACM SIGOPS Operating Systems Review, 47(1), 43-54. https://doi.org/10.1145/2433140.2433151N. Ezzati-Jivan and M. R. Dagenais, "A Framework to Compute Statistics of System Parameters from Very Large Trace Files," ACM SIGOPS Operating Systems Review, vol. 47, no. 1, pp. 43-54, 2013, doi: 10.1145/2433140.2433151