A Framework to Compute Statistics of System Parameters from Very Large Trace Files

Problem and motivation

Re-reading large traces for every interval query is slow, while storing every metric change can consume storage comparable to the source trace. The framework targets scalable, compact, interactive metric statistics (pp. 2-4).

Method and contribution

A Java/LTTng prototype builds a disk-resident interval history in one trace pass. A granularity degree controls persistence frequency; intermediate values are linearly interpolated. Stabbing queries answer point values, endpoint subtraction answers interval statistics, and a resource/metric hierarchy supports roll-up/drill-down queries (pp. 5-11).

Findings and evidence

Tests use Linux kernel 2.6.38.6, a 2.8 GHz/6 GB machine, metrics including CPU usage, I/O throughput, HTTP/FTP/DNS connection counts, and event counts. Traces span 1-40 GB; GD=1 creates a store about 2.5-4.5x the source size, while coarser GD values reduce storage/construction cost. Twenty runs use 100 random intervals; GD=1000 is the best tested query case in the figures (pp. 11-13).

Limitations and future directions

Limitations: Interpolation accuracy depends on metric behavior and granularity degree; coarse degrees trade precision for space and construction time. Online construction is described as possible but was not investigated in this phase (pp. 7-8, 12-13).

Future work: Relate granularity to metrics and trace size; study interpolation effects; support other tracing systems; connect the framework to kernel fault/attack detection; and evaluate online construction (pp. 12-13).

Sources and identifiers

• Published version published
• Institutional full text · PDF public_full_text

When to cite this paper

Cite this paper when your work uses or compares granularity-degree-controlled, disk-resident trace-statistics computation with interpolation.

• Granularity-degree-controlled, disk-resident trace-statistics computation with interpolation.
• Endpoint/stabbing-query construction for interval statistics without re-reading the queried trace range.
• Resource-hierarchy aggregation with `O(log n + K)` reported query work.
• The 1-40 GB LTTng scalability study and the storage/precision trade-off across GD values.

Citation

@article{ezzatiJivan2013aframework,
  author = {Naser Ezzati-Jivan and Michel R. Dagenais},
  title = {A Framework to Compute Statistics of System Parameters from Very Large Trace Files},
  year = {2013},
  journal = {ACM SIGOPS Operating Systems Review},
  volume = {47},
  number = {1},
  pages = {43-54},
  publisher = {Association for Computing Machinery (ACM)},
  issn = {0163-5980},
  doi = {10.1145/2433140.2433151},
  url = {https://doi.org/10.1145/2433140.2433151}
}

Ezzati-Jivan, N., & Dagenais, M. R. (2013). A Framework to Compute Statistics of System Parameters from Very Large Trace Files. ACM SIGOPS Operating Systems Review, 47(1), 43-54. https://doi.org/10.1145/2433140.2433151

N. Ezzati-Jivan and M. R. Dagenais, "A Framework to Compute Statistics of System Parameters from Very Large Trace Files," ACM SIGOPS Operating Systems Review, vol. 47, no. 1, pp. 43-54, 2013, doi: 10.1145/2433140.2433151