Combining Distributed and Kernel Tracing for Performance Analysis of Cloud Applications

Catalog abstract summary

Distributed tracing allows tracking user requests that span across multiple services and machines in a distributed application. However, typical cloud applications rely on abstraction layers that can hide the root cause of latency happening between processes or in the kernel. Because of its focus on high-level events, existing methodologies in applying distributed tracing can be limited when trying to detect complex contentions and relate them back to the originating requests. Cross-level analyses that include kernel-level events are necessary to debug problems as prevalent as mutex or disk contention, however cross-level analysis and associating events in the kernel and distributed tracing data is complex and can add a lot of overhead. This paper describes a new solution for combining distributed tracing with low-level software tracing in order to find the latency root cause better. We explain how we achieve a hybrid trace collection to capture and synchronize both kernel and distributed request events. Then, we present our design and implementation for a critical path analysis. We show that our analysis describes precisely how each request spends its time and what stands in its critical path while limiting overhead.

Source: Official MDPI full-text HTML, CC BY 4.0, reviewed 2026-08-06: https://www.mdpi.com/2079-9292/10/21/2610

Problem and motivation

Distributed request spans identify application-level paths but can miss kernel scheduling, I/O, and CPU contention that explains hidden latency.

Method and contribution

Synchronize Jaeger spans with LTTng user-space/kernel events, decode CTF with Babeltrace, and analyze request critical paths in Trace Compass with cross-layer scheduling/resource states.

Findings and evidence

HotROD and Cassandra experiments show modest tracing overhead under the stated loads. The combined analysis identifies CPU wait/preemption in a Cassandra CPU-control-group case that Jaeger-only tracing cannot explain; a 500 MB trace can be analyzed in about 15 s.

Limitations and future directions

Limitations: Evidence is remote HTML only; local PDF pages, local PDF hash, and visual PDF checks are unavailable. The evaluated services/workloads are HotROD and Cassandra, and the paper's future work identifies dashboard-triggered snapshots, work-queue/thread-pool tracking, network analysis, and multi-host extension as open areas.

Future work: Trigger snapshots from a dashboard, improve tracking of user schedulers/work queues/thread pools, add network analysis, and extend the method across multiple hosts.

Resources

• Code

Sources and identifiers

• Published version published
• Public full text · PDF public_full_text

When to cite this paper

Cite this paper when your work uses or compares synchronizing Jaeger spans with LTTng tracepoints carrying request IDs.

• For synchronizing Jaeger spans with LTTng tracepoints carrying request IDs.
• For extending thread critical paths into request critical paths with scheduler, I/O, and wait states.
• For the HotROD/Cassandra overhead measurements and 500 MB/15 s analysis result.
• For identifying CPU-control-group preemption/CPU wait as a distributed-service latency cause.

Citation

@article{ezzatiJivan2021combiningdistributed,
  author = {Loïc Gelle and Naser Ezzati-Jivan and Michel R. Dagenais},
  title = {Combining Distributed and Kernel Tracing for Performance Analysis of Cloud Applications},
  year = {2021},
  journal = {Electronics},
  volume = {10},
  number = {21},
  pages = {2610},
  publisher = {MDPI AG},
  issn = {2079-9292},
  doi = {10.3390/electronics10212610},
  url = {https://doi.org/10.3390/electronics10212610}
}

Gelle, L., Ezzati-Jivan, N., & Dagenais, M. R. (2021). Combining Distributed and Kernel Tracing for Performance Analysis of Cloud Applications. Electronics, 10(21), 2610. https://doi.org/10.3390/electronics10212610

L. Gelle, N. Ezzati-Jivan, and M. R. Dagenais, "Combining Distributed and Kernel Tracing for Performance Analysis of Cloud Applications," Electronics, vol. 10, no. 21, Art. no. 2610, 2021, doi: 10.3390/electronics10212610