【Feature Discussion】Make tb_plugin support larger traces

[lh-ycx]

Hi guys, I'm recently working on profiling LLM (like GPT-3) workloads using the Pytorch profiler. When I tried to visualize the trace using tensorboard, one major problem was that the trace is too large (typically > 1GB for one step), causing no response in the browser.

I would like to work on this problem to make the tb_plugin support larger traces. Below are a few of my concerns:

• I cannot find a doc for tb_plugin (about the code structure and the plugin's arch), which makes it difficult for me to get started.
• Could you provide some insight into this problem? like where is the possible bottleneck (the original tensorboard? the trace analysis server? the frontend browser? or something else?)
• I also noticed that "monitoring daemon for larger scale deployments" is in progress, is this solving the same problem? If yes, can I get involved?

Thanks : )

[aaronenyeshi]

Hi @lh-ycx , please feel free to contribute to the tb_plugin directory (the code is outdated, and under maintenance mode only).

• As far as I know, the documentation can be found here: https://github.com/pytorch/kineto/blob/main/tb_plugin/README.md .
• This is also a problem for Chrome traces, when using the export_chrome_trace API. One workaround is to move the traceEvents to a root JSON dict, and then open it in Perfetto. Another workaround is to turn with_stack=False, which reduces the amount of events significantly.
• My intuition is that there are too many events to show in the frontend browser. One potential solution is to sample events when we're zoomed out. This idea is because, on chrome trace, you can delete a portion of the events, and it will start loading in the frontend.

[lmelinda]

I also wonder how much work would it be to incorporate Perfetto UI into the trace tab? For all my large trace files, they load pretty fast in Perfetto UI and does not crash like tensor board profiler tab. If not this could resolve a lot of problems.