Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Step time in Overview and in Trace don't match #747

Open
somepage opened this issue Mar 28, 2023 · 2 comments
Open

Step time in Overview and in Trace don't match #747

somepage opened this issue Mar 28, 2023 · 2 comments
Labels
bug Something isn't working plugin PyTorch Profiler TensorBoard Plugin related

Comments

@somepage
Copy link

Hi all!
Trying to use torch.profiler to measure acceleration in inference of my model
https://pytorch.org/tutorials/intermediate/tensorboard_profiler_tutorial.html
I am looking at the profiling results in Tensorboard and the following question arises:
Why are the step time in the overview and the step time in the trace not the same?
Is it a bug in the torch plugin for Tensorboard?

@aaronenyeshi
Copy link
Member

There may be a bug causing a difference between the Tensorboard plugin and trace annotation. Could you please provide a screenshot of what you are seeing in both places?

@aaronenyeshi aaronenyeshi added the bug Something isn't working label May 9, 2023
@zackertypical
Copy link

image

i printed the step time in code, is about 30ms, but the overview shows 200ms.

import time
prof = torch.profiler.profile(
        schedule=torch.profiler.schedule(wait=1, warmup=1, active=5, repeat=2),
        on_trace_ready=torch.profiler.tensorboard_trace_handler('./log/resnet18'),
        # record_shapes=True,
        profile_memory=True,
        with_stack=False)
prof.start()
for step, batch_data in enumerate(train_loader):
    st = time.time()
    print("step:{}".format(step))
    if step >= (1 + 1 + 5) * 2:
        break
    train(batch_data)
    print((time.time()-st)*1000 ,"ms")
    prof.step()
prof.stop()

image

@aaronenyeshi aaronenyeshi added the plugin PyTorch Profiler TensorBoard Plugin related label Jun 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working plugin PyTorch Profiler TensorBoard Plugin related
Projects
None yet
Development

No branches or pull requests

3 participants