Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix #4183: Adjust Overall Performance runtime in HTML test report #4300

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix #4183: Adjust Overall Performance runtime in HTML test report #4300

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions lib/reporter/reporters/html.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,15 +122,19 @@ class HtmlReporter extends BaseReporter {
}

aggregateStats() {
const startTime = new Date(this.results.startTimestamp).getTime();
const endTime = new Date(this.results.endTimestamp).getTime();
Comment on lines -125 to -126
Copy link
Member

@garg3133 garg3133 Nov 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Vaibhavsahu2810 Thanks for this PR. But, were you able to figure out why the subtraction of the above two variables is giving the wrong time?

Ideally, we should try to find the root cause of the problem and try to tackle that instead of just introducing a fix. If the this.results.endTimestamp is wrong above, we should fix that because it might be causing problems at other places as well.

Also, if the tests are running in parallel, we wouldn't just want to add up the time of all the tests but instead show the time the entire test run actually took from the user's perspective (if I'm not missing something here?).

Copy link
Contributor Author

@Vaibhavsahu2810 Vaibhavsahu2810 Dec 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@garg3133 The current time calculation in aggregateStats() has a problem that , It's calculating the total time using the global start and end timestamps . This includes time between test runs and setup/teardown time that isn't part of the actual test execution.
Meanwhile, individual suite times are calculated from the actual execution time of the tests (stored in module.timeMs), which is aggregated in the createInitialResult() and aggregateEnvironments() methods.
That's why i changed the time calculation method in aggregateStats() according to aggregateEnvironments() method.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Vaibhavsahu2810 That's fine. We are supposed to show the overall time (including setup/teardown and the time spent in between the test runs) in the Overall Performance section. This section should basically show the total run time of the Nightwatch process and not the sum of the run-time of individual test cases.

Let's say you run three tests in parallel using Nightwatch and the individual run time of the tests are as follows:

  • Test 1: 30 secs
  • Test 2: 75 secs
  • Test 3: 40 secs

Then the total run time of the Nightwatch process need not be same as the sum of the run time of the three test cases (since the tests are running in parallel) but it should always be greater then the run time of the longest running test (75 secs).

This is exactly the problem in the issue where the overall performance shows a time less than the time of the longest running test, which is not possible.

let totalTimeMs = 0;

for (const envName of Object.keys(this.environments)) {
const env = this.environments[envName];
totalTimeMs += env.stats.time;
}

const stats = {
total: 0,
passed: 0,
failed: 0,
skipped: 0,
time: endTime - startTime
time: totalTimeMs
};

for (const envName of Object.keys(this.environments)) {
Expand Down