Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Take screenshot when a test fails #131

Open
gregberge opened this issue Sep 23, 2018 · 40 comments
Open

Take screenshot when a test fails #131

gregberge opened this issue Sep 23, 2018 · 40 comments

Comments

@gregberge
Copy link
Member

As suggested in #43, we would like to have a screenshot when a test fail, it is the same idea as #130.

Technical detail

  • The screenshot would be created in a .jest-puppeteer/screenshots folder.
  • The name of the screenshot would be ${fileName}-${describe}-${it}-${timestamp}.png

We could redefine it and test but I think this is not the cleanest method. Any help from Jest team to do it properly is welcome @SimenB.

@SimenB
Copy link

SimenB commented Sep 23, 2018

This is exactly the same as #130 (from Jest's perspective), right? Whether you take a screenshot or not shouldn't matter, as long as you can intercept the error reporting. Unless I'm misunderstanding?

@testerez
Copy link
Contributor

testerez commented Sep 24, 2018

Here is how I did it:

export const registerScreenshotReporter = () => {
  /**
   * jasmine reporter does not support async.
   * So we store the screenshot promise and wait for it before each test
   */
  let screenshotPromise = Promise.resolve();
  beforeEach(() => screenshotPromise);
  afterAll(() => screenshotPromise);

  /**
   * Take a screenshot on Failed test.
   * Jest standard reporters run in a separate process so they don't have
   * access to the page instance. Using jasmine reporter allows us to
   * have access to the test result, test name and page instance at the same time.
   */
  jasmine.getEnv().addReporter({
    specDone: async result => {
      if (result.status === 'failed') {
        screenshotPromise = screenshotPromise
          .catch()
          .then(() => takeScreenshot(result.fullName));
      }
    },
  });
};

@elmolicious
Copy link

+1 for a much needed feature

@kevlened
Copy link
Contributor

kevlened commented Oct 5, 2018

The jest-screenshot-reporter handles screenshots the way @testerez shows. That reporter currently assumes browser uses the webdriver api, but the logic is simple, so you could write something similar for puppeteer or add a contribution there.

@jacobweber
Copy link

@testerez Can you tell me where you're calling that code from? I was trying to do it the way jest-screenshot-reporter does, but it seems like the page gets closed before I can take the screenshot.

@testerez
Copy link
Contributor

testerez commented Oct 22, 2018

@jacobweber I just call registerScreenshotReporter() in setupTestFrameworkScriptFile (I'm not using jest-puppeteer anymore)

@jacobweber
Copy link

@testerez Thanks! Got it working.

@ricwal-richa
Copy link

@testerez On using your custom reporter, I get an error that takeScreenshot is not a valid method. I am using jest-puppeteer so I searched for this method in jasmine2 (as jest-jasmine2 used in jest-puppeteer) but could not find a reference. Can you please tell me if jasmine reporter would also need browser/page handle explicitly for this method to work ?

eg. browser.takeScreeshot() or page.takeScreenshot

I am able to make this work with puppeteer screenshot method (using that in custom defined method); but would prefer to use the in-built support for screenshot available with jasmine reporter, if any.

@testerez
Copy link
Contributor

testerez commented Dec 6, 2018

@ricwal-richa yes, takeScreenshot was not part of my code sample so it's normal you got this error. Here is the full code I use (this is typescript):

import path from 'path';
import mkdirp from 'mkdirp';

const screenshotsPath = path.resolve(__dirname, '../testReports/screenshots');

const toFilename = (s: string) => s.replace(/[^a-z0-9.-]+/gi, '_');

export const takeScreenshot = (testName: string, pageInstance = page) => {
  mkdirp.sync(screenshotsPath);
  const filePath = path.join(
    screenshotsPath,
    toFilename(`${new Date().toISOString()}_${testName}.png`),
  );
  return pageInstance.screenshot({
    path: filePath,
  });
};

export const registerScreenshotReporter = () => {
  /**
   * jasmine reporter does not support async.
   * So we store the screenshot promise and wait for it before each test
   */
  let screenshotPromise: Promise<any> = Promise.resolve();
  beforeEach(() => screenshotPromise);
  afterAll(() => screenshotPromise);

  /**
   * Take a screenshot on Failed test.
   * Jest standard reporters run in a separate process so they don't have
   * access to the page instance. Using jasmine reporter allows us to
   * have access to the test result, test name and page instance at the same time.
   */
  (jasmine as any).getEnv().addReporter({
    specDone: async (result: any) => {
      if (result.status === 'failed') {
        screenshotPromise = screenshotPromise
          .catch()
          .then(() => takeScreenshot(result.fullName));
      }
    },
  });
};

@petrogad
Copy link

petrogad commented Dec 27, 2018

Did you ever see blank screenshots and get around that issue?

Thank you for sharing!

@testerez ^

@gregberge
Copy link
Member Author

It looks like a lot of people are close to the solution. Could someone try to submit a PR to add screenshot? 🙏

@helloguille
Copy link

helloguille commented Dec 28, 2018

@testerez Thanks for the code! However, I cannot I get undefined error for the pageInstance variable.

I just put all the code (In javascript) in my setupTestFrameworkScriptFile file + a call to registerScreenshotReporter().

I modified the declaration for Vainilla as such:

export const takeScreenshot = (testName, pageInstance = page) => { ... }
And page is indeed the name of the page instance in my tests.

@RoiEXLab
Copy link

To come back at this issue, before I found this issue I had several attempts at this myself.
My goal had higher ambitions though: I wanted jest to automatically generate a screenshot before and after each individual test.
Disclaimer: I haven't really used the jest API before, so this is probably the best thing I came up with:

const path = require('path');
const fs = require('fs').promises;

const hash = (str) => {
    if (str.length === 0) {
        return 0;
    }
    let hash = 0;
    for (let i = 0; i < str.length; i++) {
        hash  = ((hash << 5) - hash) + str.charCodeAt(i);
        hash |= 0; // Convert to 32bit integer
    }
    return hash;
};

const origIt = global.it;
global.it = (description, test, timeout) => {
    const result = origIt(description, async () => {
        const basePath = path.join(result.result.testPath, '..', '__screenshots__');
        try {
            await fs.mkdir(basePath, { recursive: true });
            await page.screenshot({ path: path.join(basePath, `before-${hash(result.result.fullName)}.png`) });
        } finally {
            try {
                return await test();
            } finally {
                await page.screenshot({ path: path.join(basePath, `after-${hash(result.result.fullName)}.png`) });
            }
        }
    }, timeout);
    return result;
};

Re-defining it is kinda ugly but it works better than all my other versions. Also I ignored test for now, because in my case I don't need it.
Now with Jest 24 the file can be specified in setupFilesAfterEnv as part of an array.
I'd really like to see such a feature soon in this project, and I would create a PR as well, it's just that I'm not confident enough in my code. So if someone could nudge me into the right direction that would be great.

@xiaoyuhen
Copy link
Contributor

@RoiEXLab

Great solution!welcome a PR to solve this problem.

Now, jest-puppeteer has a bug with jest@24 and we will fix it this week as soon as possible. see #193

@celador
Copy link

celador commented Mar 4, 2019

// jest.setup.js

global.it = async function(name, func) {
  return await test(name, async () => {
    try {
      await func();
    } catch (e) {
      await fs.ensureDir('e2e/screenshots');
      await page.screenshot({ path: `e2e/screenshots/${name}.png` });
      throw e;
    }
  });
};

@redrockzee
Copy link

Use this package:
https://github.com/jeeyah/jestscreenshot

@gregberge
Copy link
Member Author

It is amazing to see that everyone has a solution and nobody wants to make a PR, welcome in Wordpress world 🤗

@lukaszfiszer
Copy link
Contributor

@petrogad check if you're not calling jestPuppeteer.resetPage() or doing something else with global page object in afterEach/afterAll. This was the reason I was getting blank screenshots.

@neoziro I should be able to submit a PR with adapted solution of @testerez in following days

@ajs139
Copy link

ajs139 commented May 1, 2019

Incase it's helpful, It seemed for me like the page was closed by the time jasmine reporter was called when using one of the above solutions, and overriding it prevents things like it.only from working. Ended up with this riff on @testerez's code, which takes a screenshot after every test (on a CI env), but only saves the screenshot if the test fails:
https://gist.github.com/ajs139/3ddc10e807ee9b94b581c80a762de587

@rostgoat
Copy link

rostgoat commented May 3, 2019

@testerez where would u even call registerScreenshotReporter ?

@testerez
Copy link
Contributor

testerez commented May 6, 2019

@rostgoat you can call it in your setupFilesAfterEnv jest script.

@rostgoat
Copy link

is this how your jest.config.js file looks @testerez?

setupFilesAfterEnv: [
    "<rootDir>/test/e2e/methods/common/screenshot.js"
  ]

how would this call the registerScreenshotReporter though?

@testerez
Copy link
Contributor

registerScreenshotReporter should be called in the file you set as setupFilesAfterEnv

@AlexMarchini
Copy link

Thought I'd share my solution to this, using jest-circus and extending the jest-puppeteer environment. It's a little messy with the sleeps, I want to find a better way to handle that, but it works for now. Maybe jest-puppeteer can integrate with jest-circus out of the box once circus has more adoption and make this better.

const fs = require('fs');
const PuppeteerEnvironment = require('jest-environment-puppeteer');
require('jest-circus');

const retryAttempts = process.env.RETRY_ATTEMPTS || 1;

class CustomEnvironment extends PuppeteerEnvironment {

  async setup() {
    await super.setup();
  }

  async teardown() {
    // Wait a few seconds before tearing down the page so we
    // have time to take screenshots and handle other events
    await this.global.page.waitFor(2000);
    await super.teardown()
  }

  async handleTestEvent(event, state) {
    if (event.name == 'test_fn_failure') {
      if (state.currentlyRunningTest.invocations > retryAttempts) {
        const testName = state.currentlyRunningTest.name;
        // Take a screenshot at the point of failure
        await this.global.page.screenshot({ path: `test-report/screenshots/${testName}.png` });
      }
    }
  }

}

module.exports = CustomEnvironment

@allimuu
Copy link

allimuu commented Jul 17, 2019

@AlexMarchini thanks for that! Perfect, I wanted to use jest-circus as well. If anyone else is wondering (or anyone has suggestions) like I was how to actually use this this is what my jest-config.js looks like:

{
    ...
    globalSetup: 'jest-environment-puppeteer/setup',
    globalTeardown: 'jest-environment-puppeteer/teardown',
    testEnvironment: './jest-environment.js',
    testRunner: 'jest-circus/runner',
    setupFilesAfterEnv: ['./jest-setup.js'],
    ...
}

So I removed preset and jest-environment.js is the above snippet. And jest-setup.js has nothing but require('expect-puppeteer'); in it.

And that seems to be working.. with jest-circus as the runner. (Note just run yarn add --dev jest-circus)

Hope this helps someone.

@favna
Copy link

favna commented Aug 13, 2019

Previously I commented here that we had a solution for work. The short version is that it was based on @testerez work. Sadly for whatever reason that was a moment of glory yesterday evening as when I started to get it into our CI/CD pipeline this morning it all crashed and burned and even refused to work locally so I deleted that comment.

That said, I ended up completely rewriting the library I made now relying on a solution very similar to @AlexMarchini / @allimuu thus relying on Jest Circus and at least it seems to work properly now.

For those interested you can find it here (bumped to v2.0.0 as it broke previous releases). Instructions and example on how to use the library are included in the README you can read on the yarnpkg / npm site. This includes a maybe more detailed (due to including code sample) version of @allimuu 's contribution.

And as before here is the actual source code behind it

sheetalkamat pushed a commit to DefinitelyTyped/DefinitelyTyped that referenced this issue Aug 19, 2019
* fix(jest-environment-puppeteer): Class & Exports

This makes the typings actually compatible to create your own test
environment and not just as part of the bigger jest-puppeteer package.
How to do this is pretty much what the test covers. One of the reasons
to do this is to write a handler where Puppeteer takes screenshots after
each failed test, as discussed [here](argos-ci/jest-puppeteer#131)

* Change to `export =` syntax

Co-Authored-By: Pranav Senthilnathan <[email protected]>

* Fix the last stuff requested
@dennismphil
Copy link

This was not as straightforward as I thought.

I found in the above methods it.only stops working. What worked for me was the below solution.

Monkey patching the Jest.it method.

// jest.config.js

module.exports = {
    ...
    setupFilesAfterEnv: [
        '<rootDir>/setupFilesAfterEnv.js',
    ]
};
// setupFilesAfterEnv.js

// Wrapper function that wraps the test function
// to take a screenshot on failure
const wrappedTest = (test, description) => {
    return Promise.resolve()
        .then(test)
        .catch(async (err) => {
            // Assuming you have a method to take a screenshot
            await takeScreenshot('some file name');

            throw err;
        });
};

// Make a copy of the original function
const originalIt = global.it;

// Modify `it` to use the wrapped test method
global.it = function it(description, test, timeout) {
    // Pass on the context by using `call` instead of directly invoking the method.
    return originalIt.call(this, description, wrappedTest.bind(this, test, description), timeout);
};

// Copy other function properties like `skip`, `only`...
for (const prop in originalIt) {
    if (Object.prototype.hasOwnProperty.call(originalIt, prop)) {
        global.it[prop] = originalIt[prop];
    }
}

// Monkey patch the `only` method also to use the wrapper method
global.it.only = function only(description, test, timeout) {
    return originalIt.only.call(this, description, wrappedTest.bind(this, test, description), timeout);
};

@noomorph
Copy link

noomorph commented Apr 3, 2020

I think this pull request jestjs/jest#9397 may come to the rescue if you are struggling with asynchronicity.

@tihuan
Copy link

tihuan commented May 1, 2020

For future readers, if you wanna get #131 (comment) to work, please make sure BOTH of your jest and jest-circus are on versions that include jestjs/jest#9397, which means both jest and jest-circus need to be v25.3.0+

Our project updated jest to v25.5.0, but jest-circus stayed at v25.2.7, resulting in Circus runner not waiting for test_fn_failure test event handler:

https://github.com/facebook/jest/pull/9397/files#diff-e77e110fa517918f09bcd38e70996fafR164

Note to self that jest and jest-circus have version parity, so always update both libraries together!

@stefanteixeira
Copy link

stefanteixeira commented May 5, 2020

In case someone wants to know how to get the test describe name when using jest-circus (#131 (comment) saves a screenshot only with the test method name), you can get it like:

const testDescription = state.currentlyRunningTest.parent.name 

I had to spend some time searching in the source code how State is defined inside jest-circus to find that information, hope it helps someone 😄

@sovoid
Copy link

sovoid commented Jun 30, 2020

I have somehow got video recording of test runs and screenshots of tests before and after each test working using Jest and Puppeteer. Please have a look if you guys find it helpful...

#361

@ajaykhobragade
Copy link

This is all quite helpful, it would be even better if somehow log would be able to generate with information and saved in directory on test failure, along with screenshot. Any help\suggestions around that?

@ajaykhobragade
Copy link

ajaykhobragade commented Aug 13, 2020 via email

@ShivamJoker
Copy link

Whats the status no PR's for taking screenshot ?

@jpourdanis
Copy link

@ricwal-richa yes, takeScreenshot was not part of my code sample so it's normal you got this error. Here is the full code I use (this is typescript):

import path from 'path';
import mkdirp from 'mkdirp';

const screenshotsPath = path.resolve(__dirname, '../testReports/screenshots');

const toFilename = (s: string) => s.replace(/[^a-z0-9.-]+/gi, '_');

export const takeScreenshot = (testName: string, pageInstance = page) => {
  mkdirp.sync(screenshotsPath);
  const filePath = path.join(
    screenshotsPath,
    toFilename(`${new Date().toISOString()}_${testName}.png`),
  );
  return pageInstance.screenshot({
    path: filePath,
  });
};

export const registerScreenshotReporter = () => {
  /**
   * jasmine reporter does not support async.
   * So we store the screenshot promise and wait for it before each test
   */
  let screenshotPromise: Promise<any> = Promise.resolve();
  beforeEach(() => screenshotPromise);
  afterAll(() => screenshotPromise);

  /**
   * Take a screenshot on Failed test.
   * Jest standard reporters run in a separate process so they don't have
   * access to the page instance. Using jasmine reporter allows us to
   * have access to the test result, test name and page instance at the same time.
   */
  (jasmine as any).getEnv().addReporter({
    specDone: async (result: any) => {
      if (result.status === 'failed') {
        screenshotPromise = screenshotPromise
          .catch()
          .then(() => takeScreenshot(result.fullName));
      }
    },
  });
};

Hey @testerez where we can call that? on protractor.conf.js ?

@andredesousa
Copy link

andredesousa commented May 1, 2021

Hi everyone.
Based on these documents, I created a setup for React with jest-puppeteer and jest-html-reporters.
For more details:

https://github.com/andredesousa/essential-react-scaffold

Basically, I cretead a new Test Environment:

class E2EEnvironment extends PuppeteerEnvironment {
  async handleTestEvent(event, state) {
    if (event.name === 'test_fn_failure') {
      const data = await this.global.page.screenshot();

      await addAttach(data, 'Full Page Screenshot', this.global);
    }
  }
}

...and I imported it into my jest.config.js file:

module.exports = {
  preset: 'jest-puppeteer',
  testRunner: 'jest-circus/runner',
  testEnvironment: '<rootDir>/environment.js',
  testMatch: ['<rootDir>/specs/**/*.spec.js'],
  reporters: [
    'default',
    [
      'jest-html-reporters',
      {
        publicPath: './reports/e2e',
        filename: 'index.html',
      },
    ],
  ],
};

Thus, when the test fails, a screenshot is attached to the report.

@madhusudhan-patha
Copy link

@ricwal-richa yes, takeScreenshot was not part of my code sample so it's normal you got this error. Here is the full code I use (this is typescript):

import path from 'path';
import mkdirp from 'mkdirp';

const screenshotsPath = path.resolve(__dirname, '../testReports/screenshots');

const toFilename = (s: string) => s.replace(/[^a-z0-9.-]+/gi, '_');

export const takeScreenshot = (testName: string, pageInstance = page) => {
  mkdirp.sync(screenshotsPath);
  const filePath = path.join(
    screenshotsPath,
    toFilename(`${new Date().toISOString()}_${testName}.png`),
  );
  return pageInstance.screenshot({
    path: filePath,
  });
};

export const registerScreenshotReporter = () => {
  /**
   * jasmine reporter does not support async.
   * So we store the screenshot promise and wait for it before each test
   */
  let screenshotPromise: Promise<any> = Promise.resolve();
  beforeEach(() => screenshotPromise);
  afterAll(() => screenshotPromise);

  /**
   * Take a screenshot on Failed test.
   * Jest standard reporters run in a separate process so they don't have
   * access to the page instance. Using jasmine reporter allows us to
   * have access to the test result, test name and page instance at the same time.
   */
  (jasmine as any).getEnv().addReporter({
    specDone: async (result: any) => {
      if (result.status === 'failed') {
        screenshotPromise = screenshotPromise
          .catch()
          .then(() => takeScreenshot(result.fullName));
      }
    },
  });
};

@jpourdanis I am trying to use this code (typescript) along with jest-cucumber, but it gives me error saying.

 ReferenceError: jasmine is not defined

      21 |      * have access to the test result, test name and page instance at the same time.
      22 |      */
    > 23 |     (jasmine as any).getEnv().addReporter({

@karlhorky
Copy link
Contributor

karlhorky commented Mar 25, 2022

Maybe the PR to WordPress Gutenberg by @kevin940726 :

WordPress/gutenberg#28449

...would provide a basis for a PR to the jest-environment-puppeteer package in this repo. Specifically, it seems to be this file that needs to be updated:

https://github.com/smooth-code/jest-puppeteer/blob/master/packages/jest-environment-puppeteer/src/PuppeteerEnvironment.js

@lucassardois
Copy link

In my use case, I would like to be able to take screenshots for all open pages of puppeteer when a test fail. Because my test needs to have some background pages running.

@gregberge
Copy link
Member Author

@lucassardois PR was done but I had to migrate to TypeScript before. Will be done in the next weeks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests