Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Intermittent ENOENT #345

Open
Maxim-Mazurok opened this issue Dec 26, 2023 · 10 comments
Open

Intermittent ENOENT #345

Maxim-Mazurok opened this issue Dec 26, 2023 · 10 comments

Comments

@Maxim-Mazurok
Copy link

Maxim-Mazurok commented Dec 26, 2023

Bug description

During GHA run it couldn't find diff for one of the pages for some reason

How to reproduce

Intermittent, saw it once so far

Expected behavior

No response

Lost Pixel information

lost-pixel logs from CI

Run lost-pixel/[email protected]
/usr/bin/docker run --name lostpixellostpixelv382_0c3f5f --label 09[7](https://github.com/Maxim-Mazurok/maxim.mazurok.com/actions/runs/7328462883/job/19956775494#step:11:8)6f5 --workdir /github/workspace --rm -e "COMPOSER_PROCESS_TIMEOUT" -e "COMPOSER_NO_INTERACTION" -e "COMPOSER_NO_AUDIT" -e "INPUT_FINALIZE" -e "INPUT_CACHE_KEY" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_ENVIRONMENT" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e "ACTIONS_RESULTS_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/maxim.mazurok.com/maxim.mazurok.com":"/github/workspace" lostpixel/lost-pixel:v3.[8](https://github.com/Maxim-Mazurok/maxim.mazurok.com/actions/runs/7328462883/job/19956775494#step:11:9).2
Environment:
WORKSPACE=/github/workspace
CI_BUILD_ID=7328462883
CI_BUILD_NUMBER=61
EVENT_PATH=/github/workflow/event.json
COMMIT_HASH=b378d5db8ca5ba04a4ee[9](https://github.com/Maxim-Mazurok/maxim.mazurok.com/actions/runs/7328462883/job/19956775494#step:11:10)9d8bc4141de877[10](https://github.com/Maxim-Mazurok/maxim.mazurok.com/actions/runs/7328462883/job/19956775494#step:11:11)8f6
COMMIT_SHA=b378d5db8ca5ba04a4ee99d8bc4[14](https://github.com/Maxim-Mazurok/maxim.mazurok.com/actions/runs/7328462883/job/19956775494#step:11:15)1de877108f6
COMMIT_REF_NAME=main
REPOSITORY=Maxim-Mazurok/maxim.mazurok.com

Version: 3.8.2
Loading project config ...
Current working directory: /github/workspace
Looking for config file: /github/workspace/lostpixel.config.(ts|js|cjs|mjs)
✅ Found config file: /github/workspace/lostpixel.config.ts
🚀 Starting Lost Pixel in 'generateOnly' mode
📂 Creating shot folders
📸 Creating shots
Removing 0 files from .lostpixel/current/
Removing 0 files from .lostpixel/difference/
Found 0 pages from external loader

=== [Page Mode] http://[17](https://github.com/Maxim-Mazurok/maxim.mazurok.com/actions/runs/7328462883/job/19956775494#step:11:18)2.17.0.1:8080 ===

Prepared 8 pages for screenshots
[1/8] Taking screenshot of '/!index ' (/!index)
[2/8] Taking screenshot of '/blog/!index ' (/blog/!index)
[3/8] Taking screenshot of '/blog/socials-calendar ' (/blog/socials-calendar)
[4/8] Taking screenshot of '/blog/typescript-excess-property-checks ' (/blog/typescript-excess-property-checks)
[5/8] Taking screenshot of '/blog/google-ads-custom-conversion-setup ' (/blog/google-ads-custom-conversion-setup)
[1/8] Screenshot of '/!index' taken and saved to '.lostpixel/current/!index.png' in 3.072s (/!index)
[6/8] Taking screenshot of '/blog/ignore-internal-traffic-in-google-analytics ' (/blog/ignore-internal-traffic-in-google-analytics)
[3/8] Screenshot of '/blog/socials-calendar' taken and saved to '.lostpixel/current/blog/socials-calendar.png' in 3.092s (/blog/socials-calendar)
[7/8] Taking screenshot of '/blog/visual-testing-percy-gha-slack ' (/blog/visual-testing-percy-gha-slack)
[5/8] Screenshot of '/blog/google-ads-custom-conversion-setup' taken and saved to '.lostpixel/current/blog/google-ads-custom-conversion-setup.png' in 3.570s (/blog/google-ads-custom-conversion-setup)
[8/8] Taking screenshot of '/blog/meta-pixel-traps-in-app-browsers ' (/blog/meta-pixel-traps-in-app-browsers)
[2/8] Screenshot of '/blog/!index' taken and saved to '.lostpixel/current/blog/!index.png' in 4.341s (/blog/!index)
[4/8] Screenshot of '/blog/typescript-excess-property-checks' taken and saved to '.lostpixel/current/blog/typescript-excess-property-checks.png' in 4.347s (/blog/typescript-excess-property-checks)
[7/8] Screenshot of '/blog/visual-testing-percy-gha-slack' taken and saved to '.lostpixel/current/blog/visual-testing-percy-gha-slack.png' in 4.6[26](https://github.com/Maxim-Mazurok/maxim.mazurok.com/actions/runs/7328462883/job/19956775494#step:11:27)s (/blog/visual-testing-percy-gha-slack)
[6/8] Screenshot of '/blog/ignore-internal-traffic-in-google-analytics' taken and saved to '.lostpixel/current/blog/ignore-internal-traffic-in-google-analytics.png' in 4.689s (/blog/ignore-internal-traffic-in-google-analytics)
[8/8] Screenshot of '/blog/meta-pixel-traps-in-app-browsers' taken and saved to '.lostpixel/current/blog/meta-pixel-traps-in-app-browsers.png' in 5.[34](https://github.com/Maxim-Mazurok/maxim.mazurok.com/actions/runs/7328462883/job/19956775494#step:11:35)7s (/blog/meta-pixel-traps-in-app-browsers)
Screenshots done!
Creating shots took 9.174 seconds
🔍 Checking differences
Comparing 8 screenshots using 'pixelmatch' as compare engine
[1/8] Comparing '/!index' (/!index)
[2/8] Comparing '/blog/!index' (/blog/!index)
[3/8] Comparing '/blog/socials-calendar' (/blog/socials-calendar)
[4/8] Comparing '/blog/typescript-excess-property-checks' (/blog/typescript-excess-property-checks)
[5/8] Comparing '/blog/google-ads-custom-conversion-setup' (/blog/google-ads-custom-conversion-setup)
[6/8] Comparing '/blog/ignore-internal-traffic-in-google-analytics' (/blog/ignore-internal-traffic-in-google-analytics)
[7/8] Comparing '/blog/visual-testing-percy-gha-slack' (/blog/visual-testing-percy-gha-slack)
[8/8] Comparing '/blog/meta-pixel-traps-in-app-browsers' (/blog/meta-pixel-traps-in-app-browsers)
[1/8] No difference found. (/!index)
[2/8] No difference found. (/blog/!index)
[3/8] No difference found. (/blog/socials-calendar)
[4/8] No difference found. (/blog/typescript-excess-property-checks)
[5/8] No difference found. (/blog/google-ads-custom-conversion-setup)
[6/8] No difference found. (/blog/ignore-internal-traffic-in-google-analytics)
[7/8] No difference found. (/blog/visual-testing-percy-gha-slack)
❌ ENOENT: no such file or directory, open '.lostpixel/difference/blog/meta-pixel-traps-in-app-browsers.png'
Sending anonymized telemetry data.
@Maxim-Mazurok Maxim-Mazurok changed the title Intermittent ENOENT in GH action Intermittent ENOENT Dec 26, 2023
@d-ivashchuk
Copy link
Collaborator

hey @Maxim-Mazurok! thanks for the reporting! This usually happens for reasons not related to lost-pixel execution, when our runner could not open a page, for example, or the page errored by itself. My usual recommendation is to first make sure that the implementation is ok and that nothing breaks when you run your app on CI, and then, by exclusion method, try to figure out what is wrong.

If it gets reproducible though - would be super happy to look into that with you and help with the fix!

@Maxim-Mazurok
Copy link
Author

Got the same when running locally:

=== [Page Mode] http://localhost:8080/ ===

Prepared 8 pages for screenshots
[1/8] Taking screenshot of '/!index ' (/!index)
[2/8] Taking screenshot of '/blog/!index ' (/blog/!index)
[3/8] Taking screenshot of '/blog/socials-calendar ' (/blog/socials-calendar)
[4/8] Taking screenshot of '/blog/typescript-excess-property-checks ' (/blog/typescript-excess-property-checks)
[5/8] Taking screenshot of '/blog/google-ads-custom-conversion-setup ' (/blog/google-ads-custom-conversion-setup)
[2/8] Screenshot of '/blog/!index' taken and saved to '.lostpixel/current/blog/!index.png' in 2.250s (/blog/!index)
[6/8] Taking screenshot of '/blog/ignore-internal-traffic-in-google-analytics ' (/blog/ignore-internal-traffic-in-google-analytics)
[3/8] Screenshot of '/blog/socials-calendar' taken and saved to '.lostpixel/current/blog/socials-calendar.png' in 2.254s (/blog/socials-calendar)
[7/8] Taking screenshot of '/blog/visual-testing-percy-gha-slack ' (/blog/visual-testing-percy-gha-slack)
[4/8] Screenshot of '/blog/typescript-excess-property-checks' taken and saved to '.lostpixel/current/blog/typescript-excess-property-checks.png' in 2.585s (/blog/typescript-excess-property-checks)
[8/8] Taking screenshot of '/blog/meta-pixel-traps-in-app-browsers ' (/blog/meta-pixel-traps-in-app-browsers)
[1/8] Screenshot of '/!index' taken and saved to '.lostpixel/current/!index.png' in 2.881s (/!index)
[5/8] Screenshot of '/blog/google-ads-custom-conversion-setup' taken and saved to '.lostpixel/current/blog/google-ads-custom-conversion-setup.png' in 2.884s (/blog/google-ads-custom-conversion-setup)
[7/8] Screenshot of '/blog/visual-testing-percy-gha-slack' taken and saved to '.lostpixel/current/blog/visual-testing-percy-gha-slack.png' in 3.012s (/blog/visual-testing-percy-gha-slack)
[6/8] Screenshot of '/blog/ignore-internal-traffic-in-google-analytics' taken and saved to '.lostpixel/current/blog/ignore-internal-traffic-in-google-analytics.png' in 3.020s (/blog/ignore-internal-traffic-in-google-analytics)
[8/8] Screenshot of '/blog/meta-pixel-traps-in-app-browsers' taken and saved to '.lostpixel/current/blog/meta-pixel-traps-in-app-browsers.png' in 3.336s (/blog/meta-pixel-traps-in-app-browsers)
Screenshots done!
Creating shots took 6.138 seconds
🔍 Checking differences
Comparing 8 screenshots using 'pixelmatch' as compare engine
[1/8] Comparing '/!index' (/!index)
[2/8] Comparing '/blog/!index' (/blog/!index)
[3/8] Comparing '/blog/socials-calendar' (/blog/socials-calendar)
[4/8] Comparing '/blog/typescript-excess-property-checks' (/blog/typescript-excess-property-checks)
[5/8] Comparing '/blog/google-ads-custom-conversion-setup' (/blog/google-ads-custom-conversion-setup)
[6/8] Comparing '/blog/ignore-internal-traffic-in-google-analytics' (/blog/ignore-internal-traffic-in-google-analytics)
[7/8] Comparing '/blog/visual-testing-percy-gha-slack' (/blog/visual-testing-percy-gha-slack)
[8/8] Comparing '/blog/meta-pixel-traps-in-app-browsers' (/blog/meta-pixel-traps-in-app-browsers)
[1/8] Difference of 126891 pixels (3.99%) found. Difference image saved to: .lostpixel/difference/!index.png (/!index)
❌ ENOENT: no such file or directory, open '.lostpixel/difference/blog/!index.png'

@d-ivashchuk
Copy link
Collaborator

d-ivashchuk commented Dec 26, 2023

Just to clarify on the above message - ENOENT: no such file or directory, open means lost-pixel did not make the screenshot of this page, means lost-pixel could not open this page properly in playwright. So first things first we need to make sure that the page you are trying to open exists & you can open it in the browser.

if you can repro this locally - awesome, please verify that /!index page is accessible by the browser 🙌

@Maxim-Mazurok
Copy link
Author

Just to clarify on the above message - ENOENT: no such file or directory, open means lost-pixel did not make the screenshot of this page, means lost-pixel could not open this page properly in playwright. So first things first we need to make sure that the page you are trying to open exists & you can open it in the browser.

if you can repro this locally - awesome, please verify that /!index page is accessible by the browser 🙌

Hmm, interesting, but I don't see errors regarding screenshot taking... And on local I do see the screenshot in current folder. It's the difference folder that is missing a file. I would assume it's some sort of a race

@d-ivashchuk
Copy link
Collaborator

that is interesting indeed! is it always this !index page or?
Is there anything special about the page?
can you run successfully locally at least once, or it fails everytime now?

@Maxim-Mazurok
Copy link
Author

On local it is the same blog/!index page and I can't get it working at all now, fails every time

On GHA it was another page - ❌ ENOENT: no such file or directory, open '.lostpixel/difference/blog/meta-pixel-traps-in-app-browsers.png'

Nothing special about the pages I think... They all are here: https://maxim.mazurok.com/

I might try remove ! from the name

@Maxim-Mazurok
Copy link
Author

Maxim-Mazurok commented Dec 26, 2023

Looks like it's related to / in the names.

I did this and it works fine now:

import { CustomProjectConfig } from 'lost-pixel';
import { getUrlsFromSitemap } from './src/helpers';
import { readFileSync } from 'fs';

export const config: CustomProjectConfig = {
  pageShots: {
    pages: getUrlsFromSitemap(readFileSync('./sitemap.xml', 'utf-8')).map(
      x => ({
        path: x.pathname,
        name: (x.pathname.endsWith('/') ? `${x.pathname}index` : x.pathname)
          .replace(/\//g, '_')
          .replace(/^_/g, ''),
      })
    ),
    baseUrl: 'http://172.17.0.1:8080', // this has to be internal IP of the docker container inside of Github Actions
  },
  generateOnly: true,
  failOnDifference: true,
};
Prepared 8 pages for screenshots
[1/8] Taking screenshot of 'index ' (index)
[2/8] Taking screenshot of 'blog_index ' (blog_index)
[3/8] Taking screenshot of 'blog_socials-calendar ' (blog_socials-calendar)
[4/8] Taking screenshot of 'blog_typescript-excess-property-checks ' (blog_typescript-excess-property-checks)
[5/8] Taking screenshot of 'blog_google-ads-custom-conversion-setup ' (blog_google-ads-custom-conversion-setup)
[2/8] Screenshot of 'blog_index' taken and saved to '.lostpixel/current/blog_index.png' in 2.259s (blog_index)
[6/8] Taking screenshot of 'blog_ignore-internal-traffic-in-google-analytics ' (blog_ignore-internal-traffic-in-google-analytics)
[3/8] Screenshot of 'blog_socials-calendar' taken and saved to '.lostpixel/current/blog_socials-calendar.png' in 2.262s (blog_socials-calendar)
[7/8] Taking screenshot of 'blog_visual-testing-percy-gha-slack ' (blog_visual-testing-percy-gha-slack)
[1/8] Screenshot of 'index' taken and saved to '.lostpixel/current/index.png' in 2.829s (index)
[8/8] Taking screenshot of 'blog_meta-pixel-traps-in-app-browsers ' (blog_meta-pixel-traps-in-app-browsers)
[4/8] Screenshot of 'blog_typescript-excess-property-checks' taken and saved to '.lostpixel/current/blog_typescript-excess-property-checks.png' in 2.830s (blog_typescript-excess-property-checks)
[5/8] Screenshot of 'blog_google-ads-custom-conversion-setup' taken and saved to '.lostpixel/current/blog_google-ads-custom-conversion-setup.png' in 2.833s (blog_google-ads-custom-conversion-setup)
[6/8] Screenshot of 'blog_ignore-internal-traffic-in-google-analytics' taken and saved to '.lostpixel/current/blog_ignore-internal-traffic-in-google-analytics.png' in 3.884s (blog_ignore-internal-traffic-in-google-analytics)
[8/8] Screenshot of 'blog_meta-pixel-traps-in-app-browsers' taken and saved to '.lostpixel/current/blog_meta-pixel-traps-in-app-browsers.png' in 3.318s (blog_meta-pixel-traps-in-app-browsers)
[7/8] Screenshot of 'blog_visual-testing-percy-gha-slack' taken and saved to '.lostpixel/current/blog_visual-testing-percy-gha-slack.png' in 3.887s (blog_visual-testing-percy-gha-slack)
Screenshots done!
Creating shots took 6.385 seconds
🔍 Checking differences
Comparing 8 screenshots using 'pixelmatch' as compare engine
[1/8] Comparing 'index' (index)
[2/8] Comparing 'blog_index' (blog_index)
[3/8] Comparing 'blog_socials-calendar' (blog_socials-calendar)
[4/8] Comparing 'blog_typescript-excess-property-checks' (blog_typescript-excess-property-checks)
[5/8] Comparing 'blog_google-ads-custom-conversion-setup' (blog_google-ads-custom-conversion-setup)
[6/8] Comparing 'blog_ignore-internal-traffic-in-google-analytics' (blog_ignore-internal-traffic-in-google-analytics)
[7/8] Comparing 'blog_visual-testing-percy-gha-slack' (blog_visual-testing-percy-gha-slack)
[8/8] Comparing 'blog_meta-pixel-traps-in-app-browsers' (blog_meta-pixel-traps-in-app-browsers)
[1/8] Difference of 126891 pixels (3.99%) found. Difference image saved to: .lostpixel/difference/index.png (index)
[2/8] Difference of 19880 pixels (2.16%) found. Difference image saved to: .lostpixel/difference/blog_index.png (blog_index)
[3/8] Difference of 30770 pixels (2.02%) found. Difference image saved to: .lostpixel/difference/blog_socials-calendar.png (blog_socials-calendar)
[4/8] Difference of 1533451 pixels (18.73%) found. Difference image saved to: .lostpixel/difference/blog_typescript-excess-property-checks.png (blog_typescript-excess-property-checks)
[5/8] Difference of 908848 pixels (14.71%) found. Difference image saved to: .lostpixel/difference/blog_google-ads-custom-conversion-setup.png (blog_google-ads-custom-conversion-setup)
[6/8] Difference of 814749 pixels (13.27%) found. Difference image saved to: .lostpixel/difference/blog_ignore-internal-traffic-in-google-analytics.png (blog_ignore-internal-traffic-in-google-analytics)
[7/8] Difference of 2182673 pixels (20.39%) found. Difference image saved to: .lostpixel/difference/blog_visual-testing-percy-gha-slack.png (blog_visual-testing-percy-gha-slack)
[8/8] Difference of 2435290 pixels (14.71%) found. Difference image saved to: .lostpixel/difference/blog_meta-pixel-traps-in-app-browsers.png (blog_meta-pixel-traps-in-app-browsers)
Comparison done!
👋 Exiting process with 8 found differences & 0 baselines to update
Sending anonymized telemetry data.

@d-ivashchuk
Copy link
Collaborator

hmm, great catch! wanna give a proper fix a spin? :D I think you are already pretty much caught the exact problem so fix might not be that complicated 🤔

@Maxim-Mazurok
Copy link
Author

Well, the exact problem is that lost-pixel doesn't work great with folders... So we could either a - replace all / in names with _ or something. Or b - make lost-pixel work well with folders.

I think option b is the best, but I don't have bandwidth for now. I am just annoyed that percy is giving me 1px diff every day, so trying out lost-pixel, if it works well for me I might consider adding folders support, but tbh probably not anytime soon as I don't have many pages for now. Anyway, hope this helps, cheers!

@d-ivashchuk
Copy link
Collaborator

yup, some legit good thoughts, thanks Maxim! we are happy to help you with your migration from Percy anytime both for personal & work projects 😉 we have bunch of people migrating & the fact that you could choose individual thresholds on per page level is usually helping a ton already with those 1-2px flakiness

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants