Error 500 with Next 13.2: pnpm symlinks breaking changes #3467

Lezzio · 2023-05-07T17:55:08Z

Before opening, please confirm:

I have checked to see if my question is addressed in the FAQ.
I have searched for duplicate or closed issues.
I have read the guide for submitting bug reports.
I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
I have removed any sensitive information from my code snippets and submission.

App Id

dq7klsyx8y65h

AWS Region

us-east-1

Amplify Hosting feature

Frontend builds

Describe the bug

Our production application stopped working. The provision, build, test and deploy would still pass and work fine. However, when accessing the domain all the users would run into an HTTP Error 500.

After intensive efforts and investigation we figured out that the error comes from our package manager pnpm. We would use the default node linker "isolated" which describes as followed:

isolated: dependencies are symlinked from a virtual store at node_modules/.pnpm.

Without any warning or notice it seems like Amplify stopped supporting symlinks the way they are used and created by pnpm (as per the logs).

We came up with a hot fix, using another node linker, the hoisted one:

hoisted: a flat node_modules without symlinks is created. Same as the node_modules created by npm or Yarn Classic. One of Yarn's libraries is used for hoisting, when this setting is used.

Apparently some serverless providers don't support symlinks. Our questions are:

How could such changes be introduced ? Was it planned on your side ?
How is it going to evolve from now on ?
What are your advice on handling this with pnpm smoothly ?

Expected behavior

We expect symlinks to keep working the way they have been so far. Therefore the node linker isolated of pnpm shouldn't cause unnoticed production issues.

Reproduction steps

Have a next js project with pnpm and its default node linker (isolated)
Build and deploy
Try accessing the domain (HTTP Error 500 should occur)

Build Settings

version: "1.0"
applications:
  - appRoot: frontend/public-portal
    frontend:
      phases:
        preBuild:
          commands:
            - nvm install
            - corepack enable
            - corepack prepare [email protected] --activate
            - export NODE_OPTIONS=--max-old-space-size=8192
            - pnpm install --virtual-store-dir frontend/public-portal/node_modules/.pnpm
            - cd ../..
            - pnpm package
            - cd frontend/public-portal
        build:
          commands:
            - env | grep -E 'LAUNCHDARKLY_KEY|MAGNOLIA_HOST|MAGNOLIA_TOKEN|NEXT_PUBLIC_STAGE|VIDEO_HOST|NEXT_PUBLIC_DOMAIN|NEXT_PUBLIC_INDICATION|NEXT_PUBLIC_LOCALE|NEXT_PUBLIC_NAME' >> .env.production
            - pnpm build --no-lint
            - (shopt -s dotglob; mv .next/standalone/frontend/public-portal/* .next/standalone/)
      artifacts:
        baseDirectory: .next
        files:
          - "**/*"
        discard-paths: "yes"

Log output

# Put your logs below this line
2023-05-06T22:05:28.103+02:00 | INIT_START Runtime Version: nodejs:16.v13 Runtime Version ARN: arn:aws:lambda:us-east-1::runtime:a071073b1da9f709b2fc2e18948f6415fceadbd37ea568079d610b67bddfaef9
-- | --
  | 2023-05-06T22:05:28.155+02:00 | LOGS Name: bootstrap State: Subscribed Types: [Platform, Function]
  | 2023-05-06T22:05:28.169+02:00 | [x-amplify-log][INFO] starting copy
  | 2023-05-06T22:05:28.177+02:00 | [x-amplify-log][INFO] copy server pages complete
  | 2023-05-06T22:05:28.181+02:00 | [x-amplify-log][INFO] copy complete
  | 2023-05-06T22:05:28.184+02:00 | [x-amplify-log][INFO] Root symlink creation complete
  | 2023-05-06T22:05:28.192+02:00 | [x-amplify-log][INFO] symlink creation complete path=.next
  | 2023-05-06T22:05:28.199+02:00 | [x-amplify-log][INFO] symlink creation complete path=.next/server
  | 2023-05-06T22:05:28.199+02:00 | [x-amplify-log][INFO] Starting Server
  | 2023-05-06T22:05:28.327+02:00 | node:internal/modules/cjs/loader:1029
  | 2023-05-06T22:05:28.327+02:00 | throw err;
  | 2023-05-06T22:05:28.327+02:00 | ^
  | 2023-05-06T22:05:28.327+02:00 | Error: Cannot find module 'styled-jsx'
  | 2023-05-06T22:05:28.327+02:00 | Require stack:
  | 2023-05-06T22:05:28.327+02:00 | - /var/task/node_modules/next/dist/build/webpack/require-hook.js
  | 2023-05-06T22:05:28.327+02:00 | - /var/task/node_modules/next/dist/server/initialize-require-hook.js
  | 2023-05-06T22:05:28.327+02:00 | - /var/task/node_modules/next/dist/server/next-server.js
  | 2023-05-06T22:05:28.327+02:00 | - /tmp/app/server.js
  | 2023-05-06T22:05:28.327+02:00 | at Function.Module._resolveFilename (node:internal/modules/cjs/loader:1026:15)
  | 2023-05-06T22:05:28.327+02:00 | at Function.resolve (node:internal/modules/cjs/helpers:114:19)
  | 2023-05-06T22:05:28.327+02:00 | at Object.loadRequireHook (/var/task/node_modules/next/dist/build/webpack/require-hook.js:38:21)
  | 2023-05-06T22:05:28.327+02:00 | at Object.<anonymous> (/var/task/node_modules/next/dist/server/initialize-require-hook.js:3:19)
  | 2023-05-06T22:05:28.327+02:00 | at Module._compile (node:internal/modules/cjs/loader:1196:14)
  | 2023-05-06T22:05:28.327+02:00 | at Object.Module._extensions..js (node:internal/modules/cjs/loader:1250:10)
  | 2023-05-06T22:05:28.327+02:00 | at Module.load (node:internal/modules/cjs/loader:1074:32)
  | 2023-05-06T22:05:28.327+02:00 | at Function.Module._load (node:internal/modules/cjs/loader:909:12)
  | 2023-05-06T22:05:28.327+02:00 | at Module.require (node:internal/modules/cjs/loader:1098:19)
  | 2023-05-06T22:05:28.327+02:00 | at require (node:internal/modules/cjs/helpers:108:18) {
  | 2023-05-06T22:05:28.327+02:00 | code: 'MODULE_NOT_FOUND',
  | 2023-05-06T22:05:28.327+02:00 | requireStack: [
  | 2023-05-06T22:05:28.327+02:00 | '/var/task/node_modules/next/dist/build/webpack/require-hook.js',
  | 2023-05-06T22:05:28.327+02:00 | '/var/task/node_modules/next/dist/server/initialize-require-hook.js',
  | 2023-05-06T22:05:28.327+02:00 | '/var/task/node_modules/next/dist/server/next-server.js',
  | 2023-05-06T22:05:28.327+02:00 | '/tmp/app/server.js'
  | 2023-05-06T22:05:28.327+02:00 | ]
  | 2023-05-06T22:05:28.327+02:00 | }
  | 2023-05-06T22:05:28.524+02:00 | LOGS Name: bootstrap State: Already subscribed Types: [Platform, Function]
  | 2023-05-06T22:05:28.543+02:00 | [x-amplify-log][INFO] starting copy
  | 2023-05-06T22:05:28.560+02:00 | [x-amplify-log][INFO] copy server pages complete
  | 2023-05-06T22:05:28.563+02:00 | [x-amplify-log][INFO] copy complete
  | 2023-05-06T22:05:28.566+02:00 | ln: failed to create symbolic link ‘/tmp/app/node_modules/node_modules’: Read-only file system
  | 2023-05-06T22:05:28.567+02:00 | ln: failed to create symbolic link ‘/tmp/app/public/public’: Read-only file system
  | 2023-05-06T22:05:28.567+02:00 | [x-amplify-log][INFO] Root symlink creation complete
  | 2023-05-06T22:05:28.583+02:00 | ln: failed to create symbolic link ‘/tmp/app/.next/static/static’: Read-only file system
  | 2023-05-06T22:05:28.583+02:00 | [x-amplify-log][INFO] symlink creation complete path=.next
  | 2023-05-06T22:05:28.584+02:00 | ln: failed to create symbolic link ‘/tmp/app/.next/server/chunks/chunks’: Read-only file system
  | 2023-05-06T22:05:28.598+02:00 | [x-amplify-log][INFO] symlink creation complete path=.next/server
  | 2023-05-06T22:05:28.598+02:00 | [x-amplify-log][INFO] Starting Server
  | 2023-05-06T22:05:28.744+02:00 | node:internal/modules/cjs/loader:1029
  | 2023-05-06T22:05:28.744+02:00 | throw err;
  | 2023-05-06T22:05:28.744+02:00 | ^
  | 2023-05-06T22:05:28.744+02:00 | Error: Cannot find module 'styled-jsx'
  | 2023-05-06T22:05:28.744+02:00 | Require stack:
  | 2023-05-06T22:05:28.744+02:00 | - /var/task/node_modules/next/dist/build/webpack/require-hook.js
  | 2023-05-06T22:05:28.744+02:00 | - /var/task/node_modules/next/dist/server/initialize-require-hook.js
  | 2023-05-06T22:05:28.744+02:00 | - /var/task/node_modules/next/dist/server/next-server.js
  | 2023-05-06T22:05:28.744+02:00 | - /tmp/app/server.js
  | 2023-05-06T22:05:28.744+02:00 | at Function.Module._resolveFilename (node:internal/modules/cjs/loader:1026:15)
  | 2023-05-06T22:05:28.744+02:00 | at Function.resolve (node:internal/modules/cjs/helpers:114:19)
  | 2023-05-06T22:05:28.744+02:00 | at Object.loadRequireHook (/var/task/node_modules/next/dist/build/webpack/require-hook.js:38:21)
  | 2023-05-06T22:05:28.744+02:00 | at Object.<anonymous> (/var/task/node_modules/next/dist/server/initialize-require-hook.js:3:19)
  | 2023-05-06T22:05:28.744+02:00 | at Module._compile (node:internal/modules/cjs/loader:1196:14)
  | 2023-05-06T22:05:28.744+02:00 | at Object.Module._extensions..js (node:internal/modules/cjs/loader:1250:10)
  | 2023-05-06T22:05:28.744+02:00 | at Module.load (node:internal/modules/cjs/loader:1074:32)
  | 2023-05-06T22:05:28.744+02:00 | at Function.Module._load (node:internal/modules/cjs/loader:909:12)
  | 2023-05-06T22:05:28.744+02:00 | at Module.require (node:internal/modules/cjs/loader:1098:19)
  | 2023-05-06T22:05:28.744+02:00 | at require (node:internal/modules/cjs/helpers:108:18) {
  | 2023-05-06T22:05:28.744+02:00 | code: 'MODULE_NOT_FOUND',
  | 2023-05-06T22:05:28.744+02:00 | requireStack: [
  | 2023-05-06T22:05:28.744+02:00 | '/var/task/node_modules/next/dist/build/webpack/require-hook.js',
  | 2023-05-06T22:05:28.744+02:00 | '/var/task/node_modules/next/dist/server/initialize-require-hook.js',
  | 2023-05-06T22:05:28.744+02:00 | '/var/task/node_modules/next/dist/server/next-server.js',
  | 2023-05-06T22:05:28.744+02:00 | '/tmp/app/server.js'
  | 2023-05-06T22:05:28.744+02:00 | ]
  | 2023-05-06T22:05:28.744+02:00 | }
  | 2023-05-06T22:05:28.748+02:00 | START RequestId: b61ff919-36f8-464e-9e0f-f0395b543727 Version: $LATEST
  | 2023-05-06T22:05:28.758+02:00 | RequestId: b61ff919-36f8-464e-9e0f-f0395b543727 Error: Runtime exited with error: exit status 1 Runtime.ExitError
  | 2023-05-06T22:05:28.758+02:00 | REPORT RequestId: b61ff919-36f8-464e-9e0f-f0395b543727 Duration: 298.65 ms Billed Duration: 299 ms Memory Size: 1024 MB Max Memory Used: 25 MB
  | 2023-05-06T22:05:46.067+02:00 | LOGS Name: bootstrap State: Already subscribed Types: [Platform, Function]
  | 2023-05-06T22:05:46.086+02:00 | [x-amplify-log][INFO] starting copy
  | 2023-05-06T22:05:46.105+02:00 | [x-amplify-log][INFO] copy server pages complete
  | 2023-05-06T22:05:46.113+02:00 | [x-amplify-log][INFO] copy complete
  | 2023-05-06T22:05:46.115+02:00 | ln: failed to create symbolic link ‘/tmp/app/node_modules/node_modules’: Read-only file system
  | 2023-05-06T22:05:46.119+02:00 | ln: failed to create symbolic link ‘/tmp/app/public/public’: Read-only file system
  | 2023-05-06T22:05:46.119+02:00 | [x-amplify-log][INFO] Root symlink creation complete
  | 2023-05-06T22:05:46.126+02:00 | ln: failed to create symbolic link ‘/tmp/app/.next/static/static’: Read-only file system
  | 2023-05-06T22:05:46.126+02:00 | [x-amplify-log][INFO] symlink creation complete path=.next
  | 2023-05-06T22:05:46.127+02:00 | ln: failed to create symbolic link ‘/tmp/app/.next/server/chunks/chunks’: Read-only file system
  | 2023-05-06T22:05:46.143+02:00 | [x-amplify-log][INFO] symlink creation complete path=.next/server
  | 2023-05-06T22:05:46.143+02:00 | [x-amplify-log][INFO] Starting Server
  | 2023-05-06T22:05:46.288+02:00 | node:internal/modules/cjs/loader:1029
  | 2023-05-06T22:05:46.288+02:00 | throw err;
  | 2023-05-06T22:05:46.288+02:00 | ^
  | 2023-05-06T22:05:46.288+02:00 | Error: Cannot find module 'styled-jsx'
  | 2023-05-06T22:05:46.288+02:00 | Require stack:
  | 2023-05-06T22:05:46.288+02:00 | - /var/task/node_modules/next/dist/build/webpack/require-hook.js
  | 2023-05-06T22:05:46.288+02:00 | - /var/task/node_modules/next/dist/server/initialize-require-hook.js
  | 2023-05-06T22:05:46.288+02:00 | - /var/task/node_modules/next/dist/server/next-server.js
  | 2023-05-06T22:05:46.288+02:00 | - /tmp/app/server.js
  | 2023-05-06T22:05:46.288+02:00 | at Function.Module._resolveFilename (node:internal/modules/cjs/loader:1026:15)
  | 2023-05-06T22:05:46.288+02:00 | at Function.resolve (node:internal/modules/cjs/helpers:114:19)
  | 2023-05-06T22:05:46.288+02:00 | at Object.loadRequireHook (/var/task/node_modules/next/dist/build/webpack/require-hook.js:38:21)
  | 2023-05-06T22:05:46.288+02:00 | at Object.<anonymous> (/var/task/node_modules/next/dist/server/initialize-require-hook.js:3:19)
  | 2023-05-06T22:05:46.288+02:00 | at Module._compile (node:internal/modules/cjs/loader:1196:14)
  | 2023-05-06T22:05:46.288+02:00 | at Object.Module._extensions..js (node:internal/modules/cjs/loader:1250:10)
  | 2023-05-06T22:05:46.288+02:00 | at Module.load (node:internal/modules/cjs/loader:1074:32)
  | 2023-05-06T22:05:46.288+02:00 | at Function.Module._load (node:internal/modules/cjs/loader:909:12)
  | 2023-05-06T22:05:46.288+02:00 | at Module.require (node:internal/modules/cjs/loader:1098:19)
  | 2023-05-06T22:05:46.288+02:00 | at require (node:internal/modules/cjs/helpers:108:18) {
  | 2023-05-06T22:05:46.288+02:00 | code: 'MODULE_NOT_FOUND',
  | 2023-05-06T22:05:46.288+02:00 | requireStack: [
  | 2023-05-06T22:05:46.288+02:00 | '/var/task/node_modules/next/dist/build/webpack/require-hook.js',
  | 2023-05-06T22:05:46.288+02:00 | '/var/task/node_modules/next/dist/server/initialize-require-hook.js',
  | 2023-05-06T22:05:46.288+02:00 | '/var/task/node_modules/next/dist/server/next-server.js',
  | 2023-05-06T22:05:46.288+02:00 | '/tmp/app/server.js'
  | 2023-05-06T22:05:46.288+02:00 | ]
  | 2023-05-06T22:05:46.288+02:00 | }
  | 2023-05-06T22:05:46.299+02:00 | START RequestId: 3f17df38-371c-4dd1-8987-69f8854bb66a Version: $LATEST
  | 2023-05-06T22:05:46.301+02:00 | RequestId: 3f17df38-371c-4dd1-8987-69f8854bb66a Error: Runtime exited with error: exit status 1 Runtime.ExitError
  | 2023-05-06T22:05:46.301+02:00 | REPORT RequestId: 3f17df38-371c-4dd1-8987-69f8854bb66a Duration: 293.29 ms Billed Duration: 294 ms Memor

Additional information

We're using a NX monorepo with the following structure:

root
- frontend
--- app
--- backoffice
--- shared
--- public-portal

The following amplify app and build specs are to build and deploy the public-portal project.

We use pnpm package defined as followed:
"package": "nx run-many --target=package --all --parallel=4",
Because we have dependencies between our projects.

The text was updated successfully, but these errors were encountered:

abepuentes · 2023-05-08T13:09:51Z

We are having the same 500 error since yesterday for different next releases as you can see at #3466. In our case we are using yarn, still same error.

ghost · 2023-05-08T18:52:41Z

Hi @Lezzio 👋🏽 thanks for raising this issue. Please set node-linker=hoisted in the .npmrc file and let us know if this unblocks you.

Lezzio · 2023-05-08T18:55:40Z

Hi @hloriii, that's the fix we figured out. It does work thanks 🙂 However, is it normal for pnpm symlinks to be broken with amplify all of sudden and is it going to be addressed ? :)

ghost · 2023-05-08T22:00:02Z

@Lezzio This modification to the .npmrc file will be necessary moving forward in order to successfully host applications using pnpm as a package manager. We are working on updating our FAQ Guide to include this information so that this extra step is known.

github-actions · 2023-05-08T22:00:44Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

github-actions · 2023-05-08T22:02:06Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

Lezzio added bug Something isn't working pending-triage labels May 7, 2023

abepuentes mentioned this issue May 8, 2023

Nextjs ^13.4.0 500 errors #3466

Closed

5 tasks

Lezzio mentioned this issue May 8, 2023

Starting today, Next.js deployed with amplify is showing an internal server error #3468

Closed

ghost added the compute label May 8, 2023

ghost self-assigned this May 8, 2023

ghost added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label May 8, 2023

github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label May 8, 2023

ghost added question Further information is requested and removed bug Something isn't working pending-triage labels May 8, 2023

ghost closed this as completed May 8, 2023

github-actions bot added the archived This issue has been locked. label May 8, 2023

github-actions bot locked and limited conversation to collaborators May 8, 2023

This issue was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error 500 with Next 13.2: pnpm symlinks breaking changes #3467

Error 500 with Next 13.2: pnpm symlinks breaking changes #3467

Lezzio commented May 7, 2023 •

edited

Loading

abepuentes commented May 8, 2023

ghost commented May 8, 2023

Lezzio commented May 8, 2023 •

edited

Loading

ghost commented May 8, 2023

github-actions bot commented May 8, 2023

github-actions bot commented May 8, 2023

Error 500 with Next 13.2: pnpm symlinks breaking changes #3467

Error 500 with Next 13.2: pnpm symlinks breaking changes #3467

Comments

Lezzio commented May 7, 2023 • edited Loading

Before opening, please confirm:

App Id

AWS Region

Amplify Hosting feature

Describe the bug

Expected behavior

Reproduction steps

Build Settings

Log output

Additional information

abepuentes commented May 8, 2023

ghost commented May 8, 2023

Lezzio commented May 8, 2023 • edited Loading

ghost commented May 8, 2023

github-actions bot commented May 8, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

github-actions bot commented May 8, 2023

Lezzio commented May 7, 2023 •

edited

Loading

Lezzio commented May 8, 2023 •

edited

Loading