Custom subdomains stopped being created for new branches (re-opened)

[tonycassara]

Before opening, please confirm:

• I have checked the custom domains troubleshooting guide.
• I have read the guide for submitting bug reports.
• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

App Id

d1h6e93qbxs4wx

Region

us-east-1

Step

Preview url with custom domain

Status

Fails to create

Issue or question

We use custom subdomains for our preview builds. E.g.; https://pr-1111.customdomain.dev. These are auto-generated for each new branch. As of yesterday evening, the preview builds are defaulting to amplify urls. E.g; https://pr-1111.cccc.amplifyapp.com. Nothing has changed on our side. Settings seem fine and if we manually add a subdomain from Domain Management page, it works.

Error message

No response

Additional information

See existing issue that was closed before. This is the exact same problem: #3774

[mcd-janan]

Hi @Jay2113

It looks like you were able to support us with this issue when it was reported in #3774.

Would you be able to assist with looping in someone to investigate this further? It's been two weeks since this issue has been opened, so we're hoping to get this resolved ASAP!

Thanks in advance,
Janan

[rickbergfalk]

More info that might help troubleshooting for folks at AWS - we're seeing a lot of old amplifyapp.com preview URLs sticking around after PRs have been merged and branch deleted. Clean up of old preview builds is not consistently happening.

[rickbergfalk]

Creating a subdomain manually via the UI silently fails with the following printed to browser console

Failed to load resource: the server responded with a status of 400 ()
main.js:1 BadRequestException: Cannot update domain while the domain is in UPDATING status.
    at Object.extractError (https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:549697)
    at constructor.extractError (https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:554486)
    at constructor.callListeners (https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:580160)
    at constructor.emit (https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:579870)
    at constructor.emitEvent (https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:565719)
    at constructor.e (https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:561236)
    at t.runTo (https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:634196)
    at https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:634402
    at constructor.<anonymous> (https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:561506)
    at constructor.<anonymous> (https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:565775)
main.js:1 key: message
main.js:1 Cannot update domain while the domain is in UPDATING status.
main.js:1 key: code
main.js:1 BadRequestException
main.js:1 key: time
main.js:1 Wed Jan 03 2024 10:36:48 GMT-0600 (Central Standard Time)
main.js:1 key: requestId
main.js:1 ca5ff3cc-e223-4f61-8195-75e4a8db1671
main.js:1 key: statusCode
main.js:1 400
main.js:1 key: retryable
main.js:1 false
main.js:1 key: retryDelay
main.js:1 33.43825921592474
main.js:1 MissingRequiredParameter: Missing required key 'domainName' in params
    at constructor.fail (https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:542013)
    at constructor.validateStructure (https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:542212)
    at constructor.validateMember (https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:542597)
    at constructor.validate (https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:541669)
    at constructor.<anonymous> (https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:516129)
    at constructor.callListeners (https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:580160)
    at s (https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:580046)
    at https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:515244
    at i (https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:484405)
    at https://a.b.cdn.console.awsstatic.com/a/v1/NYUHJ3WI5BTGE3UV5RYIM2YTZGSC5FDGRUOY65H43OBCBZXJRPKQ/vendors.js:50:484705
main.js:1 key: message
main.js:1 Missing required key 'domainName' in params
main.js:1 key: code
main.js:1 MissingRequiredParameter
main.js:1 key: time
main.js:1 Wed Jan 03 2024 10:36:58 GMT-0600 (Central Standard Time)

[rickbergfalk]

More updates - It seems our environment is inconsistently broken. A PR that is missing its custom domain does not allow for one to be manually created.

Opening a new branch and new PR results in a proper custom domain preview build.

Some branches/PRs are working. Some are not.

Would very much appreciate if someone could look into what is going on.

[tonycassara]

I just opened a new branch and the domain did not successfully create, and adding it manually does not apply to subsequent amplify builds trying to kick off our CI as a follow up task.

[arundna]

@tonycassara - Apologies for the delay! We're investigating this issue under priority and will keep you posted with updates.

[tonycassara]

@arundna Unfortunately minutes before your reply I accidentally deleted the Amplify App we were trying to fix and now we're unable to reconnect the subdomain. Here is the error: Error message: One or more of the CNAMEs you provided are already associated with a different resource.

[arundna]

@tonycassara - Could you try again after an hour? https://docs.aws.amazon.com/amplify/latest/userguide/custom-domain-troubleshoot-guide.html#i-get-a-cnamealreadyexistsexception-error

In the meantime, we have identified a fix for the subdomain issue and will update once it's complete. Thanks!

[tonycassara]

@arundna it's been nearly 2 hours, still not connecting. I see the CNAME added to Route 53, but I think the old Amplify app is not letting go of the domain.

[tonycassara]

Okay we're back up and operational, hopefully the auto branch deletion can be fixed.

[victorccccc]

Hi @tonycassara, sorry for the inconvenience. This issue has now been resolved.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

[github-actions]

This issue has been automatically locked.

[arundna]

@tonycassara We have fixed the issue and verified your app is working as expected. Can you please check and let us know if everything works? Thanks!