v3.114.0 FileArchive Issue: file not found when using a relative path of more than 2 levels above. Lambda zip: no such file or directory. #16092

lakshans · 2024-04-30T18:50:10Z

What happened?

Our CI process automatically pulls the latest version of pulumi. Seems like in the last update, we experience an issue where the lambda code reference is not finding our zip file. We have several lambda services across a few micro services, but only two of them reference a zip file that is 2 levels above the current directory. These two lambda services are failing. After troubleshooting, and ruling out any code changes, I found that reverting to the previous version works fine (3.113.3).

Example

const lambdaFunction = new lambda.Function("lambda", {
  code: new asset.FileArchive("../../lambda.zip"),
  role: lambdaRole.arn,
  handler: "newrelic-lambda-wrapper.handler",
  layers: [
    "arn:aws:lambda:ca-central-1:451483290750:layer:NewRelicNodeJS20X:10",
  ],
  vpcConfig: {
    subnetIds: [lambdaPrivateSubnetId],
    securityGroupIds: [securityGroup.id],
  },
  runtime: "nodejs18.x",
  timeout: cfg.requireNumber("LAMBDA_TIMEOUT"),
  memorySize: 2048,
  environment: {
    variables: {
      ...
    },
  },
});

Pulumi Error:

  pulumi:pulumi:Stack (my-app):
    error: Error: failed to register new resource lambda [aws:lambda/function:Function]: 2 UNKNOWN: failed to compute archive hash for "code": open ../../lambda.zip: no such file or directory

We have other lambda apps that use:

code: new asset.FileArchive("../lambda.zip")

or

code: new asset.FileArchive("lambda.zip")

without issue.

When reverting back to pulumi version 3.113.3, the above works as expected.

To reproduce:

Update cli to v3.114.0
Create a lambda function with code reference to a zip that is 2 levels above the current directory.
Run pulumi preview -> should have no such file or directory error.
Compare results to cli version v3.113.3

Output of `pulumi about`

CLI
Version 3.114.0
Go Version go1.22.2
Go Compiler gc

Plugins
KIND NAME VERSION
language nodejs unknown

Host
OS ubuntu
Version 20.04
Arch x86_64

Additional context

We will fix the version to v3.113.3 for now. What's the best approach to making sure our builds don't break to do these very frequent new versions? We need to ensure better consistency and we may have to keep versions static and update periodically. What is the standard practise others are using?

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

The text was updated successfully, but these errors were encountered:

antdking · 2024-04-30T18:59:32Z

Just been replicating this issue for someone over on Slack, and can confirm this is a v3.114.0 regression.

I suspect this PR is related: #15607

It seems the engine runs in the cwd of where pulumi is executed from, disregarding the value specified in pulumi.yaml//main.

Minimal replication for pulumi preview (using aws bucket as I couldn't find a local provider that utilises pulumi assets):

# Pulumi.yaml
name: pulumi-chdir
runtime:
  name: python
  options:
    virtualenv: venv
main: stack/
description: A minimal Python Pulumi program

# stack/__main__.py

import pulumi
import pulumi_aws
import os

stub_path = "../stub"
print(os.getcwd()) ## reports in directory of this file for both v3.113.3 and v3.114.0
asset = pulumi.FileArchive(stub_path)


# create a bucket, and upload the asset
bucket = pulumi_aws.s3.Bucket("my-bucket")
bucket_object = pulumi_aws.s3.BucketObject(
    "my-bucket-object",
    bucket=bucket.id,
    source=asset,  # engine reading this seems to be from the cwd of the command, instead of the project path specified in the Pulumi.yaml
    # make it private
    acl="private",
)

# stub/stub.py
# just a stub file

Frassle · 2024-05-01T11:26:19Z

Yup can confirm this is asset/archive hash calculations. I think it's a pretty easy fix of threading the current directory down to the right places.
This is the right fix, we can't just revert this because we want sub-programs to work one day (soon) and they necesiarlly work from a different working directory.

Fixes #16092.

…ues (#16100)  # Description  Fixes #16092. Two main changes in this PR. Firstly we add a test for assets/archives to the conformance tests, and had to fix up some small issues in the test runner to enable this (copying test data files, correcting project directory layout, handling multi-word package names). The other change is to fix the engine to respect the programs working directory when serialising/deserialising assets and archives. It does this by passing the programs working directory as a MarshalOption (n.b this option is only really needed in the engine, program SDK doesn't really need it because it doesn't change directories). ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt`  - [x] I have added tests that prove my fix is effective or that my feature works  - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change  - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version

antdking · 2024-05-03T13:46:37Z

Still facing the same problem with 3.115.0

pulumi about:

CLI          
Version      3.115.0
Go Version   go1.22.2
Go Compiler  gc

Plugins
KIND      NAME    VERSION
resource  aws     6.33.1
language  python  unknown

Host     
OS       arch
Version  "23.1.4"
Arch     x86_64

This project is written in python: executable='/home/anthony/.pyenv/shims/python3' version='3.12.3'

Current Stack: antdking/pulumi-chdir/dev

Found no resources associated with dev

Found no pending operations associated with dev

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/antdking
User           antdking
Organizations  antdking, antdking-testing
Token type     personal

Dependencies:
NAME        VERSION
pip         24.0
pulumi_aws  6.33.1
setuptools  69.5.1
wheel       0.43.0

pip show, as the pulumi sdk version doesn't seem to get detected:

$ venv/bin/pip show pulumi
Name: pulumi
Version: 3.115.0
Summary: Pulumi's Python SDK
Home-page: https://github.com/pulumi/pulumi
Author: 
Author-email: 
License: Apache 2.0
Location: /home/anthony/experiments/pulumi-chdir/venv/lib/python3.12/site-packages
Requires: dill, grpcio, protobuf, pyyaml, semver, six
Required-by: pulumi_aws

Presumably this will be fixed once providers are updated to build against 3.115.0?

Frassle · 2024-05-03T14:10:09Z

Still facing the same problem with 3.115.0

Can you post the error message you're getting?

antdking · 2024-05-03T14:39:03Z

Can you post the error message you're getting?

Previewing update (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/antdking/pulumi-chdir/dev/previews/693c06ac-5d79-4524-b43c-79eb44de87ea

     Type                 Name              Plan       Info
 +   pulumi:pulumi:Stack  pulumi-chdir-dev  create     1 error
 +   └─ aws:s3:Bucket     my-bucket         create     

Diagnostics:
  pulumi:pulumi:Stack (pulumi-chdir-dev):
    error: Program failed with an unhandled exception:
    Traceback (most recent call last):
      File "/home/anthony/experiments/pulumi-chdir/venv/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1009, in do_rpc_call
        return monitor.RegisterResource(req)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/anthony/experiments/pulumi-chdir/venv/lib/python3.12/site-packages/grpc/_channel.py", line 1160, in __call__
        return _end_unary_response_blocking(state, call, False, None)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/anthony/experiments/pulumi-chdir/venv/lib/python3.12/site-packages/grpc/_channel.py", line 1003, in _end_unary_response_blocking
        raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "failed to compute archive hash for "source": couldn't read archive path '../stub': stat ../stub: no such file or directory"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-05-03T15:38:24.914057726+01:00", grpc_status:2, grpc_message:"failed to compute archive hash for \"source\": couldn\'t read archive path \'../stub\': stat ../stub: no such file or directory"}"
    >
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/anthony/.pulumi/bin/pulumi-language-python-exec", line 191, in <module>
        loop.run_until_complete(coro)
      File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
        return future.result()
               ^^^^^^^^^^^^^^^
      File "/home/anthony/experiments/pulumi-chdir/venv/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 138, in run_in_stack
        await run_pulumi_func(run)
      File "/home/anthony/experiments/pulumi-chdir/venv/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 52, in run_pulumi_func
        await wait_for_rpcs()
      File "/home/anthony/experiments/pulumi-chdir/venv/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 114, in wait_for_rpcs
        await task
      File "/home/anthony/experiments/pulumi-chdir/venv/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1014, in do_register
        resp = await asyncio.get_event_loop().run_in_executor(None, do_rpc_call)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
        result = self.fn(*self.args, **self.kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/anthony/experiments/pulumi-chdir/venv/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1011, in do_rpc_call
        handle_grpc_error(exn)
      File "/home/anthony/experiments/pulumi-chdir/venv/lib/python3.12/site-packages/pulumi/runtime/settings.py", line 307, in handle_grpc_error
        raise grpc_error_to_exception(exn)
    Exception: failed to compute archive hash for "source": couldn't read archive path '../stub': stat ../stub: no such file or directory

antdking · 2024-05-03T14:57:07Z

I just tested with FileAsset, and this is fixed in 3.115.0 (broken in 3.114).

So it's just FileArchive that still has the working directory missing.

Frassle · 2024-05-03T18:53:37Z

Thanks I'll check that.

Frassle · 2024-05-03T18:55:45Z

Yup can confirm, we fixed file archive to handle files but the path lookup is wrong for folders. I'll add that to the tests and get it fixed, should be able to get a release out early next week with this fixed.

# Description  Fixes #16092. Last part of the fix started in #16100. That last PR fixed assets and archive files, but not archive folders. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt`  - [x] I have added tests that prove my fix is effective or that my feature works  - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change  - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version

antdking · 2024-05-06T10:52:23Z

confirmed working with v3.115.1, thanks

lakshans added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Apr 30, 2024

justinvp assigned tgummerer Apr 30, 2024

justinvp added this to the 0.104 milestone Apr 30, 2024

Frassle self-assigned this May 1, 2024

Frassle added a commit that referenced this issue May 1, 2024

Add asset/archive to conformance tests and fix engine working dir issues

5a9fb8d

Fixes #16092.

Frassle mentioned this issue May 1, 2024

Add asset/archive to conformance tests and fix engine working dir issues #16100

Merged

6 tasks

Frassle added a commit that referenced this issue May 1, 2024

Add asset/archive to conformance tests and fix engine working dir issues

4dfa055

Fixes #16092.

Frassle added a commit that referenced this issue May 1, 2024

Add asset/archive to conformance tests and fix engine working dir issues

910d918

Fixes #16092.

Frassle added a commit that referenced this issue May 1, 2024

Add asset/archive to conformance tests and fix engine working dir issues

963b347

Fixes #16092.

Frassle added a commit that referenced this issue May 2, 2024

Add asset/archive to conformance tests and fix engine working dir issues

5a1b7d4

Fixes #16092.

Frassle closed this as completed in #16100 May 2, 2024

pulumi-bot added the resolution/fixed This issue was fixed label May 2, 2024

Frassle reopened this May 3, 2024

Frassle mentioned this issue May 3, 2024

Fix folder archives in the engine #16119

Merged

6 tasks

justinvp removed the resolution/fixed This issue was fixed label May 6, 2024

justinvp unassigned tgummerer May 6, 2024

justinvp closed this as completed in #16119 May 6, 2024

pulumi-bot added the resolution/fixed This issue was fixed label May 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v3.114.0 FileArchive Issue: file not found when using a relative path of more than 2 levels above. Lambda zip: no such file or directory. #16092

v3.114.0 FileArchive Issue: file not found when using a relative path of more than 2 levels above. Lambda zip: no such file or directory. #16092

lakshans commented Apr 30, 2024

antdking commented Apr 30, 2024 •

edited

Frassle commented May 1, 2024 •

edited

antdking commented May 3, 2024 •

edited

Frassle commented May 3, 2024

antdking commented May 3, 2024

antdking commented May 3, 2024

Frassle commented May 3, 2024

Frassle commented May 3, 2024

antdking commented May 6, 2024

v3.114.0 FileArchive Issue: file not found when using a relative path of more than 2 levels above. Lambda zip: no such file or directory. #16092

v3.114.0 FileArchive Issue: file not found when using a relative path of more than 2 levels above. Lambda zip: no such file or directory. #16092

Comments

lakshans commented Apr 30, 2024

What happened?

Example

Output of pulumi about

Additional context

Contributing

antdking commented Apr 30, 2024 • edited

Frassle commented May 1, 2024 • edited

antdking commented May 3, 2024 • edited

Frassle commented May 3, 2024

antdking commented May 3, 2024

antdking commented May 3, 2024

Frassle commented May 3, 2024

Frassle commented May 3, 2024

antdking commented May 6, 2024

Output of `pulumi about`

antdking commented Apr 30, 2024 •

edited

Frassle commented May 1, 2024 •

edited

antdking commented May 3, 2024 •

edited