Help deciphering error message

[Scott-Hoefer-Spycloud]

Running into a weird error message that I can't decipher when trying to run the state machine, happens with any payload. Seems to originate from the utils.invokeLambdaWithProcessors function

{
  "errorType": "Error",
  "errorMessage": "Invocation error (running in series): 123,34,101,114,114,111,114,77,101,115,115,97,103,101,34,58,32,34,91,66,97,100,82,101,113,117,101,115,116,93,32,73,110,118,97,108,105,100,32,112,97,114,97,109,101,116,101,114,58,32,39,111,117,116,112,117,116,95,102,111,114,109,97,116,39,46,32,77,117,115,116,32,98,101,32,111,110,101,32,111,102,58,32,106,115,111,110,45,103,114,97,112,104,45,115,112,101,99,34,44,32,34,101,114,114,111,114,84,121,112,101,34,58,32,34,66,97,100,82,101,113,117,101,115,116,34,44,32,34,114,101,113,117,101,115,116,73,100,34,58,32,34,56,56,102,48,48,100,48,49,45,49,51,55,57,45,52,102,99,100,45,57,101,57,53,45,48,55,49,102,56,102,51,99,50,48,53,55,34,44,32,34,115,116,97,99,107,84,114,97,99,101,34,58,32,91,34,32,32,70,105,108,101,32,92,34,47,118,97,114,47,116,97,115,107,47,118,117,107,97,108,105,98,47,97,112,105,47,104,97,110,100,108,101,114,46,112,121,92,34,44,32,108,105,110,101,32,49,51,50,44,32,105,110,32,119,114,97,112,112,101,114,92,110,32,32,32,32,114,97,105,115,101,32,101,32,102,114,111,109,32,101,32,32,35,32,82,97,105,115,101,32,117,112,32,116,111,32,65,80,73,32,71,97,116,101,119,97,121,32,114,101,115,112,111,110,115,101,32,104,97,110,100,108,101,114,92,110,34,44,32,34,32,32,70,105,108,101,32,92,34,47,118,97,114,47,116,97,115,107,47,118,117,107,97,108,105,98,47,97,112,105,47,104,97,110,100,108,101,114,46,112,121,92,34,44,32,108,105,110,101,32,49,49,50,44,32,105,110,32,119,114,97,112,112,101,114,92,110,32,32,32,32,111,117,116,112,117,116,32,61,32,95,102,117,110,99,40,101,118,101,110,116,44,32,99,111,110,116,101,120,116,41,92,110,34,44,32,34,32,32,70,105,108,101,32,92,34,47,118,97,114,47,116,97,115,107,47,115,101,114,118,105,99,101,46,112,121,92,34,44,32,108,105,110,101,32,54,54,44,32,105,110,32,104,97,110,100,108,101,114,92,110,32,32,32,32,113,117,101,114,121,95,112,97,114,97,109,115,32,61,32,112,97,114,115,101,95,101,118,101,110,116,40,101,118,101,110,116,61,101,118,101,110,116,41,92,110,34,44,32,34,32,32,70,105,108,101,32,92,34,47,118,97,114,47,116,97,115,107,47,115,101,114,118,105,99,101,46,112,121,92,34,44,32,108,105,110,101,32,53,50,44,32,105,110,32,112,97,114,115,101,95,101,118,101,110,116,92,110,32,32,32,32,101,118,101,110,116,95,101,120,99,95,116,111,95,116,104,114,111,119,40,118,97,108,105,100,95,101,114,114,111,114,61,101,114,114,41,92,110,34,44,32,34,32,32,70,105,108,101,32,92,34,47,118,97,114,47,116,97,115,107,47,118,117,107,97,108,105,98,47,97,112,105,47,101,120,99,46,112,121,92,34,44,32,108,105,110,101,32,55,57,44,32,105,110,32,101,118,101,110,116,95,101,120,99,95,116,111,95,116,104,114,111,119,92,110,32,32,32,32,114,97,105,115,101,32,66,97,100,82,101,113,117,101,115,116,40,101,114,114,95,109,115,103,46,73,78,86,65,76,73,68,46,102,111,114,109,97,116,40,102,105,101,108,100,95,110,97,109,101,61,102,105,101,108,100,95,110,97,109,101,44,32,109,115,103,61,109,115,103,41,41,92,110,34,93,125 with payload \"{}\"",
  "trace": [
    "Error: Invocation error (running in series): 123,34,101,114,114,111,114,77,101,115,115,97,103,101,34,58,32,34,91,66,97,100,82,101,113,117,101,115,116,93,32,73,110,118,97,108,105,100,32,112,97,114,97,109,101,116,101,114,58,32,39,111,117,116,112,117,116,95,102,111,114,109,97,116,39,46,32,77,117,115,116,32,98,101,32,111,110,101,32,111,102,58,32,106,115,111,110,45,103,114,97,112,104,45,115,112,101,99,34,44,32,34,101,114,114,111,114,84,121,112,101,34,58,32,34,66,97,100,82,101,113,117,101,115,116,34,44,32,34,114,101,113,117,101,115,116,73,100,34,58,32,34,56,56,102,48,48,100,48,49,45,49,51,55,57,45,52,102,99,100,45,57,101,57,53,45,48,55,49,102,56,102,51,99,50,48,53,55,34,44,32,34,115,116,97,99,107,84,114,97,99,101,34,58,32,91,34,32,32,70,105,108,101,32,92,34,47,118,97,114,47,116,97,115,107,47,118,117,107,97,108,105,98,47,97,112,105,47,104,97,110,100,108,101,114,46,112,121,92,34,44,32,108,105,110,101,32,49,51,50,44,32,105,110,32,119,114,97,112,112,101,114,92,110,32,32,32,32,114,97,105,115,101,32,101,32,102,114,111,109,32,101,32,32,35,32,82,97,105,115,101,32,117,112,32,116,111,32,65,80,73,32,71,97,116,101,119,97,121,32,114,101,115,112,111,110,115,101,32,104,97,110,100,108,101,114,92,110,34,44,32,34,32,32,70,105,108,101,32,92,34,47,118,97,114,47,116,97,115,107,47,118,117,107,97,108,105,98,47,97,112,105,47,104,97,110,100,108,101,114,46,112,121,92,34,44,32,108,105,110,101,32,49,49,50,44,32,105,110,32,119,114,97,112,112,101,114,92,110,32,32,32,32,111,117,116,112,117,116,32,61,32,95,102,117,110,99,40,101,118,101,110,116,44,32,99,111,110,116,101,120,116,41,92,110,34,44,32,34,32,32,70,105,108,101,32,92,34,47,118,97,114,47,116,97,115,107,47,115,101,114,118,105,99,101,46,112,121,92,34,44,32,108,105,110,101,32,54,54,44,32,105,110,32,104,97,110,100,108,101,114,92,110,32,32,32,32,113,117,101,114,121,95,112,97,114,97,109,115,32,61,32,112,97,114,115,101,95,101,118,101,110,116,40,101,118,101,110,116,61,101,118,101,110,116,41,92,110,34,44,32,34,32,32,70,105,108,101,32,92,34,47,118,97,114,47,116,97,115,107,47,115,101,114,118,105,99,101,46,112,121,92,34,44,32,108,105,110,101,32,53,50,44,32,105,110,32,112,97,114,115,101,95,101,118,101,110,116,92,110,32,32,32,32,101,118,101,110,116,95,101,120,99,95,116,111,95,116,104,114,111,119,40,118,97,108,105,100,95,101,114,114,111,114,61,101,114,114,41,92,110,34,44,32,34,32,32,70,105,108,101,32,92,34,47,118,97,114,47,116,97,115,107,47,118,117,107,97,108,105,98,47,97,112,105,47,101,120,99,46,112,121,92,34,44,32,108,105,110,101,32,55,57,44,32,105,110,32,101,118,101,110,116,95,101,120,99,95,116,111,95,116,104,114,111,119,92,110,32,32,32,32,114,97,105,115,101,32,66,97,100,82,101,113,117,101,115,116,40,101,114,114,95,109,115,103,46,73,78,86,65,76,73,68,46,102,111,114,109,97,116,40,102,105,101,108,100,95,110,97,109,101,61,102,105,101,108,100,95,110,97,109,101,44,32,109,115,103,61,109,115,103,41,41,92,110,34,93,125 with payload \"{}\"",
    "    at runInSeries (/var/task/executor.js:167:19)",
    "    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
    "    at async module.exports.handler (/var/task/executor.js:65:19)"
  ]
}

[alexcasalboni]

Hey @Scott-Hoefer-Spycloud, thanks for reaching out 🙏

It looks like the function you're trying to power-tune is returning an error when invoked. And it looks like it's returning that long list of number as invocation response.

Basically, Lambda Power Tuning assumes that your function runs without raising errors. In case of errors, it assumes that something went wrong and it stops power-tuning.

A few questions:

1. Did you make sure your function runs correctly with the given input payload? (which seems to be empty)
2. Is that long list of numbers the expected output of your functions?
3. If it's the correct output, is your function expected to raise an error?
4. Are you using a pre/post-processor function too?

[Scott-Hoefer-Spycloud]

@alexcasalboni Thanks for the response

1. Yes, I removed the payload as I didn't want to upload it as part of this question, which is why I mentioned it happens with any payload. I can test invoke the lambda manually through the console and it works its only with this tool I get an error
2. No, I have no idea what that list of numbers is. I see you are altering the error message though so wondering if its coming from the tool? Is there a way to get a more descriptive error message from this tool?
3. no, not expected output and no idea what that list of numbers is or what the error even is as it seems this tool is eating it maybe?
4. No pre or post processing

[alexcasalboni]

Uh, that is interesting 🤔

The invocation results aren't really manipulated though. Since you aren't using any pre/post-processor, the list of numbers is simply invocationResults.Payload. And it's an error because it found a truthy value for invocationResults.FunctionError.

A few more questions:

1. What programming language are you using? (shouldn't matter, but might be a clue)
2. Have you enabled any particular flag/feature that might alter the Lambda invocation behavior such as response streaming?
3. Could you please share the entire state machine input? (without the input payload)

[Scott-Hoefer-Spycloud]

1. Python
2. Nothing set on the lambda that would cause issues that I can think of. Its pretty straigtforward

{
"errorType": "Error",
"errorMessage": "Invocation error (running in series): 123,34,101,114,114,111,114,77,101,115,115,97,103,101,34,58,34,50,48,50,52,45,48,52,45,48,50,84,49,52,58,52,50,58,53,48,46,48,54,53,90,32,51,55,51,101,56,48,101,50,45,49,51,97,54,45,52,49,52,48,45,98,100,98,102,45,101,50,57,52,50,54,55,49,100,50,50,50,32,84,97,115,107,32,116,105,109,101,100,32,111,117,116,32,97,102,116,101,114,32,50,57,46,48,50,32,115,101,99,111,110,100,115,34,125 with payload "...",
"trace": [
"Error: Invocation error (running in series): 123,34,101,114,114,111,114,77,101,115,115,97,103,101,34,58,34,50,48,50,52,45,48,52,45,48,50,84,49,52,58,52,50,58,53,48,46,48,54,53,90,32,51,55,51,101,56,48,101,50,45,49,51,97,54,45,52,49,52,48,45,98,100,98,102,45,101,50,57,52,50,54,55,49,100,50,50,50,32,84,97,115,107,32,116,105,109,101,100,32,111,117,116,32,97,102,116,101,114,32,50,57,46,48,50,32,115,101,99,111,110,100,115,34,125 with payload "...",
" at runInSeries (/var/task/executor.js:167:19)",
" at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
" at async module.exports.handler (/var/task/executor.js:65:19)"
]
}

Again, I can use the same payload and get a successful response outside the tool so I know thats correct

[Scott-Hoefer-Spycloud]

@alexcasalboni So I switched from a multiple weighted payload to a single payload input and it works now, no other change to the payload was made. Not sure what the issue is with weighted payloads but I can use it with just one at a time for now. Feel free to close this if you want or ask further questions. Thanks!

[alexcasalboni]

Thanks, that makes a lot of sense now 🚀

It definitely looks like an issue and we've recently changed how weighted payloads are treated recently so it could be a regression.

Could you please share with us the structure of the weighted payload you were using? (obfuscating the actual payload, if needed)

FYI @andrestoll

[Scott-Hoefer-Spycloud]

{
  "input": {
    "lambdaARN": "<my lambda arn>",
    "powerValues": [
      2048,
      4096,
      5120,
      6144,
      8192,
      10240
    ],
    "num": 50,
    "strategy": "balanced",
    "payload": [
      {
        "weight": 30,
        "payload": {...}
      },
      {
        "weight": 20,
        "payload": {...}
      }
    ]
  }

[andrestoll]

Can you check the logs of the "executor" function if the event payload for the invocation is what you expect? Is there a difference when using a weighted payload and single payload? Maybe you can share as well

[andrestoll]

Btw it seems you are passing in \"{}\" as the individual payload to the weighted payload is that what you need?

[Scott-Hoefer-Spycloud]

@andrestoll Payload isn't really empty when I run it I just can't post the contents here. I can check the executor logs and get back to you

[andrestoll]

@Scott-Hoefer-Spycloud I cannot reproduce the error, if you can let me know what caused the function to be tuned to fail we might have some more ideas.
@alexcasalboni invocationResults.Payload is a uint8 array, hence the list of numbers I suggest we include out the invocationResults.FunctionError instead

[Scott-Hoefer-Spycloud]

@andrestoll I am not sure what is causing the error as the error message is less than helpful and it works when I invoke the lambda manually with the same input. It does seem to be tied to one input whereas others work, but again I am unable to tell what the issue is without a meaningful error message. I did confirm that the weighted payload works if I remove the aforementioned input however

[andrestoll]

@Scott-Hoefer-Spycloud If you go to the log group /aws/lambda/[stack-name]-executor-[suffix] you will find more information on the invocation that caused the error and more important the payload of it.
Look for "INFO Invoking function"
also your Lambda function logs should give you some information why your function failed, can you confirm that the payload is exactly the same?

[lachriz-aws]

I was having the same issue, and for me the problem turned out to be that my Lambda function was timing out. I actually went in and modified the executor's code to dump the FunctionError property as well to the log, which gave me the value Unhandled - which then got me to triple check the logs of my Lambda function, and realize that it was timing out. Not sure if we can improve the error message in the power tuning tool to catch this scenario - or at least give a hint? :)

[alexcasalboni]

Thank you @lachriz-aws this sounds like a great step forward!

I'll make sure to investigate if that's reproducible with any function that times out 👌

[alexcasalboni]

@Scott-Hoefer-Spycloud @lachriz-aws we have a working implementation here (merging soon): #247

It will show the proper error message instead of the weird array buffer :)

[alexcasalboni]

The fix has been merged 🎉

Closing this for now, let me know if you still encounter the problem.