Skip to content

Commit

Permalink
Merge pull request #238 from ellisms/master
Browse files Browse the repository at this point in the history 
Deployment instruction revisions, removed references to UI and Lumigo
  • Loading branch information
@alexcasalboni
alexcasalboni authored Mar 25, 2024
2 parents 6272a6b + c57955e commit 12c6618
Show file tree
Hide file tree
Showing 6 changed files with 94 additions and 104 deletions.
7 changes: 7 additions & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
node_modules
jspm_packages
venv
.venv
coverage
.nyc_output

Expand All @@ -22,3 +23,9 @@ layer-sdk/src
# ignore SAM CLI created files/dirs
.aws-sam/
samconfig.toml

# ignore terraform created files/dirs
.terraform/
.terraform.*
*.tfstate*
cid.log
95 changes: 47 additions & 48 deletions README-DEPLOY.md
View file
Original file line number Diff line number Diff line change
@@ -1,21 +1,14 @@
# How to deploy the AWS Lambda Power Tuning tool

There are multiple options to deploy the tool.
There are 5 deployment options for deploying the tool using Infrastructure as Code (IaC).

If you are familiar with Infrastructure as Code, there are 4 ways for you to create all of the resources necessary for Lambda Power Tuning.

The following three options utilize [AWS CloudFormation](https://aws.amazon.com/cloudformation/) on your behalf to create the necessary resources. Each will create a new CloudFormation stack in your AWS account containing all the resources for the Lambda Power Tuning tool.
1. The easiest way is to [deploy the app via the AWS Serverless Application Repository (SAR)](#option1)
1. Manually [using the AWS SAM CLI](#option2)
1. Manually [using the AWS CDK](#option3)
1. The easiest way is to [deploy the app via the AWS Serverless Application Repository (SAR)](#option1).
1. [Using the AWS SAM CLI](#option2)
1. [Using the AWS CDK](#option3)
1. [Using Terraform by Hashicorp and SAR](#option4)
1. [Using native Terraform](#option5)

You can also [deploy manually with Terraform](#option5) by Hashicorp.

If you want to use Terraform natively (which circumvents Cloudformation), see [Option 6](#option6)

If you don't want to deal with any Infrastructure as Code tool, you can use one of the following:
1. The [Lumigo CLI](https://www.npmjs.com/package/lumigo-cli#lumigo-cli-powertune-lambda) (WARNING: deprecated)
1. The [Lambda Power Tuner UI](#option4)

Read more about the [deployment parameters here](README-INPUT-OUTPUT.md#state-machine-configuration-at-deployment-time).

Expand All @@ -28,12 +21,15 @@ You can also integrate the SAR app in your existing CloudFormation stacks - chec

## Option 2: Build and deploy with the AWS SAM CLI<a name="option2"></a>

**Note**: This method requires Docker.

1. Install the [AWS SAM CLI in your local environment](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html).

1. Configure your [AWS credentials (requires AWS CLI installed)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration):
```bash
$ aws configure
```
1. Install [Docker](https://docs.docker.com/get-docker/).
1. Clone this git repository:
```bash
$ git clone https://github.com/alexcasalboni/aws-lambda-power-tuning.git
Expand All @@ -45,24 +41,40 @@ You can also integrate the SAR app in your existing CloudFormation stacks - chec
```
[`sam build -u`](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-build.html) will run SAM build using a Docker container image that provides an environment similar to that which your function would run in. SAM build in-turn looks at your AWS SAM template file for information about Lambda functions and layers in this project.

Once the build has completed you should see output that states `Build Succeeded`. If not there will be error messages providing guidance on what went wrong.
1. Deploy the application using the SAM deploy "guided" mode:
Once the build completes successfully you will see output stating `Build Succeeded`. If the build is not successful, there will be error messages providing guidance on what went wrong.
1. Deploy the application using the guided SAM deploy mode:
```bash
$ sam deploy -g
```
[`sam deploy -g`](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-deploy.html) will provide simple prompts to walk you through the process of deploying the tool. Provide a unique name for the 'Stack Name' and supply the [AWS Region](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html#Concepts.RegionsAndAvailabilityZones.Regions) you want to run the tool in and then you can select the defaults for testing of this tool. After accepting the prompted questions with a "Y" you can optionally save your application configuration.
* For **Stack Name**, enter a unique name for the stack.
* For **AWS Region**, enter the region you want to deploy in.

Accept the defaults for all other prompts.

[`sam deploy -g`](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-deploy.html) provides simple prompts to walk you through the process of deploying the tool. The responses are saved in a configuration file, `samconfig.toml`, to be reused during subsequent deployments.

After that the SAM CLI will run the required commands to create the resources for the Lambda Power Tuning tool. The CloudFormation outputs shown will highlight any issues or failures.
SAM CLI will run the required commands to create the resources for the Lambda Power Tuning tool.

If there are no issues, once complete you will see the stack outputs and a `Successfully created/updated stack` message.
A successful deployment displays the message `Successfully created/updated stack`.
1. To delete Lambda Power Tuning, run
```bash
sam delete
```
Answer `Y` to the prompts.


## Option 3: Deploy the AWS SAR app with AWS CDK<a name="option3"></a>

1. [Install AWS CDK](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html) and [configure your AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration):

1. [Install AWS CDK](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html).
```bash
$ npm install -g aws-cdk
```

1. [Bootstrap](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_bootstrap) your account.

1. [Configure your AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration):

```bash
$ aws configure
```

Expand All @@ -83,34 +95,21 @@ You can also integrate the SAR app in your existing CloudFormation stacks - chec
})
```

Alternatively, you can use [CDK Patterns](https://github.com/cdk-patterns/serverless) to give you a pre configured project in either TypeScript or Python:
Alternatively, you can build and deploy the solution from the source in this repo. See the following pages for language-specific instructions.

```bash
# For the TypeScript CDK version
npx cdkp init the-lambda-power-tuner
### TypeScript
See the [Typescript instructions](cdk/typescript/README.md)

# or for the Python CDK version
npx cdkp init the-lambda-power-tuner --lang=python
```

1. To deploy the TypeScript version you just need to:

```bash
cd the-lambda-power-tuner
npm run deploy
```

For Python deployment, see the instructions [here](https://github.com/cdk-patterns/serverless#2-download-pattern-in-python-or-typescript-cdk).

## Option 4: Deploy via AWS Lambda Power Tuner UI<a name="option4"></a>

You can deploy and interact with Lambda Power Tuning with an ad-hoc web interface. This UI will deploy everything you need to power-tune your functions and also simplify the input/output management for Step Functions via API Gateway.

You can find the open-source project and the instructions to deploy it here: [mattymoomoo/aws-power-tuner-ui](https://github.com/mattymoomoo/aws-power-tuner-ui).
### Python
See the [Python instructions](cdk/python/README.md)

### go
See the [go instructions](cdk/go/README.md)

![Power Tuner UI](https://github.com/mattymoomoo/aws-power-tuner-ui/blob/master/imgs/website.png?raw=true)
### C\#
See the [Csharp instructions](cdk/csharp/README.md)

## Option 5: Deploy the SAR app with Terraform<a name="option5"></a>
## Option 4: Deploy the SAR app with Terraform<a name="option4"></a>

Simply add the `aws_serverlessapplicationrepository_cloudformation_stack` resource below to your Terraform code and deploy as usual through `terraform apply`.

Expand All @@ -137,11 +136,11 @@ See the [Terraform documentation](https://registry.terraform.io/providers/hashic

If you don't yet have a Terraform project, check out the [Terraform introduction](https://www.terraform.io/intro/index.html).
## Option 6: deploy natively with Terraform<a name="option6"></a>
## Option 5: Deploy natively with Terraform<a name="option5"></a>
Please see the documentation [here](terraform/Readme.md).
The Terraform modules are located in the [terraform](terraform) directory. Deployment documentation is [here](terraform/Readme.md).
## How to execute the state machine once deployed?
## How to execute the state machine once deployed
See [here](README-EXECUTE.md).
See the [execution](README-EXECUTE.md) instructions to run the state machine.
16 changes: 2 additions & 14 deletions README-EXECUTE.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# How to execute the state machine

Independently of how you've deployed the state machine, you can execute it in a few different ways. Programmatically, using the AWS CLI, AWS SDK, or Lumigo CLI. Manually, using the AWS Step Functions web console or the Lambda Power Tuner UI.
Independently of how you've deployed the state machine, you can execute it in a few different ways. Programmatically, using the AWS CLI, or AWS SDK. Manually, using the AWS Step Functions web console.

## Option 1: Execute the state machine programmatically (CLI)

Expand All @@ -10,13 +10,7 @@ Feel free to customize the `scripts/sample-execution-input.json`, and then run `

The script will start a state machine execution, wait for the execution to complete (polling), and then show the execution results.

## Option 2: Execute the state machine programmatically (Lumigo CLI)

The Lumigo CLI integration takes care of both deploying and executing the SAR app transparently.

Check it out [here](README-DEPLOY.md#user-content-option-4-deploy-with-the-lumigo-cli).

## Option 3: Execute the state machine manually (web console)
## Option 2: Execute the state machine manually (web console)

Once the state machine is deployed, you can execute it and provide an input object.

Expand All @@ -41,10 +35,4 @@ Click "**Start Execution**" again and the execution will start. In the next page

Once the execution has completed, you will find the execution results in the "**Output**" tab of the "**Execution Details**" section at the top of the page. The output will contain the optimal power configuration and its corresponding average cost per execution.

## Option 4: Execute the state machine manually (AWS Lambda Power Tuner UI)

You can deploy and interact with Lambda Power Tuning with an ad-hoc web interface. This UI will deploy everything you need to power-tune your functions and also simplify the input/output management for Step Functions via API Gateway.

You can find the open-source project and the instructions to deploy it here: [mattymoomoo/aws-power-tuner-ui](https://github.com/mattymoomoo/aws-power-tuner-ui).

![Power Tuner UI](https://github.com/mattymoomoo/aws-power-tuner-ui/blob/master/imgs/website.png?raw=true)
41 changes: 22 additions & 19 deletions README-INPUT-OUTPUT.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,28 @@
Each execution of the state machine will require an input and will provide the corresponding output.


## Deployment configuration options

The CloudFormation template accepts the following parameters:

* **PowerValues** (list of numbers): these power values will be used as the default in case no `powerValues` input parameter is provided at execution time
* **visualizationURL** (string): the base URL for the visualization tool, by default it's `lambda-power-tuning.show` but you could use your own custom tool
* **totalExecutionTimeout** (number in seconds, default=`300`): the timeout in seconds applied to all functions of the state machine
* **lambdaResource** (string, default=`*`): the `Resource` used in IAM policies; it's `*` by default but you could restrict it to a prefix or a specific function ARN
* **permissionsBoundary** (string): the ARN of a permissions boundary (policy), applied to all functions of the state machine
* **payloadS3Bucket** (string): the S3 bucket name used for large payloads (>256KB); if provided, it's added to a custom managed IAM policy that grants read-only permission to the S3 bucket; more details below in the [S3 payloads section](#user-content-s3-payloads)
* **payloadS3Key** (string, default=`*`): the S3 object key used for large payloads (>256KB); the default value grants access to all S3 objects in the bucket specified with `payloadS3Bucket`; more details below in the [S3 payloads section](#user-content-s3-payloads)
* **layerSdkName** (string): the name of the SDK layer, in case you need to customize it (optional)
* **logGroupRetentionInDays** (number, default=7): the number of days to retain log events in the Lambda log groups. Before this parameter existed, log events were retained indefinitely
* **securityGroupIds** (list of SecurityGroup IDs): List of Security Groups to use in every Lambda function's VPC Configuration (optional); please note that your VPC should be configured to allow public internet access (via NAT Gateway) or include VPC Endpoints to the Lambda service
* **subnetIds** (list of Subnet IDs): List of Subnets to use in every Lambda function's VPC Configuration (optional); please note that your VPC should be configured to allow public internet access (via NAT Gateway) or include VPC Endpoints to the Lambda service
* **stateMachineNamePrefix** (string, default=`powerTuningStateMachine`): Allows you to customize the name of the state machine. Maximum 43 characters, only alphanumeric (plus `-` and `_`). The last portion of the `AWS::StackId` will be appended to this value, so the full name will look like `powerTuningStateMachine-89549da0-a4f9-11ee-844d-12a2895ed91f`. Note: `StateMachineName` has a maximum of 80 characters and 36+1 from the `StackId` are appended, allowing 43 for a custom prefix.



Please note that the total execution time should stay below 300 seconds (5 min), which is the default timeout. You can easily estimate the total execution timeout based on the average duration of your functions. For example, if your function's average execution time is 5 seconds and you haven't enabled `parallelInvocation`, you should set `totalExecutionTimeout` to at least `num * 5`: 50 seconds if `num=10`, 500 seconds if `num=100`, and so on. If you have enabled `parallelInvocation`, usually you don't need to tune the value of `totalExecutionTimeout` unless your average execution time is above 5 min. If you have a sleep between invocations set, you should include that in your timeout calculations.


## State machine input (at execution time)

The state machine accepts the following input parameters:
Expand All @@ -25,25 +47,6 @@ The state machine accepts the following input parameters:
* **disablePayloadLogs** (boolean) If provided and set to a truthy value, suppresses `payload` from error messages and logs. If `preProcessorARN` is provided, this also suppresses the output payload of the pre-processor.
* **includeOutputResults** (boolean) If provided and set to true, the average cost and average duration for every power value configuration will be included in the state machine output.

## State machine configuration (at deployment time)

The CloudFormation template accepts the following parameters:

* **PowerValues** (list of numbers): these power values will be used as the default in case no `powerValues` input parameter is provided at execution time
* **visualizationURL** (string): the base URL for the visualization tool, by default it's `lambda-power-tuning.show` but you could use your own custom tool
* **totalExecutionTimeout** (number in seconds, default=`300`): the timeout in seconds applied to all functions of the state machine
* **lambdaResource** (string, default=`*`): the `Resource` used in IAM policies; it's `*` by default but you could restrict it to a prefix or a specific function ARN
* **permissionsBoundary** (string): the ARN of a permissions boundary (policy), applied to all functions of the state machine
* **payloadS3Bucket** (string): the S3 bucket name used for large payloads (>256KB); if provided, it's added to a custom managed IAM policy that grants read-only permission to the S3 bucket; more details below in the [S3 payloads section](#user-content-s3-payloads)
* **payloadS3Key** (string, default=`*`): the S3 object key used for large payloads (>256KB); the default value grants access to all S3 objects in the bucket specified with `payloadS3Bucket`; more details below in the [S3 payloads section](#user-content-s3-payloads)
* **layerSdkName** (string): the name of the SDK layer, in case you need to customize it (optional)
* **logGroupRetentionInDays** (number, default=7): the number of days to retain log events in the Lambda log groups. Before this parameter existed, log events were retained indefinitely
* **securityGroupIds** (list of SecurityGroup IDs): List of Security Groups to use in every Lambda function's VPC Configuration (optional); please note that your VPC should be configured to allow public internet access (via NAT Gateway) or include VPC Endpoints to the Lambda service
* **subnetIds** (list of Subnet IDs): List of Subnets to use in every Lambda function's VPC Configuration (optional); please note that your VPC should be configured to allow public internet access (via NAT Gateway) or include VPC Endpoints to the Lambda service
* **stateMachineNamePrefix** (string, default=`powerTuningStateMachine`): Allows you to customize the name of the state machine. Maximum 43 characters, only alphanumeric (plus `-` and `_`). The last portion of the `AWS::StackId` will be appended to this value, so the full name will look like `powerTuningStateMachine-89549da0-a4f9-11ee-844d-12a2895ed91f`. Note: `StateMachineName` has a maximum of 80 characters and 36+1 from the `StackId` are appended, allowing 43 for a custom prefix.


Please note that the total execution time should stay below 300 seconds (5 min), which is the default timeout. You can easily estimate the total execution timeout based on the average duration of your functions. For example, if your function's average execution time is 5 seconds and you haven't enabled `parallelInvocation`, you should set `totalExecutionTimeout` to at least `num * 5`: 50 seconds if `num=10`, 500 seconds if `num=100`, and so on. If you have enabled `parallelInvocation`, usually you don't need to tune the value of `totalExecutionTimeout` unless your average execution time is above 5 min. If you have a sleep between invocations set, you should include that in your timeout calculations.

### Usage in CI/CD pipelines

Expand Down
8 changes: 0 additions & 8 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -88,14 +88,6 @@ Website repository: [matteo-ronchetti/aws-lambda-power-tuning-ui](https://github

Optionally, you could deploy your own custom visualization tool and configure the CloudFormation Parameter named `visualizationURL` with your own URL.

## Power Tuner UI

Lambda Power Tuner UI is a web interface to simplify the deployment and execution of Lambda Power Tuning. It's built and maintained by [Matthew Dorrian](https://twitter.com/DorrianMatthew) and it aims at providing a consistent interface and a uniform developer experience across teams and projects.

![Power Tuner UI](https://github.com/mattymoomoo/aws-power-tuner-ui/blob/master/imgs/website.png?raw=true)

Power Tuner UI repository: [mattymoomoo/aws-power-tuner-ui](https://github.com/mattymoomoo/aws-power-tuner-ui)

## Additional features, considerations, and internals

[Here](README-ADVANCED.md) you can find out more about some advanced features of this project, its internals, and some considerations about security and execution cost.
Expand Down
Loading

0 comments on commit 12c6618

Please sign in to comment.