diff --git a/.gitignore b/.gitignore index 4ceb14fe..d83314cd 100644 --- a/.gitignore +++ b/.gitignore @@ -5,6 +5,7 @@ node_modules jspm_packages venv +.venv coverage .nyc_output @@ -22,3 +23,9 @@ layer-sdk/src # ignore SAM CLI created files/dirs .aws-sam/ samconfig.toml + +# ignore terraform created files/dirs +.terraform/ +.terraform.* +*.tfstate* +cid.log \ No newline at end of file diff --git a/README-DEPLOY.md b/README-DEPLOY.md index 9eb210ad..2939e3cb 100644 --- a/README-DEPLOY.md +++ b/README-DEPLOY.md @@ -1,21 +1,14 @@ # How to deploy the AWS Lambda Power Tuning tool -There are multiple options to deploy the tool. +There are 5 deployment options for deploying the tool using Infrastructure as Code (IaC). -If you are familiar with Infrastructure as Code, there are 4 ways for you to create all of the resources necessary for Lambda Power Tuning. -The following three options utilize [AWS CloudFormation](https://aws.amazon.com/cloudformation/) on your behalf to create the necessary resources. Each will create a new CloudFormation stack in your AWS account containing all the resources for the Lambda Power Tuning tool. -1. The easiest way is to [deploy the app via the AWS Serverless Application Repository (SAR)](#option1) -1. Manually [using the AWS SAM CLI](#option2) -1. Manually [using the AWS CDK](#option3) +1. The easiest way is to [deploy the app via the AWS Serverless Application Repository (SAR)](#option1). +1. [Using the AWS SAM CLI](#option2) +1. [Using the AWS CDK](#option3) +1. [Using Terraform by Hashicorp and SAR](#option4) +1. [Using native Terraform](#option5) -You can also [deploy manually with Terraform](#option5) by Hashicorp. - -If you want to use Terraform natively (which circumvents Cloudformation), see [Option 6](#option6) - -If you don't want to deal with any Infrastructure as Code tool, you can use one of the following: -1. The [Lumigo CLI](https://www.npmjs.com/package/lumigo-cli#lumigo-cli-powertune-lambda) (WARNING: deprecated) -1. The [Lambda Power Tuner UI](#option4) Read more about the [deployment parameters here](README-INPUT-OUTPUT.md#state-machine-configuration-at-deployment-time). @@ -28,12 +21,15 @@ You can also integrate the SAR app in your existing CloudFormation stacks - chec ## Option 2: Build and deploy with the AWS SAM CLI +**Note**: This method requires Docker. + 1. Install the [AWS SAM CLI in your local environment](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html). 1. Configure your [AWS credentials (requires AWS CLI installed)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration): ```bash $ aws configure ``` +1. Install [Docker](https://docs.docker.com/get-docker/). 1. Clone this git repository: ```bash $ git clone https://github.com/alexcasalboni/aws-lambda-power-tuning.git @@ -45,24 +41,40 @@ You can also integrate the SAR app in your existing CloudFormation stacks - chec ``` [`sam build -u`](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-build.html) will run SAM build using a Docker container image that provides an environment similar to that which your function would run in. SAM build in-turn looks at your AWS SAM template file for information about Lambda functions and layers in this project. - Once the build has completed you should see output that states `Build Succeeded`. If not there will be error messages providing guidance on what went wrong. -1. Deploy the application using the SAM deploy "guided" mode: + Once the build completes successfully you will see output stating `Build Succeeded`. If the build is not successful, there will be error messages providing guidance on what went wrong. +1. Deploy the application using the guided SAM deploy mode: ```bash $ sam deploy -g ``` - [`sam deploy -g`](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-deploy.html) will provide simple prompts to walk you through the process of deploying the tool. Provide a unique name for the 'Stack Name' and supply the [AWS Region](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html#Concepts.RegionsAndAvailabilityZones.Regions) you want to run the tool in and then you can select the defaults for testing of this tool. After accepting the prompted questions with a "Y" you can optionally save your application configuration. + * For **Stack Name**, enter a unique name for the stack. + * For **AWS Region**, enter the region you want to deploy in. + + Accept the defaults for all other prompts. + + [`sam deploy -g`](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-deploy.html) provides simple prompts to walk you through the process of deploying the tool. The responses are saved in a configuration file, `samconfig.toml`, to be reused during subsequent deployments. - After that the SAM CLI will run the required commands to create the resources for the Lambda Power Tuning tool. The CloudFormation outputs shown will highlight any issues or failures. + SAM CLI will run the required commands to create the resources for the Lambda Power Tuning tool. - If there are no issues, once complete you will see the stack outputs and a `Successfully created/updated stack` message. + A successful deployment displays the message `Successfully created/updated stack`. +1. To delete Lambda Power Tuning, run + ```bash + sam delete + ``` + Answer `Y` to the prompts. ## Option 3: Deploy the AWS SAR app with AWS CDK -1. [Install AWS CDK](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html) and [configure your AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration): - +1. [Install AWS CDK](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html). ```bash $ npm install -g aws-cdk + ``` + +1. [Bootstrap](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_bootstrap) your account. + +1. [Configure your AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration): + + ```bash $ aws configure ``` @@ -83,34 +95,21 @@ You can also integrate the SAR app in your existing CloudFormation stacks - chec }) ``` - Alternatively, you can use [CDK Patterns](https://github.com/cdk-patterns/serverless) to give you a pre configured project in either TypeScript or Python: +Alternatively, you can build and deploy the solution from the source in this repo. See the following pages for language-specific instructions. - ```bash - # For the TypeScript CDK version - npx cdkp init the-lambda-power-tuner + ### TypeScript +See the [Typescript instructions](cdk/typescript/README.md) - # or for the Python CDK version - npx cdkp init the-lambda-power-tuner --lang=python - ``` - -1. To deploy the TypeScript version you just need to: - - ```bash - cd the-lambda-power-tuner - npm run deploy - ``` - - For Python deployment, see the instructions [here](https://github.com/cdk-patterns/serverless#2-download-pattern-in-python-or-typescript-cdk). - -## Option 4: Deploy via AWS Lambda Power Tuner UI - -You can deploy and interact with Lambda Power Tuning with an ad-hoc web interface. This UI will deploy everything you need to power-tune your functions and also simplify the input/output management for Step Functions via API Gateway. - -You can find the open-source project and the instructions to deploy it here: [mattymoomoo/aws-power-tuner-ui](https://github.com/mattymoomoo/aws-power-tuner-ui). + ### Python +See the [Python instructions](cdk/python/README.md) + + ### go +See the [go instructions](cdk/go/README.md) -![Power Tuner UI](https://github.com/mattymoomoo/aws-power-tuner-ui/blob/master/imgs/website.png?raw=true) +### C\# +See the [Csharp instructions](cdk/csharp/README.md) -## Option 5: Deploy the SAR app with Terraform +## Option 4: Deploy the SAR app with Terraform Simply add the `aws_serverlessapplicationrepository_cloudformation_stack` resource below to your Terraform code and deploy as usual through `terraform apply`. @@ -137,11 +136,11 @@ See the [Terraform documentation](https://registry.terraform.io/providers/hashic If you don't yet have a Terraform project, check out the [Terraform introduction](https://www.terraform.io/intro/index.html). -## Option 6: deploy natively with Terraform +## Option 5: Deploy natively with Terraform -Please see the documentation [here](terraform/Readme.md). +The Terraform modules are located in the [terraform](terraform) directory. Deployment documentation is [here](terraform/Readme.md). -## How to execute the state machine once deployed? +## How to execute the state machine once deployed -See [here](README-EXECUTE.md). +See the [execution](README-EXECUTE.md) instructions to run the state machine. diff --git a/README-EXECUTE.md b/README-EXECUTE.md index 71eaa797..28e60cc9 100644 --- a/README-EXECUTE.md +++ b/README-EXECUTE.md @@ -1,6 +1,6 @@ # How to execute the state machine -Independently of how you've deployed the state machine, you can execute it in a few different ways. Programmatically, using the AWS CLI, AWS SDK, or Lumigo CLI. Manually, using the AWS Step Functions web console or the Lambda Power Tuner UI. +Independently of how you've deployed the state machine, you can execute it in a few different ways. Programmatically, using the AWS CLI, or AWS SDK. Manually, using the AWS Step Functions web console. ## Option 1: Execute the state machine programmatically (CLI) @@ -10,13 +10,7 @@ Feel free to customize the `scripts/sample-execution-input.json`, and then run ` The script will start a state machine execution, wait for the execution to complete (polling), and then show the execution results. -## Option 2: Execute the state machine programmatically (Lumigo CLI) - -The Lumigo CLI integration takes care of both deploying and executing the SAR app transparently. - -Check it out [here](README-DEPLOY.md#user-content-option-4-deploy-with-the-lumigo-cli). - -## Option 3: Execute the state machine manually (web console) +## Option 2: Execute the state machine manually (web console) Once the state machine is deployed, you can execute it and provide an input object. @@ -41,10 +35,4 @@ Click "**Start Execution**" again and the execution will start. In the next page Once the execution has completed, you will find the execution results in the "**Output**" tab of the "**Execution Details**" section at the top of the page. The output will contain the optimal power configuration and its corresponding average cost per execution. -## Option 4: Execute the state machine manually (AWS Lambda Power Tuner UI) - -You can deploy and interact with Lambda Power Tuning with an ad-hoc web interface. This UI will deploy everything you need to power-tune your functions and also simplify the input/output management for Step Functions via API Gateway. - -You can find the open-source project and the instructions to deploy it here: [mattymoomoo/aws-power-tuner-ui](https://github.com/mattymoomoo/aws-power-tuner-ui). -![Power Tuner UI](https://github.com/mattymoomoo/aws-power-tuner-ui/blob/master/imgs/website.png?raw=true) diff --git a/README-INPUT-OUTPUT.md b/README-INPUT-OUTPUT.md index e5d21157..6c4f9647 100644 --- a/README-INPUT-OUTPUT.md +++ b/README-INPUT-OUTPUT.md @@ -3,6 +3,28 @@ Each execution of the state machine will require an input and will provide the corresponding output. +## Deployment configuration options + +The CloudFormation template accepts the following parameters: + +* **PowerValues** (list of numbers): these power values will be used as the default in case no `powerValues` input parameter is provided at execution time +* **visualizationURL** (string): the base URL for the visualization tool, by default it's `lambda-power-tuning.show` but you could use your own custom tool +* **totalExecutionTimeout** (number in seconds, default=`300`): the timeout in seconds applied to all functions of the state machine +* **lambdaResource** (string, default=`*`): the `Resource` used in IAM policies; it's `*` by default but you could restrict it to a prefix or a specific function ARN +* **permissionsBoundary** (string): the ARN of a permissions boundary (policy), applied to all functions of the state machine +* **payloadS3Bucket** (string): the S3 bucket name used for large payloads (>256KB); if provided, it's added to a custom managed IAM policy that grants read-only permission to the S3 bucket; more details below in the [S3 payloads section](#user-content-s3-payloads) +* **payloadS3Key** (string, default=`*`): the S3 object key used for large payloads (>256KB); the default value grants access to all S3 objects in the bucket specified with `payloadS3Bucket`; more details below in the [S3 payloads section](#user-content-s3-payloads) +* **layerSdkName** (string): the name of the SDK layer, in case you need to customize it (optional) +* **logGroupRetentionInDays** (number, default=7): the number of days to retain log events in the Lambda log groups. Before this parameter existed, log events were retained indefinitely +* **securityGroupIds** (list of SecurityGroup IDs): List of Security Groups to use in every Lambda function's VPC Configuration (optional); please note that your VPC should be configured to allow public internet access (via NAT Gateway) or include VPC Endpoints to the Lambda service +* **subnetIds** (list of Subnet IDs): List of Subnets to use in every Lambda function's VPC Configuration (optional); please note that your VPC should be configured to allow public internet access (via NAT Gateway) or include VPC Endpoints to the Lambda service +* **stateMachineNamePrefix** (string, default=`powerTuningStateMachine`): Allows you to customize the name of the state machine. Maximum 43 characters, only alphanumeric (plus `-` and `_`). The last portion of the `AWS::StackId` will be appended to this value, so the full name will look like `powerTuningStateMachine-89549da0-a4f9-11ee-844d-12a2895ed91f`. Note: `StateMachineName` has a maximum of 80 characters and 36+1 from the `StackId` are appended, allowing 43 for a custom prefix. + + + +Please note that the total execution time should stay below 300 seconds (5 min), which is the default timeout. You can easily estimate the total execution timeout based on the average duration of your functions. For example, if your function's average execution time is 5 seconds and you haven't enabled `parallelInvocation`, you should set `totalExecutionTimeout` to at least `num * 5`: 50 seconds if `num=10`, 500 seconds if `num=100`, and so on. If you have enabled `parallelInvocation`, usually you don't need to tune the value of `totalExecutionTimeout` unless your average execution time is above 5 min. If you have a sleep between invocations set, you should include that in your timeout calculations. + + ## State machine input (at execution time) The state machine accepts the following input parameters: @@ -25,25 +47,6 @@ The state machine accepts the following input parameters: * **disablePayloadLogs** (boolean) If provided and set to a truthy value, suppresses `payload` from error messages and logs. If `preProcessorARN` is provided, this also suppresses the output payload of the pre-processor. * **includeOutputResults** (boolean) If provided and set to true, the average cost and average duration for every power value configuration will be included in the state machine output. -## State machine configuration (at deployment time) - -The CloudFormation template accepts the following parameters: - -* **PowerValues** (list of numbers): these power values will be used as the default in case no `powerValues` input parameter is provided at execution time -* **visualizationURL** (string): the base URL for the visualization tool, by default it's `lambda-power-tuning.show` but you could use your own custom tool -* **totalExecutionTimeout** (number in seconds, default=`300`): the timeout in seconds applied to all functions of the state machine -* **lambdaResource** (string, default=`*`): the `Resource` used in IAM policies; it's `*` by default but you could restrict it to a prefix or a specific function ARN -* **permissionsBoundary** (string): the ARN of a permissions boundary (policy), applied to all functions of the state machine -* **payloadS3Bucket** (string): the S3 bucket name used for large payloads (>256KB); if provided, it's added to a custom managed IAM policy that grants read-only permission to the S3 bucket; more details below in the [S3 payloads section](#user-content-s3-payloads) -* **payloadS3Key** (string, default=`*`): the S3 object key used for large payloads (>256KB); the default value grants access to all S3 objects in the bucket specified with `payloadS3Bucket`; more details below in the [S3 payloads section](#user-content-s3-payloads) -* **layerSdkName** (string): the name of the SDK layer, in case you need to customize it (optional) -* **logGroupRetentionInDays** (number, default=7): the number of days to retain log events in the Lambda log groups. Before this parameter existed, log events were retained indefinitely -* **securityGroupIds** (list of SecurityGroup IDs): List of Security Groups to use in every Lambda function's VPC Configuration (optional); please note that your VPC should be configured to allow public internet access (via NAT Gateway) or include VPC Endpoints to the Lambda service -* **subnetIds** (list of Subnet IDs): List of Subnets to use in every Lambda function's VPC Configuration (optional); please note that your VPC should be configured to allow public internet access (via NAT Gateway) or include VPC Endpoints to the Lambda service -* **stateMachineNamePrefix** (string, default=`powerTuningStateMachine`): Allows you to customize the name of the state machine. Maximum 43 characters, only alphanumeric (plus `-` and `_`). The last portion of the `AWS::StackId` will be appended to this value, so the full name will look like `powerTuningStateMachine-89549da0-a4f9-11ee-844d-12a2895ed91f`. Note: `StateMachineName` has a maximum of 80 characters and 36+1 from the `StackId` are appended, allowing 43 for a custom prefix. - - -Please note that the total execution time should stay below 300 seconds (5 min), which is the default timeout. You can easily estimate the total execution timeout based on the average duration of your functions. For example, if your function's average execution time is 5 seconds and you haven't enabled `parallelInvocation`, you should set `totalExecutionTimeout` to at least `num * 5`: 50 seconds if `num=10`, 500 seconds if `num=100`, and so on. If you have enabled `parallelInvocation`, usually you don't need to tune the value of `totalExecutionTimeout` unless your average execution time is above 5 min. If you have a sleep between invocations set, you should include that in your timeout calculations. ### Usage in CI/CD pipelines diff --git a/README.md b/README.md index d3ae025c..644c84d8 100644 --- a/README.md +++ b/README.md @@ -88,14 +88,6 @@ Website repository: [matteo-ronchetti/aws-lambda-power-tuning-ui](https://github Optionally, you could deploy your own custom visualization tool and configure the CloudFormation Parameter named `visualizationURL` with your own URL. -## Power Tuner UI - -Lambda Power Tuner UI is a web interface to simplify the deployment and execution of Lambda Power Tuning. It's built and maintained by [Matthew Dorrian](https://twitter.com/DorrianMatthew) and it aims at providing a consistent interface and a uniform developer experience across teams and projects. - -![Power Tuner UI](https://github.com/mattymoomoo/aws-power-tuner-ui/blob/master/imgs/website.png?raw=true) - -Power Tuner UI repository: [mattymoomoo/aws-power-tuner-ui](https://github.com/mattymoomoo/aws-power-tuner-ui) - ## Additional features, considerations, and internals [Here](README-ADVANCED.md) you can find out more about some advanced features of this project, its internals, and some considerations about security and execution cost. diff --git a/terraform/Readme.md b/terraform/Readme.md index 55edc254..c1081eea 100644 --- a/terraform/Readme.md +++ b/terraform/Readme.md @@ -3,42 +3,43 @@ ## Overview This deployment option is intended for those who may not be in a position to use AWS Cloudformation, in cases where you do not have access or when CloudFormation is not an approved service within your company. -## Before you start +The Terraform code is contained in the `terraform` directory of this project. All commands should be run from within this directory. -Modify the variables to target the correct AWS Account. +## Before you start +Modify the `variables.tf` file with your target AWS Account and region. ``` variable "account_id" { default = "123456789101" } +variable "aws_region" { + default = "eu-west-1" +} ``` -## Usage +## Deploy the solution ``` terraform init terraform plan terraform apply ``` +Once deployed, follow [these instructions](../README-EXECUTE.md) to run Lambda Power Tuning. + ## Deploy to multiple accounts/regions -Copy the module in `main.tf` and give it a new module name. For example: -``` -module "power_tuning" { - source = "./module" - account_id = var.account_id -} +If you're planning on deploying to multiple accounts or regions, it's recommended to adopt a folder strategy by either account or region. This will make sure you keep your statefile lightweight and plans/applies faster. -module "power_tuning_2" { - source = "./module" - account_id = var.account_id_2 -} +## Delete the solution +Run the below command to remove all resources from your account: +```bash +terraform destroy ``` - -If you're planning on deploying many, it's recommended to adopt a folder strategy by either account or region. This will make sure you keep your statefile lightweight and plans/applies faster. +Enter 'yes' at the confirmation prompt. ## Versions tested - 0.13.3 - 1.0.11 +- 1.7.3 This should provide good coverage between those versions. If there's any problems, please raise an issue.