AWS Thrifty Mod for Flowpipe

Pipelines to detect and correct misconfigurations leading to AWS savings opportunities.

Documentation

• Pipelines →

Getting Started

Requirements

Docker daemon must be installed and running. Please see Install Docker Engine for more information.

Installation

Download and install Flowpipe (https://flowpipe.io/downloads) and Steampipe (https://steampipe.io/downloads). Or use Brew:

brew install turbot/tap/flowpipe
brew install turbot/tap/steampipe

Install the AWS plugin with Steampipe:

steampipe plugin install aws

Steampipe will automatically use your default AWS credentials. Optionally, you can setup multiple accounts or customize AWS credentials.

Create a credential_import resource to import your Steampipe AWS connections:

vi ~/.flowpipe/config/aws.fpc

credential_import "aws" {
  source      = "~/.steampipe/config/aws.spc"
  connections = ["*"]
}

For more information on importing credentials, please see Credential Import.

For more information on credentials in Flowpipe, please see Managing Credentials.

Clone the mod:

mkdir aws-thrifty
cd aws-thrifty
git clone [email protected]:turbot/flowpipe-mod-aws-thrifty.git

Install the dependencies:

flowpipe mod install

Configure Variables

Several pipelines have input variables that can be configured to better match your environment and requirements.

Each variable has a default defined in its source file, e.g, s3/s3_buckets_without_lifecycle_policy.fp (or variables.fp for more generic variables), but these can be overwritten in several ways:

The easiest approach is to setup your vars file, starting with the sample:

cp thrifty.fpvars.example thrifty.fpvars
vi thrifty.fpvars

flowpipe pipeline run detect_and_correct_ebs_snapshots_exceeding_max_age --var-file=thrifty.fpvars

Alternatively, you can pass variables on the command line:

flowpipe pipeline run detect_and_correct_ebs_snapshots_exceeding_max_age --var=ebs_snapshot_age_max_days=10

Or through environment variables:

export FP_VAR_ebs_snapshot_age_max_days=10
flowpipe pipeline run detect_and_correct_ebs_snapshots_exceeding_max_age

For more information, please see Passing Input Variables

Running Detect and Correct Pipelines

To run your first detection, you'll need to ensure your Steampipe server is up and running:

steampipe service start

To find your desired detection, you can filter the pipeline list output:

flowpipe pipeline list | grep "detect_and_correct"

Then run your chosen pipeline:

flowpipe pipeline run detect_and_correct_ebs_snapshots_exceeding_max_age

By default the above approach would find the relevant resources and then send a message to your configured notifier.

However; you can request via an Input Step a corrective action to run against each detection result; this behavior is achieved by setting approvers either as a variable or for a one-off approach, by passing approvers as an argument.

Note: This approach requires running flowpipe server as it uses an input step.

flowpipe pipeline run detect_and_correct_ebs_snapshots_exceeding_max_age --host local --arg='approvers=["default"]'

If you're happy to just apply the same action against all detected items, you can apply them without the input step by overriding the default_action argument (or the detection specific variable).

flowpipe pipeline run detect_and_correct_ebs_snapshots_exceeding_max_age --arg='default_action="delete_snapshot"'

However; if you have configured a non-empty list for your approvers variable, you will need to override it as below:

flowpipe pipeline run detect_and_correct_ebs_snapshots_exceeding_max_age --arg='approvers=[]' --arg='default_action="delete_snapshot"'

Finally, each detection pipeline has a corresponding Query Trigger, these are disabled by default allowing for you to configure only those which are required, see the docs for more information.

Open Source & Contributing

This repository is published under the Apache 2.0 license. Please see our code of conduct. We look forward to collaborating with you!

Flowpipe and Steampipe are products produced from this open source software, exclusively by Turbot HQ, Inc. They are distributed under our commercial terms. Others are allowed to make their own distribution of the software, but cannot use any of the Turbot trademarks, cloud services, etc. You can learn more in our Open Source FAQ.

Get Involved

Join #flowpipe on Slack →

Want to help but don't know where to start? Pick up one of the help wanted issues:

• Flowpipe
• AWS Thrifty Mod