chef_vault_try_notify

Description

The chef_vault_try_notify cookbook provides a resource that can test if the chef-vault secrets your cookbook needs are available. If the vault items can be decrypted, the resource takes no action. If the secrets cannot be decrypted (perhaps because this is an autoscaled node and the vault has not yet been refreshed), the resource calls a ruby block which can take whatever action is appropriate to refresh the vault for the new node.

Use Case

When using chef-vault secrets in cookbooks that converge nodes created in an orchestration template or spawned from an autoscaling group, the secrets will not be encrypted for the new node. The secrets cannot be encrypted until the node has registered with your Chef server and the public key for the node is available.

The standard method of refreshing vault items for a new node is to run knife vault refresh, a command intended for use by an operator, not a program.

Members of the community have found ways around this problem, which tend to fall into three categories:

1. Bootstrap the node with a runlist that does not require chef-vault secrets, then have an operator refresh the vault items and change the run list

2. Have the cookbook test for the availability of vault secrets and either crash out or only converge resources that do not require secrets. At some point, a vault item refresh need to be performed in order to converge all resources successfully.

3. Automatically refresh vault items using some automated mechanism, either via cron or in response to a chef-client run failing. This requires that the automated mechanism have access to a private key for a Chef API user that can write to the chef-vault encrypted data bag.

The resource provided by this cookbook adapts the second approach and sets the groundwork for a more secure version of the third approach.

Usage

The LWRP supports two usage patterns

• Test / Notify / Retry / Fail
• Test / Remember / Predicate

General

To your cookbook's metadata.rb, add this line:

depends 'chef_vault_try_notify', '~> 0.1'

Then add this line to any recipe that requires chef-vault secrets:

include_recipe 'chef_vault_try_notify'

Test / Notify / Retry / Fail

In the first case, the resource tries to decrypt the given vault items. If the items cannot be decrypted, it calls the on_failure ruby block, then retries up to max_tries times, throwing an exception if the maximum attempts is exceeded.

Prior to using a chef-vault secret, declare a resource to test if the secrets are available:

chef_vault_try_notify 'web server secrets' do
  vault_items ['foo/bar', 'baz/wibble']
  on_failure do |state|
    extend ChefVaultTryNotify::Helper
    sns_notify(
      node, state,
      topic: 'arn:aws:sns:us-east-1:12345:MyTopic-ABCDE'
    )
  end
end

At compile time, this resource will attempt to decrypt the vault items bar in the foo vault and wibble in the baz vault. If they can be decrypted, it takes no further action. You can then safely retrieve your secrets:

vaultitem = chef_vault_item('foo', 'bar')

If either secret cannot be decrypted, the on_failure block is called with a state object describing the attempted decryption (number of tries so far, etc.). In the example above, the helper function sns_notify is used to send an AWS SNS notification to a topic (see below for details).

The resource will then retry the decryption, by default waiting for 10 seconds and retrying up to 30 times. Each time the decryption fails, the on_failure block will be called again with an updated state Struct.

If the secrets cannot be decrypted after a defined number of attempts, the resource will call Chef::Application.fatal! to end the Chef run.

The intent is that the notification will trigger some process to refresh the vault items, and that the subsequent attempts to decrypt will be successful.

Test / Remember / Guard

In the second case, which is enabled by setting the test_and_remember attribute of the resource to true, the resource attempts to decrypt the secrets, storing the result of the attempt in a state object. A predicate function vault_available? is provided that lets you decide whether to include a given resource or recipe based on the availability of vault items.

Prior to the point where you need the predicate, declare a resource to test if the secrets are available:

chef_vault_try_notify 'web server secrets' do
  vault_items ['foo/bar', 'baz/wibble']
  test_and_remember true
end

At some point thereafter, you can use the vault_available? predicate, either in a guard block:

file '/path/to/file' do
  content lazy { chef_vault_item('foo', 'bar')['password'] }
  only_if { vault_available?('web server secrets') }
end

Or as part of your recipe control flow:

if vault_available?('web server secrets')
  include_recipe "#{cookbook_name}::_secrets"
end

Recipes

default

The default recipe includes the chef-vault::default recipe, which installs the chef-vault gem.

It also includes the vault_available? helper into Chef::Recipe and Chef::Resource.

Attributes

There are no attributes defined by this cookbook.

Resources

chef_vault_try_notify

This resource tests if it can decrypt a set of vault items, and calls a ruby block if it cannot.

chef_vault_try_notify 'my_app' do
  vault_items ['one/two', 'three/four']
  max_tries 10
  wait_period 30
  on_failure { |state| ... }
end

The resource supports only one action: test_secrets

The resource has the following parameters:

• name: a descriptive name for this set of vault items. Defaults to the name of the resource
• vault_items: an array of strings, each describing a vault item to attempt to decrypt in the form "vaultname/itemname"
• max_tries: the maximum number of times to attempt decryption. Defaults to 30
• wait_period: how long (in seconds) to wait between attempts to decrypt. Defaults to 10
• on_failure: a ruby block that will be called if any of the vault items cannot be decrypted. It is passed a state object (described below)
• test_and_remember: enables a single-pass test-and-remember mode, in which decryption is attempted and the result remembered for use later in the vault_available? predicate. In this mode, the max_tries, wait_period and on_failure attributes are not used

Ruby Block Context

The on_failure ruby block is passed a state object that describes the attempt to decrypt the vault items. The available methods are:

• failed_vault_items: an array of vault items which could not be decrypted (structured the same as the vault_items attribute)
• tries: the number of times decryption has been attempted unsuccessfully
• max_tries: the maximum number of times decryption will be attempted before the resource fails
• wait_period: how long (in seconds) the resource waits between attempts to decrypt
• waiting_for: how long (in seconds) the resource has been waiting trying to decrypt the vault items

Helpers

This cookbook provides a helper for sending a notification to an AWS SNS topic when it cannot decrypt secrets.

These helpers are in the module ChefVaultTryNotify::Helper, which can be included into your recipe or extended into the ruby block (as in the example above).

sns_notify

sns_notify sends a notification to an AWS SNS topic. It takes as arguments the Chef node object, the state object passed to the on_failure ruby block, and a hash of options. The hash must have the following keys:

• topic: the ARN of the SNS topic

and can have the following keys:

• region: the AWS region to connect to
• credentials: the AWS credentials to use

If the region is not provided, it is constructed from EC2 Ohai data by taking the value of node['ec2']['placement_availability_zone'] and chopping off the last character.

If the credentials are not provided, then the standard AWS SDK for Ruby rules apply, which allow you to use IAM Instance Profiles. Describing these in detail is beyond the scope of this README, so please refer to the AWS documentation.

To use sns_notify, you should also depend on the aws cookbook and include the aws recipe somewhere in order to install the aws-sdk gem.

The notification send to the topic is a JSON document structured like this:

{
  "type": "chef_vault_try_notify_failure",
  "tries": 1,
  "fqdn": "foobar.example.com",
  "instance_id": "i-DEADBEEF",
  "ipaddress": "10.10.10.10",
  "ip6address": "fe80::4de:e7ff:fed4:1920",
  "macaddress": "06:DE:E7:D4:19:20",
  "chef_environment": "application1-prod",
  "max_tries": 30,
  "wait_period": 10,
  "waiting_for": 30,
  "failed_vault_items": ['foo/bar', 'baz/wibble']
}

This is (hopefully) enough information for whatever receives the notification to authenticate the request and decide whether it should refresh the vault or not.

Future Enhancements

PRs to provide helpers for other notification services, especially those of other cloud providers are welcome.

Author

James FitzGibbon - [email protected] - Nordstrom, Inc.

License

Copyright (c) 2015 Nordstrom, Inc.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Template Version

Originally created from Nordstrom Cookbook Template v0.16.2