RFC: Support testing of Habitat plans

[ksexton]

CC @eeyun @Hoverbear

Currently Habitat does not support testing that the package created matches what the package author intended. As a package author, I would like to have the ability to ensure that plan updates do not break things that I have specified in my tests.

Some possible ways this could be accomplished:

• Extend studio to support running tests with a do_test() block
• Extend hab with something like hab pkg test or hab plan test

This could also be useful in a build pipeline. A package could build successfully but not pass the test and be kept from creating the artifact.

[Hoverbear]

@ksexton and I noted this would be useful for testing things such as:

• Files which should exist do exist.
• Files contain expected contents.
• Files are appropriately changed when cfg variables are changed (via environment or otherwise).

We noted that do_test() would allow for the user to customize how they preferred to test their application, such as via test and stat or by something more such as http://inspec.io/.

This could additionally introduce something like "Test Dependencies" alongside "Build Dependencies" to facilitate the depending on said packages if necessary. These "Test Dependencies" could would not be required during the build or execution steps.

[Hoverbear]

@smith commented in the Habi-chat:

Inspec has a toml parser in its gemfile but it doesn't ship with it. You could totally have your specs inspect and modify the hab ring and do stuff like (psevdocode):

describe "my-service" do
  with_config do |cfg|
    cfg.port = 1000
  end

  it "has the port open" do
    expect(cfg.port).to be_listening
  end
end

[mickfeech]

I like the idea of a do_test() implementation

[themightychris]

Another thing that would be nice to either test by default or have a built-in helper method available to invoke from tests is running ldd over all binaries in the build output, checking that no shared library links are broken, and checking that all linked libraries are listed in pkg_deps

This was discussed more heavily in #1470. One comment from that thread was that some packages, like core/jre8, have some cruft binaries in their output you won't care about leaving broken. I suggest that in that case a plan author should either have their do_install implementation delete the binaries they don't intend to support before the test runs, or have to override do_links_test() { return 0 } to skip that test

This would make the habitat build process easier to learn since you could get clear errors about common mistakes rather than "not found" errors on a binary that's clearly there down the line, and it would increase the overall quality and reliability of packages

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.