Skip to content

Latest commit

 

History

History
154 lines (118 loc) · 6.72 KB

instructions.md

File metadata and controls

154 lines (118 loc) · 6.72 KB
Raw

How to submit conformance results

Running the tests

Each spec will provide its own testing instructions, and each will produce the following files that contain the test results and output:

  • report.html
  • junit.xml

OCI Distribution Specification

Please see instructions here.

Uploading

Prepare a PR to https://github.com/opencontainers/oci-conformance. Here are directions to prepare a pull request from a fork. In the descriptions below, $spec.X.Y refers to the spec and its major and minor version, and $dir is a short subdirectory name to hold the results for your product. Examples would be gcr or dockerhub.

Description: Conformance results for $spec/vX.Y/$dir

Contents of the PR

For simplicity you can submit the tarball or extract the relevant information from the tarball to compose your submission.

$spec/vX.Y/$dir/README.md: Description of how to reproduce your results.
$spec/vX.Y/$dir/report.html: Human-readable HTML test report.
$spec/vX.Y/$dir/junit.xml: Machine-readable JUnit test report.
$spec/vX.Y/$dir/PRODUCT.yaml: See below.

Entirely optional, but encouraged, you can also include the following files (if you have not already submitted them previously):

$spec/live/$dir/badges.md: See below.

PRODUCT.yaml

This file describes your product. It is YAML formatted with the following root-level fields. Please fill in as appropriate.

Field Description
vendor Name of the legal entity that is certifying. This entity must have a signed participation form on file with the OCI
name Name of the product being certified.
version The version of the product being certified (not the version of OCI spec).
website_url URL to the product information website
repo_url If your product is open source, this field is necessary to point to the primary GitHub repo containing the source. It's OK if this is a mirror. OPTIONAL
documentation_url URL to the product documentation
product_logo_url URL to the product's logo, (must be in SVG, AI or EPS format -- not a PNG -- and include the product name). OPTIONAL. If not supplied, we'll use your company logo. Please see logo guidelines (TODO: link)
type Is your product a distribution, hosted platform, or installer (see definitions)
description One sentence description of your offering

Examples below are for a fictional OCI implementation called Turbo Encabulator produced by a company named Yoyodyne.

vendor: Yoyodyne
name: Turbo Encabulator
version: v1.7.4
website_url: https://yoyo.dyne/turbo-encabulator
repo_url: https://github.com/yoyo.dyne/turbo-encabulator
documentation_url: https://yoyo.dyne/turbo-encabulator/docs
product_logo_url: https://yoyo.dyne/assets/turbo-encabulator.svg
type: distribution
description: 'The Yoyodyne Turbo Encabulator is a superb OCI distribution for all of your Encabulating needs.'

badges.md (Optional)

If you are running live tests (for example, in GitHub Actions), you are encouraged to include a badges.md files which contains Markdown badges pointing to your test results.

These badges will be displayed on a web-based dashboard showing live conformance results for various products.

Since you are likely testing conformance on the latest changeset to your product, this file should be submitted to the directory $spec/live/$dir (vs. $spec/vX.Y/$dir). If you have previously submitted this file, you do not need to do so again to submit conformance for a new spec version.

This file should simply contain Markdown badges, each on a new line.

Here is an example badges.md file showing 4 badges:

[![](https://github.com/myorg/myproduct/workflows/oci-pull/badge.svg)](https://github.com/myorg/myproduct/actions?query=workflow%3Aoci-pull)
[![](https://github.com/myorg/myproduct/workflows/oci-push/badge.svg)](https://github.com/myorg/myproduct/actions?query=workflow%3Aoci-push)
[![](https://github.com/myorg/myproduct/workflows/oci-content-discovery/badge.svg)](https://github.com/myorg/myproduct/actions?query=workflow%3Aoci-content-discovery)
[![](https://github.com/myorg/myproduct/workflows/oci-content-management/badge.svg)](https://github.com/myorg/myproduct/actions?query=workflow%3Aoci-content-management)

Amendment for Private Review

If you need a private review for an unreleased product, please email a zip file containing what you would otherwise submit as a pull request to [email protected]. We'll review and confirm that you are ready to be OCI Certified as soon as you open the pull request. We can then often arrange to accept your pull request soon after you make it, at which point you become OCI Certified.

Review

A reviewer will shortly comment on and/or accept your pull request, following this process. If you don't see a response within 3 business days, please contact [email protected].

Example Script

Combining the steps provided here, the process looks like this (Example: distribution-spec/v1.0):

spec_name=distribution-spec
spec_version=v1.0
prod_name=example

rm -rf tmp && git clone https://github.com/opencontainers/${spec_name}.git tmp
(cd tmp && docker build -t conformance:latest -f Dockerfile.conformance .)
rm -rf tmp results
docker run --rm \
  -v $(pwd)/results:/results \
  -w /results \
  -e OCI_ROOT_URL="https://r.myreg.io" \
  -e OCI_NAMESPACE="myorg/myrepo" \
  -e OCI_USERNAME="myuser" \
  -e OCI_PASSWORD="mypass" \
  -e OCI_DEBUG="true" \
  conformance:latest

mkdir -p ./${spec_name}/${spec_version}/${prod_name}
cp ./results/* ./${spec_name}/${spec_version}/${prod_name}/
rm -rf results

cat << EOF > ./${spec_name}/${spec_version}/${prod_name}/PRODUCT.yaml
vendor: Yoyodyne
name: Turbo Encabulator
version: v1.7.4
website_url: https://yoyo.dyne/turbo-encabulator
repo_url: https://github.com/yoyo.dyne/turbo-encabulator
documentation_url: https://yoyo.dyne/turbo-encabulator/docs
product_logo_url: https://yoyo.dyne/assets/turbo-encabulator.svg
type: distribution
description: 'The Yoyodyne Turbo Encabulator is a superb OCI distribution for all of your Encabulating needs.'
EOF

Issues

If you have problems certifying that you feel are an issue with the conformance program itself (and not just your own implementation), you can file an issue in the repository.