E2E test case fail on IBMZ

[modassarrana89]

Problem -- E2E test case fail on IBMZ. It used to work 2 - 3 months back with Quay master branch.
Error -

================================================================================================================== short test summary info ===================================================================================================================
FAILED endpoints/v2/test/test_blob.py::TestBlobPullThroughProxy::test_check_blob_exists_from_dockerhub - data.model.ManifestDoesNotExist: the requested image may not exist in the upstream registry, or the configured Quay organization credentials have ...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

From this error , it look like , its. not able pull the manifest ( as it might not exists )

How am i testing E2E -- I have checked out quay master branch on my IBMZ lpar & installed the dependency by following CI workflow. run tox command as suggested in CI workflow

tox -e py39-e2e

[flavianmissi]

Hi @modassarrana89, I just tested and was able to pull the image used in that test.
What happens if you try to pull it on the same machine where the tests are running?

podman pull docker.io/library/postgres@sha256:b2e69d778c6a0f0a18eb7de0ac7331414b7e77a6e049ced5b2d25f87fe0b1006

[flavianmissi]

I don't think you need to worry about authentication in the e2e test, the code will use an anonymous token if no user/pass is given, and that should work fine because quay.io rate limits are not at all as strict as docker hub's.

[modassarrana89]

Thanks for immediate response. How can i push the same protgress manifest to quay or can i test with any postgres image on quay . To push the image to Quay , i have to login to Quay repo.

[flavianmissi]

Perhaps you can use an existing image? such as quay.io/openshifttest/busybox:latest? You would then have to change the parameters of the test, such as tag, digest, and repository/image name.

[modassarrana89]

Thanks Flavia,
By changing org , image , repo , digest & register in test file as you have suggested , some of the e2e test case do passed as expected. For other changes , I have to analyse the failure & change accordingly

This confirms few things :

1. e2e test cases were failing on lpar due to rate limit
2. we should better move to quay.io inplace of dockerhub for e2e test casess

[root@m13lp76 quay]# tox -e py39-e2e
py39-e2e installed: alembic==1.3.3,aniso8601 @ git+https://github.com/DevTable/aniso8601-fake.git@bd7762c7dea0498706d3f57db60cd8a8af44ba90,APScheduler==3.6.3,asttokens==2.2.1,attrs==19.3.0,Authlib==1.0.0a1,aws-sam-translator==1.20.1,azure-core==1.23.1,azure-storage-blob==12.4.0,Babel==2.9.1,backcall==0.2.0,bcrypt==3.1.7,beautifulsoup4==4.11.1,bencode.py==2.1.0,bintrees==2.1.0,bitmath==1.3.3.1,black==21.12b0,blinker==1.4,boto3==1.21.42,boto3-stubs==1.14.9.0,botocore==1.24.42,cachetools==4.0.0,certifi==2022.12.7,cffi==1.14.3,cfgv==3.3.1,chardet==3.0.4,charset-normalizer==2.0.12,click==7.1.2,cnr-server @ git+https://github.com/quay/appr.git@58c88e4952e95935c0dd72d4a24b0c44f2249f5b,coverage==7.0.5,cryptography==3.3.2,Cython==3.0.0a9,DateTime==4.3,debtcollector==1.22.0,decorator==4.4.1,Deprecated==1.2.7,distlib==0.3.6,docker==5.0.3,dumb-init==1.2.2,elasticsearch==7.6.0,elasticsearch-dsl==7.0.0,exceptiongroup==1.1.0,execnet==1.9.0,executing==1.2.0,fakeredis==1.6.1,filelock==3.9.0,flake8==4.0.1,Flask==1.1.1,Flask-Cors==3.0.9,Flask-Login==0.4.1,Flask-Mail==0.9.1,Flask-Principal==0.4.0,Flask-RESTful==0.3.9,Flask-Testing @ git+https://github.com/jarus/flask-testing.git@17f19d7fee0e1e176703fc7cb04917a77913ba1a,flit_core==3.7.1,freezegun==0.3.12,funcparserlib==1.0.0a0,furl==2.1.0,futures==3.1.1,geoip2==3.0.0,gevent==21.8.0,greenlet==1.1.2,grpcio==1.46.3,gunicorn==20.1.0,hashids==1.2.0,html5lib==1.1,httmock==1.3.0,identify==2.5.13,idna==2.8,importlib-metadata==1.4.0,iniconfig==2.0.0,ipdb==0.13.11,ipython==8.8.0,iso8601==0.1.12,isodate==0.6.0,itsdangerous==1.1.0,jedi==0.18.2,Jinja2==2.11.3,jmespath==0.9.4,jsonpath-rw==1.4.0,jsonpickle==1.2,jsonpointer==2.0,jsonschema==3.2.0,kafka-python==1.4.7,keystoneauth1==3.18.0,Mako==1.2.2,MarkupSafe==1.1.1,matplotlib-inline==0.1.6,maxminddb==1.5.2,mccabe==0.6.1,mixpanel==4.5.0,mock==3.0.5,mockldap @ git+https://github.com/quay/mockldap.git@136253ca136fc7898944a4b37893a99440f7335e,more-itertools==9.0.0,moto==2.0.1,msgpack==0.6.2,msrest==0.6.21,mypy==0.950,mypy-boto3==1.14.9.0,mypy-extensions==0.4.3,ndg-httpsclient==0.5.1,netaddr==0.7.19,netifaces==0.10.9,nodeenv==1.7.0,oauthlib==3.2.1,orderedmultidict==1.0.1,os-service-types==1.7.0,oslo.config==7.0.0,oslo.i18n==3.25.1,oslo.serialization==2.29.2,oslo.utils==4.12.2,packaging==21.3,parameterized==0.8.1,parso==0.8.3,pathspec==0.10.3,pbr==5.10.0,peewee==3.13.1,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.0.1,platformdirs==2.6.2,pluggy==1.0.0,ply==3.11,pre-commit==2.20.0,prometheus-client==0.7.1,prompt-toolkit==3.0.36,protobuf==3.18.3,psutil==5.9.0,psycopg2-binary==2.9.3,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,py-bitbucket @ git+https://github.com/quay/py-bitbucket.git@3ba167408d92eb4ba2cae78c0047b8f1e8821c67,pyasn1==0.4.8,pyasn1-modules==0.2.8,pycodestyle==2.8.0,pycparser==2.20,pyflakes==2.4.0,PyGithub==1.45,Pygments==2.14.0,PyJWT==2.4.0,pymemcache==3.0.0,PyMySQL==0.9.3,pyOpenSSL==19.1.0,pyparsing==2.4.6,PyPDF2 @ https://files.pythonhosted.org/packages/3d/48/0966606e08dd93a77e839803789151ff81ac27ec6767957af8afb9588501/PyPDF2-1.27.6.tar.gz,pyrsistent==0.18.0,pytest==7.2.1,pytest-cov==4.0.0,pytest-flask==1.2.0,pytest-grpc==0.8.0,pytest-runner==6.0.0,pytest-sugar==0.9.6,pytest-timeout==2.1.0,pytest-xdist==3.1.0,python-coveralls==2.9.3,python-dateutil==2.8.1,python-editor==1.0.4,python-etcd @ git+https://github.com/DevTable/python-etcd.git@f1168cb02a2a8c83bec1108c6fcd8615ef463b14,python-gitlab==2.0.0,python-keystoneclient==3.22.0,python-ldap==3.4.0,python-magic==0.4.15,python-redis-lock==3.7.0,python-swiftclient==3.8.1,pytz==2019.3,PyYAML==5.4,raven==6.10.0,recaptcha2==0.1,redis==3.5.3,redis-py-cluster==2.1.3,rehash==1.0.0,reportlab==3.5.55,requests==2.27.1,requests-aws4auth==0.9,requests-file==1.4.3,requests-oauthlib==1.3.0,responses==0.22.0,rfc3986==1.3.2,s3transfer==0.5.1,semantic-version==2.8.4,setuptools-scm==4.1.2,six==1.14.0,sortedcontainers==2.4.0,soupsieve==1.9.5,SQLAlchemy==1.4.31,sqlalchemy-stubs==0.3,stack-data==0.6.2,stevedore==1.31.0,stringscore==0.1.0,stripe==2.42.0,supervisor==4.1.0,supervisor-logging==0.0.9,supervisor-stdout @ git+https://github.com/quay/supervisor-stdout.git@9bdf630d0d1b4a16cf8b60b96adafb1ed98e7380,termcolor==1.1.0,text-unidecode==1.3,tldextract==2.2.2,toml==0.10.2,tomli==1.2.3,toposort==1.5,tox==3.25.1,tox-docker==3.1.0,tqdm==4.64.1,traitlets==5.8.1,types-boto==2.49.2,types-cachetools==4.2.10,types-cryptography==3.3.7,types-Deprecated==1.2.3,types-docutils==0.17.1,types-enum34==1.1.1,types-freezegun==1.1.2,types-futures==3.3.1,types-geoip2==3.0.0,types-ipaddress==1.0.1,types-maxminddb==1.5.0,types-mock==4.0.3,types-protobuf==3.18.0,types-pyOpenSSL==20.0.9,types-python-dateutil==2.8.2,types-pytz==2021.3.0,types-PyYAML==5.4.12,types-redis==3.5.14,types-requests==2.25.11,types-setuptools==57.4.2,types-six==1.16.2,types-termcolor==1.1.2,types-toml==0.10.8.1,types-tzlocal==3.0.0,typing==3.7.4.3,typing_extensions==4.0.1,tzlocal==2.0.0,urllib3==1.26.9,virtualenv==20.17.1,wcwidth==0.2.6,webencodings==0.5.1,WebOb==1.8.6,websocket-client==0.57.0,Werkzeug==1.0.0,wrapt==1.13.3,xhtml2pdf==0.2.4,xmltodict==0.13.0,yapf==0.29.0,zipp==2.1.0,zope.event==4.5.0,zope.interface==5.4.0
py39-e2e run-test-pre: PYTHONHASHSEED='3721610251'
py39-e2e run-test: commands[0] | python --version
Python 3.9.7
py39-e2e run-test: commands[1] | alembic upgrade head
py39-e2e run-test: commands[2] | py.test --timeout=3600 -m e2e --cov-report=html --cov-report=term-missing -x --color no --verbose --ignore=buildman/ '' -vv
Test session starts (platform: linux, Python 3.9.7, pytest 7.2.1, pytest-sugar 0.9.6)
cachedir: .tox/py39-e2e/.pytest_cache
rootdir: /root/quay, configfile: tox.ini
plugins: xdist-3.1.0, timeout-2.1.0, sugar-0.9.6, grpc-0.8.0, flask-1.2.0, cov-4.0.0
timeout: 3600.0s
timeout method: signal
timeout func_only: False
collecting ... 
 endpoints/v2/test/test_blob.py::TestBlobPullThroughProxy.test_check_blob_exists_from_dockerhub ✓                                                                                                                                                6% ▋         
 endpoints/v2/test/test_blob.py::TestBlobPullThroughProxy.test_check_blob_exists_from_dockerhub_404 ✓                                                                                                                                           12% █▍        
 endpoints/v2/test/test_blob.py::TestBlobPullThroughProxy.test_pull_from_dockerhub ✓                                                                                                                                                            19% █▉        
 endpoints/v2/test/test_blob.py::TestBlobPullThroughProxy.test_pull_from_dockerhub_404 ✓                                                                                                                                                        25% ██▌       

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― TestManifestPullThroughProxyDockerHub.test_check_manifest_exists_from_dockerhub_by_tag ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

self = <test_manifest_pullthru.TestManifestPullThroughProxyDockerHub testMethod=test_check_manifest_exists_from_dockerhub_by_tag>

    def test_check_manifest_exists_from_dockerhub_by_tag(self):
        params = {
            "repository": self.repository,
            "manifest_ref": self.tag,
        }
        headers = self._get_auth_headers(self.repository)
        headers["Accept"] = ", ".join(
            DOCKER_SCHEMA2_CONTENT_TYPES.union(OCI_CONTENT_TYPES).union(
                DOCKER_SCHEMA1_CONTENT_TYPES
            )
        )
>       conduct_call(
            self.client,
            "v2.fetch_manifest_by_tagname",
            url_for,
            "HEAD",
            params,
            expected_code=200,
            headers=headers,
        )

endpoints/v2/test/test_manifest_pullthru.py:1035: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

client = <FlaskClient <Flask 'test.fixtures'>>, resource = 'v2.fetch_manifest_by_tagname', url_for = <function url_for at 0x3ff9390f0d0>, method = 'HEAD'
params = {'_csrf_token': 'sometoken', 'manifest_ref': 'multiarch', 'repository': 'openshifttest/busybox'}, body = None, expected_code = 200
headers = {'Accept': 'application/vnd.docker.distribution.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+pret...zih31cc73LtkHtn7BVRlEIDzSy5ko3baUTF3I2rt7Vv2fbBasYUPWZjD9z9_OxK1yr57AseYQd0aU0KoA', 'Content-Type': 'application/json'}
raw_body = None

    def conduct_call(
        client,
        resource,
        url_for,
        method,
        params,
        body=None,
        expected_code=200,
        headers=None,
        raw_body=None,
    ):
        """
        Conducts a call to a Flask endpoint.
        """
        params = add_csrf_param(client, params)
    
        final_url = url_for(resource, **params)
    
        headers = headers or {}
        headers.update({"Content-Type": "application/json"})
    
        if body is not None:
            body = json.dumps(body)
    
        if raw_body is not None:
            body = raw_body
    
        # Required for anonymous calls to not exception.
        g.identity = Identity(None, "none")
    
        rv = client.open(final_url, method=method, data=body, headers=headers)
        msg = "%s %s: got %s expected: %s | %s" % (
            method,
            final_url,
            rv.status_code,
            expected_code,
            rv.data,
        )
>       assert rv.status_code == expected_code, msg
E       AssertionError: HEAD /v2/openshifttest/busybox/manifests/multiarch?_csrf_token=sometoken: got 404 expected: 200 | b''

endpoints/test/shared.py:107: AssertionError

 endpoints/v2/test/test_manifest_pullthru.py::TestManifestPullThroughProxyDockerHub.test_check_manifest_exists_from_dockerhub_by_tag ⨯                                                                                                          31% ███▎      
====================================================================================================================== warnings summary ======================================================================================================================
.tox/py39-e2e/lib/python3.9/site-packages/peewee.py:236
.tox/py39-e2e/lib/python3.9/site-packages/peewee.py:236
.tox/py39-e2e/lib/python3.9/site-packages/peewee.py:236
.tox/py39-e2e/lib/python3.9/site-packages/peewee.py:236
.tox/py39-e2e/lib/python3.9/site-packages/peewee.py:236
.tox/py39-e2e/lib/python3.9/site-packages/peewee.py:236
.tox/py39-e2e/lib/python3.9/site-packages/peewee.py:236
  /root/quay/.tox/py39-e2e/lib/python3.9/site-packages/peewee.py:236: DeprecationWarning: "db_column" has been deprecated in favor of "column_name" for Field objects.
    warnings.warn(s, DeprecationWarning)

.tox/py39-e2e/lib/python3.9/site-packages/swiftclient/client.py:79
.tox/py39-e2e/lib/python3.9/site-packages/swiftclient/client.py:79
  /root/quay/.tox/py39-e2e/lib/python3.9/site-packages/swiftclient/client.py:79: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    if StrictVersion(requests.__version__) < StrictVersion('2.0.0') \

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================================================== short test summary info ===================================================================================================================
FAILED endpoints/v2/test/test_manifest_pullthru.py::TestManifestPullThroughProxyDockerHub::test_check_manifest_exists_from_dockerhub_by_tag - AssertionError: HEAD /v2/openshifttest/busybox/manifests/multiarch?_csrf_token=sometoken: got 404 expected: 2...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Results (22.38s):
       4 passed
       1 failed
         - endpoints/v2/test/test_manifest_pullthru.py:1024 TestManifestPullThroughProxyDockerHub.test_check_manifest_exists_from_dockerhub_by_tag
    4803 deselected
ERROR: InvocationError for command /root/quay/.tox/py39-e2e/bin/py.test --timeout=3600 -m e2e --cov-report=html --cov-report=term-missing -x --color no --verbose --ignore=buildman/ '' -vv (exited with code 1)
__________________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________________
ERROR:   py39-e2e: commands failed

[flavianmissi]

Hey @modassarrana89, can you open a PR with your changes? We can debug the test failures together more easily that way.
Also, my name is Flavian, not Flavia. Similar, but different 😅