test_pageserver_small_inmemory_layers is unstable

[jcsp]

Something went bad on the 11th/12th of December:

It fails waiting for connections to postgres, apparently in getaddrinfo:

/github/home/.cache/pypoetry/virtualenvs/non-package-mode-_pxWMzVK-py3.11/lib/python3.11/site-packages/asyncpg/connection.py:2329: in connect
    return await connect_utils._connect(
/github/home/.cache/pypoetry/virtualenvs/non-package-mode-_pxWMzVK-py3.11/lib/python3.11/site-packages/asyncpg/connect_utils.py:991: in _connect
    conn = await _connect_addr(
/github/home/.cache/pypoetry/virtualenvs/non-package-mode-_pxWMzVK-py3.11/lib/python3.11/site-packages/asyncpg/connect_utils.py:828: in _connect_addr
    return await __connect_addr(params, True, *args)
/github/home/.cache/pypoetry/virtualenvs/non-package-mode-_pxWMzVK-py3.11/lib/python3.11/site-packages/asyncpg/connect_utils.py:873: in __connect_addr
    tr, pr = await connector
/github/home/.cache/pypoetry/virtualenvs/non-package-mode-_pxWMzVK-py3.11/lib/python3.11/site-packages/asyncpg/connect_utils.py:744: in _create_ssl_connection
    tr, pr = await loop.create_connection(
/home/nonroot/.pyenv/versions/3.11.10/lib/python3.11/asyncio/base_events.py:1046: in create_connection
    infos = await self._ensure_resolved(
/home/nonroot/.pyenv/versions/3.11.10/lib/python3.11/asyncio/base_events.py:1420: in _ensure_resolved
    return await loop.getaddrinfo(host, port, family=family, type=type,
/home/nonroot/.pyenv/versions/3.11.10/lib/python3.11/asyncio/base_events.py:868: in getaddrinfo
    return await self.run_in_executor(
E   asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:
test_runner/regress/test_pageserver_layer_rolling.py:130: in test_pageserver_small_inmemory_layers
    last_flush_lsns = asyncio.run(workload(env, tenant_conf, TIMELINE_COUNT, ENTRIES_PER_TIMELINE))
/home/nonroot/.pyenv/versions/3.11.10/lib/python3.11/asyncio/runners.py:190: in run
    return runner.run(main)
/home/nonroot/.pyenv/versions/3.11.10/lib/python3.11/asyncio/runners.py:118: in run
    return self._loop.run_until_complete(task)
/home/nonroot/.pyenv/versions/3.11.10/lib/python3.11/asyncio/base_events.py:654: in run_until_complete
    return future.result()
test_runner/regress/test_pageserver_layer_rolling.py:54: in workload
    return await asyncio.gather(*workers)
test_runner/regress/test_pageserver_layer_rolling.py:46: in run_worker
    last_flush_lsn = await run_worker_for_tenant(env, entries, tenant)
test_runner/regress/test_pageserver_layer_rolling.py:31: in run_worker_for_tenant
    conn = await ep.connect_async()
test_runner/fixtures/neon_fixtures.py:276: in connect_async
    return await asyncpg.connect(**conn_options)
/github/home/.cache/pypoetry/virtualenvs/non-package-mode-_pxWMzVK-py3.11/lib/python3.11/site-packages/asyncpg/connection.py:2328: in connect
    async with compat.timeout(timeout):
/home/nonroot/.pyenv/versions/3.11.10/lib/python3.11/asyncio/timeouts.py:115: in __aexit__
    raise TimeoutError from exc_val
E   TimeoutError

However, this could be async executor starvation, if it just can't execute the task for getaddrinfo because something else is blocking the executor.

#9994 points out historical timeouts with the same python backtrace, but this became much more frequent 5 days ago

[jcsp]

Related to LFC-by-default changes?

[jcsp]

We will monitor this after addressing #9994 to see if it's still unstable