WIP: fix chromium screenshots issues with CDP

[lucywang000]

Please see #953 for the background.

• Combine asyncio event loop with twisted reactor
• implement CDP clients with websocket (which is based on asyncio) and python-chrome-devtools-protocol
• use CDP to set proper viewport for chromium engine screenshots

[lucywang000]

The new event loop is:

1. twisted reactor (asyncioreactor) layered on top of asyncio event loop (thus the qt5reactor module is no longer used)
2. The asyncio event loop is itself implemented on top of qt event loop (with asyncqt)

The new event loop works pretty well in my manual testing. However, one constant issue I'm facing is the tests in test_render_chromium.py always failed after the first test cases is passed.

Tracebacks of failed test cases

splash/tests/test_render.py:189: in test_iframes
    r = self.request({"url": self.mockurl("iframes"), "timeout": "3"})
splash/tests/test_render.py:79: in request
    return self._get_handler().request(query, endpoint, headers, **kwargs)
splash/tests/test_render.py:44: in request
    return requests.get(url, params=query, headers=headers)
/usr/local/lib/python3.6/dist-packages/requests/api.py:75: in get
    return request('get', url, params=params, **kwargs)
/usr/local/lib/python3.6/dist-packages/requests/api.py:60: in request
    return session.request(method=method, url=url, **kwargs)
/usr/local/lib/python3.6/dist-packages/requests/sessions.py:533: in request
    resp = self.send(prep, **send_kwargs)
/usr/local/lib/python3.6/dist-packages/requests/sessions.py:646: in send
    r = adapter.send(request, **kwargs)
/usr/local/lib/python3.6/dist-packages/requests/adapters.py:498: in send
    raise ConnectionError(err, request=request)
E   requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

It looks like that the spawned splash server process in pytest fixture got killed/exited somehow after the first test case is running, which does not make any sense.

I have debugged it for quite some hours, but didn't find the cause yet.

[kmike]

Hey @lucywang000! I can see these lines in logs on Travis:

2019-10-16 08:10:45+0000 [-] Log opened.
2019-10-16 08:10:45.264418 [-] Xvfb is started: ['Xvfb', ':1687858321', '-screen', '0', '1024x768x24', '-nolisten', 'tcp']
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-splash'
2019-10-16 08:10:45.310001 [-] Splash version: 3.3.1
[1456:1485:1016/081045.352910:ERROR:socket_posix.cc(144)] bind() failed: Address already in use (98)
[1456:1485:1016/081045.352970:ERROR:devtools_http_handler.cc(299)] Cannot start http server for devtools.
2019-10-16 08:10:45.359174 [-] Qt 5.13.0, PyQt 5.13.0, WebKit 602.1, Chromium 73.0.3683.105, sip 4.19.18, Twisted 19.7.0, Lua 5.2
2019-10-16 08:10:45.359533 [-] Python 3.6.8 (default, Oct  7 2019, 12:59:55) [GCC 8.3.0]
2019-10-16 08:10:45.359731 [-] Open files limit: 524288
2019-10-16 08:10:45.359848 [-] Open files limit increased from 524288 to 1048576
2019-10-16 08:10:45.367740 [-] memory cache: enabled, private mode: enabled, js cross-domain access: disabled
2019-10-16 08:10:45.489347 [-] verbosity=3, slots=5, argument_cache_max_entries=500, max-timeout=90.0
2019-10-16 08:10:45.489580 [-] Web UI: enabled, Lua: enabled (sandbox: enabled), Webkit: disabled, Chromium: enabled
2019-10-16 08:10:45.490140 [-] Site starting on 58047
2019-10-16 08:10:45.490264 [-] Starting factory <twisted.web.server.Site object at 0x7f2af48f7c18>
2019-10-16 08:10:45.490682 [-] Server listening on http://0.0.0.0:58047

These two lines are concerning:

[1456:1485:1016/081045.352910:ERROR:socket_posix.cc(144)] bind() failed: Address already in use (98)
[1456:1485:1016/081045.352970:ERROR:devtools_http_handler.cc(299)] Cannot start http server for devtools.

It seems more than 1 Splash is running (which may happen in tests - though not sure why exactly; either way, we should be supporting it, as locally running Splash tests with -n8 is nice), and they share the same port for devtools, which causes problems.

[kmike]

Do you think there is a way to support starting several Splash instances? For ports which Splash controls, in tests we use ports selected dynamically.

[kmike]

Can it be a matter of setting QTWEBENGINE_REMOTE_DEBUGGING "127.0.0.1:9050" with a different port in a pytest fixture, when starting Splash?

[kmike]

You may be able to add devtools_portnum option to SplashServer, and implement it via setting an env variable here:

splash/splash/tests/utils.py

Line 83 in a1f4488

self.proc = Popen(args, env=get_testenv())

[lucywang000]

@kmike it turned out there is a flag need to be set:

splash/splash/qtutils.py

Lines 142 to 144 in 12fff66

	# we have to set this, because otherwise the program may quit when
	# the last web view window is closed.
	_qtapp.setQuitOnLastWindowClosed(False)

This also somehow matches the strange thing I saw earlier: the tests in test_render_chromium.py always failed after the first test cases is passed. Not sure why it's not there when qt5reactor is used.

Now the tests can all pass on my local machine.

[kmike]

Great catch @lucywang000!
Port problem may need to be fixed as well, but we can take a look at it later.

[kmike]

Are you sure these failures happen after upgrade to asyncio even loop, not after adding an ENV variable which enables devtools?

[lucywang000]

Are you sure these failures happen after upgrade to asyncio even loop, not after adding an ENV variable which enables devtools?

Yeah. When I reverted my changes and the tests could all pass.