Allowing easy integration with other selenium libraries like selenium-wire

[ksdaftari]

Hello, first off great work on this amazing library, has been incredibly helpful in making our FE automated testing at our company actually easy to maintain and iterate on!

I was wondering is it currently possible or potentially thoughts on allowing it easy to integrate other selenium libraries that can help enhance the functionality seleniumbase gives. one I am thinking of is one like selenium-wire: https://github.com/wkeeling/selenium-wire (which allows a realy nice interface for inspecting the requests and responses, dummy up responses, etc). My thought of how would allow this is to pull the more core seleniumbase additional/improved interface it adds on top of the selenium webdriver and make that class. I believe given many of the other selenium libraries doing something similar, this would allow to use multiple class inheritance to add the functionality of each of them together, or perhaps allowing to specify specific webdriver (so can use one defined in these other libraries) (forget if already able to do this in seleniumbase)?

My guess this likely out of scope for this project, but wanted to bring up idea/question as thought could potentially be cool addition/capability to seleniumbase

[mdmintz]

Hello @ksdaftari, and thank you for your support! The selenium-wire integration has already been done before with SeleniumBase. I'll share with you some code that can do it:

import warnings
from seleniumbase import BaseCase
from seleniumwire import webdriver


class BaseTestCase(BaseCase):
    def setUp(self):
        super().setUp()
        self.driver.close()
        self.driver.quit()
        warnings.simplefilter("ignore", category=DeprecationWarning)
        self.driver = webdriver.Chrome()

    def test_wire(self):
        self.open('https://seleniumbase.io/w3schools/')

        for request in self.driver.requests:
            if request.response:
                print(
                    request.url,
                    request.response.status_code,
                    request.response.headers['Content-Type']
                )

Running that code with pytest will generate the following:

https://seleniumbase.io/w3schools/ 200 text/html; charset=utf-8
https://accounts.google.com/ListAccounts?gpsia=1&source=ChromiumBrowser&json=standard 200 application/json; charset=utf-8
https://seleniumbase.io/w3schools/lib/w3schools28.css 200 text/css; charset=utf-8
https://seleniumbase.io/w3schools/lib/fonts/source-code-pro-v14-latin-regular.woff2 200 font/woff2
https://seleniumbase.io/w3schools/lib/codemirror.css 200 text/css; charset=utf-8
https://seleniumbase.io/w3schools/lib/codemirror.js 200 application/javascript; charset=utf-8
https://seleniumbase.io/w3schools/lib/codemirror_jsx.js 200 application/javascript; charset=utf-8
https://fonts.googleapis.com/css?family=Montserrat 200 text/css; charset=utf-8
https://www.googletagmanager.com/gtag/js?id=UA-167313767-1 200 application/javascript; charset=UTF-8
https://seleniumbase.io/w3schools/lib/fonts/fontawesome.woff2?14663396 200 font/woff2
https://seleniumbase.io/w3schools/lib/fonts/source-sans-pro-v14-latin-600.woff2 200 font/woff2
https://fonts.gstatic.com/s/montserrat/v21/JTUHjIg1_i6t8kCHKm4532VJOt5-QNFgpCtr6Hw5aXx-p7K4KLg.woff 200 font/woff
https://www.googletagmanager.com/gtag/js?id=G-P5KFWRNLRN&l=dataLayer&cx=c 200 application/javascript; charset=UTF-8
https://www.google-analytics.com/analytics.js 200 text/javascript

As seen above, you'll get access to SeleniumBase methods, while also having access to SeleniumWire functionality. The above example will spin up one extra browser before the wire browser, but the impact should be minimal on time. I do hesitate to make selenium-wire a built-in dependency because that adds in a lot of extra weight that 99% of users won't need, but the good thing is that it's easy to use the workaround. Does that solve your issue?

[ksdaftari]

@mdmintz thanks for quick response. I do agree dont think selenium-wire should be something actually integrated within SeleniumBase but ensuring there is way for user to be able to to use within their use of SeleniumBase. Agree that ability to redefine driver likely should suffice and in flexible enough. i think what you suggest will work, I am going to give a try some time this week or next and report back in this discussion.

[SMTamim]

Hello @ksdaftari, and thank you for your support! The selenium-wire integration has already been done before with SeleniumBase. I'll share with you some code that can do it:

import warnings
from seleniumbase import BaseCase
from seleniumwire import webdriver


class BaseTestCase(BaseCase):
    def setUp(self):
        super().setUp()
        self.driver.close()
        self.driver.quit()
        warnings.simplefilter("ignore", category=DeprecationWarning)
        self.driver = webdriver.Chrome()

    def test_wire(self):
        self.open('https://seleniumbase.io/w3schools/')

        for request in self.driver.requests:
            if request.response:
                print(
                    request.url,
                    request.response.status_code,
                    request.response.headers['Content-Type']
                )

Running that code with pytest will generate the following:

https://seleniumbase.io/w3schools/ 200 text/html; charset=utf-8
https://accounts.google.com/ListAccounts?gpsia=1&source=ChromiumBrowser&json=standard 200 application/json; charset=utf-8
https://seleniumbase.io/w3schools/lib/w3schools28.css 200 text/css; charset=utf-8
https://seleniumbase.io/w3schools/lib/fonts/source-code-pro-v14-latin-regular.woff2 200 font/woff2
https://seleniumbase.io/w3schools/lib/codemirror.css 200 text/css; charset=utf-8
https://seleniumbase.io/w3schools/lib/codemirror.js 200 application/javascript; charset=utf-8
https://seleniumbase.io/w3schools/lib/codemirror_jsx.js 200 application/javascript; charset=utf-8
https://fonts.googleapis.com/css?family=Montserrat 200 text/css; charset=utf-8
https://www.googletagmanager.com/gtag/js?id=UA-167313767-1 200 application/javascript; charset=UTF-8
https://seleniumbase.io/w3schools/lib/fonts/fontawesome.woff2?14663396 200 font/woff2
https://seleniumbase.io/w3schools/lib/fonts/source-sans-pro-v14-latin-600.woff2 200 font/woff2
https://fonts.gstatic.com/s/montserrat/v21/JTUHjIg1_i6t8kCHKm4532VJOt5-QNFgpCtr6Hw5aXx-p7K4KLg.woff 200 font/woff
https://www.googletagmanager.com/gtag/js?id=G-P5KFWRNLRN&l=dataLayer&cx=c 200 application/javascript; charset=UTF-8
https://www.google-analytics.com/analytics.js 200 text/javascript

As seen above, you'll get access to SeleniumBase methods, while also having access to SeleniumWire functionality. The above example will spin up one extra browser before the wire browser, but the impact should be minimal on time. I do hesitate to make selenium-wire a built-in dependency because that adds in a lot of extra weight that 99% of users won't need, but the good thing is that it's easy to use the workaround. Does that solve your issue?

is it possible to use with SB context and uc mode and user_data_dir?

[mdmintz]

See #2145 for more details on Wire Mode. (Note that Wire Mode is not compatible with UC Mode.)

[yukeshkumarm]

Hello @mdmintz , I would like to mock multiple requests in same webpage. Current intercept example is working for one request at a time. Could you help me mocking multiple requests? I'm using seleniumbase and seleniumwire combined. Thanks to your example above!

Adding more "If" conditions will lengthen the code here. Any alternative that could help me?

def interceptor(request):
if request.url == 'https://server.com/some/path':
request.create_response(
status_code=200,
headers={'Content-Type': 'text/html'}, # Optional headers dictionary
body='Hello World!' # Optional body
)

driver.request_interceptor = interceptor
driver.get(...)

[mdmintz]

Hi @yukeshkumarm, the latest version of SeleniumBase has direct integration with selenium-wire as a pytest command-line option:
Add --wire, and then you won't need the extra setup. Then it should work with all the other default SeleniumBase command-line options that improve performance, etc. See if using the direct integration helps your situation. You can then combine that with pytest multithreading by adding -n 4 (for example), which will divide your tests into 4 separate parallel processes. Eg:

pytest --wire -n 4

[yukeshkumarm]

Thank you @mdmintz . This helps.

[mdmintz]

@ksdaftari, If you missed the announcement, there's now a direct selenium-wire integration with seleniumbase:

pytest --wire

Then self.driver.requests becomes available within the tests.

See #1574 for all the details.

You can even change the proxy settings in the middle of your tests:

self.set_wire_proxy("SERVER:PORT")
self.set_wire_proxy("socks5://SERVER:PORT")
self.set_wire_proxy("USERNAME:PASSWORD@SERVER:PORT")

---

Here's how it looks with BaseCase inheritance:

from seleniumbase import BaseCase

class TestWire(BaseCase):
    def test_wire_inside_class(self):
        self.open("https://seleniumbase.io/demo_page")
        for request in self.driver.requests:
            print(request.url)

Here's how it looks with the sb fixture:

def test_wire_with_no_class(sb):
    sb.open("https://seleniumbase.io/demo_page")
    for request in sb.driver.requests:
        print(request.url)

class TestWire:
    def test_wire_inside_class(self, sb):
        sb.open("https://seleniumbase.io/demo_page")
        for request in sb.driver.requests:
            print(request.url)

Here's how it looks with the SB Manager:

from seleniumbase import SB

with SB(wire=True) as sb:
    sb.open("https://seleniumbase.io/demo_page")
    for request in sb.driver.requests:
        print(request.url)

Here's how it looks with the Driver Manager:

from seleniumbase import Driver

driver = Driver(wire=True)
driver.get("https://seleniumbase.io/demo_page")
for request in driver.requests:
    print(request.url)

↘️
❇️ ➡️ Remember to add --wire as a command-line option for this to work! ⬅️ 🌟
↗️

[yukeshkumarm]

@mdmintz thanks a lot for sharing this information. And thanks for the quick response :).

[mdmintz]

@yeroda the --uc and --wire integrations don’t play nicely together, so pick one to use with SeleniumBase. Some of the wire features are already included with SeleniumBase, such as setting the proxy with --proxy=server:port or --proxy=user:pass@server:port.

[muhualing]

@mdmintz how can we modify the requests under uc mode? Is that possible to modify requests by add_cdp_listener? If that possible could you please show a simple example about how to modify requests?

[mdmintz]

@muhualing There's a way to get those requests. Try this example: https://github.com/seleniumbase/SeleniumBase/blob/master/examples/uc_cdp_events.py

[muhualing]

Thanks! @mdmintz I see the example is to print requests, I am not sure if it is possible that we can modify the requests and continue sending them to server and get corresponding responses.

[mdmintz]

I've seen ways to modify headers, etc: https://stackoverflow.com/a/59743954/7058266
There's also a SeleniumBase example that modifies the user agent with CDP: https://github.com/seleniumbase/SeleniumBase/blob/master/examples/user_agent_test.py
That's what I know. Check those out and see if that helps.

[muhualing]

@mdmintz Thanks! Sorry for not being clear. I assumed modifying the request body is the most needs in the common scenario. I cannot find similar example in the sharing. I will try to figure it out by checking other cdp commands. Thanks anyway!