Issue running seleniumbase in Streamlit

[Abdelgha-4]

I found in few places that google-chrome-stable or google-chrome is required to use UC mode in seleniumbase, after some research I've succeeded in installing it following this example.

I'm using Streamlit's method of installing apt-get dependencies by adding these to my packages.txt:

chromium
chromium-driver
google-chrome-stable
pkg-config
python3-dev
python3-tk
xvfb

I see in the logs that all the depenecies are installed, but seleniumbase does not detect the uc_driver (maybe because chromium-driver version is incompatible with google-chrome's version) which should not be a big issue since seleniumbase should download it anyway, but the download raises a 404 error:

Full traceback

/home/adminuser/venv/lib/python3.12/site-packages/pyautogui/_pyautogui_x11.py:284: SyntaxWarning: invalid escape sequence '\e'
  '\e': _display.keysym_to_keycode(Xlib.XK.string_to_keysym('Escape')),

Warning: uc_driver not found. Getting it now:

*** chromedriver to download = 131.0.6778.87 (Latest Stable) 

Downloading chromedriver-linux64.zip from:
https://storage.googleapis.com/chrome-for-testing-public/131.0.6778.87/linux64/chromedriver-linux64.zip ...
────────────────────── Traceback (most recent call last) ───────────────────────
  /home/adminuser/venv/lib/python3.12/site-packages/seleniumbase/core/browser_  
  launcher.py:4836[2024-12-09 20:26:24.074470]  in get_local_driver                                          
                                                                                
    4833 │   │   │   │   │   │   │   │   uc_path = LOCAL_UC_DRIVER              
    4834 │   │   │   │   │   │   │   │   uc_path = os.path.realpath(uc_path)    
    4835 │   │   │   │   │   │   │   try:                                       
  ❱ 4836 │   │   │   │   │   │   │   │   driver = undetected.Chrome(            
    4837 │   │   │   │   │   │   │   │   │   options=chrome_options,            
    4838 │   │   │   │   │   │   │   │   │   user_data_dir=user_data_dir,       
    4839 │   │   │   │   │   │   │   │   │   driver_executable_path=uc_path,    
                                                                                
  /home/adminuser/venv/lib/python3.12/site-packages/seleniumbase/undetected/__  
  init__.py:130 in __init__                                                     
                                                                                
    [2024-12-09 20:26:24.074774] 127 │   │   │   │   │   force=patcher_force_close,                          
    128 │   │   │   │   │   version_main=version_main,                          
    129 │   │   │   │   )                                                       
  ❱ 130 │   │   │   │   self.patcher.auto()                                     
    131 │   │   if not options:                                                 
    132 │   │   │   options = ChromeOptions()                                   
    133 │   │   try:                                                            
                                                                                
  /home/adminuser/venv/lib/python3.12/site-packages/seleniumbase/undetected/pa  
  tcher.py:106 in auto                                                          
                                                                                
    103 │   │   │   │   pass                                                    
    104 │   │   except FileNotFoundError:                                       
    105 │   │   │   pass                                                        
  ❱ 106 │   │   release = self.fetch_release_number()                           
    107 │   │   self.version_main = release.split(".")[0]                       
    108 │   │   self.version_full = release                                     
    109 │   │   self.unzip_package(self.fetch_package())                        
                                                                                
  /home/adminuser/venv/lib/python3.12/site-packages/seleniumbase/undetected/pa  
  tcher.py:124 in fetch_release_number                                          
                                                                                
    121 │   │   │   path += "_[2024-12-09 20:26:24.075098] %s" % self.version_main                           
    122 │   │   path = path.upper()                                             
    123 │   │   logger.debug("Getting release number from %s" % path)           
  ❱ 124 │   │   return urlopen(self.url_repo + path).read().decode()            
    125 │                                                                       
    126 │   def fetch_package(self):                                            [2024-12-09 20:26:24.075304] 
    127 │   │   """Downloads chromedriver from source.                          
                                                                                
  /usr/local/lib/python3.12/urllib/request.py:215 in urlopen                    
                                                                                
     212 │   │   _opener = opener = build_opener()                              
     213 │   else:                                                              
     214 │   │   opener = _opener                                               
  ❱  215 │   return opener.open(url, data, timeout)                             
     216                                                                        
     217 def install_opener(opener):                                            
     218 │   global _opener                                                     
 [2024-12-09 20:26:24.075862]                                                                                
  /usr/local/lib/python3.12/urllib/request.py:521 in open                       
                                                                                
     518 │   │   meth_name = protocol+"_response"                               
     519 │   │   for processor in [2024-12-09 20:26:24.076173] self.process_response.get(protocol, []):      
     520 │   │   │   meth = getattr(processor, meth_name)                       
  ❱  521 │   │   │   response = meth(req, response)                             
     522 │   │                                                                  
     523 │   │   return response                                                
     524                                                                        
                                                                                
  /usr/local/lib/python3.12/urllib/request.py:630 in http_response              [2024-12-09 20:26:24.076659] 
                                                                                
     627 │   │   # According to RFC 2616, "2xx" code indicates that the client  
     628 │   │   # request was successfully received, understood, and accepted  
     629 │   │   [2024-12-09 20:26:24.076965] if not (200 <= code < 300):                                    
  ❱  630 │   │   │   response = self.parent.error(                              
     631 │   │   │   │   'http', request, response, code, msg, hdrs)            
     632 │   │                                                                  
     633 [2024-12-09 20:26:24.077182] │   │   return response                                                
                                                                                
  /usr/local/lib/python3.12/urllib/request.py:559 in error                      
                                                                                
     556 │   │                                                                  
     557 │   │   if http_err:                                                   
     558 │   │   │   args = ([2024-12-09 20:26:24.077407] dict, 'default', 'http_error_default') + orig_arg  
  ❱  559 │   │   │   return self._call_chain(*args)                             
     560                                                                        
     561 # XXX probably also want an abstract factory that knows when it makes  
     562 # sense to skip a superclass in favor of a subclass and when it might  
                                                                                
  /usr/local/lib/python3.12/urllib/request.py:492 in _call_chain                
                                                                                
     489 │   │   handlers = chain.get(kind, ())                                 
     490 │   │   for handler in handlers:                                       
     491 │   │   │   func = getattr(handler, meth_name)                         
  ❱  492 │   │   │   result = func(*args)                                       
     493 │   │   │   if result is not None:                                     
     494 │   │   │   │   return result                                          
     495                                                                        
                                                                                
  /usr/local/lib/python3.12/urllib/request.py:639 in http_error_default         
                                                                                
     636                                                                        
     637 class HTTPDefaultErrorHandler(BaseHandler):                            
     638 │   def http_error_default(self, req, fp, code, msg, hdrs):            
  ❱  639 │   │   raise HTTPError(req.full_url, code, msg, hdrs, fp)             
     640                                                                        
     641 class [2024-12-09 20:26:24.078245] HTTPRedirectHandler(BaseHandler):                                
     642 │   # maximum number of redirections to any single URL                 
────────────────────────────────────────────────────────────────────────────────
HTTPError: HTTP Error 404: Not Found

During handling of the above exception, another exception occurred:

────────────────────── Traceback (most recent call last) ───────────────────────
  /home/adminuser/venv/lib/python3.12/site-packages/streamlit/runtime/scriptru  
  nner/exec_code.py:88 in exec_func_with_error_handling                         
                                                                                
  /home/adminuser/venv/lib/python3.12/site-packages/streamlit/runtime/scriptru  
  nner/script_runner.py:579 in code_to_exec                                     
                                                                                
  /mount/src/notifyproject/streamlit_app.py:6 in <module>                       
                                                                                
    3 [2024-12-09 20:26:24.078655] from app.main import amain                                                
    4                                                                           
    5 if __name__ == "__main__":                                                
  ❱ 6 │   asyncio.run(amain())[2024-12-09 20:26:24.078939]                                                   
    7                                                                           
                                                                                
  /home/adminuser/venv/lib/python3.12/site-packages/nest_asyncio.py:30 in run   
                                                                                
     27 │   │   loop.set_debug(debug)                                           
     28 │   │   task = asyncio.ensure_future(main)                             [2024-12-09 20:26:24.079168]  
     29 │   │   try:                                                            
  ❱  30 │   │   │   return loop.run_until_complete(task)                        
     31 │   │   finally:                                                        [2024-12-09 20:26:24.079377] 
     32 │   │   │   if not task.done():                                         
     33 │   │   │   │   task.cancel()                                           
                                                                                
  /home/adminuser/venv/lib/python3.12/site-packages/nest_asyncio.py[2024-12-09 20:26:24.079609] :98 in       
  run_until_complete                                                            
                                                                                
     95 │   │   │   if not f.done():                                            
     96 │   │   │   │   raise [2024-12-09 20:26:24.079798] RuntimeError(                                     
     97 │   │   │   │   │   'Event loop stopped before Future completed.')      
  ❱  98 │   │   │   return f.result()                                           
     99 │                                                                       
    100 │   def _run_once(self):                                                
    101 │   │   """                                                             
                                                                                [2024-12-09 20:26:24.080297] 
  /usr/local/lib/python3.12/asyncio/futures.py:202 in result                    
                                                                                
    199 │   │   │   raise exceptions.InvalidStateError([2024-12-09 20:26:24.080483] 'Result is not ready.')  
    200 │   │   self.__log_traceback = False                                    
    201 │   │   if [2024-12-09 20:26:24.080746] self._exception is not None:                                 
  ❱ 202 │   │   │   raise self._exception.with_traceback(self[2024-12-09 20:26:24.080947] ._exception_tb)    
    203 │   │   return self._result                                             
    204 │                                                                       
    205 │   def exception(self):                                                
                                                                                
  /usr/local/lib/python3.12/asyncio/tasks.py:314 in                             
  __step_run_and_handle_result                                                  
                                                                                
     311 [2024-12-09 20:26:24.081163] │   │   │   if exc is None:                                            
     312 │   │   │   │   # We use the `send` method directly, because coroutin  
     313 │   │   │   │   # don't have `__iter__` and `__next__` methods.[2024-12-09 20:26:24.081370]         
  ❱  314 │   │   │   │   result = coro.send(None)                               
     315 │   │   │   else:                                                      
     316 │   │   │   │   result = coro.throw(exc)                               
     317 │   │   except[2024-12-09 20:26:24.081556]  StopIteration as exc:                                   
                                                                                
  /mount/src/notifyproject/app/main.py:297 in amain                             
                                                                                
    294 │   )                                                                   
    295 │   [2024-12-09 20:26:24.081775] geodetic_polygon = GeodeticPolygon(location_encoded_polygon)        
    296 │                                                                       
  ❱ 297 │   listings = collect_listings(factories, selected_filters, geodetic_  
    298 │                                                                       
    299 │   if not listings:                                                    
    300 │   │   st.warning("No listings found :/, Perhaps your filters are too  
                                                                                
  /home/adminuser/venv/lib/python3.12/site-packages/streamlit/runtime/caching/  
  cache_utils.py:217 in __call__                                                
                                                                                
  /home/adminuser/venv/lib/python3.12/site-packages/streamlit/runtime/caching/ [2024-12-09 20:26:24.082321]  
  cache_utils.py:242 in _get_or_create_cached_value                             
                                                                                
  /home/adminuser/venv/lib/python3.12/site-packages/streamlit/runtime/caching/  
  cache_utils.py:299 in _handle_cache_miss                                      
                                                                                
  /mount/src/project/app/[2024-12-09 20:26:24.082514] main.py:335 in main                  
                                                                                                                
    334 ) -> list[dict[str, Any]]:                                              [2024-12-09 20:26:24.082668] 
  ❱ 335 │   return asyncio.run(_collect_listings(factories, selected_filters,   
    336                                                                         
                                         
                                                                                
  /home/adminuser/venv/lib/python3.12/site-packages/nest_asyncio.py:30 in run   
                                                                                
     27 │   │   loop.set_debug(debug)                                           
     28 │   │   [2024-12-09 20:26:24.083119] task = asyncio.ensure_future(main)                              
     29 │   │   try:                                                            
  ❱  30 │   │   │   return loop.run_until_complete(task)                        
     31 │   │   [2024-12-09 20:26:24.083339] finally:                                                        
     32 │   │   │   if not task.done():                                         
     33 │   │   │   │   task.cancel()                                           
                                                                                
  /home/adminuser/venv/lib/python3.12/site-packages/nest_asyncio.py:98[2024-12-09 20:26:24.083531]  in       
  run_until_complete                                                            
                                                                                
     95 │   │   │   if not f.done():                                            
     96 │   │   │   │   raise RuntimeError(                                    [2024-12-09 20:26:24.083745]  
     97 │   │   │   │   │   'Event loop stopped before Future completed.')      
  ❱  98 │   │   │   return f.result()                                           
     99 [2024-12-09 20:26:24.083964] │                                                                       
    100 │   def _run_once(self):                                                
    101 │   │   """                                                             
                                                                                
  /usr/local/lib/python3.12/asyncio/futures.py:202 in result                    
                                                                                
    199 │   │   │   raise exceptions.InvalidStateError('Result is not ready.')  
    200 │   │   self.__log_traceback = False                                    
    201 │   │   if self._exception is not [2024-12-09 20:26:24.084386] None:                                 
  ❱ 202 │   │   │   raise self._exception.with_traceback(self._exception_tb)    
    203 │   │   return self._result                                             
    204 │                                                                       
 [2024-12-09 20:26:24.084599]    205 │   def exception(self):                                                
                                                                                
  /usr/local/lib/python3.12/asyncio/tasks.py:314 in                             
  __step_run_and_handle_result                                                  
                                                                                
     311 │   │   │   if exc is None:                                            
     312 │   │   │   │   # We use the `send` method directly, because coroutin  
     313 │   │   │   │   # don't have `__iter__` and `__next__` methods.        
  ❱ [2024-12-09 20:26:24.084849]  314 │   │   │   │   result = coro.send(None)                               
     315 │   │   │   else:                                                      
     316 │   │   │   │   result = coro.throw(exc)                               
     317 │   │   except StopIteration as[2024-12-09 20:26:24.085028]  exc:                                   
                                                                                                                                                  
  /mount/src/connectors/crawler.py:86 in  
  crawling_context                                                              
                                                                                
     83 │   def crawling_context(self):                                         
     84 │   │   try:                                                            
     85 │   │   │   # SB is smart, it'll use xvfb (virtual display) by default  
  ❱  86 │   │   │   with SB(uc=True, uc_cdp_events=True, incognito=True) as sb  
     87 [2024-12-09 20:26:24.088152] │   │   │   │   self.sb = sb                                            
     88 │   │   │   │   yield                                                   
     89 │   │   finally:                                                        
                                                                                
  /usr/local/lib/python3.12/contextlib.py:137 in __enter__                      
                                                                                
    134 [2024-12-09 20:26:24.088470] │   │   # they are only needed for recreation, which is not possible a  
    135 │   │   del self.args, self.kwds, self.func                             
    136 [2024-12-09 20:26:24.088599] │   │   try:                                                            
  ❱ 137 │   │   │   return next(self.gen)                                       
    138 │   │   except[2024-12-09 20:26:24.088849]  StopIteration:                                           
    139 │   │   │   raise RuntimeError("generator didn't yield") from None[2024-12-09 20:26:24.089082]       
    140                                                                         
                                                                                
  /home/adminuser/venv/lib/python3.12/site-packages/seleniumbase/plugins/sb_ma  
  nager.py:1214 in SB                                                           
                                                                                
    1211 │   │   if not sb_config.multi_proxy:                                  
    1212 │   │   │   proxy_helper.remove_proxy_zip_if_present()                 
    1213 │   start_time = time.time()                                           
  ❱ 1214 [2024-12-09 20:26:24.089377] │   sb.setUp()                                                         
    1215 │   test_passed = True  # This can change later                        
    1216 │   teardown_exception = None                                          
    1217 │   if "--trace" in sys_argv:                                          
                                                                                
  /home/adminuser/venv/lib/python3.12/site-packages/seleniumbase/fixtures/base[2024-12-09 20:26:24.089912]   
  _case.py:15167 in setUp                                                       
                                                                                
    15164 │   │   │   │   │   │   self.__js_start_time = int[2024-12-09 20:26:24.090098] (time.time() * 100  
    15165 │   │   else:                                                         
    15166 │   │   │   [2024-12-09 20:26:24.090257] # Launch WebDriver for both pytest and pynose             
  ❱ 15167 │   │   │   self.driver = self.get_new_driver(                        
    15168 │   │   │   │   browser=self.browser,                                [2024-12-09 20:26:24.090417]  
    15169 │   │   │   │   headless=self.headless,                               
    15170 │   │   │   │   locale_code=self.locale_code,                         
                                                                                [2024-12-09 20:26:24.090539] 
  /home/adminuser/venv/lib/python3.12/site-packages/seleniumbase/fixtures/base  
  _case.py:4175 in get_new_driver                                               
                                                                                
 [2024-12-09 20:26:24.090658]     4172 │   │   │   │   "Valid options = {%s}" % (browser, valid_browsers)    
     4173 │   │   │   )                                                         
    [2024-12-09 20:26:24.090908]  4174 │   │   # Launch a web browser                                        
  ❱  4175 │   │   new_driver = browser_launcher.get_driver(                     
     4176 │   │   │   browser_name=browser_name,                                
     4177 │   │   │   headless=headless,                                        
     4178 │   │   │   locale_code=locale_code,                                  
                                                                                
  /home/adminuser/venv/lib/python3.12/site-packages/seleniumbase/core/browser_  
  launcher.py[2024-12-09 20:26:24.091225] :2810 in get_driver                                                
                                                                                
    2807 │   │   │   device_pixel_ratio,                                        
    2808 │   │   )                                                              
 [2024-12-09 20:26:24.091386]    2809 │   else:                                                              
  ❱ 2810 │   │   return get_local_driver(                                       
    2811 │   │   │   browser_name,                                              
    2812 │   │   │   headless,                                                  
    2813 │   │   │   locale_code,                                               
                                                                               [2024-12-09 20:26:24.091617]  
  /home/adminuser/venv/lib/python3.12/site-packages/seleniumbase/core/browser_  
  launcher.py:4848 in get_local_driver                                          
                                                                                
    4845 │   │   │   │   │   │   │   │   )                                      
    4846 │   │   │   │   │   │   │   │   uc_activated = True                    
    4847 │   │   │   │   │   │   │   except URLError as e:                      
  ❱ 4848 │   │   │   │   │   │   │   │   if cert in e.args[0] and IS_MAC:       
    4849 │   │   │   │   │   │   │   │   │   mac_certificate_error = True       
    4850 │   │   │   │   │   │   │   │   else:                                  
    4851 │   │   │   │   │   │   │   │   │   raise                              
────────────────────────────────────────────────────────────────────────────────
IndexError: tuple index out of range

Any help would be much appreciated.