uc still detected on ubuntu server

[STARKILLER-1]

def parce_ozon_rating(articles):
    results = []
    prod = True  # Set True in production
    if prod:
        display = Display(visible=0, size=(1440, 1880))
        display.start()
    driver = Driver(uc=True, headed=True)
    for index, article in enumerate(articles, start=1):
        attempts = 0
        while attempts < 2:
            try:
                driver.uc_open_with_reconnect(f"https://www.ozon.ru/product/{article}", 7) 
                try:
                    driver.uc_click("//button[contains(text(), 'Обновить')]", by="xpath")
                    time.sleep(2)
                except Exception as e:
                    pass
                try:
                    driver.find_element(By.CSS_SELECTOR, 'div[data-widget="error"]')
                    print('Страница с ошибкой, у товара нет отзывов')
                    results.append({'article': article, 
                                'rating': 0, 
                                'all_comments': 0, 
                                'questions': 0, 
                                'concrete_comments': 0})
                    break
                except:
                    pass
                elements_rating_span = WebDriverWait(driver, 7).until(EC.presence_of_all_elements_located((By.XPATH, "//span[contains(text(), ' / 5')]")))
                filtered_elements = [el for el in elements_rating_span if len(el.text) <= 8]
                if filtered_elements:
                    desired_element = filtered_elements[0]
                    rating_text = desired_element.text
                    rating_value = float(rating_text.split(" / ")[0])
                try:  
                    comments_div = WebDriverWait(driver, 3).until(EC.visibility_of_element_located((By.XPATH, "//span[text()= 'Отзывы о товаре']")))
                    comments_element = comments_div.find_element(By.XPATH, "following-sibling::span")
                    comments_text = comments_element.text.strip()
                    comments_number = int(comments_text)
                except:
                    comments_number = 0
                
                try:  
                    question_div = WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.XPATH, "//span[text()='Вопросы о товаре']")))
                    question_element = question_div.find_element(By.XPATH, "following-sibling::span")
                    question_text = question_element.text.strip()
                    question_number = int(question_text)
                except:
                    question_number = 0
                
                try: 
                    driver.find_element(By.XPATH, "//span[contains(text(), 'Этот вариант товара (')]")
                    comments_concrete_div = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.XPATH, "//span[contains(text(), 'Этот вариант товара (')]")))
                    comments_concrete_text = comments_concrete_div.text.strip()
                    comments_concrete_str = comments_concrete_text.split('(')[1].split(')')[0]
                    comments_concrete_number = int(comments_concrete_str)
                except:
                    comments_concrete_number = 0

                results.append({'article': article, 
                                'rating': rating_value, 
                                'all_comments': comments_number, 
                                'questions': question_number, 
                                'concrete_comments': comments_concrete_number})
                print(results[-1])
                break 
            except Exception as e:
                html=driver.get_page_source()
                with open('example_123.html', 'w', encoding='utf-8') as file:
                   file.write(html)
                results.append({'article': article, 
                                'rating': 0, 
                                'all_comments': 0, 
                                'questions': 0, 
                                'concrete_comments': 0})
                print(results[-1])
            finally:
                attempts += 1
    if prod:
        display.stop()
    driver.quit()
    return results

We tried some methods but it didn't make it

[mdmintz]

@STARKILLER-1 Is it working locally though?
There are plenty of known Ubuntu servers (such as GitHub Actions) where the IP range is known, and has therefore been labeled by anti-bot services as bot traffic. If local traffic is also blocked, where in your script are you getting detected?

You have lots of WebDriverWait lines that would make you detected. Use only UC Mode methods if possible; but if you need more methods, use the SB() format instead of the Driver() format. SeleniumBase also adds a lot of methods to the Driver() format (see #2200 (comment)).

[STARKILLER-1]

I have my own server on ubutu it's in my room, should i try my code there?

[mdmintz]

Probably related to the proxy you use on your Ubuntu server. Since it's working locally, the code is valid. You may need to use a different proxy. If the IP address is already labeled as bot traffic, then UC Mode can't fix that.

[STARKILLER-1]

We have 200 private proxies, and tried to use without proxies, not working

[STARKILLER-1]

I know about bad proxy addresses, you told me long time ago #2247

[mdmintz]

If I had an answer, I'd mention it. UC Mode is bypassing cloudflare for me locally, and remotely on Ubuntu depending on the proxy server.

[STARKILLER-1]

Probably related to the proxy you use on your Ubuntu server. Since it's working locally, the code is valid. You may need to use a different proxy. If the IP address is already labeled as bot traffic, then UC Mode can't fix that.

[ankushkumarpatiyal]

@mdmintz hello there, am currently using seleniumbase to scrap "truepeoplesearch" which have cloudflare detection although my script is running fine on my local machine(its a django app), but when i deployed my code on ubuntu server it cant seem to pass the bot detection, i have various methods but none seemed to help me, please if you could help :
here is my sample code for your reference :

try:
with SB(uc=True,proxy=f"{proxy_username}:{proxy_password}@{proxy_ip_port}",ad_block_on=True, headless=True) as sb:
true_people_data = truepeople_scraper(sb,true_people_url)
except:
with SB(uc=True,proxy=f"{proxy_username}:{proxy_password}@{proxy_ip_port}",ad_block_on=True, headless=True,) as sb:
true_people_data = truepeople_scraper(sb,true_people_url)

here's the function :
def truepeople_scraper(sb,url):

def click_turnstile_and_verify(sb):
    sb.switch_to_frame("iframe")
    sb.driver.uc_click("span")

def open_the_turnstile_page(sb,url):
    url = url
    try:
        sb.driver.uc_open_with_reconnect(url, 3)
        sleep(3)
        page_source = sb.get_page_source()
    except:
        sb.driver.uc_open_with_reconnect(url,3)
        sleep(3)
        page_source = sb.get_page_source()
        click_turnstile_and_verify(sb)
        page_source = sb.get_page_source()

    return page_source,url
    
    
    please tell me what am i doing wrong , i have tried (headless2 = True) even (xfvb= true) still no use

[mdmintz]

See the thread: #2842

Now, if your User-Agent looks untrustworthy, CloudFlare makes you click the CAPTCHA (which has been improved). If they detect either Selenium in the browser or JavaScript involvement in clicking the CAPTCHA, they don't let the click through. That's why pyautogui is now required for clicking the CAPTCHA if the User-Agent isn't trustworthy enough. The default user-agent set on macOS and Windows by SeleniumBase is generally good enough. On Linux, the default User-Agent might not be good enough: You may need to specify a better one to avoid needing to click the CAPTCHA in that scenario. (Or just use the pyautogui workaround for clicking it.)