Multithreading with SeleniumBase and Undetected Chrome in Linux VM Causes Random Cookie Loading Failures #3738
Labels
invalid usage
You may need to change what you're doing
UC Mode / CDP Mode
Undetected Chromedriver Mode / CDP Mode
What I am trying to do:
This code uses multithreading to automate web interactions with a search engine (via searxng_url) using SeleniumBase in headless Chrome browsers. For each question number in a list, it launches a browser, loads cookies, and performs actions defined in get_all_dumbs(). The browsers are run concurrently using ThreadPoolExecutor to speed up processing.
The issue:
As far as I know, the code works fine in Windows VM, but in Linux VM, it does not work as intended.
Setting the worker number in the Linux VM to less than 2 works, but when setting the worker to more than 1, the UC mode, as far as I understand, disconnected the driver momentarily and reconnected it again; however, for the life of me, I can not load the cookie due to the absenss of the driver. (I assume this is what caused the issue.)
Thank you for your time, and I would love to know if there is a best practice for my case.
The code:
main.py
get_all_dumbs.py
load_cookie.py
The output with the error:
Note: it is random which question triggers the error; some of the Question survived.
The text was updated successfully, but these errors were encountered: