selenium-蛋白质组学数据爬取

背景：Box driver中的数据无法自动下载，使用对应的下载链接也需要登录后才能在浏览器中使用，于是使用selenium对其进行自动下载

方法

下载chrom linux版本

参考网址：Linux 版 Google Chrome：下载和安装 (linux-console.net)

# Ubuntu,Debian,Linux Mint
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb # 下载
sudo apt install ./google-chrome-stable_current_amd64.deb # 安装
sudo apt install google-chrome-stable # 更新
sudo apt purge google-chrome-stable # 删除

# Red Hat，CentOS，Fedora
wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm # 下载
sudo dnf localinstall ./google-chrome-stable_current_x86_64.rpm # 安装
sudo dnf install google-chrome-stable # 更新
sudo dnf remove google-chrome-stable # 删除

下载chromdriver

现在最新的代码自动适配所下载的google浏览器，所以无需主动去下载

ChromeDriverManager().install()

如需要下载，步骤如下：

• 首先查看下载分goofle浏览起版本

google-chrome --version
# Google Chrome 120.0.6099.109 

随后在chromedriver镜像地址进行对应版本的chromdriver下载，如在此处即可下载120.0.6099.109 版本的driver，找到对应的linux版本即可。

镜像网址只有114版本之前的driver，若要更行的driver，则需要在https://googlechromelabs.github.io/chrome-for-testing/#stable原网址进行下载。

wget https://registry.npmmirror.com/-/binary/chromedriver/120.0.6099.109/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
mv chromedriver /usr/bin

解压即可得到二进制文件，别忘了加入环境变量中。

代码编写

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager
import os
import time

def is_download_complete(file_path, timeout=1200):
    start_time = time.time()
    while time.time() - start_time < timeout:
        if os.path.exists(file_path):
            return True
        else:
            time.sleep(30)
    return False

def login(driver, username, password):
    driver.get(login_url)
    WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.NAME, 'login'))).send_keys(username)
    driver.find_element(By.ID, 'login-submit').click()
    WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.NAME, 'password'))).send_keys(password + Keys.RETURN)

def download_files(driver, page):
    driver.get(file_url_template.format(page))
    WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "input[type='checkbox'][name='select-checkbox']")))
    checkboxes = driver.find_elements(By.CSS_SELECTOR, "input[type='checkbox'][name='select-checkbox']")
    for checkbox in checkboxes:
        driver.execute_script("arguments[0].click();", checkbox)
    WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[data-resin-target='bulkdownload']"))).click()

login_url = 'https://account.boxcn.net/login'
file_url_template = 'https://app.boxcn.net/s/u3flbp13zjydegrxjb2uepagp1vb6bj2/?page={}'

username = 'XXX'
password = 'XXX'

download_folder = "XXX"

file_name = 'Somalogic-selected.zip'
file_name_template = 'Somatich_selected_page_{}.zip'

chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-dev-shm-usage") ## 不需要可视化界面
chrome_options.add_experimental_option('prefs', {
    "download.default_directory": download_folder,
    "download.prompt_for_download": False,
    "download.directory_upgrade": True,
    "safebrowsing.enabled": True
})


driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chrome_options)


login(driver, username, password)
time.sleep(5)

for page in range(2, 166):
    destination_url = download_files(driver, page)
    # Wait for the iframe to be present
    file_path = os.path.join(download_folder, "Somalogic-selected.zip")
    new_file_path = os.path.join(download_folder, file_name_template.format(page))
    if is_download_complete(file_path):
        os.rename(file_path,new_file_path)
    else:
        print(f"Download not complete or file not found for page {page}.")
        
driver.quit()

其中，虽然在代码中设置了不需要可视化窗口，但是在xshell实际运行的过程中还是一直在不停弹出跳转信息，此时需要在链接>属性>隧道>X11转移，候选取消即可。