Tutorial to Scraping LinkedIn company profile using Python
In this article, we will scrape LinkedIn company data in Python using the Selenium and Beautiful Soup libraries.
The webpage took too long to load (due to a slow internet connection, most likely). Use time in this case. To give the webpage more time to load, use the sleep() function. Set the number of seconds to sleep to suit your needs.
Prerequisites: You will need the following items to follow this guide:
Python is already installed on your system.
Python programming basics are required.
Understanding of web scraping concepts.
Step 1: Install the Necessary Libraries Before we begin scraping, we must first install some Python libraries. Run the following commands in your terminal or command prompt:
pip install requests
pip install beautifulsoup4
pip install selenium
pip install webdriver_manager
Step 2: Configure the Web Driver LinkedIn uses JavaScript to render its pages, necessitating the use of a headless browser or a web driver to interact with the website. For this, we’ll use Selenium and the Chrome web driver. Furthermore, the webdriver_manager library aids in the automatic management of the web driver.
Get Mark John’s stories in your inbox
Join Medium for free to get updates from this writer.
Here’s an example of how to configure the Chrome web driver:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManagerdriver = webdriver.Chrome(ChromeDriverManager().install())
Step 3: Access LinkedIn. We must be logged in to access LinkedIn’s data. We can automate this process using Selenium. To log in programmatically, you must enter your LinkedIn account credentials. In the code snippet below, replace the placeholders with your credentials:
# Navigate to the LinkedIn login page
driver.get(‘https://www.linkedin.com/login')# Enter your email address and password
driver.find_element_by_id(‘username’).send_keys(‘your_email@example.com’)
driver.find_element_by_id(‘password’).send_keys(‘your_password’)
# Submit the login form
driver.find_element_by_css_selector(‘.login__form_action_container button’).click(
Step 4: Go to a LinkedIn Page. Once logged in, go to any LinkedIn page you want to scrape. To scrape data from a specific user’s profile, for example, use the following code snippet:
profile_url = ‘https://www.linkedin.com/in/example-profile'
driver.get(profile_url)
Step 5: Using Beautiful Soup, extract data. Now that we’ve arrived at the desired LinkedIn page, we can extract the relevant data with Beautiful Soup, a popular Python web scraping library. We’ll identify and parse the HTML elements that contain the data we want to extract.
Here’s an example of extracting a user’s profile name and headline:
from bs4 import BeautifulSoup
# Get the page source
page_source = driver.page_source
# Parse the HTML using Beautiful Soup
soup = BeautifulSoup(page_source, ‘html.parser’)
# Extract the name and headline
name = soup.find(‘li’, {‘class’: ‘inline t-24 t-black t-normal break-words’}).text.strip()
headline = soup.find(‘h2’, {‘class’: ‘mt1 t-18 t-black t-normal break-words’}).text.strip()
# Print the extracted data
print(‘Name:’, name)
print(‘Headline:’, headline)
Step 6: Gather More Information Other data from LinkedIn profiles, such as work experience, education, skills, and more, can be extracted using the same method. Examine the HTML source of the LinkedIn page you’re interested in and select the elements to target with Beautiful Soup.
Step 7: Dealing with Pagination and Iterating Through Multiple Profiles LinkedIn frequently employs pagination to display multiple profiles. To scrape data from multiple profiles, use appropriate code logic to navigate through the pagination and iterate over each profile. This could entail locating the “Next” button, clicking it, and then repeating the scraping process.
Conclusion
In this article, we looked at how to scrape data from LinkedIn with Python. We went over the necessary steps, such as installing the necessary libraries, configuring the web driver, logging into LinkedIn, navigating to specific pages, and extracting data with Beautiful Soup. Remember to use web scraping responsibly, follow legal requirements, and respect individuals’ privacy.
