<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by NithilaRaja on Medium]]></title>
        <description><![CDATA[Stories by NithilaRaja on Medium]]></description>
        <link>https://medium.com/@nithilaraja07?source=rss-c25b04c97e63------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Q7zo5ms4N81ltWFYT5nujg.jpeg</url>
            <title>Stories by NithilaRaja on Medium</title>
            <link>https://medium.com/@nithilaraja07?source=rss-c25b04c97e63------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 23:31:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nithilaraja07/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Essential things to make WebScraping easy(Selenium/java)]]></title>
            <link>https://medium.com/@nithilaraja07/essential-things-to-make-webscraping-easy-selenium-java-675b97736ac8?source=rss-c25b04c97e63------2</link>
            <guid isPermaLink="false">https://medium.com/p/675b97736ac8</guid>
            <dc:creator><![CDATA[NithilaRaja]]></dc:creator>
            <pubDate>Wed, 20 Apr 2022 18:57:30 GMT</pubDate>
            <atom:updated>2022-04-20T18:57:30.064Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Web Scraping</strong></p><p>WebScrapping is used to extract the information from the requested URL using the tools and converting it into the needed format for the user.</p><p><strong>Selenium/Java/Eclipse IDE</strong></p><p>We are going to use the MAVEN build tool for working with Java/Selenium to scrape the website.</p><p><strong>1.HeadlessBrowser</strong></p><p>Using a headless browser you can’t see the UI, it runs in the background and gives the needed result. As there won’t be any need to see the functionality in web scraping this will be more useful as you don’t need to wait for images to get loaded.</p><p><strong><em>Dependency needed in pom.xml</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*QUS5fxkhUHW4wOejDjEQLQ.png" /><figcaption>Headless browser dependecy</figcaption></figure><p><em>Sample program using Headless browser</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qNieUUJXeAeno3wy-HhQJg.png" /><figcaption>Headless Browser</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/1*dyL0bCJbQqsEm6enjXMbKg.png" /><figcaption>Console output</figcaption></figure><p>So without opening the UI we can get the title of the page. Also, you can select BrowserVersion with the options below:-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*rijli2ApzBN3wLO0nZJdXw.png" /><figcaption>browseroption to be selected</figcaption></figure><p><strong>2.PageLoadStartegy</strong></p><p>PageLoadStartegy can be used in Webscrapping to make the document ready according to your needs.</p><p><em>Example:-</em></p><p>a)If you need your page to load completely before scrapping use <strong>PageLoadStrategy.</strong> <strong>NORMAL.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1YCEdaqsw6S4mP28_1yFCw.png" /><figcaption>SetPageLoadStrategy.NORMAL</figcaption></figure><p>b)If you just want to scrape the page discard the image use <strong>PageLoadStrategy.</strong> <strong>Eager</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ncK4Zd2OdPjlB4Fg4iy8-Q.png" /><figcaption>SetPageLoadStrategy.Eager</figcaption></figure><p><strong>Eager</strong>-Will find the elements even before 100% page has been rendered(Time-Saving).</p><p>3.<strong>WebDriverManager</strong></p><p>Worried about the changing version in chrome/firefox/Edge/IE.</p><p>Voila!!! Use WebDriverManager don’t worry about downloading the Browser and updating it.</p><p>Add this dependency in POM.XML</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*1g0tx5LUWvoQklxz33w4Kw.png" /></figure><p>Instead of this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*WZFyJVuZxDoIBPQmXqZOpw.png" /></figure><p>No path needed to initialize the browser</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*P2mROKMRIP8PVT0tUyIQQA.png" /></figure><p><strong>4. Validate the broken Images</strong></p><p>If your going to scrape the images first validate them, If it’s broken image by checking the valid Httpstatus code=200.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f4u30KHmxHsfcxJDMf56HQ.png" /></figure><p>By checking the status code response, we can come to a conclusion about the images we are able to scrape.</p><p><strong>5.Check the Broken Links</strong></p><p>If you are in need of scrapping the broken links ,check with the statuscode to get an idea.</p><p>Sample code:-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*Lp6mpg5LYvh0OAbuTvU9hQ.png" /><figcaption>Status code for valid is 200</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*oHL7rmYaa8BBe8LttisUfA.png" /><figcaption>Status code for invalid link is set to 400</figcaption></figure><p>So if you’re trying to set the status code to 400 for invalid and valid=200. You can find the broken links.</p><p><strong>6.Pagination</strong></p><p>If you are unaware of the page number in the websites your going to scrape Use the pagination method .</p><p>Sample code:-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wsHilAW3UEGetX6EscavZw.png" /><figcaption>Paginaiton</figcaption></figure><p><strong>7.Dynamic XPATH</strong></p><p>Sometimes the standard way we cannot find the elements, So in that case try to write it with Dynamic XPath using XPath axes like a child, parent, ancestor, sibling, etc.,</p><p>Keep these methods before scrapping and make your scrapping easy before you start.</p><p><strong><em>Happy Scrapping!!!!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=675b97736ac8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>