<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tim Voronov on Medium]]></title>
        <description><![CDATA[Stories by Tim Voronov on Medium]]></description>
        <link>https://medium.com/@ziflex?source=rss-2c6a37bee044------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*LkbyE_GLr4ZwQshb.jpg</url>
            <title>Stories by Tim Voronov on Medium</title>
            <link>https://medium.com/@ziflex?source=rss-2c6a37bee044------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 20:57:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ziflex/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Say “Hello” to Ferret — a modern web scraping tool]]></title>
            <link>https://medium.com/@ziflex/say-hello-to-ferret-a-modern-web-scraping-tool-5c9cc85ba183?source=rss-2c6a37bee044------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c9cc85ba183</guid>
            <category><![CDATA[data-mining]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[dsl]]></category>
            <dc:creator><![CDATA[Tim Voronov]]></dc:creator>
            <pubDate>Fri, 08 Feb 2019 12:41:18 GMT</pubDate>
            <atom:updated>2019-05-04T17:44:17.123Z</atom:updated>
            <content:encoded><![CDATA[<h3>Say “Hello” to Ferret — a modern web scraping tool</h3><h4>Making the Internet queryable</h4><p>As a developer, you’ve probably created a couple of web scrapers to get some data from the Internet during your career. That data could be images for an AI project or hotel prices from booking websites for a new travel mobile app or something else. But something that needs data.</p><p>It’s a relatively easy task to write one or two web scrapers if the amount of required data is small. But it’s a completely different level of complexity when you need to scrape big amount of data and, moreover, do it frequently.</p><p>This is what happened to me. A while ago, I started a side project which was heavily data driven. The project required to gather data about cosmetological brands and its products, the ingredients of these products and their effect on human’s skin. Not huge, but pretty large data set.</p><p>The first impulse was to write bunch of scrapers using some scraping framework. And this is what was done. After a while, I started facing with issues I’m going to talk about next and came up with a solution you will see after.</p><h3>The problem</h3><p>There are many problems that you need to solve building a solid and scalable scraping system. Thankfully, many of them can be solved by picking a good library or framework. But there are few, that I could not solve and they were pretty important to me:</p><ul><li>Every time you start, you have to write some amount of boilerplate code which might be annoying. Even with a good scraping library or framework — you have to write some glue code.</li><li>Web pages tend to evolve and change their markup. It leads to frequent code updates and re-deploys.</li><li>Nowadays, more and more websites use dynamic page rendering and old plain HTTP GET request is not a solution any more. It gets worse if the pages require some user interaction to get the data you need.</li></ul><p>Of course, these problems are solvable, but wouldn’t it be nice if we didn’t have to solve them again and again every time we write scraping code?</p><p>So, I started looking a for a tool that would solve the issues for me, but unfortunately, I could not find any. Instead I created one.</p><h3>The solution</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*06mYhTdfqmr4QI_K" /></figure><p><em>Logo by </em><a href="https://medium.com/@cfriel"><em>Chris Friel</em></a></p><p>And this is how <a href="https://github.com/MontFerret/ferret">Ferret</a> was born, a modern web scraping tool with a declarative query language.</p><p>Ferret is a DSL with a runtime that represents pretty powerful and extensible declarative query language.</p><p>It provides very high level solution that lets users focus on data they are looking for instead of writing all that glue code.</p><p>It’s embeddable. Even though, it comes with a CLI, it was designed to be easily embedded into applications from the beginning.</p><p>And of course, it’s super extensible. You can register not only your own functions, but also create custom types that Ferret runtime will recognize.</p><p>The overall idea is simple, you write a query using the declarative query language focusing only on data you need, and Ferret handles the rest of the work for you.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fg9gWv1PODi8%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dg9gWv1PODi8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fg9gWv1PODi8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b97be1f71aa609e411967a38d7b783a4/href">https://medium.com/media/b97be1f71aa609e411967a38d7b783a4/href</a></iframe><h4>Show me the code</h4><p>Let’s see how it works.</p><p>For example, to scrape a list of trending repositories on GitHub, you can use the following query:</p><pre>LET page = DOCUMENT(&quot;<a href="https://github.com/trending">https://github.com/trending</a>&quot;)</pre><pre>FOR row IN ELEMENTS(page, &quot;ol.repo-list li&quot;)<br>    LET name = INNER_TEXT(row, &quot;div:nth-child(1)&quot;)<br>    LET description = INNER_TEXT(row, &quot;div:nth-child(3)&quot;)</pre><pre>    RETURN { name, description }</pre><p><em>NOTE: If you think that the query language looks familiar — you are probably right, the FQL (Ferret Query Language) is heavily inspired by </em><a href="https://docs.arangodb.com/3.4/AQL/index.html"><em>AQL (ArangoDB Query Language)</em></a><em>.</em></p><p>There are not many things happening above, but they solve 2 of our 3 problems: boilerplate code and frequent markup changes. We will discuss how to solve the 3rd later.</p><p>Let’s take a closer look.</p><p>The first problem gets solved automatically by the language itself — Ferret abstracts away all infrastructural glue code from a user. All heavy lifting is done under the hood. All a user needs to do is to describe what data they need to scrape, and what shape of the data should be as an output. Declarative languages allow you to express yourself in a very readable way.</p><p>The second problem gets solved by the fact that FQL is an interpreted language. Script updates do not require you to redeploy your host server. You can easily change it on your production server without any harm.</p><p>Now, let’s discuss what is happening in our script in order to get better understanding of the tool.</p><p>First of all, we tell Ferret what page to load by calling <em>DOCUMENT</em> function. Ferret loads the page and creates its representation in memory as an object. Then we assigned the returned document object to a variable. Yes, in FQL it’s possible to define variables which is an extremely useful feature.</p><p>Once the document is loaded, we use the <em>ELEMENTS </em>function to find all list entries and iterate over them using FQL keywords <em>FOR IN, </em>the construction which is familiar to many developers<em>. </em>Notice, for the search we use regular CSS selector like we would use in a browser.</p><p>Inside the <em>FOR</em> loop body we get inner text of elements of a row using the helper function <em>INNER_TEXT. </em>And again we use variables and CSS selector for simplicity and more efficiency.</p><p>After getting inner text of all elements we are interested in, we use <em>RETURN </em>statement with an object literal.<em> </em>Those who are familiar with JavaScript will recognize the syntax — an object literal with property shorthand. In other words, on each iteration we create an object with 2 properties: name and description. The <em>RETURN</em> statement works as aggregator, and the result of the query is an array of objects.</p><p>That’s how easy and fast you can scrape the Internet.</p><h4>Dynamic pages</h4><p>The example above solves only 2 problems, and we still cannot scrape pages that are rendered dynamically.</p><p>What if we need to get a list of top songs from SoundCloud? If we execute the following query, we will get an empty array:</p><pre>LET doc = DOCUMENT(&#39;https://soundcloud.com/charts/top) <br>LET tracks = ELEMENTS(doc, &#39;.chartTrack__details&#39;)</pre><pre>FOR track IN tracks<br>    RETURN {<br>        artist: TRIM(INNER_TEXT(track, &#39;.chartTrack__username&#39;)),<br>        track: TRIM(INNER_TEXT(track, &#39;.chartTrack__title&#39;))<br>    }</pre><p>In order to fix the query we need to do 2 things:</p><ul><li>Run Chrome or Chromium either in <a href="https://hub.docker.com/r/alpeware/chrome-headless-trunk">Docker</a> or with<em> — remote-debugging-port=9222 </em>argument.</li><li>Add a boolean argument to the DOCUMENT function — this informs Ferret that the page document is dynamic, which causes to use different driver for that.</li></ul><p>Here is an updated version of the query:</p><pre>LET doc = DOCUMENT(&#39;https://soundcloud.com/charts/top&#39;, true)</pre><pre>WAIT_ELEMENT(doc, &#39;.chartTrack__details&#39;, 5000)</pre><pre>LET tracks = ELEMENTS(doc, &#39;.chartTrack__details&#39;)</pre><pre>FOR track IN tracks<br>    RETURN {<br>        artist: TRIM(INNER_TEXT(track, &#39;.chartTrack__username&#39;)),<br>        track: TRIM(INNER_TEXT(track, &#39;.chartTrack__title&#39;))<br>    }</pre><p>You may notice, that there is a new line of code: <em>WAIT_ELEMENT(doc, ‘.chartTrack__details’, 5000)</em>. Since everything is dynamically rendered, we need to be sure that the data we need is in place. That’s why we block the execution until the table with tracks appears on the page or the operation times out. The signature is pretty similar to what we have seen above with an extra timeout argument.</p><p>Once the data is loaded and rendered, we are ready to iterate over all table rows and parse them.</p><p>As you can see, the way how we had written the query didn’t change much, but at the same time we need to take into consideration some nuances of dynamic pages and give Ferret some hints how to handle them.</p><h4>Page interactions</h4><p>Ok, the above examples may not look very impressive and cannot be a real reason to build such a relatively complex solution like Ferret.</p><p>In order to turn the Internet into the Database of the Web we need to be able to interact with the Web like a real user. Clicks, data inputs or just mouse events — all these regular actions need to be supported.</p><p>And Ferret supports them.</p><p>Let’s imagine, that we are building a website that aggregates product prices from all popular sales platforms like Amazon and eBay.</p><p>In order to get a list of products and their information we need to implement the following algorithm:</p><ul><li>Type a search criteria into an input box</li><li>Click “Search” button</li><li>Wait when the page gets loaded and the data gets rendered</li><li>Iterate over the results</li><li>Move to a next page</li><li>Repeat step #3</li></ul><p>Let’s try it out on Amazon.com with <a href="https://github.com/MontFerret/ferret/blob/master/examples/pagination.fql">this script</a>.</p><p><em>NOTE: Unfortunately, Medium has very limited control of fonts, so the query does not really look readable here.</em></p><p>At this point, most of the constructions of the language should be familiar, so I will just highlight what has not been covered before.</p><p>First of all, it’s <a href="https://github.com/MontFerret/ferret/blob/master/examples/pagination.fql#L3">data input</a> and submitting the form <a href="https://github.com/MontFerret/ferret/blob/master/examples/pagination.fql#L4">by clicking the search button</a>. Notice, we use parameters (words starting with @ symbol) in this script which. Their values are define either via CLI or a context in embedded mode (I will cover how to embed Ferret to you application in next blog post).</p><p><a href="https://github.com/MontFerret/ferret/blob/master/examples/pagination.fql#L5">Once the results page gets loaded</a>, we iterate over a range of page numbers, using another parameter. Ideally, you need to extrapolate amount of pages and use this number, but I use the parameter here for simplicity.</p><p>On each iteration, we navigate to the target page (except 1st page) <a href="https://github.com/MontFerret/ferret/blob/master/examples/pagination.fql#L18">by clicking the pagination button</a> and again wait till it gets loaded. After that we just parse the result table.</p><p>As you can see, most of the actions are similar to what we do as regular users. Also, it’s a good demonstration of how web crawlers can be easily implemented using FQL.</p><p><a href="https://github.com/MontFerret/ferret/tree/master/examples">Here</a> you can find more examples and use cases. PR are welcome.</p><h3>Who needs it?</h3><p>Very reasonable and popular question.</p><p>Well, first of all I need it.</p><p>As for others, I can only suggest that this tool <strong>might be</strong> useful for data scientists, ML/AI researchers, QA specialists and anyone who needs to scrape a website without setting up a new Python/Node/etc project.</p><h3>What next?</h3><p>The project is still growing and one of the main milestones is to stabilize API and provide rich functionality out of the box. Besides the Ferret itself, there are some other projects that are planned to be built:</p><h4><a href="https://github.com/MontFerret/ferret-server">Ferret Server</a></h4><p>As you may noticed, Ferret is just a library that comes with a handy CLI. But it does not yet provide a production grade solution for complex scraping. That’s what Ferret Server aims to solve — provide a web scraping platform. The platform that would allows you to persists scripts and scraped data in a database, executes scripts manually or with a scheduler, notify 3rd party systems and etc. The project has started but it’s in very early stage.</p><h4><a href="https://github.com/MontFerret/ferret/issues?utf8=%E2%9C%93&amp;q=is%3Aopen+label%3Aarea%2Fdrivers+Create+in%3Atitle+">More HTML drivers</a></h4><p>Having more HTML drivers will let QA and developers easily test their web apps in different web browsers using same query.</p><h4><a href="https://github.com/MontFerret/ferret-registry">Ferret Registry</a></h4><p>See it as NPM registry for FQL scripts where people can share and explore scripts which will make data scraping even easier!</p><h4>Jupyter kernel</h4><p>I’m not really well familiar with Jupyter, but I know that many data scientists use it, so I figured that it could be useful to integrate Ferret with it.</p><p>So, if you have some spare time and want to contribute to an open source project — you are very welcome!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c9cc85ba183" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evolution of encapsulation in JavaScript]]></title>
            <link>https://medium.com/@ziflex/evolution-of-encapsulation-in-javascript-eb01d3128fae?source=rss-2c6a37bee044------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb01d3128fae</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Tim Voronov]]></dc:creator>
            <pubDate>Thu, 13 Apr 2017 20:51:25 GMT</pubDate>
            <atom:updated>2017-04-13T20:51:25.859Z</atom:updated>
            <content:encoded><![CDATA[<h3>OOP</h3><p>Encapsulation is one of four major principles of OOP (<strong>Encapsulation</strong>, <strong>Abstraction</strong>, <strong>Inheritance, Polymorphism</strong>). Encapsulation restricts direct access to an object’s data. In many classic object-oriented programming languages like Java and C# this principle is supported at a language level by constructs like theprivate keyword, for example.</p><p>However, JavaScript is not an OOP language and in order to implement some of these principles, it needs to apply some tricks. Today I want to talk about some of them which allow for the implementation of encapsulation.</p><h3>Naming convention</h3><p>The first way to work around the lack of encapsulation supported by the language is using naming convention, usually underscore as a prefix. All private members have the prefix and should not be accessed directly:</p><pre>class User {<br>    constructor(name) {<br>        this._name = name; <br>    }<br>    <br>    getName() {<br>        return this._name;<br>    }</pre><pre>    setName(value) {<br>        this._name = value;<br>    }</pre><pre>    _checkNameType(value) {<br>        if (value &amp;&amp; typeof value !== &#39;string&#39;) {<br>            throw new TypeError(&#39;Invalid type of &quot;name&quot;&#39;);<br>        }<br>    }<br>}</pre><p>This approach has several benefits. It’s very simple, fast and memory-leak free.</p><p>However, naming convention based encapsulation also has several drawbacks. Since, it’s just a convention, it does not provide any real restrictions against the access of internal object state and, as a result, there is no way to prevent someone from abusing your classes’ implementations.</p><h3>Closure</h3><p>For the most part naming convention works fine for applications with a small team. However it does not work so well for bigger applications with a lot of developers on board and/or for public frameworks and libraries, where restricted access to internal data is crucial. Closure based encapsulation keeps private data secure by declaring it inside the closure of an object during object initialization. Then access to the data is implemented by functions which have an access to the same closure:</p><pre>// Factory based with plain JS object<br>function User(name) {<br>    let _name = name;<br>    <br>    return {<br>        getName() {<br>            return _name;<br>        },<br> <br>        setName(value) {<br>            _name = value;<br>        }<br>}</pre><pre>// ES6 Class based<br>class User {<br>    constructor(name) {<br>        let _name = name;</pre><pre>        this.getName = () =&gt; {<br>            return _name;<br>        };</pre><pre>        this.setName = (value) =&gt; {<br>            _name = value;<br>        };<br>    }<br>}</pre><p>This approach has several benefits. It gives real data restriction, there is no way to get direct access to an internal object’ state and it works in all browsers (you may use function based constructor in order to support older browsers).</p><p>However, closure based encapsulation has also several drawbacks. First of all, it’s a more memory consuming approach, since every object has its own closure and new instances of each function. When we create functions as a part of a class/prototype definition, every instance reuses them from its prototype. However in this case, we have to create new functions during object initialization dynamically for each instance. Even though their names are the same, they are different for the JS runtime. Another main drawback is an increased risk for memory leakage. JavaScript garbage collectors are not so good at freeing memory from closures, and therefore, it’s a potential place of memory leakage. That’s why, if your objects have a short lifetime, you need to provide functionality for manual memory clean up (like setting all variables to null).</p><h3>ES6 Symbol</h3><p>Symbols are a new primitive type in ES6. This data type can be used as the key for an object property when the property is intended to be private, for the internal use of a class or an object type:</p><pre>const _NAME = Symbol(&#39;name&#39;);</pre><pre>class User {<br>    constructor(name) {<br>        this[_NAME] = name;<br>    }</pre><pre>    getName() {<br>        return this[_NAME];<br>    }</pre><pre>    setName(value) {<br>        this[_NAME] = value;<br>    }<br>}</pre><p>This approach has several benefits. It’s rather simple and fast. Similar to closure based approach, it gives full restriction of direct access to the internal object’s state (although, you can apply some kind of meta programming by using getOwnPropertySymbols to get all object’s symbols). It also doesn’t have any issues with potential memory leaks, since symbol instances are part of an object.</p><p>But ES6 Symbol based approach has also some drawbacks. It’s only compatible with the latest browsers, if you want to support older browsers, you will need to use shims. Also, it needs to hide references to symbol instances in order to protect internal data and make it private for real; perfect place for it is ES6 module closure.</p><h3>Conclusion</h3><p>I have used ES6 Symbols for more than 2 years in all my projects and they work like a charm — it’s easy to read and understand. Symbols are also fast and memory safe in terms of leaks. Until the proposal which defines the use of private fields gets approved, and we see any Babel implementations of it, I recommend using ES6 Symbols.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb01d3128fae" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>