<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ricardo Araújo on Medium]]></title>
        <description><![CDATA[Stories by Ricardo Araújo on Medium]]></description>
        <link>https://medium.com/@rjbma?source=rss-810d4100bcc7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*DHHOCgslwqvJN10U.jpg</url>
            <title>Stories by Ricardo Araújo on Medium</title>
            <link>https://medium.com/@rjbma?source=rss-810d4100bcc7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 04:10:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rjbma/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Scraping the web: experiments with functional programming in javascript — part II]]></title>
            <link>https://medium.com/front-end-weekly/scraping-the-web-experiments-with-functional-programming-in-javascript-part-ii-ab57d172f467?source=rss-810d4100bcc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab57d172f467</guid>
            <category><![CDATA[scraping]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[functional-programming]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Ricardo Araújo]]></dc:creator>
            <pubDate>Thu, 28 Dec 2017 12:19:11 GMT</pubDate>
            <atom:updated>2017-12-28T12:19:11.794Z</atom:updated>
            <content:encoded><![CDATA[<p>In <a href="https://medium.com/@rjbma/scraping-the-web-experiments-with-functional-programming-in-javascript-part-i-56ec7c40b297">Part I</a> we went through the fundamental constructs of functional programming, at least the ones that we’ll now be using to build our scraper. Now, we can start building it!</p><h3>What we want to build</h3><p>So, our first goal is to build a scraper to obtain movie information from <a href="http://www.imdb.com/">IMDB</a>, stuff like title, summary, director’s name, etc. After that, we’ll want to generalize our scraper and turn it into a library so we can use it to do scrape <em>any</em> web page.</p><p>OK, <em>some</em> web pages.</p><p>Yeah, it will probably be useless.</p><h3>Libraries we’ll use</h3><p>If you read part I, you probably already guessed that one of the libraries we’ll use is <a href="http://ramdajs.com/">Ramda</a>. It’s a utility library that’s specifically designed for functional programming. Other excellent alternatives would be <a href="https://github.com/lodash/lodash/wiki/FP-Guide">Lodash/fp</a> or <a href="https://sanctuary.js.org/">Sanctuary</a> (and, no doubt, many others I don’t even know about).</p><p>As cool as it is, Ramda does not provide us the basic types that we talked about previously: <strong>Maybe</strong>, <strong>Either</strong> and <strong>Future</strong> (note however that we’ll be using <strong>map</strong> and <strong>chain</strong> from Ramda). We’ll use <a href="https://sanctuary.js.org/">Sanctuary</a> for <strong>Maybe</strong> and <strong>Either</strong>, and <a href="https://github.com/fluture-js/Fluture">Fluture</a> for <strong>Future</strong>.</p><p>For fetching the web page we’ll just use <a href="https://github.com/request/request">Request</a>. Do note that this means our scraper will not work for pages with dynamic content, specifically it won’t work for single page apps (SPAs). SPAs will typically load a static HTML page and then let Javascript kick in and load the bulk of the page dynamically. But for that to work you need a web browser (or something that can mimic one, like <a href="http://phantomjs.org/">PhantomJS</a>). With <strong>Request</strong> tough, we will only do a single request to the server, meaning we won’t get past the static HTML page. Bummer.</p><p>So, <strong>Request </strong>will help us get the HTML for a specific URL. For extracting information from elements of that HTML we will use <a href="https://github.com/cheeriojs"><strong>Cheerio</strong></a>, which lets us use jQuery selectors to get the values we need.</p><p>In summary, these will be our external imports:</p><pre>import request from &#39;request&#39;<br>import cheerio from &#39;cheerio&#39;<br>import R from &#39;ramda&#39;<br>import S from &#39;sanctuary&#39;<br>import Future from &#39;fluture&#39;</pre><p>Most of the times, I will not be describing the functions that I use from these libraries. Hopefully, their names will be more or less self-explanatory; if that’s not the case, you are hereby encouraged to take a detour and check the online documentation for them.</p><h3>The main scraper function — part I</h3><p>Our main function should receive an URL and give back an object that describes a movie. Let’s start by defining the type signature of this function:</p><pre>scrapeUrl :: String -&gt; Object</pre><p>That seems overly generic, not very descriptive, and honestly not all that helpful. For example, if we pass in a random string, are we really expecting to get back information about a movie? And what about the shape of our Object? Let’s do better:</p><pre>type alias Url = String</pre><pre>type alias Movie = {<br>  title: String,<br>  summary: String,<br>  year: String,<br>  director: String<br>}</pre><pre>scrapeUrl :: Url -&gt; Movie</pre><p><em>(Mind you that these are just comments, it’s not valid Javascript, so we can basically write what we want here, just as long as it proves helpful to us, and to whoever has to read our code.)</em></p><p>Although this is much better, it still has some problems. For example, supposing that we pass in an Url, is that any guarantee that we’ll be able to get a movie description back? Of course not! Lots of things can go wrong, especially if you give it <a href="http://www.everydayim.com/">this</a>.</p><p>So, if we give it an Url to the wrong page, we want our function to fail. It’s a good opportunity to use our friend <strong>Either</strong>! Our function will either give us back a <strong>Movie</strong> or an <strong>Error</strong>.</p><pre>type alias Url, Movie...</pre><pre>// we could also use something like <a href="https://github.com/hapijs/boom">Boom</a>, for example<br>type alias Err = { msg: String }</pre><pre>scrapeUrl :: Url -&gt; Either Err Movie</pre><p>OK, so immediately we now know that our scraper can fail <em>synchronously</em>. But if our network connection is unavailable? Or the website is down? That’s an <em>asynchronous </em>error that we are not contemplating in our code. For that, we can use <strong>Future</strong>:</p><pre>type alias Url, Movie, Future...</pre><pre>scrapeUrl :: Url -&gt; Future Err (Either Err Movie)</pre><p>Ouch! This is getting ugly fast.</p><p>So, it’s kind of cool to know that our function can fail due to several problems, but seems to me like there’s too much detail there.</p><p>Honestly, when using this function I only need to know if it failed or not. I don’t really care if it failed due to network, or invalid website, or whatever! Let’s ditch the <strong>Either</strong> and keep only the <strong>Future </strong>(more on this later):</p><pre>type alias Url, Movie, Future...</pre><pre>scrapeUrl :: Url -&gt; Future Err Movie</pre><h3>Adapting the external libraries</h3><p>OK, so after all this work we got a function signature that takes an <strong>Url</strong> and gives back a <strong>Future</strong>. The implementation should be simple: (1) use <strong>Fetch</strong> to the HTML and (2) use <strong>Cheerio</strong> to turn that HTML into a <strong>Movie</strong> object.</p><p>The problem is that <strong>Fetch</strong> and <strong>Cheerio</strong> don’t use the functional style that we’re imposing in our program, and as such, they won’t compose easily with our own functions. We’ll have to build adapters for the parts of those libraries that we want to use; a functional API of sorts.</p><p>In case of <strong>Fetch</strong>, we just want to always get the HTML wrapped in a <strong>Future</strong>. <strong>Fetch</strong> allows us to work both with callbacks and promises. Here I chose callbacks:</p><pre>// type alias Html = String</pre><pre>// getHtml :: Url -&gt; Future Err Html<br>export const getHtml = url =&gt; Future((rej, res) =&gt; {<br>  const opts = { method: &quot;GET&quot;, uri: url };</pre><pre>  request(opts, (error, response, body) =&gt; {<br>    if (error) {<br>      rej(error);<br>    } else {<br>      res(body);<br>    }<br>  });<br>});</pre><p>Now we never have to deal with callbacks again! For <strong>Cheerio</strong>, it’s going to take a little bit more work because we need it to perform several operations for us when scraping a web page. At a minimum we need to:</p><ol><li>from an <strong>Html</strong> string we want to build a <strong>Cheerio</strong> object (let’s name that type <strong>DOM</strong>)</li><li>from <strong>DOM</strong> and a jQuery selector (let’s name that <strong>Selector</strong>) we want to get a list of elements</li><li>from <strong>DOM</strong> and a <strong>Selector</strong> we want to get a single element (<strong>Just</strong> the first one, if it exists; <strong>Nothing </strong>otherwise)</li><li>from a <strong>DOM</strong> and the name of an attribute, we want to get the value of that attribute</li><li>from <strong>DOM </strong>we want to get its text content</li></ol><p>That’s a lot of functions for us to adapt Cheerio for our functional programming style:</p><pre>// type alias Html = String<br>// type alias DOM = Object // unknown structure, defined by Cheerio<br>// type alias Selector = String</pre><pre>// 1. loadDom :: Html -&gt; DOM<br>const loadDom = cheerio.load.bind(cheerio)</pre><pre>// 2. selectAll :: Selector -&gt; DOM -&gt; List DOM<br>// get all the elements that match the selector<br>export const selectAll = R.curry((sel, dom) =&gt; {<br>  const res = dom(sel)</pre><pre>  // we want all selected elements to also be valid Dom elements<br>  // that way, we&#39;ll be able to query them the same way a full doc<br>  return R.map(cheerio, res.toArray())<br>})</pre><pre>// 3. selectFirst :: Selector -&gt; DOM -&gt; Maybe DOM<br>// get the first selector that matches<br>export const selectFirst = R.curry((sel, dom) =&gt;<br>  R.compose(S.toMaybe, R.head, selectAll(sel))(dom)<br>)</pre><pre>// 4. attr :: String -&gt; Dom -&gt; Maybe String<br>export const attr = R.curry((attrName, dom) =&gt;<br>  R.compose(R.map(R.trim), S.toMaybe, dom.attr.bind(dom))(attrName)<br>)</pre><pre>// 5. text :: Dom -&gt; String<br>export const text = dom =&gt; R.trim(dom.text())</pre><p>Note how we use <strong>selectAll</strong> in order to define <strong>selectFirst</strong>, and wrap the result in a <strong>Maybe</strong> simply because it may not exist. Note that <strong>S.toMaybe</strong> wraps a value in a <strong>Just</strong>, or returns <strong>Nothing</strong> if we give it <strong>null</strong> or <strong>undefined</strong>.</p><h3>Scraping a single movie</h3><p>With our new <strong>Cheerio</strong> functional API, we are now ready to turn a <strong>DOM</strong> representation into a <strong>Movie</strong>. Similarly to what we did with <strong>scrapeUrl</strong>, let’s start with the function signature:</p><pre>// decodeMovie :: DOM -&gt; Either Err Movie</pre><p>That looks neat. Parsing a movie from any old DOM element can obviously fail, so this should return an <strong>Either</strong>.</p><blockquote><strong>Side note</strong>: notice one recurring pattern here? our functions can return values <em>wrapped </em>inside some functor like <strong>Either</strong>, <strong>Future </strong>or <strong>Maybe</strong> whenever it makes sense. That’s not a problem because we already know how to compose them with other functions: we’ll use <strong>map</strong> and <strong>chain</strong>. That’s our main reason to use functors and monads: being able to compose our compose our functions!</blockquote><blockquote>However, very rarely will our functions take these wrapped values as arguments; we want our functions to be as simple as possible. If I say <strong>decodeMovie</strong> takes a DOM element, it means it must be called with one no matter what. If that DOM element may not be there and it’s wrapped in a <strong>Maybe</strong>, then <strong>map</strong> will make sure this function will not get called. This allows our functions to as simple as they can be, without needing to check if the <strong>Maybe </strong>is a <strong>Just </strong>or a <strong>Nothing</strong>, for example. That kind of plumbing is handled before the function gets called, and best of all: we don’t have to handle it ourselves.</blockquote><p>We already defined a <strong>Movie</strong> as an object with four fields: <strong>title</strong>, <strong>summary</strong>, <strong>year</strong> and <strong>director</strong>. Let’s assume that all these fields are required except for the <strong>year</strong>. This means that if any of the other fields isn’t found by our scraper, we want it to fail (meaning <strong>decodeMovie</strong> will return a <strong>Left</strong>), otherwise it will succeed. In particular, if the <strong>year </strong>isn’t found for some reason the function will succeed, but that field will be an empty string.</p><p>Let’s start by building a small function that uses our new <strong>cheerio </strong>API for getting the text of a <em>required</em> element. We should also specify the error message, in case that element can’t be found.</p><pre>// required :: Err -&gt; Selector -&gt; Either Err String<br>const required = R.curry((err, selector) =&gt; R.compose(<br>  S.maybeToEither(err), // Either Err String<br>  R.map(C.text), // Maybe String<br>  C.selectFirst(selector)) // Maybe Dom<br>)</pre><p>Remember that <strong>R.compose</strong> reads from right-to-left, on in this case bottom-up</p><blockquote><strong>Tip</strong>: if you don’t like this particular ordering, you can also use <strong>R.pipe</strong> which is exactly the same as <strong>R.compose</strong> but the functions are specified in the reverse — perhaps more natural? — order.</blockquote><blockquote>Also, I included some comments to make it clear what type we get at the end of each line. I will be doing this in most functions that define this type of pipeline from now on.</blockquote><p>So, this function selects the first element that matches the given jQuery selector and extracts its text content. However, remember that the selector function from our <strong>cheerio</strong> API returns <strong>Maybe</strong>s instead of <strong>Either</strong>s, which makes sense on its own, but it’s not what we’re looking for here. Fortunately, our functional toolset (in this case, <strong>Sanctuary</strong>) gives us the means to easily convert a <strong>Maybe</strong> to an <strong>Either</strong>. We just need to provide what we want to put in the <strong>Left </strong>of the <strong>Either </strong>when our <strong>Maybe</strong> has <strong>Nothing</strong> (hopefully that’s not too confusing by now), which in this case is our error value.</p><blockquote><strong>Side note</strong>: Whenever we’re transforming one functor into another (in this case, from <strong>Maybe</strong> to <strong>Either</strong>) that is called a <a href="https://en.wikipedia.org/wiki/Natural_transformation">natural transformation</a> (seriously though, don’t open that link). It’s possible to define natural transformations for a lot of functors, such as from <strong>Either</strong> to <strong>Future</strong>, <strong>Either</strong> to <strong>Maybe</strong>, etc.</blockquote><blockquote>It’s very interesting to compare <strong>map</strong> and <strong>chain</strong> with natural transformations: whereas both <strong>map</strong> and <strong>chain</strong> allow us to change the value that’s inside the functor/monad, but not change the functor itself, a natural transformation does just the opposite: the wrapped value stays the same, and the wrapper is changed.</blockquote><p>Now for a function that does the same for an optional field:</p><pre>// optional :: String -&gt; Selector -&gt; String<br>const optional = R.curry((defaultValue, selector) =&gt; R.compose(<br>  S.fromMaybe(defaultValue), // String<br>  R.map(C.text), // Maybe String<br>  C.selectFirst(selector)) // Maybe Dom<br>)</pre><p>Notice there is no need to return an <strong>Either</strong> here: the function cannot fail, worst case scenario it should just return the <strong>defaultValue</strong>.</p><p>Now we can finally start building our movie decoder:</p><pre>// decodeMovie :: DOM -&gt; Either Err Movie<br>const decodeMovie = dom =&gt; {<br>  const obj = {<br>    title: required(&#39;what the hell is it named?&#39;, <br>                    &#39;.title_block .title_wrapper h1&#39;)(dom),<br>  <br>    summary: C.required(`Don&#39;t know what it&#39;s about!`,<br>                        &#39;.summary_text&#39;)(dom),</pre><pre>    year: optional(&#39;&#39;, &#39;.title_block #titleYear a&#39;)(dom),<br>  <br>    director: required(&#39;Could not find director&#39;,<br>                   &#39;.plot_summary span[itemprop=director]&#39;)(dom),</pre><pre>}</pre><p>This is now the shape of our object:</p><pre>{<br>  title: Either Err String,<br>  summary: Either Err String,<br>  year: String,<br>  director: Either Err String,<br>}</pre><p>Clearly not what we want. To solve this, we need to first make sure that all our fields are <strong>Either</strong>s:</p><pre>const decodeMovie = dom =&gt; {<br>  const obj = {<br>    title: ...,<br>    summary: ...,<br>    year: S.Right(optional(&#39;&#39;, &#39;.title_block #titleYear a&#39;)(dom)),  <br>    director: ...,<br>}</pre><p>Easy part done. We know we always have a value for the <strong>year</strong>, so we can just wrap in a <strong>Right</strong>. Now we need to somehow unwrap our individual fields and wrap our whole object in an <strong>Either</strong>. The idea being that if at least one of our fields is a <strong>Left</strong>, then the whole object will be wrapped in a <strong>Left</strong>. Else, it will be wrapped in a <strong>Right </strong>(success!).</p><p>Good news first: every time we have a value wrapped inside two special kinds of functors (for the record, traversable and applicative) we can flip the order in which they wrap that value (pretty obscure, I know, but tremendously useful in many cases, as we’ll soon see).</p><p>For this, we can use <strong>Ramda</strong>’s <strong>sequence</strong> function. Example straight from the documentation (here we want to “flip” an <strong>Array </strong>with a bunch or <strong>Maybe</strong>s inside, to get a <strong>Maybe</strong> with an <strong>Array</strong> inside it):</p><pre>R.sequence( Maybe.of, [Just(1), Just(2), Just(3)] );<br>//=&gt; Just([1, 2, 3])</pre><pre>R.sequence(Maybe.of, [Just(1), Just(2), Nothing]); <br>//=&gt; Nothing</pre><p>The first argument is a function that teaches <strong>R.sequence</strong> how to build a <strong>Maybe</strong>. And <strong>R.sequence</strong> just knows how to traverse an <strong>Array</strong>! Note that if there’s a <strong>Nothing</strong> somewhere, then the whole thing will be a <strong>Nothing</strong>. The same thing happens with <strong>Either/Left</strong>, or with <strong>Future</strong>: one failed <strong>Future </strong>is all it takes for <strong>R.sequence</strong> to give us back another failed <strong>Future</strong>.</p><p>Now for the bad news: object (i.e., our <strong>Movie</strong>) is not a valid traversable, and so <strong>R.sequence</strong> doesn’t know how to traverse its fields. Fortunately, it’s pretty easy to do it ourselves: we can turn the object’s values into an array, map <strong>R.sequence </strong>to it, and zip the object back up:</p><pre>// sequenceObject :: (* -&gt; f *) -&gt; Object (f *) -&gt; f (Object *)<br>// This is basically R.sequence, but for objects<br>const sequenceObject = R.curry((appl, obj) =&gt; {<br>  // e.g. obj = {title: Maybe(1), summary: Maybe(2), year: Maybe(3)}<br>  <br>  const keys = R.keys(obj) <br>  // e.g. [&#39;title&#39;, &#39;summary&#39;, &#39;year&#39;]</pre><pre>  const wrappedValues = R.values(obj) <br>  // e.g.[Maybe(1), Maybe(2), Maybe(3)]</pre><pre>  const unwrappedValues = R.sequence(appl, wrappedValues) <br>  // e.g. Maybe([1,2,3])</pre><pre>  return R.map(R.zipObj(keys))(unwrappedValues) <br>  // e.g. Maybe({ title: 1, summary: 2, year: 3 })</pre><pre>})</pre><p>Notice that, if <strong>unwrappedValues</strong> becomes a <strong>Nothing</strong>, the last <strong>R.map</strong> that zips the object back up would be a no-op and return <strong>Nothing</strong>. We didn’t even have to check for failures. Sweet!</p><p>Now we can finish our <strong>decodeMovie</strong> function:</p><pre>// decodeMovie :: DOM -&gt; Either Err Movie<br>const decodeMovie = dom =&gt; {<br>  const obj = {<br>    title: required(&#39;what the hell is it named?&#39;, <br>                    &#39;.title_block .title_wrapper h1&#39;)(dom),<br>  <br>    summary: C.required(`Don&#39;t know what it&#39;s about!`,<br>                        &#39;.summary_text&#39;)(dom),</pre><pre>    year: S.Right(optional(&#39;&#39;, &#39;.title_block #titleYear a&#39;)(dom)),<br>  <br>    director: required(&#39;Could not find the director&#39;,<br>                   &#39;.plot_summary span[itemprop=director]&#39;)(dom),</pre><pre>  }</pre><pre>  return sequenceObject(S.of(S.Either), obj)<br>}</pre><h3>The main scraper function — part II</h3><p>Now, we’re finally ready to build the <strong>scrapeUrl</strong> function we defined earlier:</p><pre>// scrapeUrl :: Url -&gt; Future Err Movie<br>const scrapeUrl = <br>  R.compose(<br>    R.chain(U.eitherToFuture),<br>    R.map(decodeMovie),<br>    R.map(C.loadDom),<br>    U.getHtml<br>  )<br></pre><p>By now, this should look familiar. Our function takes an URL, loads the result into a Dom object, decodes the movie, and flattens the result so that we don’t get an <strong>Either</strong> inside a <strong>Future</strong> (notice that <strong>U.eitherToFuture </strong>returns a <strong>Future</strong>; if we used <strong>map</strong> instead of <strong>chain</strong> here we would have gotten a <strong>Future </strong>inside another <strong>Future</strong>). So it defines a pipeline with the following types:</p><pre>URL =&gt; Future Html =&gt; Future Dom =&gt; Future (Either Movie) =&gt; Future Movie</pre><p>To use this we just have to <a href="https://github.com/fluture-js/Fluture#fork">fork</a> the <strong>Future</strong>:</p><pre>scrapeUrl(&#39;<a href="http://www.imdb.com/title/tt0089695">http://www.imdb.com/title/tt0089695</a>&#39;)<br>  .fork(console.error, console.log)</pre><p>Here, we’re just logging the results to the console.</p><h3>Generalising our scraper</h3><p>Now we have a scraper that works for IMDB movie pages. What if we want to get all the movies from a specific actor? We need to get the list of movies from the actor page, which means we have to scrape that page. Let’s change our <strong>scrapeUrl</strong> function to be able to adapt to any page:</p><pre>// scrapeUrl :: (Dom -&gt; Either Err a) -&gt; Url -&gt; Future Err a<br>const scrapeUrl = R.curry((strategy, url) =&gt; <br>  R.compose(<br>    R.chain(U.eitherToFuture),<br>    R.map(strategy),<br>    R.map(C.loadDom),<br>    U.getHtml<br>  )(url)<br>)</pre><p>In the spirit of <a href="https://youtu.be/E8I19uA-wGY?t=1097">parameterizing all thing</a>, we’ll pass a function to our scraper that knows how to turn some <strong>Dom</strong> into the data we need (or an error). Note how the type of this data must match with the data inside the <strong>Future</strong>.</p><p>By the way, we’ve been using <strong>R.compose</strong> a lot to build our functions, but that’s not the only option we have to build our function pipelines. Using <strong>R.compose</strong> (or <strong>R.pipe</strong>) promotes a programming style known as <a href="https://en.wikipedia.org/wiki/Tacit_programming"><em>point-free style</em></a>, where we can define our functions without ever naming their arguments. However, many people tend to think this code is difficult to read and understand (like that’s of any importance…). It’s perfectly OK to write our functions explicitly instead:</p><pre>const scrapeUrl = R.curry((strategy, url) =&gt; {<br>  const html = U.getHtml(url)<br>  const dom = R.map(C.loadDom, html)<br>  const res1 = R.map(strategy, dom)<br>  const res2 = R.chain(U.eitherToFuture, res1)<br>  return res2<br>}</pre><p>Of course, having to name all the things sucks sometimes, as my last two constants show… Luckily, there’s an even better, and at least equally readable alternative, at least when we just need to <strong>map</strong> or <strong>chain</strong> stuff: our beloved types <strong>Maybe</strong>, <strong>Either</strong> and <strong>Future</strong> (as well as any other functor or monad) are also javascript objects that have these functions available as methods (<em>hard-core functional programmer rolls eyes</em>):</p><pre>const scrapeUrl = R.curry((strategy, url) =&gt;<br>  U.getHtml(url)<br>    .map(C.loadDom)<br>    .map(strategy)<br>    .chain(U.eitherToFuture))</pre><p>This looks very clean and natural, at least in javascript. In some other languages, where every function is curried by default, we would use the pipe operator (usually |&gt; or &lt;|) to do something similar. For example, in <strong>Elm</strong> it could look something like this:</p><pre>getHtml url<br>  |&gt; Task.map loadDom<br>  |&gt; Task.map strategy<br>  |&gt; Task.andThen eitherToFuture<br></pre><pre>-- <strong>Task</strong> is equivalent to <strong>Future</strong><br>-- <strong>andThen</strong> is equivalent to <strong>chain</strong><br>-- <strong>Elm</strong> doesn&#39;t have type classes, so that means there isn&#39;t a single <strong>map</strong> interface. I.e., there&#39;s a <strong>Task.map</strong>, <strong>List.map</strong>, <strong>Maybe.map</strong>, etc.</pre><p>Anyway, just use whatever whatever option works best for you. Indeed, you can even mix and match styles:</p><pre>const scrapeUrl = R.curry((strategy, url) =&gt;<br>  U.getHtml(url)<br>    .map( R.compose(strategy, C.loadDom) )<br>    .chain(U.eitherToFuture))</pre><p>By the way, we haven’t really covered this, but here we used one of the <a href="https://en.wikibooks.org/wiki/Haskell/The_Functor_class">functor laws</a> (<em>creepy music plays in the background</em>) that says that composing two mapped functions is the same as mapping one composed function. This is <strong>always</strong> true, and we can apply this law blindly, without even knowing what those functions do. Really cool stuff: imagine you have a large array and you want to apply two functions over it. You can just compose those two functions and map the new function on the array once. In some languages, the compiler can actually perform these optimizations on the code automatically!</p><h3>Scraping a list of movie URLs</h3><p>OK, back to reality. We can know scrape any page, so now we’re ready to get a list of URLs of movies for a single actor. That should be fairly straightforward:</p><pre>// decodeActorMovies :: Dom -&gt; Either Err (List Url)<br>const decodeActorMovieUrls = dom =&gt; {</pre><pre>  const actorMoviesSelector = &#39;#filmography #filmo-head-actor + .filmo-category-section .filmo-row b a&#39;</pre><pre>  const arr = R.compose(<br>    R.sequence(S.of(S.Either)), <br>    // Either Err (List String)</pre><pre>    R.map(S.maybeToEither(&#39;Invalid Url&#39;)), <br>    // List (Either Err String)</pre><pre>    R.map(R.map(x =&gt; &#39;http://www.imdb.com&#39; + x)), <br>    // List (Maybe String)</pre><pre>    R.map(C.attr(&#39;href&#39;)), <br>    // List (Maybe String)</pre><pre>    C.selectAll(actorMoviesSelector) <br>    // List Dom<br>  )(dom)</pre><pre>  return arr<br>}</pre><p>We select all links for movies where this person has acted with some CSS selector, and get its <strong>href</strong> attribute. At this point we get a list of maybes, so we know we’ll have to <strong>mapmap</strong>!</p><p>Then, since those URLs are relative, we have to turn them into absolute URLs by prefixing the server address, then we turn the <strong>Maybe</strong>s into <strong>Either</strong>s because we want to specify an error message in case something goes wrong.</p><p>Finally, we just use our friend <strong>sequence </strong>in order to “flip” the result from a <strong>List</strong> of <strong>Eithers</strong> into an <strong>Either</strong> of <strong>List</strong>.</p><h3>Putting it all together</h3><p>Finally, we’re ready to scrape all the movies of one actor in one go:</p><pre>const actorPage = &#39;http://www.imdb.com/name/nm0000241&#39;</pre><pre>scrapeUrl(decodeActorMovieUrls, actorPage)<br>  // Future Err (List Url)</pre><pre>  .map(R.map(scrapeUrl(decodeMovie)))<br>  // Future Err (List (Future Err Movie))</pre><pre>  .chain(R.sequence(Future.of))<br>  // Future Err (List Movie)</pre><pre>  .fork(e =&gt; {<br>    // e :: Err<br>    console.error(e)</pre><pre>  }, x =&gt; {<br>    // x :: List Movie<br>    console.log(x)</pre><pre>  })</pre><blockquote><strong>By the way</strong>, notice we opted to use method chaining here instead <strong>R.compose</strong>. It really is just a matter of preference.</blockquote><p>We first scrape the URLs from the page, then we scrape each one of those movie pages. Then all that’s left is to use <strong>R.sequence</strong> and <strong>chain </strong>to get the data in the format we want, and fork the <strong>Future</strong>. Since <strong>Future</strong>s are lazy, it’s only when we fork that the requests are performed on the network. Up until <strong>fork</strong>, we’ve only been dealing with pure functions. That is one of the mantras of functional programming: since you can’t live without side effects, you should at least try to push them to the edges of your program.</p><p>It’s interesting to note that we only forked once, and that single fork is responsible to set everything else in motion. In particular, we only needed a single <strong>fork</strong> to perform the request that fetches the list of movie URLs <em>and</em> each subsequent request for the details of each movie.</p><blockquote><strong>Side note</strong>: By using sequence the way that we do here, if the scraper fails even if for a single movie, then the whole thing will fail. We get nothing back. Which kinda sucks.</blockquote><blockquote>One way of solving this is to transform each individual <strong>Future</strong> into an <strong>Either</strong>, and then handle errors and successes accordingly. More information <a href="https://github.com/fluture-js/Fluture#parallel">here</a>.</blockquote><h4>Performance</h4><p>If we run this for an actor that stars in a lot of movies, we’ll see that the whole process is somewhat slow. That happens because of the way <strong>R.sequence</strong> works: it goes sequentially through the list of URLs and handles each one at a time! Surely, we can speed things up if only we can parallelize some of those requests.</p><p>Fortunately, <strong>Fluture</strong> comes to the rescue: <a href="https://github.com/fluture-js/Fluture#parallel"><strong>Fluture.parallel</strong></a> works just like <strong>R.sequence</strong>, but takes an extra argument that dictates how many futures can be running at the same time. For example, if we specify 10, <strong>Fluture</strong> will start running 10 futures, and as soon as one finishes it will spin up another one. And so on, until they’re all finished. We basically get this parallel functionality for free, almost without changing our code. Pretty neat!</p><pre>scrapeUrl(decodeActorMovieUrls, actorPage)<br>  .map(R.map(scrapeUrl(decodeMovie)))<br>  .chain(R.parallel(10))<br>  .fork(console.error, console.log)</pre><p><strong>Very important</strong>: remember not to set this value too high because you don’t want to put too much stress on the server. Scrape responsibly!</p><h3>Conclusion</h3><p>Developing software (as well as solving almost any other problem) has always been about decomposing some problem into smaller pieces, smaller problems that are easier to solve on their own, and then composing those solutions back together.</p><p>What I find nice about this functional programming style is that it almost forces us to write very small, very focused, and thus very reusable functions. Consequently, these functions are also very easy to test, (almost) without any mocking (we didn’t go into testing, but you can check the GitHub <a href="https://github.com/rjbma/curry-digger">repo</a> for some examples). These functions become the basic building blocks of many potentially different applications.</p><p>All in all, this was a really fun <em>little</em> project. It wasn’t all roses though, as there were some bumps along the way, such as</p><ul><li>Javascript is not type checked. And although many people don’t like it, I think it really would help in this case. Even for such a simple problem like building our scraper, I must admit I found myself struggling from time to time to make sure the types align correctly when composing functions. A type checker would prove valuable here, but due to the dynamic nature of Javascript, and heavy use of currying in this programming style, I doubt compile-time type-checking (like <a href="https://www.typescriptlang.org/">TypeScript</a> of <a href="https://flow.org/">Flow</a>) will exist in the near future for this kind of thing. Note however that <a href="https://sanctuary.js.org/#type-checking">Sanctuary</a> does provide run-time type checking in case you want to try it out.</li><li>Sometimes, especially when using point-free style, if an error occurs, the stack trace will be completely obfuscated. Worst case scenario, you will have no idea where the error has occurred! For more information on this problem and how to solve it, take a look at this <a href="https://hackernoon.com/partially-applied-curried-functions-could-obfuscate-the-javascript-stack-trace-84d66bd8032e">article</a>.</li></ul><h3>Link-throwing</h3><p>Here are some useful links:</p><ul><li>the GitHub <a href="https://github.com/rjbma/curry-digger">repo</a> for this post</li><li><a href="https://egghead.io/courses/professor-frisby-introduces-composable-functional-javascript"><strong>Professor Frisby Introduces Composable Functional JavaScript</strong></a> — free course on <a href="http://egghead.io/">Egghead.io</a>. This is a quick paced introduction to this stuff (and more), and really fun to watch. I think this was the first time I ever saw anything about this topic on javascript, and it made me want to look for more.</li><li><a href="https://github.com/MostlyAdequate/mostly-adequate-guide">Mostly adequate guide to FP (in javascript)</a> — free online book, also from <strong>Professor F. Risby</strong> aka <a href="https://medium.com/u/a3129fead82b">Brian Lonsdorf</a>. Hell, just watch the guy’s videos on <a href="https://www.youtube.com/results?search_query=brian+lonsdorf">YouTube</a>.</li><li><a href="http://adit.io/posts/2013-04-17-functors,_applicatives,_and_monads_in_pictures.html"><strong>Functors, Applicatives, And Monads In Pictures</strong></a> — this stuff, in pictures. Nice!</li><li>If you have the time and curiosity (which you should), check out Bartosz Milewski’s free course <a href="https://www.youtube.com/watch?v=I8LbkfSSR58&amp;list=PLbgaMIhjbmEnaH_LTkxLI7FMa2HsnawM_">Category Theory for programmers</a> on YouTube. You won’t see much code in it, but it really puts these things in a different perspective.</li></ul><p>And of course, <strong>learn other languages</strong>. Let’s be honest. This hasn’t been the most idiomatic javascript ever written. The fact is that this stuff looks and works a lot better in other programming languages.</p><p>Just look at the amount of effort that’s been put into trying to turn javascript into a more FP-friendly language:</p><ul><li><a href="https://facebook.github.io/immutable-js/"><strong>Immutable.js</strong></a><strong> </strong>— immutable data structures</li><li><strong>TypeScript</strong> and <strong>Flow</strong> — compile-time type checking</li><li><strong>Ramda, Sanctuary, Folktale, Ramda-fantasy, …</strong>— functional style utility libraries, algebraic data types implementations, etc.</li><li>To some extent, even <strong>React</strong> and <strong>Redux</strong> are also trying to bring stuff from the FP community to the front-end</li><li>and many others, I’m sure</li></ul><p>Don’t get me wrong: these libraries are great! The thing is, most purely functional languages give you this stuff and more for free. It’s worth checking them out, if nothing else for learning purposes.</p><ul><li><strong>Haskell</strong>: <a href="http://learnyouahaskell.com/">Learn You a Haskell for Great Good</a> seems to have become the de facto standard in learning both Haskell and functional programming. Give it a try!</li><li><a href="http://elm-lang.org/"><strong>Elm</strong></a>: It’s becoming one of the programming languages of choice for the front-end. Also be sure to check out <a href="https://clojurescript.org/">ClojureScript</a>, <a href="http://www.purescript.org/">PureScript</a> or <a href="https://reasonml.github.io/">Reason</a>.</li><li><strong>F#</strong>: I’ve never coded a single line of F#, but <a href="https://fsharpforfunandprofit.com/"><strong>F# for fun and profit</strong></a> has some really good content, and most of it can be easily translated into other functional languages. Specifically, stuff about domain-driven design and making it impossible for our software to even represent illegal states. The site is run by <a href="https://medium.com/u/c842d836f25b">Scott Wlaschin</a> who as also some great introductory videos about functional programming on YouTube: <a href="https://www.youtube.com/watch?v=E8I19uA-wGY">1</a> and <a href="https://www.youtube.com/watch?v=Up7LcbGZFuo">2</a>;</li><li>If you’re on the JVM, you can check-out <a href="https://www.scala-lang.org/">Scala</a> and <a href="https://clojure.org/">Clojure</a> in order to have access to the gigantic Java ecosystem</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab57d172f467" width="1" height="1" alt=""><hr><p><a href="https://medium.com/front-end-weekly/scraping-the-web-experiments-with-functional-programming-in-javascript-part-ii-ab57d172f467">Scraping the web: experiments with functional programming in javascript — part II</a> was originally published in <a href="https://medium.com/front-end-weekly">Frontend Weekly</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scraping the web: experiments with functional programming in javascript — part I]]></title>
            <link>https://medium.com/front-end-weekly/scraping-the-web-experiments-with-functional-programming-in-javascript-part-i-56ec7c40b297?source=rss-810d4100bcc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/56ec7c40b297</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[scraping]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[functional-programming]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Ricardo Araújo]]></dc:creator>
            <pubDate>Thu, 28 Dec 2017 12:18:39 GMT</pubDate>
            <atom:updated>2017-12-28T12:18:39.456Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently I needed to scrape some data for a personal project. Of course, there’s a lot of options that I could use for this, from helper libraries for various programming languages to full-blown applications designed specifically for this purpose.</p><p>However, this seemed like a good opportunity to learn something new, so I decided to implement the scraper in javascript using functional programming techniques described in the excellent <a href="https://www.gitbook.com/book/drboolean/mostly-adequate-guide">Professor Frisby’s Mostly Adequate Guide to Functional Programming</a>. This book is written by <a href="https://medium.com/u/a3129fead82b">Brian Lonsdorf</a> and is available online for free. Check it out! You should also see his free <a href="https://egghead.io/courses/professor-frisby-introduces-composable-functional-javascript">egghead.io</a> video course that moves at a faster pace. <strong>Great stuff</strong>.</p><p>Since this will be a long post, I will be dividing it into two parts. Part I will describe (well, mostly enumerate) some fundamental aspects of functional programming, all the way from composing pure functions to describing some useful functors and monads. Then, in <a href="https://medium.com/@rjbma/scraping-the-web-experiments-with-functional-programming-in-javascript-part-ii-ab57d172f467">Part II</a>, we’ll be using that good stuff to incrementally build our scraper.</p><h3>Some fundamentals</h3><p>Before we start coding there are some concepts regarding functional programming that we should grasp, or else we would be moving way too slow. I will be quickly describing these concepts in this section.</p><h4>Pure functions</h4><p>These are functions that:</p><ul><li>given the same set of inputs, always produce the same output. This means they cannot depend on any outside state (i.e., anything that was not on its set of inputs). As a consequence, a pure function without any inputs could also be called a constant!</li><li>They cannot produce any side-effect, like changing some outside state (outside of the function, I mean), writing to a database, or even log to the console.</li></ul><p>Pure functions have some advantages over impure ones, such as:</p><ul><li>easier to test, because we don’t need to mock anything. Provide the inputs, test for expected outputs.</li><li>no side-effects means that you can always safely call the function without thinking for a second if it will destroy the world (or worse, your app). It’s a liberating feeling: no matter how many times I call this function, the result will always be the same, and it won’t destroy or change anything. As a matter a fact, I can safely <a href="https://en.wikipedia.org/wiki/Memoization">memoize</a> this function and never look back!</li></ul><p>We’ll be writing <strong>most</strong> of our functions as pure functions, and try to delay the use of impure functions as much as possible. Note that we always want our program to produce some side-effects (such as show print some output, produce a file, or whatever…), so it would be impossible to use only pure functions.</p><h4>Composing</h4><p>Composing is basically taking two functions and creating a new one that executes the first two in order. For example, given:</p><pre>const add1 = x =&gt; x+1</pre><pre>const mult3 = x =&gt; x*3</pre><p>We can build a new function that does both:</p><pre>const add1mult3 = x =&gt; mult3(add1(x))</pre><pre>add1mult3(5)  // 18</pre><p>We’ll be using <strong>Ramda</strong>’s <strong>compose</strong> function for this (we could also easily make our own. That’s actually a cool exercise!):</p><pre>const add1mult3 = R.compose(mult3, add1)</pre><p>Note that order matters! <strong>R.compose</strong> reads from right to left, so <strong>add1</strong> will be executed first, and <strong>mult3</strong> will be executed with the result from <strong>add1</strong>.</p><p>One (cool?) property of this version is we didn’t even have to come up with a name for the parameter (<strong>x</strong> in the case of the “manual” version). We just make sure the types of both functions match (i.e., the return type of the first is what the second function is expecting) and compose away.</p><p>Composing functions (either this way or the “manual” way) is <strong>the way </strong>to manage the complexity of software. We decompose big and complex problems into smaller ones and build small functions in order to solve those smaller problems. We then compose those functions (i.e., call them in the correct order) to solve the big problems.</p><p>Two additional remarks about <strong>R.compose</strong>:</p><ul><li>we’re not limited to composing two functions at a time. Guess what this does:</li></ul><pre>const add5mult3 = R.compose(mult3, add1, add1, add1, add1, add1)</pre><ul><li>one very important constraint of function composition done this way is that <strong>each function must receive one parameter</strong>, no more, no less. <strong>WHAT?!</strong> That seems rather limiting. Do you really expect me to code functions such as <strong>add1</strong>, <strong>add2</strong>, <strong>add3</strong>. Of course not, for solving that we need:</li></ul><h4>Currying</h4><p>Currying a function creates a new function whose parameters needn’t be supplied all at once. If we supply all them, then the function is immediately executed, as usual. However, if we supply fewer parameters than needed by the function, then a new function is returned, one that is expecting the missing parameters. For example, consider a function that takes three parameters:</p><pre>const volume = (a, b, c) =&gt; a * b * c</pre><pre>// we&#39;ll curry the function using Ramda.curry<br>const curriedVolume = R.curry(volume)</pre><pre>curriedVolume(1,2,3)    // 6<br>curriedVolume(1,2)(3)   // 6<br>curriedVolume(1)(2)(3)  // 6</pre><p>The first time I read about currying I thought “So what? Why da hell would I want to do that?!”. At first sight it seemed rather useless. However, currying is extremely important in order to be able to compose functions with more than one parameter. Let’s look at our previous example with curry in our toolbox:</p><pre>const add = R.curry( (x,y) =&gt; x+y )<br>const mult = R.curry( (x,y) =&gt; x*y )<br>const add1mult3 = R.compose(mult(3), add(1))<br>add1mult3(5)  // 18</pre><p>Much nicer! Now, our functions are much more general, and so our chances of reusing them (by <strong>composing</strong> them with other functions) are far greater.</p><p>One extremely important thing to pay attention to with respect to currying is the order of the parameters when we are defining our curried functions. The <em>main data parameter</em>, if you will, that we want to pass to our function should always be the last parameter, because that will be the thing that gets passed from the previous function in a composition.</p><p>As an example, imagine we’re defining a function that returns the value of a field on a given object. This function has two parameters:</p><ul><li>the name of the field that we want to fetch from the object. This is <strong>not </strong>the <em>data </em>that we’ll pass to the function, but more of a <em>configuration parameter</em>: for example, if we want to, we can create another function that is set up to fetch the <strong>name </strong>field of any object we throw at it</li><li>the object that holds the field. Most of the time, this will be the <em>data </em>that we’ll pass to the function; it’s the variable part, not a <em>configuration parameter</em>.</li></ul><p>With that in mind, we should define this function like this:</p><pre>const getProp = R.curry( (field, obj) =&gt; obj[field] )</pre><p>And we can use it in a composition that reverses the name of some guy:</p><pre>const guy = { name: &#39;Ritchie&#39; }<br>const getName = getProp(&#39;name&#39;)<br>const reverseName = compose(R.reverse, getName)<br>reverseName(guy)  // eihctiR</pre><p>If we had chosen to place obj as the first parameter, we’d be screwed right now, because we couldn’t have <em>configured </em>our function to fetch the name field of any object using currying.</p><blockquote><strong>Side note:</strong> it actually is possible in <strong>Ramda</strong> to <em>configure </em>parameters other than the first using <a href="http://ramdajs.com/docs/#__">R.__</a>. Sometimes we’ll have to, because deciding which parameter is the <em>main data</em> and which ones are <em>configuration parameters</em> may not be simple, and may depend entirely on the context in which we use the function.</blockquote><p>By the way, two of the cooler features about <strong>Ramda</strong> is that its functions are automatically curried, and the order of their parameters is carefully crafted so that they can easily be curried in most situations (once again, depending on the context in which we’re using these functions, that may not always be the case). Other libraries such as <a href="https://github.com/lodash/lodash/wiki/FP-Guide"><strong>Lodash/fp</strong></a> or <a href="https://sanctuary.js.org/"><strong>Sanctuary</strong></a> (and probably others) do the same, so feel free to use any of those.</p><h4>Mapping</h4><p>Probably every javascript developer knows the map function. It simply takes an array and a function and <em>iterates</em> through the array <em>applying the function</em> to each of its elements. In the end, it returns a <em>new array, with the same number of elements</em>, that contains the results of applying the function.</p><p>However, at least for the purpose of this post, <strong><em>map</em></strong><em> will no longer be about iteration;</em> <strong><em>map </em></strong><em>will be about applying a function to a value inside some context</em>.</p><p>The context is what determines <strong>if</strong> and <strong>how</strong> the function is applied. In the case of an array, the array itself is the context and it determines that the function must be applied to all of its elements. Also, if the array is empty, the function is never applied to any values.</p><p>Array is just one such context with which we can use <strong>map </strong>with. There are lots of other interesting contexts, some of which we’ll use for our web scraper, namely <strong>Maybe</strong>, <strong>Either</strong> and <strong>Future</strong>. All these contexts hold (or wrap) some value (like our array which actually holds a bunch of them) but the really interesting thing about them is the way they control how functions get applied on that value.</p><p>One very important about <strong>map</strong> is that it always returns a context of exactly the type. Meaning:</p><ul><li>mapping over an <strong>Array</strong> always produces another array with the exact same number of elements. Note that the values inside the new array may well be of a different type (for example, we can create an array of numbers from an array of strings by mapping the function <strong>toString</strong>), but the new array will always have the same number of elements. No more, no less.</li><li>mapping over a <strong>Maybe</strong> will always produce another maybe (we’ll briefly discuss <strong>Maybes</strong> and other types next)</li><li>and so on</li></ul><p>A type (or context) that can be mapped over is also called a <em>functor</em>.</p><h4>Chaining</h4><p>The <strong>chain</strong> function is very similar to <strong>map</strong> in the sense that it applies a function to a value inside some context. The difference between the two is the type of function that is mapped and how that mapping occurs.</p><p>Consider the following scenario: we have an array of purchase order (PO) numbers. For each order, we want to get its line items, extract the price from each item, and calculate the sum for all POs. This is what we got:</p><pre>// this returns an array of line items. a line item is an object that looks like:<br>// { id: 1, name: &quot;socks&quot;, price: &quot;3&quot;, ... }<br>const getLineItems = poNumber =&gt; ...</pre><pre>// get the price of one line item<br>// this is the same as using <strong>R.prop(&#39;price&#39;)<br></strong>const extractPrice = li =&gt; li.price</pre><pre>// all the PO numbers<br>const poNumbers = [1, 23, 11, 45]</pre><p>The important part here is that getLineItems is a function that returns an array. If we try to map it to our array of poNumbers, we get an array of arrays:</p><pre>const lineItems = R.map(getLineItems, poNumbers)<br>// --&gt; [ [{id:1,price:3,...} , {id:2,price:56,...}]<br>       , [{id:10,price:42,...} , {id:292,price:4,...}]<br>       , [{id:43,price:2,...} , {id:456,price:5000,...}]<br>       ]</pre><p>In this case, it would be better if we got a flatten array because we want to just treat each line item equally, no matter which PO it belongs to. We just want to extract their prices and sum the whole thing.</p><p>We can use <strong>chain </strong>to do that. Like <strong>map</strong>, <strong>chain</strong> also operates on these contexts we’ve been talking about, but takes a function that returns a value that’s also inside the <em>same kind of context</em>. In our example, we use chain on an array, and give it a function that also returns an array. What chain does is <em>collapse </em>the two arrays into one:</p><pre>const lineItems = R.chain(getLineItems, poNumbers)<br>// --&gt; [ {id:1,price:22,...} , {id:1,price:56,...}<br>       , {id:10,price:42,...} , {id:292,price:4,...}<br>       , {id:43,price:2,...} , {id:456,price:5000,...}<br>       ]</pre><p>From there, we can easily complete our requirement:</p><pre>const prices = R.map(extractPrice, lineItems)<br>const total = R.sum(prices)</pre><p>Or, the whole thing in a more concise way (remember that compose reads from right-to-left):</p><pre>const getLineItems = poNumber =&gt; ...<br>const poNumbers = [1, 23, 11, 45]</pre><pre>const calc = R.compose(<br>    R.sum,                     // returns the total price<br>    R.map(R.prop(&#39;price&#39;)),    // returns an array of prices<br>    R.chain(getLineItems))     // returns an array of line items</pre><pre>calc(poNumbers)</pre><p>This way we don’t have to make up names for the intermediate values. Whether it reads better or worse is probably a matter of opinion.</p><p>This is a rather convoluted example, but <strong>chain</strong> is really extremely helpful in many situations where we don’t want to have a value inside two contexts (e.g, array of array, maybe inside maybe, etc.), as we’ll see later.</p><p>Other possible names for <strong>chain</strong> are <strong>bind</strong> and <strong>flatMap</strong>. Also, a type (or context) that can be chained over in this way is also called a <em>monad</em>.</p><h3>Types</h3><p>We’ve been talking about contexts (or types) that contain values, and on which we can <strong>map</strong> and <strong>chain</strong> functions. We also know that <strong>Array </strong>is one of those types. In this section we’ll describe three other types that we’ll use to build our scraper: <strong>Maybe</strong>, <strong>Either</strong> and <strong>Future</strong>.</p><h4>Maybe</h4><p>The <strong>Maybe</strong> type represents a value that may or may not be there. It can only have one of two possible values:</p><ul><li>it can be <strong>Nothing</strong>, in which case it doesn’t have anything else associated with it. If we map or chain over <strong>Nothing</strong>, we always get <strong>Nothing </strong>back</li><li>or it can be <strong>Just</strong> <em>something</em>. In this case, it wraps another value (which can be anything, a string, a number, or even the whole state of your application). If we map over <strong>Just</strong> <em>something</em>, we get back another <strong>Just</strong> wrapping whatever was the result of applying the function to the original value. Note that if we chain over <strong>Just</strong> we can get back a <strong>Nothing</strong>, though!</li></ul><p>Or, in code:</p><pre>// &#39;a&#39; means any type of value, such as string, number, object, etc.<br>// &#39;|&#39; reads as OR<br>// Maybe a = Nothing | Just a</pre><pre>const add1 = x =&gt; x + 1<br>R.map(add1, Just(5))     // Just(6)<br>R.map(add1, Nothing)     // Nothing</pre><p>We are going to be using Maybes to replace null and undefined in our code. Every time there’s a field somewhere that may not have a value, or a function that may return an empty or non-existent value, we’ll use a Maybe.</p><p>The advantage of <strong>Maybe</strong> over <strong>null</strong> or <strong>undefined</strong> is that we can safely apply functions on empty <strong>Maybes</strong> (i.e., <strong>Nothings</strong>) without it blowing up on us. And when it’s time to get the value inside the <strong>Maybe</strong>, we <em>have</em> to take into account the possibility that there is no value, forcing us to handle that case.</p><p><strong>Either</strong></p><p><strong>Either</strong> is similar to <strong>Maybe</strong> in the sense that it has only two possible values: a <strong>Left</strong> and a <strong>Right</strong>. However, in an <strong>Either</strong> both of those values wrap another value:</p><pre>// Either a b = Left a | Right b</pre><pre>const add1 = x =&gt; x + 1<br>R.map(add1, Right(5))              // Right(6)<br>R.map(add1, Left({msg: &quot;Ouch!&quot;}))  // Left(&quot;Ouch!&quot;)</pre><p>The <strong>Right</strong> value is basically the same as <strong>Just</strong>: it represents the “regular” case, one where the functions we pass to <strong>map</strong> will actually get applied.</p><p>The <strong>Left</strong> value, on the other hand, is like the <strong>Nothing</strong> because it ignores the function we try to map on it, and just returns the value it already had.</p><p>Why is this useful? We are going to use <strong>Either</strong> to avoid throwing and catching exceptions in our scrapper! Every time we code a <em>synchronous </em>function that may fail, that function will return an <strong>Either</strong>. That way, if the function fails in the middle of a large composition of functions, any subsequent <strong>mapping</strong> or <strong>chaining</strong> will be no-ops.</p><p>And, in the end, we will be forced to look inside the <strong>Either</strong> to check if all went well or if we had an error. This means that, we can still <strong>choose</strong> to ignore errors that happen in our program, but that will be a conscious decision on our part; it won’t be because we simply forgot to handle the error case!</p><h4>Futures</h4><p>Whereas <strong>Either</strong> will be used to handle <strong>synchronous</strong> functions that may fail, <strong>Future</strong> will be used to handle any <em>asynchronous </em>function (which, of course, can always fail).</p><p>A <strong>Future</strong> is like a <strong>Promise</strong>, with some very important differences:</p><ul><li>A <strong>Promise</strong> is <em>eager</em>, which means it will start doing its stuff (e.g., network connection) right after it’s created. A <strong>Future</strong> is always lazy: you create it, map or chain over it several times and it doesn’t do anything. Only when you call a special method called <strong>fork</strong>, will it actually do its thing</li><li>A <strong>Promise</strong> has a <strong>then</strong> method that’s used to chain operations on the promise. From the <strong>then</strong> method you can return either another promise or a “regular” value. The next method on the chain will receive the value from the previous value. So the <strong>then </strong>method on the promise behaves like our <strong>map</strong> function(in case we return a “regular” value on <strong>then</strong>) or like the <strong>chain</strong> function (in case we return another promise on <strong>then</strong>)</li></ul><p>Before we finally start our scraper, there’s one last thing to talk about that will help us gather important insight about our functions without looking at their code, or even their name: type signatures!</p><h3>Type signatures</h3><p>We’ll use type signatures extensively in our scraper, specifically to describe our functions. A type signature will let us know immediately the inputs and the output of a function.</p><p>As an example, let’s consider the following type signature.</p><pre>// add :: Number -&gt; Number -&gt; Number</pre><p>This means there’s a function named <strong>add</strong>, that takes two numbers and returns another number. Or we could also read it as a function <strong>add</strong> that takes a number and returns another function that takes a number and returns a number (we won’t read functions that way, but it’s good to recognize this is as <strong>currying</strong>!)</p><p>Now, for our purposes we can implement this function in any of the following ways:</p><pre>// uncurried version<br>const add = (x, y) =&gt; x + y</pre><pre>// curried version (with standard arrow functions)<br>const add = x =&gt; y =&gt; x + y</pre><pre>// curried version (using Ramda)<br>const add = R.curry( (x, y) =&gt; x + y )</pre><p>Mind you that neither of those function are identical (can you spot the differences in the ways each version can be used?), but we can treat them as roughly the same function. We’ll use the first version if we think we probably don’t need currying (or if the function has only one value, of course). And we’ll use the third version if we know that we might need currying.</p><p>Type signatures are really helpful because they tell a lot about our functions even without needing to look at their body. For example, can you guess which function this is?</p><pre>??? :: (Number -&gt; String) -&gt; Array Number -&gt; Array String</pre><p>So, this takes a function that transforms a number into a string, and an array of numbers. And it puts out an array of strings. This is a specialized version of our best friend, the <strong>map</strong> function! A more generic approach would be:</p><pre>mapArray :: (a -&gt; b) -&gt; Array a -&gt; Array b</pre><p>(here, <strong>a</strong> and <strong>b</strong> are <em>type variables that </em>can be replaced by any two types). Of course, we’ve already seen that <strong>map </strong>is not at all about arrays, so:</p><pre>mapMaybe :: (a -&gt; b) -&gt; Maybe a -&gt; Maybe b</pre><p>Or, for completeness, in the most general type signature:</p><pre>map :: Functor f =&gt; (a -&gt; b) -&gt; f a -&gt; f b</pre><p>Here, Functor f is just a constraint that says that <strong>f </strong>must be a <strong>Functor</strong>. Remember that <strong>Array</strong>, <strong>Maybe</strong>, <strong>Either</strong> and <strong>Future</strong> are all functors. So, in the type signature above, you can just replace <strong>f</strong> by one of those and get <strong>mapArray</strong>, <strong>mapMaybe</strong>, etc.</p><p>As an aside, here <strong>Functor</strong> is also called a <em>type class</em> because it’s saying: any type that wants to join the club needs to implement a function named <strong>map </strong>that has this signature. It’s important to understand that each specific functor implements <strong>map</strong> differently: <strong>Arrays</strong> must iterate, <strong>Maybes</strong> must ignore <strong>Nothing</strong>, etc. However, the map function must obey the <a href="https://en.wikibooks.org/wiki/Haskell/The_Functor_class">functor laws</a>.</p><p>That’s it for part I! Head on over to <a href="https://medium.com/@rjbma/scraping-the-web-experiments-with-functional-programming-in-javascript-part-ii-ab57d172f467">Part II</a>, which contains the fun bit, where we actually build the scraper using the stuff we’ve just been over.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56ec7c40b297" width="1" height="1" alt=""><hr><p><a href="https://medium.com/front-end-weekly/scraping-the-web-experiments-with-functional-programming-in-javascript-part-i-56ec7c40b297">Scraping the web: experiments with functional programming in javascript — part I</a> was originally published in <a href="https://medium.com/front-end-weekly">Frontend Weekly</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[elm-pull-dialog.md]]></title>
            <link>https://medium.com/@rjbma/elm-pull-dialog-md-e6d0be228d02?source=rss-810d4100bcc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/e6d0be228d02</guid>
            <category><![CDATA[elm]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Ricardo Araújo]]></dc:creator>
            <pubDate>Thu, 17 Aug 2017 14:37:46 GMT</pubDate>
            <atom:updated>2017-08-17T14:37:46.987Z</atom:updated>
            <content:encoded><![CDATA[<h3>Implementing a modal dialog in Elm</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*YGbg9S-ejkWVQzcBz241MA.png" /><figcaption><strong>User</strong>: you’re blocking my view / <strong>Modal</strong>: I am the view!</figcaption></figure><p>I like modal dialogs. I (ab)use them for all sorts of things, like showing messages, side menus, additional info on some value, etc.</p><p>Most of the times, I just use a framework such as <strong>Bootstrap</strong> or <strong>Semantic UI</strong> to show them. That usually makes sense because the project already uses that framework, and of course you want to maintain a consistent look-and-feel.</p><p>I’ve been reading a lot about <strong>Elm</strong> lately, and now I’ve decided to use it for building a small personal project (along with some other stuff <em>I’ve been learning and not really using</em> — but that’s for another post). Of course it wasn’t long before I had the need for some modal dialogs. But this time, instead of pulling in some framework in Elm to render modal dialogs, I decided I should implement one from scratch.</p><h4>tl;dr</h4><p>This is a learning exercise. It probably won’t be of much use to you. Sorry… <strong>But fear not</strong>: people may not yet be fatigued of the Elm eco-system, however there are already many options to chose from:</p><ul><li><a href="http://elm-ui.info/">elm-ui</a></li><li><a href="https://debois.github.io/elm-mdl/">elm-mdl</a></li><li><a href="http://elm-bootstrap.info/">elm-bootstrap</a></li><li><a href="https://github.com/krisajenkins/elm-dialog">elm-dialog</a></li><li><em>probably a lot more</em></li></ul><h3>Some guidelines</h3><ul><li>The markup should be as simple as possible</li><li>The CSS classes assigned to each element should be fully customisable</li><li>No inline CSS</li><li>Can’t have internal state (duh!)</li><li>Should have a backdrop. Clicking it should (<em>read</em>: issue a message to) close the dialog</li><li>Only message it produces should be for closing the dialog. Meaning that the content of the dialog is completely up to you</li><li>Some basic animations when opening or closing (<em>read</em>: CSS transitions only)</li><li>Using it shouldn’t be too different than using any other Elm Html element function, meaning it should take at the very least:</li><li>a list of properties (for the top-level element)</li><li>a list of children elements (the content)</li></ul><h3>Html structure</h3><p>This is the simplest markup I could think of:</p><pre>&lt;div class=&quot;pullDialog pullDialog--top pullDialog-isOpen&quot;&gt;<br>  &lt;div class=&quot;pullDialog-container&quot;&gt;<br>    &lt;!-- stuff goes here --&gt;<br>  &lt;/div&gt;<br>  &lt;div class=&quot;pullDialog-backdrop&quot;&gt;&lt;/div&gt;<br>&lt;/div&gt;</pre><p>I’m using <a href="https://maintainablecss.com/">Maintainable CSS</a> for naming classes here, but really could have used any other strategy. I just happened to read about it recently, and thought I’d take it for a spin.</p><p>Here, pullDialog is the name of the <strong>module</strong> (or <strong>block</strong> if you&#39;re using <a href="http://getbem.com/introduction/">BEM</a>); it contains everything else. When the dialog is open, it should be assigned the pullDialog-isOpen <strong>state</strong>. Also, we&#39;re assigning it the pullDialog--top modifier to say that the dialog will be pulled in from the top of the screen.</p><p>The pullDialog-container <strong>component</strong> (or <strong>element</strong> in BEM) will contain every children element we&#39;ll throw at our dialog. Initially, this component will be rendered empty and off-canvas; it will be transitioned into the screen when the dialog is opened.</p><p>Finally, the pullDialog-backdrop component will become visible when we open the dialog. If clicked, a message should be issued to close the dialog.</p><h3>Let’s do this thing!</h3><p>Unsurprisingly, this will just be a bunch of functions. Better yet, this will consist one main function and a bunch of helper functions.</p><p>The main function will be responsible for rendering the modal, given some configuration parameters, a flag indicating whether the dialog is open or closed, and the aforementioned list of Html attributes and children elements.</p><pre>view : Config msg -&gt; Bool -&gt; List (Html.Attribute msg) -&gt; List (Html msg) -&gt; Html msg<br>view cfg isOpen attributes children =<br>    ...</pre><p>And the Config type alias will be</p><pre>type alias Config msg =<br>    { mainClass : String<br>    , openClass : String<br>    , containerClass : String<br>    , backdropClass : String<br>    , direction : PullDirection<br>    , closeMsg : msg<br>    }</pre><p>Here, closeMsg is the message that we want to issue when/if the user clicks the backdrop. Note that elm-modal doesn&#39;t maintain any state, not even the state that determines if the modal is visible or not. So, it&#39;s up to you to handle this message and update that flag in your model.</p><p>Also, every modal must appear from somewhere:</p><pre>type PullDirection<br>    = Top<br>    | Right<br>    | Bottom<br>    | Left</pre><p>So, in our previous example, Config would be:</p><pre>{ mainClass = &#39;pullDialog&#39;<br>, openClass = &#39;pullDialog-isOpen&#39;<br>, containerClass = &#39;pullDialog-container&#39;<br>, backdropClass = &#39;pullDialog-backdrop&#39;<br>, direction = Top<br>, closeMsg = WhateverYouDefine<br>}</pre><p>Our main view function is really simple:</p><pre>view : Config msg -&gt; Bool -&gt; List (Html.Attribute msg) -&gt; List (Html msg) -&gt; Html msg<br>view cfg isOpen attributes children =<br>    let<br>        isOpenClass =<br>            if isOpen then<br>                class cfg.openClass<br>            else<br>                class &quot;&quot;<br><br>        allAttrs =<br>            class cfg.mainClass<br>                :: isOpenClass<br>                :: attributes<br>    in<br>        div allAttrs<br>            [ div [ class cfg.containerClass ] children<br>            , div [ class cfg.backdropClass, onClick cfg.closeMsg ] []<br>            ]</pre><h3>A helping hand</h3><p>Building a Config object every time seems tedious. If we&#39;re using <strong>MaintainableCSS</strong>, we can have a helper function that parameterizes the module name:</p><pre>maintainableCssConfig : String -&gt; PullDirection -&gt; msg -&gt; Config msg<br>maintainableCssConfig moduleName pullDirection closeMsg =<br>    let<br>        direction =<br>            case pullDirection of<br>                Top -&gt;<br>                    &quot;--top&quot;<br><br>                Right -&gt;<br>                    &quot;--right&quot;<br><br>                Bottom -&gt;<br>                    &quot;--bottom&quot;<br><br>                Left -&gt;<br>                    &quot;--left&quot;<br>    in<br>        { mainClass = moduleName ++ &quot; &quot; ++ moduleName ++ direction<br>        , direction = pullDirection<br>        , openClass = moduleName ++ &quot;-isOpen&quot;<br>        , containerClass = moduleName ++ &quot;-container&quot;<br>        , backdropClass = moduleName ++ &quot;-backdrop&quot;<br>        , closeMsg = closeMsg<br>        }</pre><p>And if we want all our modals to be named the same by default, we can use a default naming function:</p><pre>defaultConfig : PullDirection -&gt; msg -&gt; Config msg<br>defaultConfig =<br>    maintainableCssConfig &quot;pullDialog&quot;</pre><h3>Wait a minute! Aren’t we missing something?</h3><p>We still need some CSS. Here’s something that works for the classes in our example above:</p><pre>.pullDialog-container {<br>    border: 1px solid #333;<br>    padding: 10px;<br>    background-color: white;<br>    border-radius: 0px 0px 0px 0px;<br>    position: fixed;<br>    overflow: hidden;<br>    transition: transform 0.2s ease-out;<br>    z-index: 200;<br>}<br><br>.pullDialog--top .pullDialog-container {<br>    top: 0px;<br>    left: 0px;<br>    width: 100%;<br>    border-width: 0px 0px 1px 0px;<br>    transform: translate(0%, -100%);<br>}<br>.pullDialog--top.pullDialog-isOpen .pullDialog-container {<br>    transform: translate(0%, 0%);<br>}<br><br>.pullDialog--right .pullDialog-container {<br>    top: 0px;<br>    right: 0px;<br>    height: 100%;<br>    border-width: 0px 0px 0px 1px;<br>    transform: translate(100%, 0%);<br>}<br>.pullDialog--right.pullDialog-isOpen .pullDialog-container {<br>    transform: translate(0%, 0%);<br>}<br><br>.pullDialog--bottom .pullDialog-container {<br>    bottom: 0px;<br>    left: 0px;<br>    width: 100%;<br>    border-width: 1px 0px 0px 0px;<br>    transform: translate(0%, 100%);<br>}<br>.pullDialog--bottom.pullDialog-isOpen .pullDialog-container {<br>    transform: translate(0%, 0%);<br>}<br><br>.pullDialog--left .pullDialog-container {<br>    top: 0px;<br>    left: 0px;<br>    height: 100%;<br>    border-width: 0px 1px 0px 0px;<br>    transform: translate(-100%, 0%);<br>}<br>.pullDialog--left.pullDialog-isOpen .pullDialog-container {<br>    transform: translate(0%, 0%);<br>}<br><br>.pullDialog-backdrop {<br>    position: fixed;<br>    height: 100%;<br>    top: 0px;<br>    left: 0px;<br>    width: 100%;<br>    background: rgba(0,0,0,0);<br>    z-index: 100;<br>    display: none;<br>}<br>.pullDialog-isOpen .pullDialog-backdrop {<br>   background: rgba(0,0,0,0.4);<br>   display: block;<br> }</pre><p>Some remarks:</p><ul><li>the pull direction affects where the content will be pulled from, and also some basic styling for the container</li><li>the -isOpen class determines whether the container is on canvas and the backdrop is visible</li></ul><h3><strong><em>Bonus</em></strong> : show a dialog when something is selected</h3><p>So far, we’ve been showing the modal depending on a boolean value. This is useful for static content like menus, or confirmation messages.</p><p>A different, but also common, use case is when a user selects some value and we want to present additional information about it; or we want to present a set of actions on that value that couldn’t/shouldn’t be visible before. In other words, we want to present the dialog depending on the value of a Maybe:</p><ul><li>if it’s a Nothing, then we want to hide the modal</li><li>if it’s a Just, then we want to show the modal. Also, its content probably depends on the value that&#39;s in Just</li></ul><p>In code, that would be:</p><pre>maybeView : Config msg -&gt; Maybe a -&gt; List (Html.Attribute msg) -&gt; (a -&gt; List (Html msg)) -&gt; Html msg<br>maybeView cfg val attrs childrenFn =<br>    case val of<br>        Nothing -&gt;<br>            view cfg False attrs [ text &quot;&quot; ]<br><br>        Just a -&gt;<br>            view cfg True attrs (childrenFn a)</pre><p>Some remarks about this solution:</p><ul><li>We replaced our boolean flag with a Maybe</li><li>The modal’s content can no longer be a list of HTML elementes. Since it depends on the Maybe&#39;s value, it must now be a function. <em>Makes sense</em>.</li><li>We always render the modal, even when, theoretically, we didn’t have to. This is because we need them for the CSS transitions to kick in</li></ul><h3>Conclusion</h3><p>The end result is actually pretty simple to use. As a user of the library, most of the code you have to write is the CSS to give a personalised look and feel to your modals.</p><p>The whole thing is available on <a href="https://github.com/rjbma/elm-modal">GitHub</a>, and as an Elm <a href="http://package.elm-lang.org/packages/rjbma/elm-modal/1.0.0/Modal">package</a>. Here’s also an example <a href="https://ellie-app.com/43f723B8fpka1/3">Ellie</a> app:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.ellie-app.com%2F43f723B8fpka1%2F3&amp;url=https%3A%2F%2Fellie-app.com%2F43f723B8fpka1%2F3&amp;image=https%3A%2F%2Fdbamidicc4dm6.cloudfront.net%2Fimages%2Fellie-announcement%2Fellie_profile_pic_2.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=ellie-app" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/b0ea37b510ebc35659cabf7d979f8ca4/href">https://medium.com/media/b0ea37b510ebc35659cabf7d979f8ca4/href</a></iframe><p>If this was useful to you in anyway, I’d love to read your comments below, as well as any suggestions for improvements!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e6d0be228d02" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>