<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pedro Costa on Medium]]></title>
        <description><![CDATA[Stories by Pedro Costa on Medium]]></description>
        <link>https://medium.com/@pedrodc?source=rss-5bb7d15a6e16------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*yRhknqWApiwrdSwfrD9iYg.jpeg</url>
            <title>Stories by Pedro Costa on Medium</title>
            <link>https://medium.com/@pedrodc?source=rss-5bb7d15a6e16------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 09:38:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pedrodc/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Implementing a Blockchain Oracle on Ethereum]]></title>
            <link>https://medium.com/@pedrodc/implementing-a-blockchain-oracle-on-ethereum-cedc7e26b49e?source=rss-5bb7d15a6e16------2</link>
            <guid isPermaLink="false">https://medium.com/p/cedc7e26b49e</guid>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[ethereum]]></category>
            <dc:creator><![CDATA[Pedro Costa]]></dc:creator>
            <pubDate>Thu, 20 Jun 2019 16:56:53 GMT</pubDate>
            <atom:updated>2019-06-20T17:05:16.178Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>TLDR: In this post, I explain how you can build an Ethereum blockchain oracle. This service will query APIs, upon request by smart-contracts, and add the requested information to the blockchain. The described approach allows for having multiple competing parties deploying the same oracle and having the same say in the result. But can easily be converted into a simpler oracle to serve a single stakeholder.</blockquote><h3>Background</h3><p>Smart Contracts in Ethereum can power a wide range of applications, however, and due to the nature of the blockchain, smart contracts lack an essential feature: <em>Internet Connectivity.</em></p><h4>Why? you may ask</h4><p>The Ethereum blockchain is designed to be entirely deterministic, meaning that if someone downloads the whole network history and replays it they should always end up with the same state. Determinism is necessary so that nodes can come to a consensus.</p><p>However, the internet is not deterministic, a smart contract querying an API at a certain point in time, cannot be guaranteed that later querying the same API will get the same result. APIs and data on the web change. Therefore, by nature smart contracts lack connectivity.</p><h4>So Oracles, right?</h4><p>The name oracle comes from the fact that oracles, historically speaking were sources of the truth. And that’s what we need.</p><p>Oracles are services that insert data on the blockchain to be used by smart contract. By adding a transaction with the required information to the blockchain the smart contract can run and always obtain the same information since it is retrieving it from the blockchain.</p><h3>Solution</h3><p>We will create an oracle service that can query JSON APIs and retrieve a single value from the API response. The oracle will save all the requests and answers and will have a predefined set of stakeholders.</p><p>These are the accounts running the node.js service that queries the APIs and returns a response to the oracle. The oracle also has a minimum number of equal responses that it must receive in order to confirm that the answer provided is valid.</p><p>This way, if competing parties depend on the oracle to power their contracts and one of the parties (nodes) becomes rogue or tries to manipulate the result, it can’t because they agreed on a predefined quorum of equal answers.</p><h3>Architecture</h3><p>This oracle will comprise two components. The on-chain oracle (a smart contract) and the off-chain oracle service (node.js server).</p><p>The on-chain oracle is a smart-contract that has a public function, <strong>createRequest</strong>, that receives the URL, to query, and the attribute to retrieve. And then, launches an event to alert the off-chain oracle of a new request.</p><p>The off-chain oracle is composed of several node.js services deployed by different parties that will query the API and return to the contract the response.</p><p>The on-chain oracle then verifies if the minimum number of equal responses has been reached and if so emits an event saying that it has achieved consensus on the value so that the client smart-contract that queried the oracle knows that it has its response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cs3w9iFmhIfkyg3Kg_FzFw.png" /></figure><h3>On-chain Oracle Implementation</h3><p>We define the oracle contract with the agreed terms: Minimum Quorum and Total Oracles. For this example, there are three stakeholders, and for achieving consensus 2 out of 3 must provide the same answer.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee78b43dba64e092267a20f5f4417f95/href">https://medium.com/media/ee78b43dba64e092267a20f5f4417f95/href</a></iframe><p>Then we add the Request Struct, which will hold the requests:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d1eb9a585ca9f5a588fca0e9b557d7b/href">https://medium.com/media/1d1eb9a585ca9f5a588fca0e9b557d7b/href</a></iframe><p>Now we can create the public function, <strong>createRequest</strong>, that client smart contracts (any contract that wants to use the oracle service) will call:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f0d5c29030aa30acb9efebd25d505e72/href">https://medium.com/media/f0d5c29030aa30acb9efebd25d505e72/href</a></iframe><p>This function contains an important part of the agreement between the stakeholders. The addresses of the accounts that are trusted to take part in the final solution. And will emit the event, <strong>NewRequest </strong>that will be listened by the off-chain oracles.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/74b7fd61af11c5906cbf5d4c5b3a1439/href">https://medium.com/media/74b7fd61af11c5906cbf5d4c5b3a1439/href</a></iframe><p>Upon listening to this event the off-chain oracles will call the public function <strong>updateRequest</strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4ecbaaaa856665b8b1ec2c6b07398cba/href">https://medium.com/media/4ecbaaaa856665b8b1ec2c6b07398cba/href</a></iframe><p>This function will first check if the caller is one of the predefined addresses. Then it will check it the oracle hasn’t voted and if so it will save the oracle answer. Then it will check if that answer has been provided by at least the required minimum quorum. If so, then we have agreed on a result and will emit an event, <strong>UpdatedRequest</strong>, to alert the client contract of the result.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/82c736beb48bc8d7b5e11e3239c40a51/href">https://medium.com/media/82c736beb48bc8d7b5e11e3239c40a51/href</a></iframe><p><a href="https://github.com/pedroduartecosta/ethereum-oracle/tree/master/on-chain-oracle">HERE </a>you can find the full oracle code and instructions on how to use truffle to create the boilerplate.</p><h3>Off-chain Oracle Implementation</h3><p>This is the simpler part, it’s any service that can listen to the emitted blockchain events and query APIs.</p><p>The off-chain oracle listens to the events emitted by the on-chain oracle, using web3, and queries the requested API, parses the retrieved JSON for the requested key and calls the public function <strong>updateRequest</strong>.Since it is not the focus of the post and it’s a simple service, I will not go further into the details of the implementation of the off-chain service.</p><p><a href="https://github.com/pedroduartecosta/ethereum-oracle/tree/master/off-chain-oracle">HERE </a>you can find the code for this service that should be deployed by all the stakeholders.</p><h4>Summing up</h4><p>This implementation allows to not depend on a single party to be the source of truth, by being the only one querying the API, but to have multiple parties agreeing on a result. It is also a very flexible service since it can query any public JSON API, allowing to be used in a plethora of use cases.</p><p>All the code can be found <a href="https://github.com/pedroduartecosta/ethereum-oracle">HERE</a>.</p><h3>👋 Thanks for reading!</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cedc7e26b49e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Big Data Machine Learning Spark Application for Flight Delay Prediction]]></title>
            <link>https://medium.com/@pedrodc/building-a-big-data-machine-learning-spark-application-for-flight-delay-prediction-4f9507cdb010?source=rss-5bb7d15a6e16------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f9507cdb010</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Pedro Costa]]></dc:creator>
            <pubDate>Tue, 29 Jan 2019 16:06:27 GMT</pubDate>
            <atom:updated>2020-12-29T13:21:37.723Z</atom:updated>
            <content:encoded><![CDATA[<p>In this tutorial, I explain how to develop a Spark application that allows creating a machine learning model for a real-world problem, using real-world data: Predicting the arrival delay of commercial flights.</p><p>To find the best model, several machine learning techniques were used and provided with several inputs combinations. The machine learning methods explored are <strong>Linear Regression</strong>, <strong>Random Forest Trees</strong> and <strong>Gradient-Boosted Trees</strong>. All the code will be written in <strong>Scala</strong>.</p><p>The full code for this application and how to run it can be found in the GitHub repo: <a href="https://github.com/pedroduartecosta/Spark-PredictFlightDelay">https://github.com/pedroduartecosta/Spark-PredictFlightDelay</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ysjey9exDXqiN3Lx" /><figcaption>Photo by <a href="https://unsplash.com/@chuttersnap?utm_source=medium&amp;utm_medium=referral">chuttersnap</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Problem</h3><p>We want to be able to predict, based on historical data, the arrival delay of a flight using only information available before the flight takes off. Since the target variable (ArrDelay - arrival delay) is a numerical value we will use regression algorithms only, I’ll explain this further below.</p><h3>Dataset</h3><p>The data used was published by the US Department of Transportation and can be downloaded <a href="http://stat-computing.org/dataexpo/2009/the-data.html">here</a>. It compromises almost 23 years worth of data, and analysing such amount of data requires big data tools such as Spark.</p><h3>Setting up the project</h3><p>For those who are new to Spark, I provide a tutorial on how to set up the project and its dependencies using <strong>SBT</strong>.</p><p>The tutorial can be found here: <a href="https://medium.com/@pedrodc/setting-up-a-spark-machine-learning-project-with-scala-sbt-and-mllib-831c329907ea">https://medium.com/@pedrodc/setting-up-a-spark-machine-learning-project-with-scala-sbt-and-mllib-831c329907ea</a></p><h3>Exploratory Data Analysis</h3><p>The most important part of a data science project is to perform a good analysis of the problem and the data we have. Since most variables are numerical we will first create a correlation matrix, of the numerical variables, to understand if we can find any correlation between them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ml0R4pHnQR9xxHdewZX_DA.png" /></figure><h4>Correlation explained</h4><blockquote>A correlation describes the strength of association between two attributes. The closer the value is to 1 or -1, the more positively or negatively the variables are correlated. A correlation of zero means that that the two attributes are not correlated at all. A coefficient of +1 indicates that the two attributes have a perfect correlation, meaning every change in one attribute is accompanied by an equivalent change in the other attribute in the same direction. A coefficient of -1 indicates that every change in one attribute is accompanied by the opposite change in the other attribute. —based on the book Data Science by MIT Press</blockquote><p>Looking at the correlation matrix we can see that the DepDelay (departure delay) it highly correlated to arrival delay, so the departure delay might be the most indicative attribute of a flight delay. Looking a bit further we can also see a weak, but existing correlation, between the TaxiOut and arrival delay. Therefore we will be looking further into these two variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YWhGtpPldmkTas9SFkcNNw.png" /><figcaption>Scatterplot between departure delay and arrival delay from the 2008 dataset.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3yT7HOrwgtr9_S6-PTHMlA.png" /><figcaption>Scatterplot between taxi-out duration and arrival delay from the 2008 dataset.</figcaption></figure><p>These two plots allow us to better understand the correlation value presented before.</p><h3>Implementation</h3><p>Any implementation mentioned and not presented in code here, it is because the code should be trivial and can be found in full in the repository.</p><h4>Loading the data</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a05c92e42f7fbdd6d29f113ac58260c/href">https://medium.com/media/8a05c92e42f7fbdd6d29f113ac58260c/href</a></iframe><p>So first we read the data from the CSV file using the spark SQLContext. To load multiple CSV files, simply provide a path such as: /mydata/*.csv. This will load the data into a DataFrame which is a distributed collection of data organized into named columns. So if we are running the spark application in a cluster this will distribute the data randomly across the cluster.</p><p>Then we use the .drop() function to drop all the variables that are only known after the flight took off and also variables that are too correlated between themselves and can be redundant to the model may be causing the model to overfit. We also drop cancelled flights and variables that are not correlated to the arrival delay.</p><h4>Processing the data</h4><p>The data processing highly depends on the machine learning algorithm implementation and on the experiment being done. The main difference will be in what variables are dropped and if we use nominal (categorical) values, such as, the Airport of origin and destination.</p><p>To use the categorical variables in regression algorithms, further processing was needed. These variables were converted to numerical values by applying a <a href="https://spark.apache.org/docs/latest/ml-features.html#stringindexer"><strong>StringIndexer</strong></a> and <a href="https://spark.apache.org/docs/latest/ml-features.html#onehotencoderestimator"><strong>OneHotEncoder</strong></a>. More information on these two methods and why there were used can be found on the links.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/67343f659d8923c5531d3aa80166b7b9/href">https://medium.com/media/67343f659d8923c5531d3aa80166b7b9/href</a></iframe><h4>Training the data and obtaining a model</h4><p>We found attributes that demonstrate a high linear correlation, departure delay with arrival delay, which can be seen in the first scatterplot. Therefore the first method we will test on is <strong>Linear Regression</strong>. This is a very fast and simple algorithm which provides good results in these circumstances.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d04a4c0e61247abdee9fa4bd1b599dfe/href">https://medium.com/media/d04a4c0e61247abdee9fa4bd1b599dfe/href</a></iframe><p>Here we create a pipeline to process all the transformations on the data. This allows us to define all the steps and only run them when the function .fit() is called to train the model. The pipeline compromises the following steps: converting the categorical values (if it applies) to numerical ones, then assembling all in a features vector and running the machine learning algorithm to obtain a model.</p><p>The code for running <strong>Random Forest Trees </strong>and <strong>Gradient-Boosted Trees </strong>machine learning algorithms<strong> </strong>can be found in the repository and these methods integrate seamlessly in this pipeline. They base their implementation on decision trees and require much more space and time to run but might provide better results when the features and target variable relation is not linear. More information can be found <a href="https://spark.apache.org/docs/latest/ml-classification-regression.html#random-forests"><em>here</em></a>.</p><p>Before the data is used, it must first be split in training and testing sets, a split of 70% was used, in order to later evaluate the resulting model.</p><blockquote>The golden rule for evaluating models is that models should never be tested on the same data they were trained on. — Data Science, MIT Press</blockquote><h4>Optimizations</h4><p>We use the <em>TrainValidationSplit </em>method to split the training data in training and validating data. The validating data is used to optimize the algorithm parameters. A <em>ParamGridBuilder</em> is used to define the parameters range in which the machine learning algorithm will be trained on. The best performing set of parameters will be then used to obtain the final model.</p><h4>Evaluating the resulting model</h4><p>Two main metrics were used to evaluate the model’s performance. R squared and Root Mean Square Error(RMSE).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0de59a1ad55d5f4acdf6db2011db56f3/href">https://medium.com/media/0de59a1ad55d5f4acdf6db2011db56f3/href</a></iframe><p><strong>RMSE</strong>, measures the difference between values predicted by a model and the values observed. Considering this, the lower RMSE value is, the better is the model.</p><p><strong>R Squared</strong>, also known as the coefficient of determination, it is a statistical measure of how close the data is to the fitted regression line. It assesses the goodness of fit of our regression model. The closer the value is to 1, the better is the model.</p><h3>Final considerations</h3><p>In the tests performed, the Linear Regression algorithm performed better than the other two algorithms based on decision trees. This can be due to the linear relation of the features to the target variable used. The results and further analysis can be found in the report in the repository.</p><p>Spark is a very powerful tool, the code and application here demonstrated can run on a cluster without any alteration, allowing to test huge datasets of data with more complex algorithms.</p><h3>👋 Thanks for reading!</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f9507cdb010" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up a Spark machine learning project with Scala, sbt and MLlib]]></title>
            <link>https://medium.com/@pedrodc/setting-up-a-spark-machine-learning-project-with-scala-sbt-and-mllib-831c329907ea?source=rss-5bb7d15a6e16------2</link>
            <guid isPermaLink="false">https://medium.com/p/831c329907ea</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[bts]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Pedro Costa]]></dc:creator>
            <pubDate>Tue, 08 Jan 2019 22:43:30 GMT</pubDate>
            <atom:updated>2019-01-08T22:48:38.690Z</atom:updated>
            <content:encoded><![CDATA[<p>In this tutorial, we will set up a <a href="https://spark.apache.org/">Spark</a> Machine Learning project with <a href="https://en.wikipedia.org/wiki/Scala_(programming_language)">Scala</a>, <a href="https://spark.apache.org/mllib/">Spark MLlib</a> and <a href="http://scala-sbt.org/">sbt</a>.</p><blockquote>sbt is an open-source build tool for Scala and Java projects, similar to Java’s <a href="https://en.wikipedia.org/wiki/Apache_Maven">Maven</a> and <a href="https://en.wikipedia.org/wiki/Apache_Ant">Ant</a>.</blockquote><p><em>sbt requires the Java Development Kit 8 (JDK 8), so if you don’t have it installed follow </em><a href="http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html"><em>this</em></a><em> link to install.</em></p><h3>Installing sbt:</h3><p>On Ubuntu:</p><pre>$ echo &quot;deb https://dl.bintray.com/sbt/debian /&quot; | sudo tee -a /etc/apt/sources.list.d/sbt.list</pre><pre>$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823</pre><pre>$ sudo apt-get update</pre><pre>$ sudo apt-get install sbt</pre><p>For other Linux distributions: <a href="https://www.scala-sbt.org/download.html">https://www.scala-sbt.org/download.html</a></p><p>On Mac using <a href="https://brew.sh/">homebrew</a>:</p><pre>$ brew install sbt</pre><h3>Creating the project</h3><p>To quickly start your project we will use a <a href="https://github.com/scala/scala-seed.g8">Gitter8 bootstrap template</a>. This will create the necessary folder structure and project files.</p><pre>$ sbt new sbt/scala-seed.g8</pre><p>You can quickly check if everything is working by changing directory into your newly created project and running sbt:</p><pre>$ cd [my-project]</pre><pre>$ sbt</pre><p>Inside the sbt shell use the command <em>run</em> to run the template project:</p><pre>$ sbt:myproject&gt; run</pre><p>This should return a simple <em>hello</em> message.</p><h3>Adding Spark and Spark MLlib</h3><p>The default template already includes a scalaTest dependency. Now we will add Spark core and Spark MLlib.</p><p>In your project folder root you can find your <em>build.sbt</em> configuration file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I-k_iEWh-7iAg65XY42qpA.png" /></figure><p>Add the last two lines,</p><pre><em>libraryDependencies += sparkCore,<br>libraryDependencies += sparkMLlib</em></pre><p>, to include the spark core and the spark MLlib dependency.</p><p>Then we need to specify what these dependencies are in ./project/Dependencies.scala</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0PrzK2ufGe8h9K7J-X8hTw.png" /></figure><pre><em>// </em><a href="https://mvnrepository.com/artifact/org.apache.spark/spark-core"><em>https://mvnrepository.com/artifact/org.apache.spark/spark-core</em></a><em><br></em>lazy val sparkCore = “org.apache.spark” %% “spark-core” % “2.4.0”</pre><pre><em>// </em><a href="https://mvnrepository.com/artifact/org.apache.spark/spark-mllib"><em>https://mvnrepository.com/artifact/org.apache.spark/spark-mllib</em></a><em><br></em>lazy val sparkMLlib = “org.apache.spark” %% “spark-mllib” % “2.4.0”</pre><h3>That’s it!</h3><p>Now you can code your machine learning project with Spark and MLlib in the source folder and run with sbt.</p><pre>$ sbt</pre><p>Inside the sbt shell use the command <em>run.</em></p><pre>$ sbt:myproject&gt; run</pre><h3>👋 Thanks for reading!</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=831c329907ea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Roadmap to Serverless Computing]]></title>
            <link>https://medium.com/@pedrodc/roadmap-to-serverless-computing-c8a3e20cdbcf?source=rss-5bb7d15a6e16------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8a3e20cdbcf</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[lambda]]></category>
            <dc:creator><![CDATA[Pedro Costa]]></dc:creator>
            <pubDate>Thu, 09 Aug 2018 13:10:06 GMT</pubDate>
            <atom:updated>2018-08-23T09:09:26.329Z</atom:updated>
            <content:encoded><![CDATA[<p>Serverless is the latest stage of On-Demand Economy applied to software development.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NTnZAIXW9BNqfHAQ" /><figcaption>“Close-up of colorful pins in a world map” by <a href="https://unsplash.com/@delfidelarua7?utm_source=medium&amp;utm_medium=referral">delfi de la Rua</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A bit of history on how we got here before explaining what serverless is. In the early days of cloud computing, companies had to set up their infrastructure for their services and applications, including mounting the racks with the servers and configuring all the underlying network and security. Emerging from the necessity of an abstract layer and quicker response from conception to production (TTM — time to market), Infrastructure as a Service was born.</p><p><strong>IaaS (Infrastructure as a Service)</strong>, allowed companies to request the required underlaying infrastructure from other providers, releasing them of huge upfront costs in material and personnel to set up all their services.</p><p>But they still had to configure operating systems, software licenses, some network settings and so on. So we got to the next stage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fYWLB7Wwp-JqolVc" /><figcaption>Photo by <a href="https://unsplash.com/@samuelzeller?utm_source=medium&amp;utm_medium=referral">Samuel Zeller</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Platform as a service (PaaS)</strong>, now customers can focus on developing their solutions, knowing that they have a platform already configured to run those workloads, in minutes, without having to manage networking infrastructure, servers, storage and so on. Therefore a customer can, for example, launch a PostgreSQL or MySQL database almost instantly without having to install anything by simply configuring the database on a web interface.</p><p>But still, this was not enough, because even though with PaaS, SaaS (Software as a Service) or even containerisation (e.g. Docker) were are giving a high-level abstraction of the underlying infrastructure most of the times we still needed to tackle availability problems and manage some applications restrictions and decide on how many instances/containers we needed or how much memory and cores should be used to run the application smoothly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1UeFGvOXUS9wlyQp" /><figcaption>Photo by <a href="https://unsplash.com/@liamhenry?utm_source=medium&amp;utm_medium=referral">Liam Henry</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>So out of the micro-services model and the breakdown of monolithic applications, <strong>serverless was born</strong>.</p><p>With highly decoupled applications we were able to go a level upwards in abstraction layer and only think about the code and what output we want to have for a certain input, regardless of the number of requests.</p><p>Quick example, in order to process large amounts of incoming data, a good developer traditionally would send these requests to a queue to be processed by our instances once they were free to do so. With lambda, you can launch as many calls as we want without ever thinking about application and infrastructure limitations. And if we think about cost, with providers such as AWS you pay only for the amount of time your code is running rounded up to the nearest 100ms, so that is efficiency!</p><p>Serverless presents a true on-demand service that starts when receives a request and shuts down as soon as that request is processed, leveraging only the resources needed for as long as they are needed and nothing more. A high-level abstraction such as this also allows for more fault tolerant and secure applications since it cuts the underlying dependencies between services while making sure you don’t have servers running on half or even less of their computational capacity, while paying for the whole computation power.</p><p><strong>Wrapping up</strong>, serverless takes advantage of a truly on-demand approach and can benefit a lot from economies of scale, since the same server can serve thousands of customers by using virtual independent processes to launch each lambda function allowing for a better distribution and sharing of resources.</p><h3>In more detail</h3><p>In practice, a lambda function is simply an anonymous function that receives an input, it can be from a IoT device sending some data, a queue, an instance, a database trigger, and API endpoint and so on that is processed and generates an output, which again can be a trigger to a next step.</p><p>Example of a lambda function in node 8.10:</p><p>exports.handler = async (event) =&gt; {<br> // TODO implement<br> return ‘Hello from Lambda!’;<br>};</p><p>You can have a full operating application only using serverless. Services like AWS provide you with all the necessary tools, for example, you can use Cognito for authentication, S3 for static hosting, Lambda for any required processing, DynamoDB for database and so on, all stateless services and with well documented SDKs ready to use. I don’t want to get into to much detail on specifics but hope I got your interest in serverless and go ahead and give it a try!</p><h3>Promising services</h3><p><em>Aurora Serverless (Serverless Database):</em> <a href="https://aws.amazon.com/rds/aurora/serverless/">https://aws.amazon.com/rds/aurora/serverless/</a></p><p><em>Serverless Docker Deployments: </em><a href="https://zeit.co/blog/serverless-docker?utm_source=newsletter&amp;utm_medium=email&amp;utm_campaign=now_docker_beta">https://zeit.co/blog/serverless-docker?utm_source=newsletter&amp;utm_medium=email&amp;utm_campaign=now_docker_beta</a></p><p><em>The AWS Serverless Platform (describes a wide range of serverless services):</em> <a href="https://aws.amazon.com/serverless/">https://aws.amazon.com/serverless/</a></p><p>Thanks to <a href="https://medium.com/u/1f7ed93df485">Edgar Passos</a> and <a href="https://medium.com/u/2c75fd80bd3b">Bernardo Belchior</a> for the article revision.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8a3e20cdbcf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why make a summer internship at all?]]></title>
            <link>https://medium.com/@pedrodc/why-make-a-summer-internship-at-all-23983e6d8093?source=rss-5bb7d15a6e16------2</link>
            <guid isPermaLink="false">https://medium.com/p/23983e6d8093</guid>
            <category><![CDATA[porto]]></category>
            <category><![CDATA[summer-internships]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[internships]]></category>
            <dc:creator><![CDATA[Pedro Costa]]></dc:creator>
            <pubDate>Sun, 29 Jul 2018 20:08:12 GMT</pubDate>
            <atom:updated>2018-07-29T20:08:12.530Z</atom:updated>
            <content:encoded><![CDATA[<p>There were three reasons that lead me to search for a summer internship:</p><ul><li>First of all, I wanted to put in practice what I’ve learned at the university so far, put my skills to the test.</li><li>Secondly, learn new things, new technologies, find out what is actually used in the market to foster the real services that we use every day.</li><li>At last, acquire professional experience, working in a professional environment using development methodologies such as Agile.</li></ul><h4>So what to do?</h4><p>Quite easy actually, I just sent e-mails to a few companies, in Porto, asking if they were interested in me working for a couple of months and attached my CV.</p><p>Et voilá, Bitmaker presented itself. On the first interview, at the Founders Founders Caffée, they asked for my interests, background in programming and we discussed a few of their projects. I end up choosing to integrate the team that was developing a web platform for benefits management. It was the first time I heard of using a functional programming language in web development for a real use case scenario and that caught my attention.</p><h4>The first two days</h4><p>On the first two days, I was presented to the team and given access to their code repository, company e-mail, slack, etc. But most importantly they gave me two books, <a href="https://pragprog.com/book/phoenix/programming-phoenix">Programming Phoenix</a> and <a href="https://pragprog.com/book/elixir/programming-elixir">Programming Elixir</a>, at first I was a bit lost and went just through the books trying to learn as fast as I could about the framework and the functional language elixir. The more I read the more interesting Phoenix and Elixir seemed to be, Phoenix is amazing with a lot of features and simple routing system powered by plugs (aka middlewares).</p><h4>The fun part</h4><p>On the third day, I told Tiago Fernandes (CTO of Bitmaker) that I was confident enough to get my hands on the project and start contributing to it. And this is where things became interesting. Bitmaker takes Agile quite serious, Tiago had all the user stories on Jira, all in the correct format that we are told to use at university but never thought that companies actually took it that seriously.</p><p>Almost every day, the whole team had a daily scrum session at 11 am, one by one we told what we had done and what we were going to do, and this was critical to keep everyone motivated, there was a great environment where if one was struggling with some feature others would share their thoughts and offer help after the session.</p><p>The agile method worked so well that every day when I came to the office I didn’t even have to ask what I had to do, just opened Jira and started working. Special thanks to Paulo and Daniel who were always keen on helping no matter how many times a day I asked for and there was always time for breaks at the ping pong table and coffee whenever work required.</p><p>Every month we went out to have “Francesinha” for a team lunch so there was really a pleasant and welcoming environment.</p><p>The thing that I actually enjoyed the most was the meetings at the whiteboard. Whenever the team had to make a decision about the project, Tiago always invited everyone (even the intern) to join him around the whiteboard and think how to solve the issue, it was really a collaborative environment. My opinion was always heard and taken into account as well as everyone else. Tiago always had in mind problems such as scalability and performance and he was always open to discuss any development on the project with me.</p><h4>Was it worth the time?</h4><p>With hundred percent certainty, I can say it was. I learned a lot about web development, how to properly work with a REST API backend and the Phoenix Framework powered by the Elixir language as well as how the agile methods work and how to distribute tasks across a team and efficiently communicate and support each other. Both skills were very important later in a university project where we used Phoenix and Elixir for developing a web platform and where we were evaluated by the proper use of the Agile methods.</p><h3>Bonus</h3><p>Bitmaker keeps a nice shelf with excellent books such as “Computer Networks” and “Distributed Systems: Principles and Paradigms” both by Andrew Tanenbaum and “Building Microservices” by Sam Newman. The later I took it home for the duration of the internship and I definitely advise everyone to read! As well as the previous two which I had the opportunity to study at university.</p><h3>Summing it up</h3><p>It was a rewarding experience, surrounded by a great CTO and team, with exceptional books, free coffee, ping pong partners for the breaks, excellent whiteboard discussions and an outstanding choice of framework and language. Not to forget, the design team and CEO which were both friendly and always keen to help!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23983e6d8093" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why read papers/articles?]]></title>
            <link>https://medium.com/@pedrodc/why-read-papers-articles-a7f74eedc004?source=rss-5bb7d15a6e16------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7f74eedc004</guid>
            <category><![CDATA[economist]]></category>
            <category><![CDATA[paper]]></category>
            <category><![CDATA[articles]]></category>
            <dc:creator><![CDATA[Pedro Costa]]></dc:creator>
            <pubDate>Sun, 29 Jul 2018 20:06:18 GMT</pubDate>
            <atom:updated>2018-07-29T20:06:18.210Z</atom:updated>
            <content:encoded><![CDATA[<p>Because you should. Ok, there’s more to it, but basically they shortly (usually) address issues in a detailed manner that allows you to comprehend the logic behind some subject and that realization is worth the time. Or, they simply give you an overview of a specific topic. Either way, you’ll gain much from it, you just have to choose those that are of your interest.</p><p>The list below is intended to not address papers/articles that are too complex or in depth. They are rather a good start!</p><p><a href="https://pedroc.typeform.com/to/X54NIz">Please, click here to make a suggestion or comment!</a></p><h4>My list</h4><ul><li><a href="http://fermatslibrary.com/s/bitcoin">(Link)</a> <strong>Bitcoin: A Peer-to-Peer Electronic Cash System</strong> by Satoshi Nakamoto</li><li><a href="http://www0.cs.ucl.ac.uk/staff/M.Handley/papers/only-just-works.pdf">(Link)</a> <strong>Why the Internet only just works</strong> by Mark Handley (Professor of Networked Systems at UCL)</li><li><a href="http://book.mixu.net/distsys/single-page.html">(Link)</a> <strong>Distributed Systems: for fun and profit</strong> by Mikito Takada</li><li><a href="https://onlinelibrary.wiley.com/doi/full/10.1111/poms.12838">(Link)</a> <strong>Big Data Analytics in Operations Management</strong> by Choi, Wallace, and Wang</li><li><a href="https://www.economist.com/economics-brief/2016/08/20/prison-breakthrough">(Link)</a> <strong>Game theory: Prison breakthrough (Nash equilibrium)</strong> by The Economist</li><li><a href="https://www.1843magazine.com/content/ideas/ian-leslie/non-cogito-ergo-sum">(Link)</a> <strong>Non cogito, ergo sum: Sometimes thinking is a bad idea</strong> by Ian Leslie (1843 Magazine by The Economist)</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7f74eedc004" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Books I’ve read and books I intend to read]]></title>
            <link>https://medium.com/@pedrodc/books-ive-read-and-books-i-intend-to-read-74e46449f6a0?source=rss-5bb7d15a6e16------2</link>
            <guid isPermaLink="false">https://medium.com/p/74e46449f6a0</guid>
            <category><![CDATA[books]]></category>
            <dc:creator><![CDATA[Pedro Costa]]></dc:creator>
            <pubDate>Sun, 29 Jul 2018 20:04:37 GMT</pubDate>
            <atom:updated>2018-07-29T20:04:37.582Z</atom:updated>
            <content:encoded><![CDATA[<p>Here lies a list of books I’ve read and some that I intend to read in a near future. Please feel free to contact me with suggestions. I will try, in time, to give a small opinion on each one.</p><p>There are some books that I’ve read such as Inferno by Dan Brown and other by some Portuguese authors that I don’t feel like having on this list for now.</p><p><a href="https://pedroc.typeform.com/to/fzU8JU">Please, click here to make a suggestion or comment!</a></p><h4>Books I’ve read</h4><ul><li><strong>Harry Potter</strong> (All of them)</li><li><strong>The Art of War</strong> by Sun Tzu</li><li><strong>The Prince</strong> by Niccolò Machiavelli</li><li><strong>How to Win Friends and Influence People</strong> by Dale Carnegie</li><li><strong>Building Microservices</strong> by Sam Newman</li><li><strong>Computer Networks</strong> by Andrew S. Tanenbaum</li><li><strong>Distributed Operating Systems</strong> by Andrew S. Tanenbaum</li><li><strong>The Richest Man In Babylon</strong> by George S. Clason</li><li><strong>The Communist Manifesto</strong> by Karl Marx and Friedrich Engels.</li><li><strong>The Negotiation Book</strong> by Steve Gates</li><li><strong>Animal Farm</strong> by George Orwell</li></ul><h4>Now reading</h4><ul><li><strong>12 Rules for Life</strong> by Jordan B Peterson</li></ul><h4>Next in line</h4><ul><li><strong>1984</strong> by George Orwell</li><li><strong>The 4-hour Work Week</strong> by Tim Ferriss</li><li><strong>The Wealth of Nations</strong> by Adam Smith</li><li><strong>Tao Te Ching</strong> by Lao-Tzu</li><li><strong>The Republic</strong> by Plato</li><li><strong>Inferno</strong> by Dante Alighieri</li><li><strong>The Origin of Species</strong> by Charles Darwin</li><li><strong>The Rules of Management</strong> by Richard Templar</li><li><strong>Don Quijote de La Mancha</strong> by Miguel de Cervantes (When my Spanish allows it)</li><li><strong>Think and Grow Rich</strong> by Napoleon Hill</li></ul><h4>Opinion on books I’ve read</h4><p>In time…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=74e46449f6a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Three economical models that rule modern tech giants]]></title>
            <link>https://medium.com/@pedrodc/three-economical-models-that-rule-modern-tech-giants-a79471926b24?source=rss-5bb7d15a6e16------2</link>
            <guid isPermaLink="false">https://medium.com/p/a79471926b24</guid>
            <category><![CDATA[scale]]></category>
            <category><![CDATA[customer]]></category>
            <category><![CDATA[economics]]></category>
            <category><![CDATA[tech]]></category>
            <dc:creator><![CDATA[Pedro Costa]]></dc:creator>
            <pubDate>Fri, 20 Jul 2018 23:33:17 GMT</pubDate>
            <atom:updated>2018-07-23T16:01:18.871Z</atom:updated>
            <content:encoded><![CDATA[<h4>On-Demand Economy, Economies of scale and Customer-Centric Economy</h4><p>In this post, I am going to present you my views on the On-Demand Economy, Economies of Scale and Customer-Centric Economy, by giving real examples of how giants like Amazon, Netflix and Uber got where they are today. You don’t require any technical background to go through this post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1av-FeudtQff3JWJ" /><figcaption>Photo by <a href="https://unsplash.com/@jakobowens1?utm_source=medium&amp;utm_medium=referral">Jakob Owens</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>On-Demand Economy</h3><blockquote><em>The On-Demand Economy is defined as the economic activity created by technology companies that fulfil consumer demand via the immediate provisioning of goods and services.</em></blockquote><blockquote>— Business Insider</blockquote><p>This trend started with software companies like Amazon, who allowed any customer, either small startups or large companies and federal governments like the United States Government, to request computing capacity for exactly the amount they need and only for as long as needed.</p><p>In the past, these companies had to provision their own servers, by acquiring in advance computing capacity that they might need in the foreseeable future. This led to having to pay for resources that would not be used during most of the time and that would not sustain huge peaks in demand without high investments and accurate forecasts. Amazon revolutionized the market of IaaS (Infrastructure as a Service) with this on-demand approach.</p><p>Now, you can see this being adopted by all other types of business. Take transportation with Uber, housing with AirBnB, offices rental with WeWork and entertainment with Netflix. Netflix allows you to have the content you want to see when you want to see. You don’t wait for the next week to see the next episode, you don’t have commercial breaks showing you publicity, you get what you want when you want. Another example, WeWork, which allows small companies and big ones (e.g. General Motors and Samsung) to quickly set up offices in 70+ cities worldwide without having to build their own, saving both time and money.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cq4viM-9-X22BM2P" /><figcaption>“A drone shot of stacks of large shipping containers in a port” by <a href="https://unsplash.com/@chuttersnap?utm_source=medium&amp;utm_medium=referral">chuttersnap</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Economies of Scale</h3><blockquote>Economies of scale refer to reduced costs <em>per unit</em> that arise from increased <em>total</em> output of a product. —<a href="https://www.investopedia.com/academy/?aca_ref=header_home_link_2">Investopedia</a></blockquote><p>WeWork brings us to the subject of Economies of Scale. Their model is based on having a large portfolio of offices locations (250 as of the moment of this writing) which allows them to rent them for a lower price than any company could get if it had to build all the infrastructure, security and maintenance. And the more they grow the more the customers benefit, as a result their cost per desk has been decreasing over the past years.</p><blockquote>From $10,888 per desk in 2015 to $5,631 per desk in 2017, and is expected to reach roughly $4,000 per desk this year.— The Economist</blockquote><p>Although this is not at all a new business model, Amazon Web Services has been decreasing the price of their services since its launch in 2006. Their constant growth as resulted cheaper services. Putting it simply, imagine having the costs of security, maintenance, power and cooling of 10 servers compared to a million servers, it is easy to understand that the price per server is cheaper the more servers you have.</p><p>This model has led to the exponential growth of companies such as Uber and Netflix that, sometimes aggressively, try to take over the monopoly for a specific market. They present a service that is cheaper and of higher quality than the ones existing at the moment. But that is not sustainable in its current form, they required rounds of great investment to keep expanding until they model becomes viable by scale. It’s a very risky all in play that requires the constant convincing of the shareholders that they should keep investing in an unprofitable business for the sake of long-term profitability.</p><p>To better understand a well designed long-term profitability model I highly advise reading Jeff Bezos <a href="http://media.corporate-ir.net/media_files/irol/97/97664/reports/Shareholderletter97.pdf">1997 letter to shareholders</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y3lA-vVcbJ-GXIta" /><figcaption>Photo by <a href="https://unsplash.com/@rawpixel?utm_source=medium&amp;utm_medium=referral">rawpixel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Customer Centric Economies</h3><blockquote>We start with the customer and we work backwards. — Jeff Bezos</blockquote><p>The two previous economic models are quite simple, to grasp and understand, and even envision. They can be seen being applied to a large set of companies. But the one model that I believe that has the most long-term benifits is Amazon’s Customer-Centric vision. At first, it seems just a normal company slogan but after seeing Jeff Bezos famous napkin model it struck me that by focusing on the customers and their needs is how you maintain a long-term standing that will directly benefit the company in the long run.</p><p>When amazon.com opened their website to other retailers to sell on their website, they probably were called crazy. “Are they just going to let the competition products being sold right next to their products?” Some thought that allowing other retailers to sell would just crumble Amazon’s business, they could sell the same products at cheaper prices on Amazon’s platform, who would want that?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*iMMSSuwNSa-f04nnBbjGxQ.jpeg" /><figcaption>Bezos Flying Wheel</figcaption></figure><p>This is a short-term vision of the play. By allowing others to sell on their platform, Amazon increased the variety of products they were selling. A customer that can get everything is looking for in the same platform, is a happy and a returning customer. This leads to more sellers on the platform, which results on more selection, leading to an increase in customer satisfaction, that then again creates more traffic and consequently more sellers. We can see here a spiral effect that in the end results on the growth of the platform. And as we’ve seen from economies of scale the bigger the platform the cheaper the prices and therefore an increase in customer satisfaction.</p><p>Another example is AWS, Amazon Web Services. It was born from the fact that the company had already made large investments in development and research for the underlying infrastructure that supported amazon.com website retail business. By opening their services to the public, and allowing other customers to use them for a fee, their services grew exponentially. By listening to their customers’ needs they now have the infrastructure that supports their website tested and deployed globally as the biggest and more advanced and with more services than any other cloud infrastructure while making money out of it.</p><h3>Summing up</h3><p>This three models are extremely coupled. The scale of your business impacts directly your ability to serve your customers on-demand and by consulting your customers and therefore achieving a customer centric relation the more you attract new customers and grow. Nevertheless, to achieve such scale you require large amounts of investment and most importantly a clear vision of your long-term goals.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a79471926b24" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to set up a NodeJS website with Authentication, Back office and Blog]]></title>
            <link>https://medium.com/@pedrodc/how-to-set-up-a-nodejs-website-with-authentication-back-office-and-blog-1a0b78ab4d42?source=rss-5bb7d15a6e16------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a0b78ab4d42</guid>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[node]]></category>
            <category><![CDATA[keystonejs]]></category>
            <dc:creator><![CDATA[Pedro Costa]]></dc:creator>
            <pubDate>Sun, 10 Sep 2017 14:40:07 GMT</pubDate>
            <atom:updated>2017-09-10T14:40:56.622Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<p>In this tutorial we’re going to build a NodeJS website with the following features: — User authentication — Back office — Blog (optional)</p><p>The first to parts of the tutorial are meant to quickly set you up and running. The third part is a more in depth explanation of some points I find important, so feel free to read them later as you need.</p><p>We’ll be using <a href="http://keystonejs.com/">KeystoneJS</a> , which is a promising framework, based on <a href="http://expressjs.com/">ExpressJS</a>, that allows for a quick setup of the previous features.</p><p><strong>Prerequisites:</strong> — <a href="https://nodejs.org/en/">Node.js 0.10+</a></p><h3>Initial set-up</h3><p><strong>Install the generator</strong> You’ll be using the KeystoneJS generator made with Yeoman. In your terminal run:</p><pre>$ npm install -g generator-keystone</pre><p><strong>Create a folder for your project</strong></p><pre>$ cd my-test-project</pre><p><strong>Run the generator</strong></p><pre>$ yo keystone</pre><p>The generator will ask you a few questions about what features you’d like to include.</p><p>The templates question may cause some confusion: *Would you like to use Pug, Nunjucks, Twig or Handlebars for templates? [pug | nunjucks | twig | hbs] ? *If you’re starting from scratch I recommend <a href="https://pugjs.org/api/getting-started.html">Pug</a> it’s simple and powerful, though if you’re porting an existing project, <a href="http://handlebarsjs.com/">Handlebars</a>(hbs) may be easier because you can use the existing html without any change.</p><p>I also strongly advice saying yes to extra comments in the code question.</p><p>Once finished your project will have an admin UI and all the dependencies installed.</p><p>So you just created your first KeystoneJS project, congrats!</p><p>To run your project you first have to connect to a database, so let’s continue.</p><h3>Getting started</h3><p><strong>Project Structure</strong></p><p>At first the amount of folders and files may seem overwhelming so let’s break it.</p><pre>|--models<br>|  Your application&#39;s database models<br>|--public<br>|  Static files (css, js, images, etc.) that are publicly available<br>|--routes<br>|  |--views<br>|  |  Your application&#39;s view controllers<br>|  |--index.js<br>|  |  Initialises your application&#39;s routes and views<br>|  |--middleware.js<br>|  |  Custom middleware for your routes<br>|--templates<br>|  |--layouts<br>|  |  Base .pug layouts go in here<br>|  |--mixins<br>|  |  Common .pug mixins go in here<br>|  |--views<br>|  |  Your application&#39;s view templates<br>|--updates<br>|  Data population and migration scripts<br>|--package.json<br>|  Project configuration for npm<br>|--keystone.js<br>|  Main script that starts your application</pre><p>Don’t worry we’ll explore more in depth some of these files later.</p><p><strong>Database set-up (the tricky part)</strong></p><p>The first thing you want to do is connect to a database. To make it easier I advise using a Database-as-a-Service product such as <a href="https://mlab.com/">mlab</a>.</p><p>After creating a free account create a new db by clicking on “+ Create new” then the Sandbox option (free), select Europe as the region and give it a name.</p><p>Now you have to setup the admin user of your newly created db. So click on the db and then on the tab Users and create a new user.</p><p>Next, copy the database URI which is at the top of the page (ex: mongodb://:@ds157853.mlab.com:57835/yournewdb) to the bottom of the .env file at the root of your project, like so: undefinedMONGO_URI=mongodb::@ds157853.mlab.com:57835/yournewdb (The .env should never be uploaded to your github repository!).</p><p>Notice that I had the ‘MONGO_URI=’ to the beginning. Also don’t forget to change and to the user and password you’ve just created at mLab.</p><p>Finally let’s see our project running, in your terminal type:</p><pre>$ node keystone</pre><p>Then open <a href="http://localhost:3000/">http://localhost:3000</a> to view it in your browser and voylá!</p><p>Now you may be wondering, that’s nice but what just happened?</p><p>Well when launching for the first time the script in /updates/0.0.1-admin.js will create your first User. Which is an admin. So if you sign-in in your website with the credentials you set up in the generator you’ll be able to manage all your website’s database related data such as users and blog posts. Admins have access to the keystone back office while regular Users may only have access to certain parts of your website, that’s up to you.</p><h3>Now you have a working web app and may feel free to explore the rest by yourself !</h3><p>There’s more to it so in the next part I will explain a bit more some of the features of the framework.</p><h3>The next part (aka. Understanding KeystoneJS)</h3><p><strong>Models</strong> So let’s start with the models folder, here we define our database models, such as the users or the blog posts in this example. In User.js define the user and add properties such as name, email, password and so on and set several properties. For example you can make the e-mail be unique and say which fields are required. You can get a more in depth understanding <a href="http://keystonejs.com/docs/database/">here</a>.</p><p><strong>keystone.js</strong> This is your app’s starting point. And the one of the two places where you can customize the Admin UI (aka back office).</p><p>The init function you’ll have something like:</p><pre>keystone.init({<br>	&#39;name&#39;: &#39;NIAEFEUP Website&#39;,<br>	&#39;brand&#39;: &#39;NIAEFEUP Website&#39;,<br>	&#39;less&#39;: &#39;public&#39;,<br>	&#39;static&#39;: &#39;public&#39;,<br>	&#39;favicon&#39;: &#39;public/favicon.ico&#39;,<br>	&#39;views&#39;: &#39;templates/views&#39;,<br>	&#39;view engine&#39;: &#39;pug&#39;,<br>	&#39;auto update&#39;: true,<br>	&#39;session&#39;: true,<br>	&#39;auth&#39;: true,<br>	&#39;user model&#39;: &#39;User&#39;,<br>});</pre><p>If you want to change the keystone logo you have to append to the function the following line:</p><pre>&#39;signin logo&#39;: [&#39;../../images/logo_aefeup.png&#39;, 230, 100],</pre><p>I also usually add the following lines in order to redirect to the home page when signing out/in and override the ‘/keystone’ in the url bar.</p><pre>&#39;signin redirect&#39; : &#39;/&#39;,<br>	&#39;signout redirect&#39; : &#39;/&#39;,<br>	&#39;signin url&#39; : &#39;/signin&#39;,<br>	&#39;signout url&#39; : &#39;/signout&#39;,</pre><p>That’s it about keystone.js everything else you’ll find well explained in the code comments or in the documentation.</p><p><strong>Routes</strong> This is probably the most important part. Here you define the behaviour of your webapp. The middleware.js is a starting point to some features in your app such as flash messages, enforcing user auth and so on. I’ll focus on the initLoclas function. Here you set the nav links of your page and to which view they’ll point.</p><p>But where the magic happens is in the index.js. Here you define the routes to your website, but what are the routes? Simply the links in the url bar that point to the desired page.</p><p>You can pass non mandatory arguments like this:</p><pre>app.get(&#39;/blog/:category?&#39;, routes.views.blog);</pre><p>or mandatory arguments such:</p><pre>app.get(&#39;/blog/post/:post&#39;, routes.views.post);</pre><p>or even none:</p><pre>app.get(&#39;/&#39;, routes.views.index);</pre><p>You can also enforce the user to be authenticated by adding one more parameter (‘middleware.requireUser’):</p><pre>app.get(&#39;/profile&#39;, middleware.requireUser, routes.views.profile);</pre><p>In the <strong>views</strong> folder is where you actually define the logic for each page and render the desired template.</p><p><strong>Templates</strong></p><p>First you have the ‘layouts’ folder, this is where you define the default layout of your website, the html header, body and footer. So this is the general layout and inside the body there’s only the header. The body content is defined in the views folder and will be displayed accordingly to url you’ve typed as defined in the routes. The views folder contains the body content for each page in our website. You can also implement your own version of errors page such as 404 and 500 by adding these templates in the errors folder.</p><p><strong>Public Assets</strong></p><p>Keystone will serve any static assets you place in the public directory. For example your css files or your js scripts and libs (bootstrap, jquery, etc) This path is specified in keystone.js by the static option.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a0b78ab4d42" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>