<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dimitris Dovinos on Medium]]></title>
        <description><![CDATA[Stories by Dimitris Dovinos on Medium]]></description>
        <link>https://medium.com/@dimitris.dovinos?source=rss-ee42848291ef------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*G8CDfuBUxErcHZ-s.</url>
            <title>Stories by Dimitris Dovinos on Medium</title>
            <link>https://medium.com/@dimitris.dovinos?source=rss-ee42848291ef------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 22:01:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dimitris.dovinos/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Basic encryption with AWS KMS (part 1 of 2)]]></title>
            <link>https://medium.com/codefully-io/basic-encryption-with-aws-kms-part-1-of-2-10bd4c079771?source=rss-ee42848291ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/10bd4c079771</guid>
            <category><![CDATA[key-management-service]]></category>
            <category><![CDATA[encryption]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[ruby]]></category>
            <dc:creator><![CDATA[Dimitris Dovinos]]></dc:creator>
            <pubDate>Sat, 22 Feb 2020 16:57:16 GMT</pubDate>
            <atom:updated>2020-02-22T16:57:16.144Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*Sgks7y9MR3TR3B2NtnOkbA.png" /></figure><p>Encryption is no longer a feature for the select few. Data breaches are common and government snooping, or just plain old snooping is also on the rise. We, the software developers, need to advise our customers to use encryption on all possible levels and not just rely on the fact that we use safe platforms for deploying applications and storing data.</p><p>Encryption allows us to have control of our data now and hopefully in the future too. Judging from <a href="https://www.theguardian.com/commentisfree/2019/oct/15/encryption-lose-privacy-us-uk-australia-facebook">Snowden’s articles</a> it looks like encryption is the last line of defense against organizations like the NSA. If the NSA struggles against cryptographic mechanisms, then your average hacker or hacker organization does not stand a chance. Which brings us to algorithms:</p><p><a href="https://en.wikipedia.org/wiki/Advanced_Encryption_Standard"><strong>AES-256</strong></a></p><p>It is silly trying to come up with an algorithm for encrypting data when your job is to put together software. This is an intellectual task best left to mathematicians and the spy agencies around the world (in particular, <a href="https://www.nsa.gov/">NSA</a> for USA, <a href="https://www.gchq.gov.uk/">GCHQ</a> for UK, and <a href="https://en.wikipedia.org/wiki/Unit_8200">unit 8200</a> for Israel). So don’t be too smart and accept that you have to use whatever is available out there. These guys have done their homework and you kind of have to accept the result of their efforts as the de facto truth. After all, we are dealing with maths, which are under the scrutiny of the academic community, which is not necessarily always connected to any of these institutions. Do some research and you will find that there is consensus (as of early 2020) as to which algorithms are safe and which are not. Go with that consensus.</p><p>The standard for symmetric encryption is the AES algorithm. There are libraries in pretty much every language that implement the algorithm. Take care regarding the library that you use and make sure that the implementation is correct. You cannot necessarily go over all the code of your libraries, but you have to avoid libraries that are released by individual developers. Let’s look at Ruby: You could get an AES implementation by a random GitHub developer or you could use the <a href="https://docs.ruby-lang.org/en/master/OpenSSL.html">OpenSSL</a> library released by the people who maintain Ruby itself. The choice is clear. This is too serious a matter to allow any room for doubt. Your tools must be up to date and maintained by the best.</p><p><strong>How it works</strong></p><p>The concept is trivially simple (perhaps a major understatement). Imagine you have a database table with the details of military personnel. This is plaintext information that you have to protect. By ‘protecting’ I mean that nobody should get access other than authorized users, and no-one should be able to alter the information without being detected.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4be673a1dc4aa45f7e97f5203ba99f05/href">https://medium.com/media/4be673a1dc4aa45f7e97f5203ba99f05/href</a></iframe><p>and here is the encrypted version:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/403eba93ed1510ecb5715f06b6a12125/href">https://medium.com/media/403eba93ed1510ecb5715f06b6a12125/href</a></iframe><p>The way to move from plaintext (unencrypted) to ciphertext (encrypted) is the <strong>key</strong>, which is nothing more than a string of characters. The same key is used both for encrypting and for decrypting and this is why the algorithm is called <a href="https://en.wikipedia.org/wiki/Symmetric-key_algorithm">symmetric</a>; As opposed to <a href="https://en.wikipedia.org/wiki/Public-key_cryptography">asymmetric</a>, where you use separate keys for encrypting and decrypting. In this case, the key was:</p><blockquote>4VZS/oCY73fTENgF8LyLq00B+bsXEHLlAFpRhoDXTKc=</blockquote><p>So now we are in a position where we can put together the ‘ecnrypt’ and ‘decrypt’ methods and we have managed to safeguard our data. Which brings us to the next problem of what do we do with the key. The key should ideally be easily accessible since for every row we pull from the database we will need it. Here are some alternatives:</p><p><strong>Keep it with the client (e.g. ask for it in every call to the webserver)</strong></p><p>Pros:</p><ul><li>You are no longer responsible for the key</li><li>The customer has full control. They can stop supplying the key and the data will no longer be accessible to anyone.</li></ul><p>Cons:</p><ul><li>You have to identify the client. In the case of a military personnel database, the client could be the solder, or the person using the computer, or the platoon leader. It boils down to who owns the data, which may not always be clear.</li><li>You have to manage the safe delivery of the key from the client’s computer to your server. Sending the key from the browser will expose it to several pieces of equipment that manage internet traffic and are prone to attacks or owned by actors that cannot be trusted. You will constantly expose the key to 3rd parties. (Yes, there will be https but still, there are ways to expose your data)</li><li>Should there be one key for each person, or is there one key for all, or is there one key per department? Sharing the key between different users is totally not secure.</li><li>The client will soon tire of having to supply the key on every request. In fact, it is not realistic asking for the key more than once every few hours.</li></ul><p>Soon the disadvantages of this approach pile up and make the system both unusable and insecure. It is very nice though that the client has the option of <strong>revoking</strong> the key and making their data inaccessible. This would have been very useful if the client was a company that stored its files on the system. It would allow the company to instantaneously make the data inaccessible.</p><p><strong>Keep it inside the database right next to the data</strong></p><p>Pros:</p><ul><li>You can encrypt decrypt with minimal overhead since you will get the key in the same database query that you get the data with.</li><li>You no longer expose the key to unnecessary journeys over the internet.</li><li>You do not bother the user with supplying the key on every request.</li><li>You have flexibility and control over which data you encrypt with which key. Should you decide you want one key per database row then you can do it.</li></ul><p>Cons:</p><ul><li>The system is no more secure than it used to be!! In the case your database is accessed by a malicious actor then they have both the key and the encrypted data.</li></ul><p>The disadvantage of this approach is so critical that it renders it unusable. But there are some great advantages to having the key close to the data. If only I could keep it <strong>close to the data</strong> and yet make it <strong>inaccessible</strong> in the case the database is compromised…</p><p><strong>Keep it with a Key Management Service!</strong></p><p>We will look into this approach on the next post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10bd4c079771" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codefully-io/basic-encryption-with-aws-kms-part-1-of-2-10bd4c079771">Basic encryption with AWS KMS (part 1 of 2)</a> was originally published in <a href="https://medium.com/codefully-io">codefully.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to communicate your MVP to the development team]]></title>
            <link>https://medium.com/codefully-io/how-to-communicate-your-mvp-to-the-development-team-a9cae360809e?source=rss-ee42848291ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/a9cae360809e</guid>
            <category><![CDATA[software]]></category>
            <category><![CDATA[mvp]]></category>
            <category><![CDATA[communication]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Dimitris Dovinos]]></dc:creator>
            <pubDate>Fri, 08 Nov 2019 11:59:14 GMT</pubDate>
            <atom:updated>2019-11-11T11:23:52.209Z</atom:updated>
            <content:encoded><![CDATA[<p>You may have come up with the greatest idea of the century which will bring you bags of money (or just save the world in general) and afford you the freedom to finally make your own, independent decisions. This is absolutely fantastic but as you have already guessed it is not enough. There is hardly anything more common than a great plan that never materialized. The shipwrecks of half baked attempts abound in the sea of entrepreneurship. As an exercise try to count the domains that you have bought over the years and compare them to the ideas that made it to actual projects.</p><p>Turning an idea to a piece of software requires two skills. Both are relatively easy to pick up and are useful under any circumstances that you need to interface with a technical team:</p><h3><strong>Learn to scope your project:</strong></h3><p>Before discussing the scoping of a project I would like to ask you to dig up your developer hat. We need to pick the brain of a software engineer and understand how she thinks. Great developers are geeks obsessed with their craft. They are engineers who love to create. Their ideal situation is when they can generate truly useful (or just impressive) tools with elegant code. Left to their own devices, engineers will create, or at least aim for, projects like Zaha Hadid’s <a href="https://en.wikipedia.org/wiki/Zaha_Hadid#Library_and_Learning_Center,_Vienna_University_of_Economics_and_Business,_Vienna,_Austria_(2008%E2%80%932013)">Vienna University</a> library with walls sloping at 35 degrees.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4ePqlNfChVUNH8MjLBEckw.jpeg" /></figure><p>It is a stunning building and one would have to consider the technical difficulty and cost behind such a feat. If the budget was no issue, then this looks like an engineer’s dream come true.</p><p>Something similar happens with software projects. Engineers have all these fantastic ideas and tools which they are dying to put to use.</p><p>Remove the developer hat for a moment and go back to your business persona. Left unchecked, you would probably also aim for the most feature-rich product possible. In fact, you and the software engineer, are creating a disaster; an accident waiting to happen. An over-ambitious project that requires infinite resources.</p><p><em>Back to reality</em></p><p>Truth is that nobody has an infinite amount of money or time. All our endeavors are constrained by the resources available. Time and money are top of my list, but I would also consider motivation and intellect. Your aim should be to identify what are the basic elements that without them the project would not be <strong>your project</strong>: Define the <strong>M</strong> in <strong>M</strong>VP. Your project should be ambitious but it should also be realistic and confined within the means available. Since we are using architecture for illustrating concepts, here is a design that I believe captures better the spirit:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5RyrbFPdRMAq8MI1y-j-Aw.jpeg" /><figcaption>The Betty and Gordon Moore Library in Cambridge.</figcaption></figure><p>The above building still looks impressive but it is also down to earth, has clear boundaries and it looks like it still gets the job done. Both buildings are libraries, both are functional, yet it is clear which one is the most impressive and also most expensive.</p><p><em>How to build the wall</em></p><p>Applications can always be improved and augmented. The demand for new features is endless, both from the business and the technical perspective. I have yet to see a piece of software that is still being used and all parties involved have agreed that it does not require further development. The only reason for stopping development or maintenance — which is also low pace development - is the death of a project. This fact, the bottomless appetite for more features, should be recognized as a dangerous pitfall for a new project.</p><p>You don’t have the luxury of endless development. You therefore have to limit your application’s scope. You should look into building a fence that determines the boundaries of what you want to achieve. The scoping of the project will help you to plan and dedicate resources to where it matters the most.</p><p>There are two steps for determining the limits of your system:</p><ol><li>Which are the user types? Identify the key players. In most systems, there are at least two types: the admins and the regular users.</li><li>How are the users interacting with the application? List all the activities that each user category will be doing within the application. (call each activity a <em>high-level use case</em>)</li></ol><p>Once you have the high-level use cases you have already established a perimeter of your application. The list of use cases can be further refined so that only the essential ones remain. Remember that this is an <strong>M</strong>VP (Minimum …).</p><p>The previous process, formally known as requirements elicitation, is complex and requires lots of expertise to get right. Your goal is not to reach a high level of detail, but to describe in broad strokes what is the basic functionality of the application.</p><h3>Learn to communicate with the developer</h3><p>The second step in creating the MVP is interfacing with the developers. There is no clear boundary between requirements and development. In fact, experienced developers will normally participate in the process of defining the application requirements, and requirements will change frequently during development. Even if this is the case, it is reasonable to be able to prepare some of the work before reaching the developer. This is a great way to get a head start and minimize any uncertainty with respect to the direction of the project. A list of use cases is an excellent first step toward effectively communicating your needs to the developer. As an extra bonus, having a list of requirements will allow the developers to jump faster to building the project rather than spend time discussing vague ideas.</p><p><em>Not all use cases are born equal</em></p><p>Use cases translate to features. For example, entering a user’s personal details translates to a ‘form’ with fields. The developer will interpret your request for data entry as an HTML form. Is this an expensive or a not so expensive feature? You would expect that in most technologies putting together a form is rather straightforward. A form with a few fields, like first name and last name, is rather simple.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*rqXUMTVPRBxjB-AeR7vEKg.png" /><figcaption>Simple form for signing up to Gmail</figcaption></figure><p>Both yourself and your developer have a clear understanding that a form is easy. You want though to stand out from the competition so you realize that you are not too certain of the fields that you would like to record for the user. Let’s use as an example an application that stores data on entertainers (singers, dancers, stand-up comedians, actors e.t.c). Each entertainer has unique talents (singing ability, acting, trapeze skills ) and each one of these skills may need a different description. You may even want to record short videos or audio for each user. Using as guide your experience from earlier you contact your developer and explain that the form needs to have a few more fields, but not for all users. This is starting to get more complex and complex usually means expensive. At this point, it is crucial that technical people and business people can see eye-to-eye.</p><p>The developer needs to understand your need and should explain that this is getting complicated, e.g. time-consuming and therefore expensive. From the business perspective, it has to be clear that this is not an essential feature, at least initially, and that time should not be wasted on implementing a fancy feature that can be completed later (not at the MVP stage). A good compromise may be that a free text field is introduced to store relevant information. This is the type of compromise that should be reached after communication between the developers and the business owner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*pFKGJB055QtION9xmbJguQ.jpeg" /><figcaption>complex features can cost!</figcaption></figure><p>The alternative would have been to set the developer down a long and pointless path and allocate resources in the wrong area. In other words, spend money without getting a result that suits the MVP.</p><p>The ability to quickly and effectively communicate with the developers cannot be overstressed. Good communication skills are not a ‘nice-to-have’ feature. They are essential ammunition in the armory and in many cases are even more important than technical skills. If a developer is not familiar with a particular technology, then it is only a matter of time for her to gain the knowledge. Poor communication skills, on the other hand, are difficult to improve in general and near impossible to acquire in the short timeframe of an MVP.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a9cae360809e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codefully-io/how-to-communicate-your-mvp-to-the-development-team-a9cae360809e">How to communicate your MVP to the development team</a> was originally published in <a href="https://medium.com/codefully-io">codefully.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Search engines as weapons and why Europe should build one.]]></title>
            <link>https://medium.com/codefully-io/search-engines-as-weapons-and-why-europe-should-build-one-9c142e427f5e?source=rss-ee42848291ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c142e427f5e</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[internet]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[european-union]]></category>
            <category><![CDATA[search-engines]]></category>
            <dc:creator><![CDATA[Dimitris Dovinos]]></dc:creator>
            <pubDate>Tue, 20 Aug 2019 15:36:27 GMT</pubDate>
            <atom:updated>2019-08-20T15:36:27.505Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4zPFJMZ6o5lT1uh-Rklrww.jpeg" /><figcaption>Money well spent? (image by <a href="https://unsplash.com/@the_don_photography">DON JACKSON-WYATT</a>)</figcaption></figure><p>Google has the lion’s share of the search engine market. There are regions, such as China, where this is not the case, but for the most part, Google rules the market. This is a bizarre situation for any product, which is precisely what Google’s search engine is. The right to use Google’s algorithms is bought in exchange for your data. You have made a rather peculiar purchase, where instead of money you have offered your information. Regardless of the rules of the transaction, the fact remains that you are buying a service — a product. Whether the price is right it remains to be seen.</p><p>Try and think of another product or service, with the exception of certain drugs, that is global and yet is a monopoly. There is an alternative source for nearly everything that you use or consume. Of course, there are state monopolies, e.g. a single electricity utility, but in this case, a government has decided to manage their power supply the best way they saw fit. At the government level, there are options for multiple sources of power and even power imports from other countries. There is no global monopoly on power. Since power has strategic value, a government may treat this as a public good that deserves extra attention. Similar arguments can be put forward for health, education, transport, and defense. These are all services that are too large to be handled by a corporation and too important to be left in the hands of a few powerful individuals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OB1w_X77qUQpzCa7xYodoQ.jpeg" /><figcaption>The Google (Image by <a href="https://unsplash.com/@pawel_czerwinski">Paweł Czerwiński</a>)</figcaption></figure><p>Yet, the one tool that we use dozens of times a day, for tasks ranging from the trivial (looking for cat gifs) to the most important (take your pick — many people even research medical symptoms) is in the hands of a few shareholders of a single corporation. We are all very happy to use this tool because we inadvertently believe the following, special, rule:</p><p><em>The corporation (Google) will always look out for the interests of its users</em>.</p><p>Corporations look out for the interests of their owners and perhaps their clients, not their users. Don’t forget that just because you use the search engine, you don’t qualify as a client. This rule is made up. It is a convenient cover story to justify a very useful service. It helps us sleep easy.</p><blockquote>The advertisers pay Google’s bills, and therefore they are the clients. The users are products dressed up as pseudo-clients.</blockquote><p>Don’t get me wrong. I have nothing against Google. I wish I came up with the idea and I honestly consider myself lucky to be living in a time that such a tool exists. I argue that this is such a tremendously significant tool that we have to have credible alternatives. We need alternative solutions not just because of the model that drives Google (you are the product — yet you assume to be the client) but also because Google is just a company. All companies eventually fail and google will be no exception.</p><h3>Mars — the only alternative</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vl3-YpFAY-Go5Qht_7dbOQ.jpeg" /><figcaption>Take me to Mars now! (Image by <a href="https://unsplash.com/@billjelen">Bill Jelen</a>)</figcaption></figure><p>Google’s size is probably enough to put anybody off from considering the possibility of starting a competitive venture. Honestly, why bother? Any attempt is doomed to fail or just get bought (which results in a monopoly again). People with lots of resources opt for projects like colonizing Mars, rather than get involved in a struggle to come up with a better search engine. I guess the probability of spending time on Mars is higher than building a new search engine. The odds don’t look that great, hence people steer clear.</p><p>So this is a dead-end, and it looks like the world has given up. It seems that the common wisdom is along the lines of: ‘Well, we have a search engine, let’s thank our lucky stars and pretend that it will always be there to serve us — oh, and it won’t bias the results to serve a special interest’. Something similar happened with banks. The banks did offer a great service until their owners decided to take greater risks, sell us more products (loans) and then finally fail. Then they held their ‘get out of goal card’: too big to fail! Therefore should be rescued (by the same people they let down). If google is not too big to fail then I don’t know what is.</p><h3>Weaponized information</h3><p>The sole purpose of a weapon is to wield power. Rockets, guns, and slingshots are all created not so much for destroying living things, but for forcing others to follow our will. A weapon’s strength lies not in its destructive power, but in its potential for destruction. That potential can be achieved with means other than a traditional weapon. Information can be weaponized and used for forcing somebody’s hand in a similar way as a traditional weapon. Look at how political events are influenced by social media. We have reached a point where corporations have information on our moves, our friends, our likes and dislikes. The information is not necessarily used against us, but it is recorded. It is a loaded gun waiting to go off.</p><p>Weaponized information does not have to be employed in the traditional sense of forcing individuals to follow a certain action. This is too crude a method with hardly any moral capital. There is a more subtle way of exercising power: Appeal to the masses. Demagogues have done this for millennia, and now we do not need a charismatic individual any longer. We have direct, personalized access to all individuals. Knowing users’ preferences and through trial and error (call it machine learning or any other fancy term) it is possible to influence groups of like-minded people. The problem of power has shifted from the arena of demagogues to that of mathematics. <em>Appealing to the masses legitimizes the cause since the true motives are hidden behind the veil of democracy.</em> If half of the population voted for a complete idiot willing to just follow the money, then<em> it must be right</em>. After all, it was a democratic process. The fact that these were vulnerable groups that were bombarded with propaganda and over simplistic arguments is only a footnote.</p><p>Since data can be used as a weapon, then perhaps we ought to look at it the same way we look a defense.</p><h3>Defense, a good example of money spent just because we decided so.</h3><p>As far as I know, most countries organize their armies at the national level. It is not that common to come across an army that belongs to an individual, and if you do, then there is something seriously wrong (run fast now!). Also, nobody owns a fighter jet as an individual. In fact, there are not that many countries that produce fighter jets. Buying a fighter jet is not just a purchase of equipment, it is an alliance between the two parties. Military power is so important that governments plan it at the highest level. The defense expenditure ranks as high as education and health. This is because we, the citizens, recognize that having freedom from our potential enemies (a rather non-tangible good in many countries of the west) requires sacrifices of a high order. <em>There is the political will to spend money</em> and therefore budgets are allocated.</p><p><em>There is no political will to secure our data resources</em> and therefore no money is allocated towards it. I do not have the faintest idea how much it would take to put together a decent search engine, but I assume that a few billion dollars should do it. Who has the cash reserves and the motivation to pursue the task though? Frankly, it seems almost too large an undertaking for a single country, but probably a decent size project for the European Union.</p><h3>The European Achilles heel</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_FTCj-ftygbXs5QYBy7MWQ.jpeg" /><figcaption>Liked by many, loved by none (Image by <br>Christian Wiediger)</figcaption></figure><p>Europe appears to lag in every technological respect compared to its transatlantic cousin. Although Tim Berners-Lee, a Brit, invented the world wide web in Europe, the Americans soon took over and turned an idea into a world economy. Europe lacks the American risk aptitude, is a fragmented market and does not have a strong central government. Bureaucracy is the norm, followed by bickering, followed by failure. It does though possess a budget that allows for grand projects and can muster enough support to spend what is required for a project of the search engine magnitude. I am not sure though that there are politicians sufficiently informed and motived to rise to such a task. I’d love to talk to whoever makes such decisions!</p><h3>Treat data resources like defense</h3><p>I am certain that should google go down <strong>right now</strong> the world would still carry on. The same holds for the internet in general, the power grids and even health care. The only thing that would stop the world is a meteorite or a black hole that decided to gobble us up. So what would happen if you lost your ability to search the internet or if all the data that you have provided was weaponized? The answer evades me. I would expect though that a major disruption to the economy combined with a political shift and the emergence of an alternative search engine would be a likely outcome. Similarly, if you did not have an army, the invasion from your hostile neighbor would most likely result in casualties, perhaps the loss of identity but nonetheless, the world would not end and power would eventually be restored. This is a relaxed approach to getting invaded that nobody can accept. The risk of losing the power to rule yourself, and others, drives the defense spending. Why is it that the dependence of the world economy on a single company does not trigger alarm bells when the mere suspicion of political power loss would result in an almost unlimited defense spending?</p><h3>History not to the rescue</h3><p>Defense budgets are history driven. Past experience has shown that whoever has the technological edge can rule the world. The data-ammunition is a relatively new resource. We do not have any great examples of what happened to a nation / civilization once they lost control of their data (or do we with the election of Trump and possibly Brexit?). Perhaps we need a small disaster so that there can be an entry in the history annals which marks the first major data breach incident. Then we may reconsider and come up with the resources.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c142e427f5e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codefully-io/search-engines-as-weapons-and-why-europe-should-build-one-9c142e427f5e">Search engines as weapons and why Europe should build one.</a> was originally published in <a href="https://medium.com/codefully-io">codefully.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Glue for loading data from a file to the database (Extract, Transform, Load)]]></title>
            <link>https://medium.com/codefully-io/aws-glue-for-loading-data-from-a-file-to-the-database-extract-transform-load-fe6b722e11b8?source=rss-ee42848291ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe6b722e11b8</guid>
            <category><![CDATA[import-export]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[glue]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Dimitris Dovinos]]></dc:creator>
            <pubDate>Mon, 15 Apr 2019 08:17:26 GMT</pubDate>
            <atom:updated>2020-04-20T07:48:54.219Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*6ELZ2fC4LRn7NktB_vqgWQ.png" /></figure><p>I have spent a rather large part of my time coding scripts for importing data from a file into the database. It is a common feature of an application to ask the user to upload a file with data. A usual use case is sales numbers. My approach up to now was to try and avoid at all costs the file upload and provide a form instead. This is fine for a low volume of data, but once there is more than a couple of rows then an upload makes sense.</p><p>The upload comes with lots of problems of its own. To name a few:</p><ol><li>Data validation. Users will upload data that may not conform to what you have been expecting. For example, you may have asked for csv and the user instead uploaded tab delimited (or excel!). Or, you have been expecting the data to be in a particular order but the user switched the columns’ order around. The list is endless and therefore the problem is also a hard one to solve.</li><li>Transforming the data into the format you want. Dates should be turned into a date object of some sort, numbers should be converted to numbers etc. There may also be logic that needs to be applied so that the data is converted to something that is useful for the application. For example, a product SKU may need to be supplemented by an id that is internal to your database or an email may need to be converted to lowercase or to a user id.</li><li>Work load. If this is a web application then it needs to remain responsive. This is not going to be the case if the user asks to have 1M rows to be processed. You need therefore to start thinking about extra workers and queues.</li></ol><p><a href="https://aws.amazon.com/glue/">AWS Glue</a> solves part of these problems.</p><h3>What is AWS Glue?</h3><p>It is a ‘wrapper’ service that sits on top of an Apache Spark environment. I don’t have a clue what <a href="https://spark.apache.org/">Apache Spark</a> does, and I guess this is the beauty of Glue: I can use the service without needing to be an expert.</p><h3>What does it do for the developer?</h3><p>It does 3 things:</p><ol><li>Glue <strong>discovers the schema</strong> of your data. You provide it with a sample file (I will walk you through it) and it figures out for itself how the data is structured. The data could be csv, json, xml, or custom (grok — see <a href="https://github.com/influxdata/telegraf/tree/master/plugins/parsers/grok">github</a> and <a href="https://logz.io/blog/logstash-grok/">logz.io</a>). This is the job of the <a href="https://docs.aws.amazon.com/glue/latest/dg/add-crawler.html"><strong>Crawler</strong></a>. You can also manually make changes to the detected schema. This is done just once (unless your data keeps changing shape — in which case you need to run the crawler again in order to update the schema).</li><li>Based on the data schema and its source/destination, Glue <strong>will help you create a script</strong> (a <a href="https://docs.aws.amazon.com/glue/latest/dg/add-job.html">job</a>) for importing the data, transforming it and then loading it to a database. The <em>job</em> is the heart of the service, and Glue does a good job (no pun intended) of getting you started without any prior knowledge. Glue acts like a wizard which helps you generate a piece of code. The job creation part should be a one-off.</li><li>Provided that you have established a <strong>schema</strong>, Glue can run the <strong>job</strong>, <strong>read the data</strong> and <strong>load</strong> it to a database like Postgres (or just dump it on an s3 folder). This is the step that needs to be repeated every time there is a new upload.</li></ol><h3>Sample data</h3><p>There are many great sources available. I found some interesting examples in <a href="http://insideairbnb.com/get-the-data.html">Airbnb</a> and also in <a href="https://registry.opendata.aws/">AWS open data</a>. I ended up with the ‘<a href="http://data.insideairbnb.com/united-states/nc/asheville/2019-02-17/visualisations/listings.csv">listings</a>’ of Airbnb. It is a csv file with a few thousand rows (for the city of Athens) and about a dozen fields (columns), a rather typical example of what most applications have to deal with. The data file was stored in an s3 bucket named <em>gluedatasource</em>. The name is obviously irrelevant, but we will use it in a minute.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/67f09de7a68afbd503dc3d440ecefa36/href">https://medium.com/media/67f09de7a68afbd503dc3d440ecefa36/href</a></iframe><p>The columns include numbers, strings, coordinates, and dates.</p><h3>The big picture</h3><p>The plan is to upload my data file to an S3 folder, ask Glue to do it&#39;s magic and output the data to an RDS Postgres. The source and sink could have been different but this seemed like a workflow that fits well with a web application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*dM8H_NgBbQyC7HnUf4k3FA.png" /><figcaption>Workflow for importing data from a CSV to a Database</figcaption></figure><h3>Crawl it!</h3><p>Firstly Glue has to crawl the file in order to discover the data schema. We need to create a crawler.</p><p>Select to create a new crawler and then give it a name:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*Lu2qOl2tvmO1LXn2W7WqSw.png" /></figure><p>Define the path from which the crawler will get the file(s) which it should analyze. In this case, it is the top path of an S3 bucket:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*awc4rktsIHKMiQB-Auy6mQ.png" /></figure><p>The crawler will need access to the bucket, so there should be an IAM role that allows it. Chances are you won’t have one ready, so go ahead and create a new one. I called mine AWSGlueServiceRole-deletemeRole since I wanted to throw it away later:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*DM2zdGZH7ZCHtIiwnOI2bg.png" /></figure><p>Decide when will the job run. I wanted it to run on demand:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*ppX_TxvFQ2qEQMzLRw14DQ.png" /></figure><p>Decide where the crawler will store the data it will generate. The crawler’s output is a schema of the data that it analyzes. The schema is probably a json file like <a href="https://gist.github.com/dimitrisdovinos/1db50e5ec547b8d277f482dc6f940497">this one</a>. On the top of the menu of the left there is a link for the databases that can be used for the schema. These are not databases in the traditional sense. It is just a way to organize your information. Perhaps the naming is not that great. I selected sampledb (which I created earlier).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*0loSOLsbyO2SMrbc23jo9w.png" /></figure><p>Finally, review and complete the crawler creation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*w-sBxRkBSk6bdikjVYFDcg.png" /></figure><p>Once the crawler is in place then select it and run it. It should return with something similar to this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*p50DpEMMumV4AIQmfRKOfA.png" /><figcaption>Detected schema</figcaption></figure><p>The schema reflects the file. Here is the file again for an easy comparison:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/67f09de7a68afbd503dc3d440ecefa36/href">https://medium.com/media/67f09de7a68afbd503dc3d440ecefa36/href</a></iframe><h3>Prepare for the ‘job’!</h3><p>Glue now has a schema for the data that we want to import. It still does not know though what to do with the data. We want to load the data to a database so we have to create a job for doing that. The job is a piece of code (I assume the kind of code that Spark users would be familiar with) that Glue helps you generate.</p><p>Before getting there you need to allow Glue access to your db. For the connection to take place you need to have the db inside a VPC (creating a VPC is beyond the scope of this post, so I won’t bother with details) with a security group that allows access to Glue.</p><p>Select the security group from the dashboard of the RDS database:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eMcVZXVJmyxwrkG0g5D3wA.png" /><figcaption>RDS dashboard</figcaption></figure><p>Then change the rules (inbound and outbound) to allow for Glue to access the db. For the purpose of this experiment set ALL TCP traffic from all sources (0.0.0.0/0) as both an incoming and an outbound rule. The is <strong>not safe</strong>. Change it for your production setup to appropriate values.</p><h3>RDS connection</h3><p>Glue requires that you create a connection to your database (the data sink) so that it knows how to connect to it. Select Connections (in the Databases Menu of Glue) and ‘Add connection’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*MYTpyGAV_IxrDdkuC77Bvg.png" /></figure><p>Then enter the database name, your username, and the password.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*Qehi36WYC-LXiGYG5DdVcg.png" /></figure><p>Review and submit</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*fM5W95nnGzzusYLXddVY4Q.png" /></figure><h3>S3 endpoint</h3><p>There is a similar issue with S3. You need to allow access to S3 which implies the creation of a <a href="https://docs.aws.amazon.com/vpc/latest/userguide/vpc-endpoints.html">VPC endpoint</a>. Navigate to the VPC where your RDS database resides and select Endpoints -&gt; Create Endpoint. Basically, you are going to allow connections to be established between the VPC resources and the S3 bucket (S3 buckets do not live inside a VPC).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qPzFj4UMN4m2D2jRvr2nGA.png" /></figure><h3>Start the job wizard</h3><p>Here is the juicy part! Start the job wizard. Nearly everything is automatically populated. Make sure to go for python and for ‘A proposed script generated by AWS’:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZSQAl_laOnO23m-TuCvPKQ.png" /></figure><p>Then select where is the file that you want to parse (the crawler has automatically created a source (in Databases -&gt; Tables) which you can use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*tv0RGQVx-BNRwDQD2r23YA.png" /></figure><p>and choose where should the output be written to (you created earlier a connection which should show up here now).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*QHq9uQ_iVYUO0Uc2-r7sXA.png" /></figure><p>Drum roll…. The moment you have been waiting for. Map the data to the database table. On the left, you see the columns as they emerge from the crawling. On the right, you can decide what the output will be like or you could drop/add more columns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o3v2JcgvKMQOSNjOWOpj1Q.png" /></figure><p>Then Glue will generate the job code. The end result should look similar to this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1t-Rdc9oFCqElpK4z2ahYw.png" /><figcaption>Code for completing the job.</figcaption></figure><p>This is what you have been working for! This is the script that explains to Glue how to load your data and then map it and save it to the database. Run the script and you should have a new table in the database populated with your data. After running the script check that the new table has the correct schema. Fortunately, in my example, it did look good so I did not have to write a single line of SQL!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*FBDHVCtjs0yfK-lSGeqgvw.png" /><figcaption>Postgres table, as created (and populated) by Glue.</figcaption></figure><h3>Cost</h3><p>It is all relative. Each time you run a job there is a minimum charge of $0.44. My jobs were small so I was always hitting the minimum charge (about 10K rows). I think that you have to decide for yourself if this is cost effective. Consider though the alternative: spend time developing — reusing code for performing the import and allocate resources and logic for background jobs. If you have to deal with large amounts of data (e.g. more than you would want to upload over a browser) then it seems that you also have to deal with handling large files. Overall, I’d say that Glue is cost-effective given the massive savings in development time, and it would probably be a no-brainer for larger files (I would consider anything above 5Mb to be larger than what a web app has to deal with).</p><h3>It would be nice if…</h3><p>Glue is a developers’ tool. Before using it in any application you would have to build a layer wrapping its functionality. A tricky part that I would love to have a solution for (that does not involve me writing it), is the data validation. When users upload data they need instant feedback along with ways to fix/ignore the data at the moment of the upload. Currently, we do this with <a href="https://reactjs.org/">React</a> and various other mostly front end approaches. This is hard work which I would love to have done for me!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe6b722e11b8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codefully-io/aws-glue-for-loading-data-from-a-file-to-the-database-extract-transform-load-fe6b722e11b8">AWS Glue for loading data from a file to the database (Extract, Transform, Load)</a> was originally published in <a href="https://medium.com/codefully-io">codefully.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Authentication with AWS Cognito, React and express.]]></title>
            <link>https://medium.com/codefully-io/authentication-with-aws-cognito-react-and-express-b3acf17d37e6?source=rss-ee42848291ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/b3acf17d37e6</guid>
            <category><![CDATA[cognito]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[jwt]]></category>
            <category><![CDATA[authentication]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Dimitris Dovinos]]></dc:creator>
            <pubDate>Thu, 11 Apr 2019 18:04:02 GMT</pubDate>
            <atom:updated>2019-04-11T18:04:02.398Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*m5Wc6pyKsGIf2SnYbUH3PQ.png" /><figcaption>AWS Cognito logo (from freeicons.io)</figcaption></figure><p>At <a href="https://codefully.io/">codefully.io</a> we try to use as much as possible low cost (technically and economically) — high-performance and low maintenance solutions. Here is the setup and the background behind using AWS <a href="https://aws.amazon.com/cognito/">Cognito</a>.</p><h4>The need</h4><p>We wanted to put together a <a href="https://reactjs.org/">React</a> front end application which would initially be launched on a web client but would eventually work also with Android and iOS (<a href="https://facebook.github.io/react-native/">React Native</a> ). We decided to decouple the front end from the back end for maximum flexibility. The back end is based on <a href="https://nodejs.org/en/">node</a> and <a href="https://expressjs.com/">express</a> with <a href="http://docs.sequelizejs.com/">sequelize</a> for persistence, whereas the front end is a <a href="https://reactjs.org/">React</a> application. While designing the application we were keeping in mind that eventually, we would need to support native clients. Having an API based back-end is a great solution towards that end.</p><p>An API backend allows for interaction with the application from any possible platform (web browser, native, another application etc). All the clients can be independent but all do require a single common piece of functionality: <em>authentication</em>. Here comes AWS’s <a href="https://aws.amazon.com/cognito/">Cognito</a> to the rescue. Cognito manages the sign in and sign up process as well as any other aspect of authentication. It works for a <a href="https://docs.aws.amazon.com/cognito/latest/developerguide/setting-up-the-javascript-sdk.html">javascript application</a> (our case just now) as well as for an <a href="https://docs.aws.amazon.com/cognito/latest/developerguide/walkthrough-using-the-ios-sdk.html">iOS</a> or an <a href="https://docs.aws.amazon.com/cognito/latest/developerguide/setting-up-android-sdk.html">Android</a> App. It also integrates with <a href="https://docs.aws.amazon.com/cognito/latest/developerguide/facebook.html">Facebook</a> and <a href="https://docs.aws.amazon.com/cognito/latest/developerguide/google.html">Google</a> authentication (as well as <a href="https://docs.aws.amazon.com/cognito/latest/developerguide/amazon.html">Amazon’s</a> own provider) and other <a href="https://docs.aws.amazon.com/cognito/latest/developerguide/saml-identity-provider.html">SAML providers</a>. It does all you may ever ask from an authentication system, and being an AWS product it comes at a low <a href="https://aws.amazon.com/cognito/pricing/">cost</a> and guaranteed long term support.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*yTadjB_IhI_RaxghTcX2-g.png" /><figcaption>A standard architecture for authenticating with Cognito</figcaption></figure><p><a href="https://aws.amazon.com/cognito/">Cognito</a> future proofed our design and allowed us to focus on the development of the business features rather than on crucial but without much value for the end customers aspects, such as the authentication. Nobody is going to give you praise for a safe and smooth authentication setup, yet everybody expects it and will complain if it does not work flawlessly.</p><p><strong>How </strong><a href="https://aws.amazon.com/cognito/"><strong>Cognito</strong></a><strong> works for a web client based application.</strong></p><p>The sign-up process is not that complicated and we can leave it for later (part II). The sign-in is more interesting. First you have to setup a <a href="https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-identity-pools.html">User Pool</a>. I will assume that you are familiar with some of the basics of Cognito. If not, then the <a href="https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-identity-pools.html">AWS documentation</a> is a good starting point. The only part where I struggled was setting up the App client portion. I am including a screenshot just in case there are others that that need some help.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LEXxUDrM8slujOuuRwputA.png" /><figcaption>Cognito Application Client settings</figcaption></figure><p><strong>Sign-in</strong></p><p>Once a user reaches your site then you will redirect them to the Cognito URL that is available in the <a href="https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-domain.html">Domain name</a> section. This will be something like:</p><pre><a href="https://my-application.auth.us-east-1.amazoncognito.com/"><strong>https://my-application.auth.us-east-1.amazoncognito.com</strong></a></pre><p>unless you have setup your own domain. Cognito will then present a sign in form, which can be customized via the <a href="https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-ux.html">UI customization</a> section. You can see that in our case we have allowed for a logo and have performed other minor modifications to the card. Overall you do not have a lot of control but you have just enough to make the sign-in form look relevant to your application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*H_zD4sWeZPFtkW5bInlrNg.png" /><figcaption>Cognito sign in</figcaption></figure><p>Provided that the user enters correctly their credentials then she will be redirected to your site. Cognito will call a URL on your site with a parameter that includes the token or <strong>code</strong>. For example:</p><pre><a href="https://mywebsite.com/auth/callback?code=3ba07b14-4e90-4e52-9309-7867b01dc8ed">https://my-website.com/auth/callback?code=3ba07b14-4e90-4e52-9309-7867b01dc8ed</a></pre><p>Your application must figure out how to convert this code “3ba07b14–4e90–4e52–9309–7867b01dc8ed” into a user and log them in. You need to get the user information that is behind the code, which is available at the <a href="https://docs.aws.amazon.com/cognito/latest/developerguide/token-endpoint.html">TOKEN endpoint</a>. The next method does just that. It performs a POST call to the token endpoint using <a href="https://www.npmjs.com/package/axios">axios</a> for the communication. The debug info can be of course ignored or removed. The URLs are based on a config file which could be fed with parameters from the environment (the config file can be found at the end).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bd9de8e166771fc2deca7fe81f5f2dc4/href">https://medium.com/media/bd9de8e166771fc2deca7fe81f5f2dc4/href</a></iframe><p>The axios POST object will look something like this (straight from the log output):</p><pre>AWS oauth2/token request parameters: {&quot;method&quot;:&quot;post&quot;,&quot;url&quot;:&quot;<a href="https://collide-v2.auth.us-east-1.amazoncognito.com/oauth2/token">https://my-cognito-url.auth.us-east-1.amazoncognito.com/oauth2/token</a>&quot;,&quot;data&quot;:&quot;grant_type=authorization_code&amp;client_id=7uanuaesvjunxxxxxxxxxxx&amp;code=<a href="https://mywebsite.com/auth/callback?code=3ba07b14-4e90-4e52-9309-7867b01dc8ed">3ba07b14-4e90-4e52-9309-7867b01dc8ed</a>&amp;scope=profile&amp;redirect_uri=http%3A%2F%2Fmy-website.com%2Fauth%2Fcallback&quot;,&quot;auth&quot;:{&quot;username&quot;:&quot;7uanuaesvjunxxxxxxxxxxx&quot;,&quot;password&quot;:&quot;xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&quot;}}</pre><p>and the response will be similar, but much longer than this:</p><pre>{&quot;id_token&quot;:&quot;LOTSANDLOTSOFCHARACTERSIDTOKEN&quot;,<br>&quot;access_token&quot;:&quot;LOTSANDLOTSOFCHARACTERS&quot;,<br>&quot;refresh_token&quot;:&quot;LOTSANDLOTSOFCHARACTERS&quot;,<br>&quot;expires_in&quot;:3600,<br>&quot;token_type&quot;:&quot;Bearer&quot;}</pre><p>All the information regarding the user is inside the id_token. You can copy paste the contents of the id_token at <a href="https://jwt.io/">jwt.io</a> and you will see all the different pieces of information that come back from <a href="https://aws.amazon.com/cognito/">Cognito</a>. At this point, you have the user but you have not verified that the sender of the information is indeed AWS. You can verify and extract the user information using the following method. The input to <em>getEmailFromCode</em> is the code that was sent to the callback URL (“3ba07b14–4e90–4e52–9309–7867b01dc8ed”) after the user successfully entered their credentials. <em>getEmailFromCode</em> includes a call to <em>awsCallTokenEndpoint</em>. (probably best to include both methods in the same file)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/44e0ce379b7710a9c8b683f20a7c22db/href">https://medium.com/media/44e0ce379b7710a9c8b683f20a7c22db/href</a></iframe><p>The above method still requires logic for handling failures in the verification process and for dealing with caching of calls to the <a href="https://aws.amazon.com/premiumsupport/knowledge-center/decode-verify-cognito-json-token/">jwksUrl</a>. Neither is implemented here.</p><p><strong>JWT token</strong></p><p>With the user information, it is up to you to decide what to do next. In our case, we wanted to create a <a href="https://jwt.io/">jwt token</a> and forward it to the front end.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ae0c58bbbc8e7422abc5c776983c7c3/href">https://medium.com/media/7ae0c58bbbc8e7422abc5c776983c7c3/href</a></iframe><p>The user object would have been retrieved from your database based on the data that you got from <em>getEmailFromCode</em>. This last part is not really <a href="https://aws.amazon.com/cognito/">Cognito</a> related, but it does give an idea of where the authentication process ends.</p><p>Finally, I am including the config file. Again this is not relevant to <a href="https://aws.amazon.com/cognito/">Cognito</a> but it shows a way of accessing variables that should not be embedded in the code (passwords, API keys):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/098ed80a1d78cfc169b4dddbd8e87083/href">https://medium.com/media/098ed80a1d78cfc169b4dddbd8e87083/href</a></iframe><p><strong>Case sensitivity — the (minor?) catch</strong></p><p>Try registering with auser@codefully.io and then try again a<strong>U</strong>ser@codefully.io. <em>Horror of horrors!!</em> Congnito has registered two different users. I am sure there is a standard that Cognito follows and that this is not a random decision. Nonetheless, it did shake my confidence when I figured this out and realized that I have to work around the issue. There are discussions on the matter with solutions of varying complexity, but I wish that AWS would make case sensitivity a setup option. I don’t see why I need to implement logic for dealing with case sensitivity of emails (or usernames) when everybody treats emails as case insensitive.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b3acf17d37e6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codefully-io/authentication-with-aws-cognito-react-and-express-b3acf17d37e6">Authentication with AWS Cognito, React and express.</a> was originally published in <a href="https://medium.com/codefully-io">codefully.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[React forms (validation) with Formik and Material-UI]]></title>
            <link>https://medium.com/codefully-io/react-forms-validation-with-formik-and-material-ui-1adf0c1cae5c?source=rss-ee42848291ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/1adf0c1cae5c</guid>
            <category><![CDATA[formik]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[material-ui]]></category>
            <category><![CDATA[yup]]></category>
            <dc:creator><![CDATA[Dimitris Dovinos]]></dc:creator>
            <pubDate>Fri, 05 Apr 2019 10:23:56 GMT</pubDate>
            <atom:updated>2019-04-05T10:24:31.438Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*Dqna3OonSq_N_FvqiDrNXg.png" /></figure><p>The React documentation on how to deal with <a href="https://reactjs.org/docs/forms.html">forms</a> leaves a lot to the imagination. And perhaps this is fine since forms could potentially be business-logic-heavy. React’s job is to help with the UI and not with the underlying business logic.</p><h3>The problem</h3><p><em>On a single page React app, which uses </em><a href="https://material-ui.com/"><em>Material-UI</em></a><em> components, introduce a contact form</em>.</p><p>My goals were:</p><ol><li>keep using Material-UI components like <a href="https://material-ui.com/demos/text-fields/#text-fields">&lt;Textfield&gt;</a></li><li>have very basic validation which I don’t have to write from scratch ( ’cause I am lazy and I make mistakes — or I may want to extend it in the future-or both)</li><li>make the form a component that I could reuse on my next project (again, ’cause I am lazy)</li></ol><p>The validation part was what got me scratching my head the most. All I needed was to validate an email address and mark a couple of fields as required. Keeping in mind that the app had no back end, it seemed obvious that <a href="https://developer.mozilla.org/en-US/docs/Learn/HTML/Forms/Form_validation">built-in HTML5 form validation</a> should be enough. I am sure there is a way to do this but I just did not find it (please enlighten me, dear reader!).</p><p>Long story short, I figured that if you want to keep all the goodies that come from Material-UI and remain within the React way of doing things, then you have to compromise and increase the complexity a little bit. There are <a href="https://www.npmtrends.com/final-form-vs-formik-vs-react-form-vs-react-redux-form">js libraries</a> that deal with forms which addressed my problem. The React documentation suggests <a href="https://github.com/jaredpalmer/formik">Formik</a>, which I found to be the most relevant. Formik stood out because of its simplicity and also because of the video of its author (Jared Palmer) which made a lot of sense. Not the most scientific way of choosing tools, but when you want results fast, you have to also go with gut feeling. Formik <em>did smell right!</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FoiNtnehlaTo%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DoiNtnehlaTo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FoiNtnehlaTo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e3d3259835ec28ab6a3b2c362802e6c6/href">https://medium.com/media/e3d3259835ec28ab6a3b2c362802e6c6/href</a></iframe><p><em>Validation</em></p><p>There are a few <a href="https://www.npmtrends.com/ajv-vs-joi-vs-yup">js libraries</a> for validating data, but <a href="https://github.com/jquense/yup">yup</a> seemed very simple and was also recommended by Formik. Once formik was introduced, it was very easy to plug any validation library, so I was not too concerned about extensively researching it. I am certain that <a href="https://github.com/epoberezkin/ajv">ajv</a> and <a href="https://github.com/hapijs/joi">joi</a> would have done for my task (which was very simple).</p><p><em>Presentation</em></p><p>We want to stick with Material-UI yet we want to plug in some business logic that takes care of validation and error messages. The <a href="https://material-ui.com/api/text-field/">TextField API</a> is designed so that it allows <strong>us</strong> <strong>to control the data</strong> it displays while <strong>it</strong> <strong>deals with the presentation</strong> of the data. The API allows for the <em>value, </em>the <em>onChange, </em>and the<em> onBlur</em> event handlers, the <em>helperText </em>and the <em>error </em>flag to be controlled from outside the component.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05e7ee8cec49cd1afef3b39e044a4116/href">https://medium.com/media/05e7ee8cec49cd1afef3b39e044a4116/href</a></iframe><p>Formik deals with how the data will be passed around the form (and most importantly through the validation). Join the 2 and you have a <strong>great looking form with validation</strong> that is easy put together and maintain. Most importantly, the presentation layer (Material-UI) is completely separate from the form handling layer (Formik). We could easily swap either without affecting any.</p><p>Here is the code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/865aafef902ac07d663ae426f7dd6e67/href">https://medium.com/media/865aafef902ac07d663ae426f7dd6e67/href</a></iframe><p>Before copy — pasting please notice just one thing: <em>the form attributes (email, name, comment) are not stored in my component’s state!!!! </em>Sure, I keep a couple of variables in the state (open, isSubmitionCompleted), but both have to do with the presentation of the component. Formik has encapsulated all the form attributes. Kind of cool.</p><p>As a footnote: the endpoint is an <a href="https://aws.amazon.com/blogs/architecture/create-dynamic-contact-forms-for-s3-static-websites-using-aws-lambda-amazon-api-gateway-and-amazon-ses/">AWS lambda function</a> which generates an email with the contact form data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1adf0c1cae5c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codefully-io/react-forms-validation-with-formik-and-material-ui-1adf0c1cae5c">React forms (validation) with Formik and Material-UI</a> was originally published in <a href="https://medium.com/codefully-io">codefully.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Loading data to PostgreSQL]]></title>
            <link>https://medium.com/codefully-io/loading-data-to-postgresql-a3b687ee6416?source=rss-ee42848291ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3b687ee6416</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Dimitris Dovinos]]></dc:creator>
            <pubDate>Mon, 01 Apr 2019 13:53:40 GMT</pubDate>
            <atom:updated>2019-04-05T12:04:25.492Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yOPtEemXKmh4yEOxGmpebA.jpeg" /></figure><p>Loading data into an existing database is part of the lifecycle of the maintenance or the original setup of an application. At <a href="https://codefully.io/">codefully.io</a> we have to deal with a variety of data types and sizes. We work with Postgres and we employ different types of techniques and tools depending on the size, quality, and type of data.</p><h3>Tools</h3><ol><li>Text editors. I use <a href="https://www.vim.org/">vim</a> and <a href="https://www.sublimetext.com/">sublime</a>. <a href="https://atom.io/">Atom</a> seems to struggle with large datasets (a few thousand lines). I have not tested <a href="https://code.visualstudio.com/">VisualStudio</a> yet.</li><li><a href="http://postgresguide.com/utilities/psql.html">Psql</a> command line. I rarely use anything more complex that <a href="http://postgresguide.com/utilities/psql.html">Psql</a>. There are plenty of clients (<a href="https://www.dbvis.com/">DbVisualizer</a>, <a href="https://www.navicat.com/en/">Navicat</a>, etc) but they all seem to add bulk to the process and rarely provide more information that the <a href="http://postgresguide.com/utilities/psql.html">Psql</a> console.</li></ol><h3>File Organization</h3><p>Every change that we make to the database, no matter the size (a few kB or a few GB), we always keep it under version control. We manage several databases and we have a <a href="https://git-scm.com/">git</a> repository for each database where we store each change. Version control is not that important in itself in this case. SQL scripts tend to be run and forgotten — as opposed to code that needs maintenance. Being able to share the migrations with other developers and store them is the great advantage of using a version control system like <a href="https://git-scm.com/">git</a> (or <a href="https://www.mercurial-scm.org/">mercurial</a> occasionally). Each migration is stored under a separate folder with a date and a brief description. The folder includes files, each of which is usually a migration of a single table, or of a particular area of the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/423/1*XIUwsvgqoPTit8C_PyXSrg.png" /><figcaption>File system of database migrations</figcaption></figure><p>Having a system for the migrations can prove to be a lifesaver. Keeping track of what you have already done will help you debug issues down the road. Hopefully, you will never have to revisit old migrations but it is not unheard of!</p><h3>The data is too big!!</h3><p>Data size is a matter of perspective. For an application that holds blog entries, you may consider a few thousand entries to be too many. A few thousand rows may be a minute’s worth of data for an application that tracks sales of small items. I consider any file that you cannot visually inspect and cannot, therefore, make manual changes to, to be too big. The techniques and tools are similar whether you deal with one hundred rows or 100 million rows:</p><p><strong>Lots of memory</strong> and hard drive access speed. Our setups (Linux) have at least 16GB of RAM and fairly fast solid state hard drives. I think that this is a minimum requirement for any development environment.</p><p>You need to select the <strong>correct editor</strong>. <a href="https://www.sublimetext.com/">Sublime</a> is great for handling a few hundred thousand rows (50–100 MB). The editor will allow you to do some basic operations to the file: edit the top or bottom area and quick visual inspection in case something really sticks out (for example a large gap in the data). Most modern editors support multiple cursors which is a very useful tool for manipulating a text file. Beyond that, you need to look into vim<a href="https://www.vim.org/">.</a></p><p>Search and replace operations are best done with <a href="https://www.vim.org/">vim</a> and <a href="https://regexr.com/">regular expressions</a>. I find <strong>regex</strong> a pain but I have to admit that there is nothing that comes even close for dealing with big files and complex changes. My favorite examples:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/179/1*ffqxu-GyCLXKmvDsTp62sQ.png" /><figcaption>replace consecutive tabs with nulls.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/157/1*YhYfFisE1uYdU7sEFqSXmA.png" /><figcaption>Replace tabs at the end of a file with nulls.</figcaption></figure><p>These two examples are lifesavers. Hit the colon sign in <a href="https://www.vim.org/">vim</a> and you will be in command mode. You can then enter the top one (<strong>%s/\t\t/\t\\N\t/g</strong>) for replacing consecutive tabs with nulls, or the one below (<strong>%s/\t$/\t\\N/g</strong>) for replacing tabs at the end of a file.</p><p>Replacing millions of entries is not a problem with <a href="https://www.vim.org/">vim</a>. The amount of changes that you need to do depends on how well formed is your initial data. Usually, you will need to remove empty entries with nulls ( \N ), or perhaps remove some special characters that Postgres does not like.</p><h3>The data is (always) badly formed!</h3><p>The data has come from the sales department (or some other area that does not care about you!). The sale people are of course oblivious to your table structure and you realize that in order to reach something that is close to your schema you will need lots and lots of changes. For example:</p><ol><li><strong>Manipulation and concatenation/splitting of strings</strong>: the input data has the first and the last name joined together but you store them separately! (we will analyze this one further down)</li><li><strong>Calculations based on the input data</strong>: For example, you need to know how much each item costs in a multi-pack , but the input data includes the total price and the number of items in the pack.</li><li><strong>References to other tables</strong>: For example, each row references a user but the input data includes just the user&#39;s email (not the foreign key).</li></ol><p>All these are common problems that involve at least an extra processing step. Luckily you have the best tool available for the job: Postgres itself!</p><p>Postgres offers <a href="https://www.postgresql.org/docs/9.3/sql-createtable.html"><strong>temporary tables</strong> </a>which are “automatically dropped at the end of the session”. They are great for doing work that should not be a permanent part of the database. Imagine that you have a sales table which includes the first and last name of the customer as well as the total expenditure. The accounting department has sent you the following data, and your job is to enter it into your database.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e9d478bf28d54c0e8e82856b3db01495/href">https://medium.com/media/e9d478bf28d54c0e8e82856b3db01495/href</a></iframe><p>Your table looks like (enter \d sales in psql to get a description):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*wSR8V1mzvdb_-EmU1APeBQ.png" /><figcaption>Target table.</figcaption></figure><p>Your original data have the first and the last name in a single string, while you need them separated. Create a temporary table that holds the data that is to be loaded. Base the new table on the existing table, but also include some extra columns for data that needs to be further processed.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8039f3bd4af1e94a143af718b4c0cb9a/href">https://medium.com/media/8039f3bd4af1e94a143af718b4c0cb9a/href</a></iframe><p>The <em>t_data</em> table will serve as an intermediate table for loading data, processing it, copying it to the actual database table and then throwing it away.</p><p>Notice that we have added a couple of extra columns (<em>customer_first_name</em> and <em>customer_last_name</em>) which we will populate with data from the <em>customer</em> column.</p><p>First, we need to load the data to the table. We can use the following SQL script.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/de5249a894a0421aadba35af389241a5/href">https://medium.com/media/de5249a894a0421aadba35af389241a5/href</a></iframe><p>I usually prefer to use <a href="https://www.postgresql.org/docs/9.2/sql-copy.html">tabs</a> as a delimiter (which is the default), but for the purpose of this post, it is easier to use commas. Once the script has run, you will have 3 rows in the temp table. There will still be no first and last name data, so we need to extract this from the existing data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*lZmHWPoOUA351Y_dzGlMTA.png" /></figure><p>Next, we need to set the first name</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/497904d0793915095e213e3f2c631c26/href">https://medium.com/media/497904d0793915095e213e3f2c631c26/href</a></iframe><p>And similarly the last name</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e6f05c48fe314a084bc4a7f1eb0cafc6/href">https://medium.com/media/e6f05c48fe314a084bc4a7f1eb0cafc6/href</a></iframe><p>now the table looks closer to the finished product:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*Df0ed4sCPhWfVjUZYJ6G7w.png" /></figure><p>There is one last step for entering the data to the sales table:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/67c2f5fbf92244ec08062150e1f3e6d6/href">https://medium.com/media/67c2f5fbf92244ec08062150e1f3e6d6/href</a></iframe><p>Clearly, this was too small a dataset to bother with a temporary table and multiple sql scripts. The approach though is sound and it can easily be expanded to millions of rows.</p><p>Hope this helps, and please provide feedback if you would like to cover other postgres issues.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3b687ee6416" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codefully-io/loading-data-to-postgresql-a3b687ee6416">Loading data to PostgreSQL</a> was originally published in <a href="https://medium.com/codefully-io">codefully.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>