<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Federico Tomassetti on Medium]]></title>
        <description><![CDATA[Stories by Federico Tomassetti on Medium]]></description>
        <link>https://medium.com/@ftomassetti?source=rss-debe1b49278d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*jbS0T_5CRJOcAv6ek0fpqA.jpeg</url>
            <title>Stories by Federico Tomassetti on Medium</title>
            <link>https://medium.com/@ftomassetti?source=rss-debe1b49278d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 16:10:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ftomassetti/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The complete guide to (external) Domain Specific Languages]]></title>
            <link>https://medium.com/@ftomassetti/the-complete-guide-to-external-domain-specific-languages-aa2e0427e17?source=rss-debe1b49278d------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa2e0427e17</guid>
            <category><![CDATA[domain-specific-languages]]></category>
            <category><![CDATA[dsl]]></category>
            <dc:creator><![CDATA[Federico Tomassetti]]></dc:creator>
            <pubDate>Mon, 20 Feb 2017 07:25:00 GMT</pubDate>
            <atom:updated>2017-03-01T13:36:35.541Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/0*nhusZZo099E1HXbj." /></figure><p>This guide will show you:</p><ul><li>the <a href="https://tomassetti.me/domain-specific-languages#what_are_dsls"><strong>what</strong></a>: after a definition we will look into examples of DSLs</li><li>the <a href="https://tomassetti.me/domain-specific-languages#why_adopting_dsls"><strong>why</strong></a>: what are the concrete benefits you can achieve using DSLs</li><li>the <a href="https://tomassetti.me/domain-specific-languages#how_create_dsls"><strong>how</strong></a>: we will discuss the different ways to build a DSLs and what are the success factors</li></ul><p>After that you will get a list of <a href="https://tomassetti.me/domain-specific-languages#resources">resources</a> to learn even more: books, websites, papers, screencasts.</p><p>This is the an abridged version of the most <a href="https://tomassetti.me/domain-specific-languages/">complete resource on Domain Specific Languages</a> out there. I hope you will enjoy it!</p><h3>What are Domain Specific Languages?</h3><p>Domain Specific Languages are languages created to support a particular set of tasks, as they are performed in a specific domain.</p><p>You could be familiar with the typical programming languages (a.k.a. General Programming Languages or GPLs). They are tools good enough to create all sort of programs, but not really specific to anything. They are like hammers: good enough for many tasks, if you have the patience and ability to adapt them, but in most cases you would be better off using a more specific tool. You can open a beer with an hammer, it is just way more difficult, risky and lead to poorer results that using a specific tool like a bottle opener.</p><blockquote><em>Languages are tools to solve problems and Domain Specific Languages are specific tools, good to solve a limited set of problems.</em></blockquote><p>Ok, but what does it mean in practice? How do DSLs look like? Let’s see plenty of examples.</p><h3>5 Examples of how Domain Specific Languages</h3><p>Domain Specific Languages can serve all sort of purposes. They can be used in different contexts and by different kinds of users. Some DSLs are intended to be used by programmers, and therefore are more technical, while others are intended to be used by someone who is not a programmer and therefore they use less geeky concepts and syntax.</p><p>Domain Specific Languages can be extremely specific and being created only to be used within a company. I have built several of this kind of DSLs myself, but I am not allowed to share them. I can instead list several examples of public DSLs which are used by millions of persons.</p><h4>1. DOT — A DSL to define graphs</h4><p>DOT is a language that can describe graphs, either directed or non directed.</p><p>digraph graphname { yellow -&gt; orange -&gt; red; orange -&gt; green; }</p><p>From this description images representing these graphs can be generated. To do that you use a program named graphviz, which works with the DOT language. From the previous example you would get this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/189/0*CEGNW36vlLlbLTQn." /></figure><p>An image generated using the dot DSL</p><p>The language permits also to define the shape of the nodes, their colors and many other characteristics. But the basics are pretty simple and almost everyone can learn how to use it in a matter of minutes.</p><h4>2. PlantUML — A DSL to draw UML diagrams</h4><p>PlantUML Can be used to define UML diagrams of different kinds. For example we can define a sequence diagram.</p><p>@startuml actor MyUser actor CustomerCare database database MyUser -&gt; CustomerCare : Ask a refund CustomerCare -&gt; database : Verify the data CustomerCare -&gt; MyUser : Issue a refund @enduml</p><p>MyUser -&gt; CustomerCare : Ask a refund</p><p>CustomerCare -&gt; database : Verify the data</p><p>CustomerCare -&gt; MyUser : Issue a refund</p><p>From this definition we can get a picture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/308/0*0wuBya5LLPkg3dmH." /></figure><p><em>Image generated with the PlantUML DSL</em></p><p>With a similar syntax different kinds of diagrams can be defined like class diagrams or use case diagrams. Using a textual DSL to define UML diagrams have several advantages: it is easier to version, and it can be modified by everyone without special tools. A DSL like this one could be used during a meeting to support a discussion: as the participants argue different diagrams can be quickly defined and the corresponding images generated with a click.</p><h4>3. Sed — A DSL to define text transformation</h4><p>On UNIX-like operating systems (Linux, Mac, BSD, etc.) there are a set of command line tools, each one accepting instructions in their own format. This format can be considered a DSL that permits to specify the tasks to be executed. For example <em>sed</em> executes the text transformations indicated using its own DSL.</p><p>Do you want to replace the word “Jack” with the word “John”?</p><p>Or do you want to delete all the lines of a file from line 10 until the word “stophere” is found?</p><p>The technical level necessary to become proficient with this kind of DSLs is elevated. However many advanced computer users could learn the basic to execute common operation on files. Small things that they could need to do every day and that currently are doing manually. For example, you could have e-mail templates containing placeholders like “{FIRST_NAME}” and replace them with the proper test with one of this commands.</p><h4>4. Gawk — A DSL to print and process text</h4><p>Like sed, gawk is another UNIX utility accepting commands in its own language. For example you could print all the lines of a given file which are longer than 80 characters:</p><p>Or count the lines in a file:</p><p>The UNIX philosophy is to use several or these little utilities and combine them to perform the most amazing and complex tasks. For example you could take an input file, transform it with <em>sed</em> and then print selected parts using <em>gawk</em>.</p><h4>5. Gherkin — A DSL to define functional tests</h4><p><a href="https://cucumber.io/docs/reference">Gherkin</a> is a DSL for defining functional tests. It has a very flexible syntax that makes it look almost like free text. Basically developers, analysts and clients can sit around a table and define some scenarios. These scenarios will be then executable as tests, to verify whether the application meet the expectations.<br> Here it is how we could define the expectations for withdrawing from an ATM:</p><p>Scenario: Verify withdraw at the ATM works correctly Given John has 500$ on his account When John ask to withdraw 200$ And John inserts the correct PIN Then 200$ are dispensed by the ATM And John has 300$ on his account</p><p>Scenario: Verify withdraw at the ATM works correctly</p><p>Given John has 500$ on his account</p><p>When John ask to withdraw 200$</p><p>And John inserts the correct PIN</p><p>Then 200$ are dispensed by the ATM</p><p>And John has 300$ on his account</p><p>I really like this DSL because the bar for using it is very low. This DSL however requires a developer to define some code using a GPL. How it works in practice is that a developer define specific commands like: “{name} has {amount}$ on his account” and define the code that execute this command in the GPL chosen for the project (Ruby, Java, or others are supported). Once the developers have created these commands, specific to the application of interest, all users can use them while defining their functional tests. It is also possible to start in the other way: first you write your scenarios, as you want, trying to capture the requirements and only later developers map each command to a corresponding function in a GPL.</p><p>In other words, this DSL is great for hiding the real code behind a surface that everyone can understand and everyone can contribute to. It is much better to sit at a table and discuss with a bank representative using the example we have displayed than showing him the hundreds of lines of Java which correspond to those commands, right?</p><h3>So what can we use DSLs for?</h3><p>After looking at these examples we can derive that DSLs can be used for a variety of goals:</p><ul><li>define commands to be executed: like sed or gawk</li><li>describe documents or some of their specific aspects: like html, latex or CSS</li><li>define rules or processes: like BPMN or Actulus</li></ul><p>These are just some typical usages, but DSLs can be used for so many other reasons.</p><p>It is also important to notice that DSLs focus on one specific aspect of a system and often it makes sense to combine several of them to describe the different facets of a product. For example HTML and CSS are used together to describe the content and the style or a document, XML and XPath are used to define data and how to traverse that data. OCL is used to define constraints on UML models.</p><blockquote><em>You could think of designing not one DSL but a family or interrelated DSLs.</em></blockquote><p>Ok, at this point you should have some understanding of how a DSL can look like and what it can be used for. Now let’s see<em> why</em> you should use one and then how to build one.</p><h3>DSLs vs GPLS: 5 Advantages of using Domain Specific Languages</h3><p>You could be asking yourself the following question:</p><blockquote><em>Why using a specific, limited language instead of a generic, powerful one?</em></blockquote><p>The short answer is that Domain Specific Languages are limited in the things they can do, but because of their specialization they can do much more in their own limited domain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/0*nGri_YgDeA3-VXqf." /></figure><p>Let’s be concrete and see five separate advantages:</p><ol><li>We can analyze them much better: while it is impossible in practice to guarantee that a program written in C or in Java will respect certain characteristics (like not ending in an infinite loop) we can perform all sort of analyses when we use DSLs. Precisely because they are limited in what they can do they are easier to analyze.</li><li>They are more safe. Less things can possibly go wrong when using a DSL. When is the last time you had a Null Pointer Exception when working with HTML or SQL? Exactly, never. This is very important if we are doing something critical like dealing with the health of someone or his money.</li><li>When there are errors those are errors specific to the domain, so that they are easier to understand. They are domain specific: so errors are not about null pointers, they are about things that a domain expert can understand.</li><li>It also means that the interpretation is easier, so bringing them to a new platform is easy. The same applies to simulators. The same HTML documents we could open on a PDA in 2000 can now be open on an iPad pro.</li><li>We can teach them more easily: they are limited in scope so less time and less training are needed to master them simply because there is less stuff to study.</li></ol><h3>Why Adopting Domain Specific Languages?</h3><p>Ok, we have seen what Domain Specific Languages are and how they differ from GPL, now we should understand <strong><em>why </em></strong>we should consider adopting them.<strong><em> What are the real benefits?</em></strong><br> Domain Specific Languages are great because:</p><ol><li>They let you <em>communicate with domain experts</em>. Do you write medical applications? Doctors do not understand when you talk about <em>arrays</em> or <em>for-loops</em>, but if you use a DSL that is about <em>patients, temperature measures </em>and<em> blood pressure </em>they could understand it better than you do</li><li>They let you focus on the important concepts. They hide the implementation or the technical details and expose just the information that really matters.</li></ol><p>They are great tools to support reasoning on specific domains and all the other advantages derive from that. Let’s look at them in details.</p><h3>Communication with domain experts</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/0*7ZrbzIQ0gIw8BLqb." /></figure><p>In many contexts you need to build software together with domain experts who are not themselves developers.</p><p>For example:</p><ul><li>You could build medical applications and need to communicate with doctors to understand the treatment a companion software should suggest</li><li>You could build marketing automation software. You would need the marketing people to explain you how to identify clients matching a certain profile, to offer them a particular deal</li><li>You could build software for the automotive industry. You would need to communicate with the engineers to understand how to control the brakes</li><li>You could build software for accountants. You need to represent all the specific tax rules to apply in a given context and you would need an accountant to explain them to you</li></ul><p>Now, the problem is that these domain experts do not have a background in software development and the way of communicating of developers and those domain experts can be very different, because they speak different languages.</p><blockquote><em>Developers talk about software, while domain experts talk about their domain.</em></blockquote><p>By building a DSL we build a language to communicate between developers and domain experts. This is not too dissimilar to what is described in <a href="https://www.amazon.com/Domain-Driven-Design-Tackling-Complexity-Software/dp/0321125215">Domain Driven Design</a>. The difference here is that we want to create a language understood by developers, domain experts and also by the software that will be able to execute the instructions specified in the DSL.</p><p>Now, the holy grail would be to create a language, give it to domain experts and have them go away and write their queries or logic alone. In practice, usually DSLs do not achieve that but prove very useful anyway: a typical interaction consists in having a domain expert describes what it wants to a developer, and the developer can immediately write down that description using a DSL. The domain expert could at this point <strong>read it and criticize it</strong>.</p><p>Typically non-developers do not have the analytical skill to formalize a problem, but they can still <strong>read it and understand it</strong>, if it is written in a DSL using a lingo familiar with the user. Also, these tools could use simulators or run queries on the fly so that the domain expert can look not only at the code itself, but also at the result.</p><p>These kinds of interactions in practice can have a <strong>very short turnaround</strong>: code can be written during a meeting or within days. While typically when using GPL the turnaround is measured at the very least in weeks, if not months or years.</p><p>By using DSLs you can very frequently:</p><ul><li>having domain experts read or write high level tests. Like requirements which are executable</li><li>when doing co-development with domain experts developers can get feedback at a very fast pace</li></ul><p>So the answer to the question:</p><blockquote><em>Can domain experts (not programmers) write DSLs alone?</em></blockquote><p>Is, of course, “it depends”. But definitely DSLs permit to have <strong>domain experts involved in the development process</strong>. Reducing dramatically feedback cycles and reducing exponentially the risks of disalignments. You know, when you have developers talking a couple of time with the domain experts, than they walk away and come back to show their solution to domain experts. And those experts stare the solution and declare it to be absolutely, completely, irremediably wrong. You can avoid that, by using a DSL.<br> So it is important to understand the advantages of that, and at the same time be realistic. You see, some decades ago there were enthusiast suggesting that all sort of people could write queries in SQL autonomously:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/0*RBeSJbgJskEU3qsG." /></figure><p>Forty years later it appears pretty clear that housewives are not going to write SQL.<br> The point is that many DSLs require to formalize processes in a way that demand significant analytic skills. Those skills are easily found in developers, but are not so common in people with a non scientific background.</p><h3>Focus and productivity</h3><p>The fact that the DSLs abstract some technical details, to focus on the knowledge to capture, has important consequences.</p><p>On one hand, it make the investments in the code written using DSLs something that <strong>mantain value over time</strong>. As the technology change you can change the interpreter processing DSL code, but the DSL code can stay the same. An HTML page written 20 years ago can still be opened using devices that no one was able to imagine 20 years ago. The browsers in the meantime have been completely rewritten multiple times. Then the logic can be ported to new technologies.</p><p>I want to share a story about a company I have worked with. This company has created its own DSL to define logic for accounting and tax calculations. They started building this DSL 30 years ago and at that time they used to generate console applications. Yes, applications that run in consoles of 80×25 cells. I worked with them re-engineering the compiler and the same code of their DSL is now used to generate reactive web applications. How this happened? Because the <strong>DSL captured only the logic, the really valuable part of the programs</strong>, an asset extremely important for the company. The technical details were abstracted in the compiler. So we just had to change the DSL compiler to preserve the value of the logic and make it usable in a more modern context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/0*v2Nagq7V_VWqLlYz." /></figure><p>This teach us that:</p><blockquote><em>Domain logic is what has value and should preserved, while technology change over time</em></blockquote><p>By using a DSL we can decouple domain logic and technlogy and make them evolve separately.</p><p>Another advantage of hiding technical details is productivity. Think about the time spent thinking about deallocating memory or choosing implementation of a list would perform best for the case at hand. That time has a poor ROI. With a DSL instead you just focus on the relevant parts of the problem and get it solved.</p><h3>How to create Domain Specific Languages</h3><p>Wonderful, you have seen why Domain Specific Languages are so cool and what benefits they can bring you.<br> Now there is only one question:</p><blockquote>How do we build DSLs?</blockquote><p>Let’s see how by looking at:</p><ul><li>what tools you can use to build DSLs</li><li>what are the most important success factors</li><li>which skills do you need</li></ul><p>There are different ways to build a DSL. The goal here is to build a language, with tool support, while keeping the effort reasonable. We are not building the next Java or C# so we are not going to pour tens of man years at building an extra complex compiler or an IDE with tons of features. We are going to build a useful language, with good tool support with an investment that can be substained by a small company.</p><p>So I am going to show you the menu and you can pick your own choice. If you need help you can look at the <a href="https://tomassetti.me/domain-specific-languages#comparison">comparison</a> I prepared at the end of this list.</p><p>The approaches are divided in three groups, depending on the kind of DSLs you want to build:</p><ul><li><a href="https://tomassetti.me/domain-specific-languages#textual_languages">textual languages</a></li><li><a href="https://tomassetti.me/domain-specific-languages#projectional_editors">projectional editors</a></li><li><a href="https://tomassetti.me/domain-specific-languages#graphical_languages">graphical languages</a> (if you are interested into look into the <a href="https://tomassetti.me/domain-specific-languages/#graphical_languages">complete guide to domain specific languages</a>)</li></ul><p>You probably know what textual and graphical languages are but you may not have encountered projectional editors before. They are pretty interesting things so you should probably take a look.</p><h3>Textual languages</h3><p>These are the most classical languages. Most practicioners will not even conceive other kinds of languages. Admittedly we are all used to work with textual languages. They are easier to support and can be used in all sort of contexts. However to use them productively I think that a specific editor is mandatory. Let’s see how to build textual languages and supporting tools.</p><h4>A pragmatic do-it-yourself approach</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/0*cWZQsgmJ9hbILNT3." /></figure><p><strong>Roll up your own solution: </strong>You could reuse a parser generator like ANTLR (or Lex &amp; Yacc if you are an old-style guy) and write the rest of the processing yourself. This is doable but it requires knowing what your doing. If you don’t know where to start you can take a look at my <a href="https://leanpub.com/create_languages">book on building languages</a>.</p><p>I am sure you want to read it so I do now want to spoil it too much, but the path is more or less this:</p><ol><li>You define the lexer and parser grammar using ANTLR. You do not know ANTLR? No problem, here it is a nice <a href="https://tomassetti.me/parsing-any-language-in-java-in-5-minutes-using-antlr-for-example-python/">tutorial on ANTLR</a>. In this blog there are many other <a href="https://tomassetti.me/category/language-engineering/antlr">articles about ANTLR</a>.</li><li>You transform the parse tree produced by ANTLR in a format easier to work with. So you get the model of your code.</li><li>You resolve reference, build validation, and implement a typesystem as a set of operations on the model of your code. It sounds complex, but if you know what you are doing you can get it done in a few hundreds of lines of code.</li><li>Finally you either interpret or compile the model of your code</li><li>You build a simple editor for your language. For how to do that look for tutorials on this blog or into the book. I tend to use an <a href="https://github.com/ftomassetti/kanvas">hackable editor</a> I built myself. I named it Kanvas</li></ol><p>What I like of this approach is that you are in control of what is happening. You can change the whole system, evolve it and it is simple enough that you can really understand it. Sure with this approach you are not going to get a super complex IDE with tens of refactoring operation. Or you are not going to get those things for free at least.</p><h4>Xtext</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*WPNuYWltt_3D753L." /></figure><p>Xtext is a solid solution to build textual languages. In practice you define your grammar in a way similar to what you would do with ANTLR but instead of getting just a parser you get a nice editor. This editor is by default an Eclipse plugin. It means that you will be able to edit the files written in your DSL inside Eclipse. If you know how the Eclipse platform works you can create an RCP application: i.e., a stripped down version of Eclipse basically supporting only your language and removing a bunch of stuff that would not be useful to your users.</p><p>So Xtext give you an editor and a parser. This parser produces for you a model of your code using the Eclipse Modeling Framework (EMF). So it basically means that you have to study this technology. I remember the long days reading the <a href="https://www.amazon.com/Eclipse-Modeling-Framework-Frank-Budinsky/dp/0131425420">EMF book</a> as one of the most boring, mind-numbing experiences I have ever had. I also remember asking questions on the Eclipse forums and not getting any answer. I have open bug reports and I have received answers <em>three years after </em>(I am not joking). So it was disheartening at first but over time the community seems to be improved a lot. Right now the material available on the <a href="http://www.eclipse.org/Xtext/">Xtext website</a> is incomparably better than it used to be and the book from Lorenzo Bettini helped a lot (more on it later).</p><p>The editors generated by Xtext can be deeply customized, if you know what you are doing. You can get away with minor changes with a reasonable effort, but if you want to do advanced stuff you need to learn the Eclipse internals and this is not a walk in the park.</p><p>Recently Xtext escaped the “Eclipse trap” by adding the possibility of generating editors also for IntelliJ IDEA and… the web! When I first found out this I was extremely excited. Me, as many other developers, switched to IntelliJ some years ago and I was missing a way to easily build editors for IntelliJ IDEA. So this was great news, even if Xtext has been created to work with Eclipse so the support for IntelliJ IDEA is not as mature and battle-tested as the one for Eclipse. I did not try yet the web editor, but from what I understood it generates a server side application which is basically an headless Eclipse and on the client side it generates three different editors based on three technologies (each one with a different level of completeness). The one fully supported is <a href="https://orionhub.org/">Orion</a>, an Eclipse project. While the other two are the well-known <a href="https://codemirror.net/">CodeMirror</a> and <a href="https://ace.c9.io/">ACE</a>.</p><p>You may want to check out this list of <a href="http://www.eclipse.org/Xtext/community.html">projects implemented with Xtext</a> to get an idea of what is possible to achieve using Xtext.</p><h4>Textual languages: other tools</h4><p>If I had to build a textual language in most cases I would go for one of the two approaches defined earlier: either my do-it-yourself approach or using Xtext. That said there are alternatives on which I think it makes sense to keep an eye on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/265/0*pYcb7RzgGk5loSlY." /></figure><p>textX is a Python framework inspired by Xtext. You can define the grammar of your language with a syntax very, very close to the one used by Xtext. textX does not use EMF or generate code but it use instead the metaprogramming power of Python to define classes in memory. While it seems nice and easy to use, textX does not generate editor support like Xtext, so that is a major difference.</p><p>If you want to get a better feeling of how textX works take a look at this video.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/423/0*zUkzApCKGMm9rfm3." /></figure><p>There are other tools like Spoofax. I did not use it myself so I cannot vouch for it. It is more academic stuff than an industrial-grade language workbench, so I would suggest a bit of caution.</p><p>Spoofax can be used inside Eclipse. It is based on a set of DSLs to use to create other DSLs.</p><p>If you want to look into Spoofax you may want to look at this free short book from Eelco Visser named <a href="http://declare-your-language.metaborg.org/">Declare Your Language</a>.</p><h3>Projectional editors</h3><p>Projectional editors are extremely powerful and exciting but they are unfamiliar to many users. I can give you the theory bit, and throw at you a definition, but if you really want to understand them watch the video below.</p><p>A projectional editor is an editor that show a projection of the content stored on file. The user interacts with such projection and the editor translates those interactions to changes to the persisted model. When you use a text editor you see characters, you add or delete characters and characters are actually saved on disk. In a projectional editor you could edit tables, diagrams and even what it looks like text, but those changes would be persisted in some format different from what you see on screen. Maybe in some XML format, maybe in a binary format. The point is that you can work with those files only inside their special editor. If you think about it, this is the case also for all the graphical languages: you see nice pictures, you drag them around, connect lines and in the end the editor save some obscure format, not the nice pictures you see on the screen. The point with projectional editors is that they are much more flexible than your typical graphical language. You can combine different notations and support all sort of representation you need for your case.</p><p>Confused? That is expected, watch the video below, watch many more videos and things will appear clear at some time.</p><p>You could also take a look at this explanation of <a href="https://martinfowler.com/bliki/ProjectionalEditing.html">projectional editing</a> written by Martin Fowler.</p><h4>Jetbrains MPS</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/0*ESEI4k6Cl2_wpxVL." /></figure><p>Jetbrains MPS is a tool that I have been using for some years, working on it daily on most of last 12 months. It is an extremely powerful tool and it is the most mature projectional editor available out there. It is no accident: Jetbrains has invested significantly on developing it over more than a decade.</p><p>Do you want to see how it looks like? Watch the video.</p><p><a href="https://www.youtube.com/watch?v=I8EEhXJSJYI">https://www.youtube.com/watch?v=I8EEhXJSJYI</a></p><p>Now, it means you need to be ready to embrace Jetbrains MPS and to invest a significant amount of time to properly learn it (or hire someone like me). However if you are ready to do the investment it can simply revolutionize your processes.</p><h4>Intentional Platform</h4><p>This one is the mysterious and intriguing one.</p><p>Charles Simonyi is the man who designed Excel, one of the first space tourists and one of the richiest men on earth. In 1995 he wrote the revolutionary paper <a href="https://www.microsoft.com/en-us/research/publication/the-death-of-computer-languages-the-birth-of-intentional-programming/">The Death of Computer Languages, The Birth of Intentional Programming</a>. In this paper he presents a new, more intuitive way of programming, fundamentally based on projectional editors. He starts working at Microsoft on these ideas, but in 2002 he leaves Microsost to cofound Intentional Software. Since then the publicly available information on their tool, the <em>Intentional Platform </em>has been extremely scarce. They have published a few papers and given a few presentations.</p><p>I have heard from people that have used it that it has a lot of potential but it is not there yet. For sure I would love to put my hands on it. The closest you can get is to read this somehow old <a href="https://martinfowler.com/bliki/IntentionalSoftware.html">review of the Intentional Platform</a> from Martin Fowler. Maybe one day even us mortals will have the possibility to know more about this legendary tool.</p><p>As far as I know they work with selected companies but for now their tool is not publicly available.</p><h4>Whole Platform</h4><p>This is a good Language Workbench too frequently overlooked. While it has been used in production for many years in a large company in Italy, it is lacking a little bit on the documentation and marketing side. Yes, I know it is sad that pure engineering awesomeness is not enough to gain popularity. Anyway if you want to know more about it you can read my post on <a href="https://tomassetti.me/getting-started-with-the-whole-platform-grammars/">getting started with the whole platform</a>.</p><p>There are a few concepts that I find quite interesting. I am not an expert on the Whole Platform: I have just played with it and talked with his authors.</p><p>One aspect that I find really interesting and different with respect to the other Language Workbenches is that the Whole Platform supports quite well working with existing formats, and evolve from existing processes to more advanced Language-Engineering approaches. For example, it is quite easy to define grammars for existing formats in order to parse them.</p><p>The following is an example of a grammar to parse JSON but the same approach has been used to parse very complex formats used in the banking domain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/0*zVArIs5E5Lf2VC_N." /></figure><p>Whole Platform — Grammar definition</p><p>Another idea I love about the Whole Platform is the Pattern Language: the possibility to take a model (in this sense a piece of Java code is a model) and define variability points that could be filled with values from another model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/0*KOyiYdXpSK0AVbnX." /></figure><p>Whole Platform — Pattern Language</p><p>Also, the Whole Platform is quite rich and it supports also graphical languages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U5kG3156fdjghJoS." /></figure><p>Whole Platform — Graphical Language</p><p><a href="https://www.linkedin.com/in/riccardosolmi/">Riccardo Solmi</a> and <a href="https://www.linkedin.com/in/enricopersiani/">Enrico Persiani</a> are the minds behind the Whole Platform and you should probably talk to them if you are interested in using this Language Workbench.<br> <em>The images of the Whole Plargorm I have used are released unfer the CC Attribution 2.0 license (https://creativecommons.org/licenses/by/2.0/)</em></p><h3>What do I need to make my DSL succeed?</h3><p>There are just two things that will seem obvious but are not:</p><ol><li>You need your users to use it</li><li>You need your users to get benefits from using it</li></ol><p>To achieve this you will need to build the right tool support and adopt the right skills. We are going to discuss all of this in this section.</p><h3>Get users to use it</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/0*Z6uMbeaMa6odu3Yz." /></figure><p>The first point means that you need to win the support of users. During my PhD I conducted a survey on the reasons why DSLs are not adopted. There are several causes, but one important factor is <strong>resistance from users, especially when they are developers</strong>.</p><p>If you target a DSL to developers, they could resist because they feel they are not in control as when they use a General Purpose Language (GPL). They could also fear that a DSL lowers the bar, being simpler to use than, let’s say, Java. In addition to that, as all innovations, a new DSL is threatening for seasoned developers because it reduces the importance of some of their skills, like the vast experience in dealing with the quirks of the current GPL you are using at your company.</p><p>If your DSL is intended to non-developers generally it is easier to win their support. Why? Because you are giving them a superpower: <strong>the ability to do something on their own</strong>. They could use a DSL to automatize a procedure that previously was done manually. Maybe before the DSL was available the only possibility for them to do something was bothering some developer to write custom code. Now they get a DSL which means more power and independence because of it. Still they could resist adopting it if they perceive it as too difficult or if they feel it does not match their way of thinking.</p><p>To me the key as a DSL designer in this case is being <strong>humble and listen</strong>. Listen to the developers, work on capturing their experience and embedding it in the design of the DSL or the tooling around it. Involve them in the design of the DSL. When talking with your users, technical or not, communicate that the DSL will be a tool for them, designed to support them and derived by their understanding of the domain at hand. When designing DSLs the cowboy approach does not work, you need to succeed as a team or not succeed at all.</p><h3>Give benefits to users</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/0*S8osloZWWfsWn7yP." /></figure><p>If you get the support of users and people start using it you win only if they get a significant advantage from using the DSL.</p><p>We have discussed the importance of a DSL as communication tool, as a medium to support co-design. This is vital.</p><p>In addition to this, you can increase significantly the productivity of your users by building first-class tool support.</p><p>A few examples:</p><ul><li>a great editor with syntax highlighting and auto completion: so that learning the language and using it feel like a breeze</li><li>great error messages: a DSL is an high level language and error can be very significant for users</li><li>simulators: nothing helps users as the possibility to interact with a simulator and see the results of what they are writing</li><li>static analysis: in some contexts the possibility to analyze the code and reassure against possible mistake is a big win</li></ul><p>These are a few ideas but more can be adopted, depending on the specific case. Specific tools support for specific languages.</p><h3>Tool support: why we do not care about internal Domain Specific Languages</h3><p>There is one factor that is frequently overlooked and this is tool support.</p><p>Many practicioners think that the only relevant thing is the syntax of your language or what it permits to express, with everything else being a detail.</p><p>This is just fundamentally wrong because the tool support can multiply exponentially the productivity when using any language, especially a DSL. This is a crucial aspect to consider, because the language should be designed considering tool support.</p><p>Because when building Domain Specific Languages, if you want to get serious, you have to build good tool support. Tool support is an essential key in delivering value.</p><p>I recorded this short video on this very subject.</p><p><a href="https://vimeo.com/199672478">https://vimeo.com/199672478</a></p><p>Tool support is the reason why internal domain specific languages (i.e., fake DSLs) are irrelevant: they do not have any significant tool support.</p><p>When using some host languages you can bend them enough of getting some sort of feeling of having your own little language, but that is it. There are some languages that are flexible enough to give a sort of decent … like ruby. With other languages you get very poor results. I feel pity for the people trying to build “internal DSLs” for languages like Scala or Java. The worst of all is lispers. I understand their philosophy “if you want to solve a problem in LISP, first you create your LISP dialect and then solve it using it”. I understand and I think it is a great technique. Just let’s not pretend this is a real DSL. This is not something you can share and work with closely with domain experts. It can be your trick to be more productive, but that is it.</p><p>You see those ridicuosouly long chains of method calls and you hear someone presenting those as Domain Specific Language. That makes me feel a mix of two emotions:</p><ul><li>pity for him and his users</li><li>rage for the confusion it creates. Real DSLs are very different and they can bring real benefits. Stop mixing them with this… thing</li></ul><p>Just build a real DSL, so an external DSL!</p><h3>Resources</h3><p>Now that you have seen what DSLs can bring you and you have an idea how to build them you should be happy.</p><p>But you are not, you want more, you want to understand better DSLs, you want to learn everything about them.</p><p>Well, I do not know about everything, but definitely I can give you some pointers. I can tell you where to find the good stuff.</p><p>Let’s see what we can find:</p><ul><li><a href="https://tomassetti.me/domain-specific-languages#books">books</a></li><li><a href="https://tomassetti.me/domain-specific-languages#other_resources">websites and papers</a></li><li><a href="https://tomassetti.me/domain-specific-languages#companies">companies</a></li></ul><h3>Books</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/236/0*aGZKCw4UCQU6FIhQ." /></figure><p>I would start suggesting to read the <a href="http://voelter.de/dslbook/markusvoelter-dslengineering-1.0.pdf">DSL Engineering</a> by <a href="http://voelter.de/">Markus Völter</a>. This PDF version of the book is donation-ware. So just read it and donate. Alternatively you can find the printed version on Amazon.</p><p>The book start with an introduction part: it is very useful to set your terminology straight. After that it comes the DSL design part, focusing on different aspects separately. If you do not have direct access to an expert to teach you how to design DSLs reading this part of this book is the best alternative I can recommend (together with as much practice as you can, of course).</p><p>Then it comes the part about implementation: remember that Markus has a PhD, but he is first of all someone who gets things done so this part is very well written, with examples based on Xtext, Spoofax and MPS.</p><p>Part IV is about scenarios in which DSLs are useful. Given this is based on his large experience in this field there are a lot of interesting comments.</p><p>I had the occasion to work with Markus. I used to admire him a lot before meet him and I now I admire him even more. He is simply the best one on this field, so if you can learn something from him do it. Read his books, watch his presentations, follow his projects. It will be a good way to invest your time. He is lately working on Jetbrains MPS stuff, so you should follow what is going on with <a href="http://mbeddr.com/">mbeddr</a> and <a href="http://github.com/iETS3/iets3.opensource">IETS3</a>. Mbeddr is both a set of plugins for MPS and an implementation of the C language in MPS with special domain-specific extensions to support development of embedded software. IETS3 is instead an expression language built in MPS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/0*sPL1tf9jo3smV7WC." /></figure><p>Martin Fowler is a very famous thought leader and bestseller author. I really admire his clarity. He is the author of <a href="https://martinfowler.com/books/dsl.html">Domain Specific Languages</a>, a book about both internal and external DSLs.</p><p>I find the mental models presented in the book quite useful and elegant. In practice however I find internal DSLs irrelevant, so I am interested in only some portions of this book.</p><p>There are 15 chapters dedicated specifically to external domain specific languages. While those chapters are organized around implementation techniques there are comments and remarks from which you can learn some design principles.</p><p>I think the sections on the alternative computational models and code generation are very valuable. You will have an hard time finding an exploration to these topics at this level of detail anywhere else.</p><p>The book is 7 years old and the techniques may have evolved since the book was written, but the vast majority of the considerations presented in the book are still valid. And of course they are thoughtful and well explained, as you would expect from Martin Fowler.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/190/0*PXfvRhGYpNaxJNy3." /></figure><p>If you are interested in textual languages and in particular on ANTLR you should definitely look into the <a href="https://pragprog.com/book/tpdsl/language-implementation-patterns">Language Implementation Patterns</a> from Terence Parr. I like the author, I like the publisher (the Pragmatic Bookshelf) and unsurprisingly I love the book.</p><p>The book starts discussing different parsing algorithms. If you like to learn how stuff works you should take a look at these chapters. Then there are chapters about working with the Abstract Syntax Tree, extracting information, transforming it. This is the kind of stuff you need to learn if you want to become a Language Engineer. Also, there chapters on resolving references, building symbol tables or implementing a typesystem. These are the foundations to learn how to process the information expressed in your DSL.</p><p>Finally Terence explains you how to use the information you have processed by building an interpreter or a code generator. At this point you end your journey, having seen how to build a useful language from start to finish. This book will give you solid basis to learn how to implement DSLs. The only thing missing is a discussion on how to design DSLs, but this is not the goal of this book.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/171/0*ksv7z14v1B8TaLH_." /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/171/0*vcVISVmkyjhOyFXC." /></figure><p>If you are looking into MPS there are not many resources around. It could make sense to buy the two books from Fabien Campagne on the <a href="http://campagnelab.org/publications/our-books/">MPS Language Workbench</a>. They explain in details all the many features of MPS (admittedly some are a bit obscure). If I would have to find one thing missing is more advices on language design. These books are very good references to learn how MPS works, but there is not much guidance on how to combine these features to get your results. One reason for that is that MPS is an extremely powerful tool, which can be used in very different ways so it is not easy to give general directions.</p><p>Volume I explains separately the different aspects of a language: how to define the structure (the metamodel), how to define the editors, the behavior, the constraints, the typesystem rules and so on. Most of the chapters are in reference-manual style (e.g. the chapter <em>The Structure AspectStructure In Practice</em>). Everything you need to learn to get started and build real languages with MPS is explained in Volume I.</p><p>Volume II is mostly about the advanced stuff that you can safely ignore at the beginning. I suggest looking into this book only when you feel comfortable with all the topics explained in Volume I. If you have never used MPS before it will take some time. Volume II explaine you how to use the build framework to define complex building configurations, it gives you an overview of all the different kinds of testing you may want to use for your languages. It also show you how to define custom aspects for your language or custom persistence.</p><p>I have bought the Google Play version but they are available also in print form.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/243/0*nNLYfiFBogcIU9mj." /></figure><p>On Xtext there is a quite practical and enjoyable book from Lorenzo Bettini: <a href="https://www.packtpub.com/application-development/implementing-domain-specific-languages-xtext-and-xtend-second-edition">Implementing Domain-Specific Languages with Xtext and Xtend</a>. I wrote a <a href="https://tomassetti.me/review-second-edition-implementing-domain-specific-languages-xtext-xtend/">review</a> on the second edition of this book: if you want to read my long-form opinion of that book you can visit the link.</p><p>If you want to learn how to write textual languages with good tool support you could start following a couple of tutorials on Xtext and then jump to this book. This book will explain you everything you need to know to use Xtext to build rather complex editors for your language. The only caveat is that Xtext is part of a complex ecosytem so if you really want to become an expert of Xtext you need to learn EMF and Xtend. The book does a good job in teaching you what you need to know to get started on these subjects but you may have to complete your education with other resources too, when you want to progress.</p><p>What I like about this book is that is not a reference manual, but it contains indications and opinions on topics like Scoping or building a typesystem rules (the author has some significative experience on this specific topic). Also, the author is interested in best practices so you will read his take on testing and continuos integration. The kind of stuff you should not ignore if you are serious about language engineering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WFIlJN92DgGsj6BP." /></figure><p>If you are interested in DSLs in general you can take a look at <a href="https://www.manning.com/books/dsls-in-action">DSLs in Action</a><em>. </em>The book is interesting but I have two comments:</p><ol><li>It focus way too much on internal DSLs, which are, as we all know, not the real thing</li><li>They have misspelled my name. -1 point for that</li></ol><p>Specifically on external DSLs there is not much: the author briefly discuss Xtext and then spend a chapter on using Scala parser combinators to build external DSLs. That would not have been my first choice. So if you are interested in learning how to implement an external DSL do not pick this book. However if you want a gentle introduction to the topic of DSLs, if you are a degenerate who prefers internal DSLs instead of external DSLs, or if you want to read every available resource on DSLs this book would be a good choice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/0*FlUzqWgl8bex3zDS." /></figure><p><a href="https://www.amazon.com/Domain-Driven-Design-Tackling-Complexity-Software/dp/0321125215">Domain Driven Design</a> is a relevant and important book to read. Because you need skills to understand the domain, in order to be able to represent it in your language, and to design your language so that it can capture it. Now, I should probably just praise this book and stress how much I have enjoyed it. Unfortunately I tend to err on the honesty side so I warn you: this is one of the most boring books I have ever read. It is important, it is useful, it is great but it just so plain and long. It stayed on my night stand for months.</p><p>What you should get how of this book is the importance of capturing the Domain in all of your software artifacts. The book stress the importance of building a common language to be shared among the stakeholders. This is completely and absolutely relevant if you want to build Domain Specific Languages. What is not part of this book is how to map this domain model to a language. For that part you should refer to the other books, specific to DSLs design. This book is a good complement to any of them.</p><h3>Websites and papers</h3><h3>Companies</h3><p>I design and implement Domain Specific Languages for a living but instead of tell you how great I am, I will instead list other companies that you could work with.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*oC-tAu_DGkZcqUwY." /></figure><p>The first, obvious name is Itemis. They are a German company with small offices in France and Switzerland. They employ some of the best in the field: I have met and worked with Markus Völter and Bernd Kolb and I am seriously impressed by the level at which they work. They have worked with so many companies and have done so many projects that the list will be scarely too long. I would just say that in the later years they have done amazing work using Jetbrains MPS. They have created the <a href="http://mbeddr.com/">mbeddr project</a> and from it they have derived a set of utilities, named the mbeddr platform, which contributed enormously to the growth of MPS. They have contributed early and significantly to the Language Workbench community, so if you need some help on a DSL you should seriously consider working with them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*-ONkofUKBXGxLMR6." /></figure><p>TypeFox is another German company. I have interviewed one of the founders some time ago. Several Xtext core committers, including the project lead, are involved in the company so as you can imagine they have some serious competencies on Xtext and the EMF platform in general. If you need Xtext training or consulting I would consider them the top-choice. They are also a Solution Member of Eclipse and I would expect them to be able to build complex Eclipse-based solutions. If you want to hear more you can read this <a href="https://tomassetti.me/interview-to-jan-kohnlein-on-typefox-dsls-and-xtext/">interview to Jan Köhnlein</a>. Jan is one of the founders of the company and we talked a few months after the company was created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*qvtn7t7n32Qa5dzI." /></figure><p>Jetbrains is obviously the company behind Jetbrains MPS. Recently they started offering training on MPS. I asked about this during my <a href="https://tomassetti.me/interview-to-vaclav-pech-on-jetbrains-mps-the-community-and-the-future/">interview at Vaclav Pech</a>. They offer training, either basic or advanced, on their premises in Prague or on-site.</p><p>At the moment I do not think they offer MPS consulting (for that you can talk to me or to Itemis).</p><h3>Conclusions</h3><p>There are many reasons why you should really consider Domain Specific Languages. I have seen companies benefit enormously from DSLs. Most of the people I have worked with used DSLs as a key differentiator that helped them increase productivity by 10–20 times, reduce time-to-market and feedback cycles, increase the longevity of their business logic and much more.</p><p>Aside from the practical benefits I find the topic extremely fascinating. Most of all I feel that by building DSLs we build powerful tools that help other people do their job. As language designers we act as enablers, our languages can be used by skilled professionals to achieve great things and this is an amazing feeling for me.</p><p><em>Originally published at </em><a href="https://tomassetti.me/domain-specific-languages/"><em>tomassetti.me</em></a><em> on February 20, 2017.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa2e0427e17" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Best posts on Language Engineering]]></title>
            <link>https://medium.com/@ftomassetti/best-posts-on-language-engineering-21a7c0798632?source=rss-debe1b49278d------2</link>
            <guid isPermaLink="false">https://medium.com/p/21a7c0798632</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[dsl]]></category>
            <category><![CDATA[xtext]]></category>
            <category><![CDATA[antlr]]></category>
            <category><![CDATA[jetbrains]]></category>
            <dc:creator><![CDATA[Federico Tomassetti]]></dc:creator>
            <pubDate>Tue, 17 Jan 2017 08:50:00 GMT</pubDate>
            <atom:updated>2017-01-25T19:26:01.413Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/560/0*rB0pUnCHQWJz6X0s." /></figure><p>In the past years, and in the last 12 months in particular I wrote tens of posts, some of which got a good response from you (thank you!).</p><p>I thought it could be useful to create a list of the most relevant posts for each topic. It could help in particular new readers.</p><h3>ANTLR</h3><p>A lot of people love to read about ANTLR! I did not anticipated writing so much about ANTLR, however many posts involve ANTLR but are used to show specific techniques, like building syntax highlighting. I plan to write more about tools built around ANTLR, like editors for the web or the desktop (such as <a href="https://github.com/ftomassetti/kanvas">Kanvas</a>).</p><ul><li><a href="http://tomassetti.me/parsing-any-language-in-java-in-5-minutes-using-antlr-for-example-python/">Parsing any language in Java in 5 minutes using ANTLR: for example Python</a> how to setup ANTLR and take advantage of the many grammars already existing</li><li><a href="http://tomassetti.me/antlr-and-the-web/">ANTLR and the web: a simple example</a> how to generate a JavaScript parser from an ANTLR grammar and used it in the web</li><li><a href="https://tomassetti.me/getting-started-with-antlr-building-a-simple-expression-language/">Getting started with ANTLR: building a simple expression language</a> how to write a lexer using ANTLR and Kotlin</li><li><a href="http://tomassetti.me/getting-started-with-antlr-in-csharp/">Getting started with ANTLR in C#</a> how to write a lexer and parser ANTLR and C#</li><li><a href="https://tomassetti.me/how-to-create-an-editor-with-syntax-highlighting-dsl/">How to create an editor with syntax highlighting for your language using ANTLR and Kotlin</a> how to derive automatically an editor with syntax highlighting from an ANTLR grammar</li></ul><h3>JavaParser &amp; JavaSymbolSolver</h3><p>I wrote several more posts about JavaParser, discussing how specific part of JavaParser were implemented. I plan to work more on writing tutorials on JavaParser this year. First of all version 3.0 was just released, then there are other features coming: I have just completed support for Javadoc processing and I am working on lexical preservation. A little voice is even telling me about a book on JavaParser…</p><ul><li><a href="http://tomassetti.me/getting-started-with-javaparser-analyzing-java-code-programmatically/">Getting started with JavaParser: analyzing Java Code programmatically</a> showing the typical ways JavaParser can be used and applying those to answer some of the most common questions about JavaParser (taken from StackOverflow)</li><li><a href="http://tomassetti.me/resolve-method-calls-using-java-symbol-solver/">Resolve method calls in Java code using the JavaSymbolSolver</a> introduction to JavaSymbolSolver, the JavaParser extension to resolve symbols. It shows how to use it to find the method definition corresponding to a certain method invocation.</li></ul><h3>Static analysis</h3><p>This is in interesting topic for me, connected also to my interest to JavaParser. This year my brother Gabriele contributed a few articles on static analysis in the C# world, while I tend to focus more on the JVM world, with excursion to other platforms (like Python).</p><ul><li><a href="http://tomassetti.me/getting-started-roslyn/">Getting started with Roslyn: transforming C# code</a> Roslyn is a very interesting framework that can be used to parse, analyze and transform C# code. In this post we show you how to perform simple transformation to C# code</li><li><a href="https://tomassetti.me/python-reflection-how-to-list-modules-and-inspect-functions/">Python reflection: how to list modules and inspect functions</a> we combine several reflection techniques to build a model of all Python packages, functions and classes</li><li><a href="https://tomassetti.me/generate-diagrams-csharp/">Generate diagrams from C# source code using Roslyn</a> we show how to analyze C# code and build an internal model. From that model we generate a few diagrams</li></ul><h3>Jetbrains MPS</h3><p>I have been using Jetbrains MPS on daily basis and I did quite a few interesting things with it. I should write it more about it this year.</p><ul><li><a href="https://tomassetti.me/antlr-and-jetbrains-mps-parsing-files-and-display-the-ast-usign-the-tree-notation/">ANTLR and Jetbrains MPS: Parsing files and display the AST using the tree notation</a> it shows how to combine ANTLR and Jetbrains MPS. We use ANTLR to parse existing files and import them in MPS</li><li><a href="http://tomassetti.me/interview-to-vaclav-pech-on-jetbrains-mps-the-community-and-the-future/">Interview to Vaclav Pech on Jetbrains MPS: the community and the future</a> Vaclav is a technical leader at Jetbrains, involved with MPS for several years. He was kind enough to answer all the questions I had on the development of MPS. Quite an interesting reading</li></ul><h3>Other related to languages</h3><p>There are so many other things I would like to talk about which are broadly related to languages. Interviews to very interesting people like Jan or Vaclav are high on my list. Also discussing generation, for example targeting the LLVM or interpretation are topics that I think could interest some of you.</p><ul><li><a href="https://tomassetti.me/interview-to-jan-kohnlein-on-typefox-dsls-and-xtext/">Interview with Jan Köhnlein on TypeFox, DSLs and Xtext</a> Jan is one of the most well-known committers of the Xtext project. He recently created a company named TypeFox, focusing on building DSLs using Xtext. We discuss about Xtext and his company</li><li><a href="https://tomassetti.me/generating-bytecode/">Generating bytecode</a> parsing is fun, I know, but after having parsed code you need to generate something from it. Bytecode is a very good choice. In this post we explain how to generate bytecode.</li></ul><h3>In the end…</h3><p>There are topics that I love, and I will keep writing about those even if I am the only one reading those posts. However I also try to be useful here, so if there are topics related to language engineering that you feel are not explained well feel free to reach me (write to me at federico at tomassetti dot me, use the <a href="https://tomassetti.me/contact-me-about-a-project/">contact page</a>, ping me <a href="https://twitter.com/ftomasse">on twitter</a>) and let me know.</p><p><em>Originally published at </em><a href="https://tomassetti.me/bests-posts-on-language-engineering-2016/"><em>tomassetti.me</em></a><em> on January 17, 2017.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21a7c0798632" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[On the quest for the right project management tool: Jira, Trello, Asana, Redmine]]></title>
            <link>https://medium.com/@ftomassetti/on-the-quest-for-the-right-project-management-tool-jira-trello-asana-redmine-b18420672030?source=rss-debe1b49278d------2</link>
            <guid isPermaLink="false">https://medium.com/p/b18420672030</guid>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[productivity]]></category>
            <dc:creator><![CDATA[Federico Tomassetti]]></dc:creator>
            <pubDate>Tue, 21 Jun 2016 12:50:31 GMT</pubDate>
            <atom:updated>2016-11-16T20:33:13.391Z</atom:updated>
            <content:encoded><![CDATA[<p>I have used many different project management tools but I have to say that I have not yet found one which really makes me 100% happy to use it.</p><p>I have played with many tools and used some of them extensively and I was growing a bit frustrated about this. In the good old times I have also hacked a couple of tools myself. Now that I am older (and maybe wiser) I decided to sit down and thing about this problem.</p><h3>Activities involving a project management tool</h3><p>First of all I tried to examine for which activities I feel I need a project management tool. Here it is the list:</p><ol><li><strong>Planning sessions: </strong>I brainstorm ideas, review existing tickets and plan work for the week while talking over Skype with other people. To be efficient during planning sessions I need an easy way to identify tickets (“ok, so I am going to close X-123, right?”, “Y-34 is urgent”) and it helps to have the possibility to quickly create tasks, without having to go through one dialog box (or more) for each new ticket.</li><li><strong>Scheduling sessions: </strong>at the beginning of the day or the week, alone or talking with others, I want to take a look at my open tickets and decide in which order I am going to work on them. I want to be able to assign due dates and priority levels. Ideally I would like also to be able to order the tickets.</li><li><strong>Coding sessions: </strong>while coding I want to open my tickets and. add comments. Possibly I want to assign the ticket to someone else if I need his comments to move on. I need an identifier to be used in branches and commits. It helps also to be able to quickly mark the ticket as done and see on what else I need to work. During the coding sessions I should not do any scheduling, that should be done upfront.</li></ol><h3>Features I am looking for</h3><p>The right project management tool is partially a question of feeling: you want something with an easy to use interface, sometimes responsive and intuitive. However I tried to identify a short list of features that matter to me:</p><ol><li><em>(Critical) </em>Issues should have a <strong>short ID</strong>, something that can be used when discussing about them and to tag commits. It may seem a minor thing but for me this basic feature is fundamental</li><li>It should be possible to <strong>quickly create a list of tasks</strong>, from one single screen. This is important during planning sessions to be able to sketch a list of tasks without having to go through several pages.</li><li>It should be possible to <strong>specify due dates</strong>. It should be also possible to sort tasks by that due date and in general to see which tasks are about to expire.</li><li>It should be possible to <strong>specify priority levels</strong>. Not all tasks are created equal. It is true that ordering issues is a partial substitute for that but I think that the possibility to assign priority levels help when planning.</li><li>The <strong>number of fields</strong> of an issue should be limited. I do not want to have to find the useful information in a sea of useless fields. And I do not want absolutely to have to fill fields which are irrelevant to me.</li><li>It should be possible to <strong>assign tickets</strong> to project members. Basic stuff</li><li>It should be possible to <strong>comment tickets</strong>. Other really basic stuff</li><li><em>(Optional) </em>It should be possible to <strong>find</strong> all the open issues for a project. Creating issues is part of process but querying, filtering and examine issues list is another very important activity.</li><li><em>(Optional)</em> It should be possible to <strong>order</strong> issues. I would like to define an order for issues, so that when I am working I can just pick the issue on top and keep crunching issues and move forward.</li><li><em>(Optional)</em> It should be easy to get a list of <strong>closed tasks</strong></li></ol><p>Now let’s take a look at a few candidates.</p><h3>Jira</h3><p>I have used Jira while working at several companies.</p><p>One thing that can be said about Jira is that it is <em>complete.</em></p><p>Another thing that can be said is that it is <em>bloated.</em></p><p>Did you ever have the feeling Jira asks you to go through screens which are completely irrelevant? Have you ever had the feeling there are just too many fields?</p><p>Now, it is true that it can be heavily configured but this seems not to be a trivial task. I have seen several people who have used Jira for years being regularly confused by it and wasting a lot of time finding the screens which they did not use daily. When I was working at a certain company (in Ireland) we had to ask our Jira administrator (in Germany) to configure Jira for us. I think this was a very clever way to add even more red tape. Useless to say: we wasted several days just to have Jira configured <em>for each project</em>. No good.</p><h3>Trello</h3><p>Now, Trello is fun to use and very good to get started. I am using it and I have found a plugin to show the short-ID of cards. It is not perfect: you need to refresh the page after inserting the card: it is a bit annoying but not a deal-breaker.</p><p>It is very easy and fun to quickly add a few cards and there is no red-tape at all. It is absolutely a pleasure to use while planning. It becomes less nice when you need to examine what you have done in previous weeks. Do you archive the lists of tasks done last week? Do you keep a huge list of “Done” tasks?</p><p>Also, you cannot add priorities to tasks and while you can insert due dates I could not find a way to easily see tasks which are about to expire or to get a notification that a task is expired. I just feel that the tool is not helping while managing, everything is up to my goodwill and attention.</p><h3>Redmine</h3><p>Redmine is the open-source competitor. The default interface seems reasonable but a bit ugly. However you can add a lot of plugins and the <em>easyredmine</em> large plugin seems to produce a very nice product. It remains a bit slow and sluggish but I think it has all the feature I would be looking for.</p><p>The only hosting I found for redmine starts at 29 Euro per month (and only for the annual billing).</p><h3>Asana</h3><p>Asana is to me something in between Trello and Jira. It seems good but in my opinion it fails to deliver. It is very easy to create a list of tasks and you can assign due dates. However it feels not very structured. It seems unstructured like Trello with an interface similar to Jira.</p><p>Wrike and Azendoo seem similar to Asana.</p><p>While these tools are interesting they are not contendant to me because they miss simple IDs for items. Some of them have long UUIDs associated to tasks, but this is not something I could use to mark commits or discuss about tasks. I tried doing without IDs but it just complicates things. I also tried assigning IDs manually, writing them in the title of the issue but this is rather awkward and you can end up having duplicates.</p><h3>Comparison</h3><p>There are other contenders like Taiga (it seems very rigid to me) and Basecamp (I did not spend too much tome on it). For now I have focused on Jira, Trello and Redmine. Asana, Wrike and Azendoo have been excluded because they do not assign human readable IDs to task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jSEJvtt3yp6vxgPGiM3hpQ.png" /></figure><p>My totals are:</p><ul><li>Jira: 7.5</li><li>Trello: 6.5</li><li>Redmine: 9</li></ul><p>When I say Redmine I actually mean Easyredmine or Redmine with some plugins like AgileDwarf.</p><h3>Conclusions</h3><p>Jira is not perfect and sometimes I hate it. I think everyone hates it a little bit. However it is something that works. It has a good hosting offer and it has a nice REST Api which I could use to implement quick creation of issues for planning sessions. It has also a CLI based on that REST API (this CLI comes with an additional plugin which is not free of charge).</p><p>I like also Redmine and I think that with a few additions I could get something rather good. What I do not like is the lack of hosting options.</p><p>I plan to focus on Jira and Redmine and see what kind of workarounds I can find for their problems. After that I will need to think about the deployment: use something from the cloud? Host it on my servers?</p><p>And you? What tools do you use to stay on top of your activities?</p><p><em>Originally published at </em><a href="http://tomassetti.me/on-the-quest-for-the-right-project-management-tool-jira-trello-asana-redmine/"><em>tomassetti.me</em></a><em> on June 21, 2016.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b18420672030" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Parsing any language in Java in 5 minutes using ANTLR: for example Python]]></title>
            <link>https://medium.com/@ftomassetti/parsing-any-language-in-java-in-5-minutes-using-antlr-for-example-python-e4670266274a?source=rss-debe1b49278d------2</link>
            <guid isPermaLink="false">https://medium.com/p/e4670266274a</guid>
            <category><![CDATA[gradle]]></category>
            <category><![CDATA[android]]></category>
            <dc:creator><![CDATA[Federico Tomassetti]]></dc:creator>
            <pubDate>Wed, 24 Feb 2016 08:00:40 GMT</pubDate>
            <atom:updated>2016-11-16T20:31:23.765Z</atom:updated>
            <content:encoded><![CDATA[<p>I like processing code for several purposes, like static analysis or automated refactoring. The interesting part to me is to reason on the models you build from the Abstract Syntax Tree (AST). To get there you need a way to get the AST from your source files. This can be done easily using ANTLR and the collection of complete grammars available here: <a href="https://github.com/antlr/grammars-v4">https://github.com/antlr/grammars-v4</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*0wsFozq_iuUOnsJc." /></figure><p><em>Thank you folks for all the grammars!</em></p><p>We are just going to take the one for Python 3, which should work fine also for Python 2. If we will need to do minor adjustment we can easily do that starting from this base.</p><h3>Getting the grammar</h3><p>First things first: let’s get the grammar.</p><p>Just visit <a href="https://github.com/antlr/grammars-v4">https://github.com/antlr/grammars-v4</a> and take the grammar you need. Most grammars have a very permissive license.</p><p>There are tens of grammars for languages such as R, Scala, Python, Swift, PHP and many others. There is also one for Java but for Java you prefer to use <a href="https://github.com/javaparser/javaparser">JavaParser,</a> am I right?</p><p>Just copy the grammar into your new project, under <em>src/main/</em>antlr</p><h3>Setting up the project using Gradle</h3><p>Now we are going to setup a build script with Gradle.</p><p>We will use the ANTLR4 plugin from <a href="https://github.com/melix/antlr4-gradle-plugin">melix</a>, because I find it more flexible of the one described in the <a href="https://docs.gradle.org/current/userguide/antlr_plugin.html">official documentation</a>.</p><p>We will generate the code in a specific package (<em>me.tomassetti.pythonast.parser</em>) and therefore in a directory derived from that package (build/generated-src/me/tomassetti/pythonast/parser).</p><pre>buildscript {<br>    repositories {<br>        maven {<br>            name &#39;JFrog OSS snapshot repo&#39;<br>            url  &#39;<a href="https://oss.jfrog.org/oss-snapshot-local/&#39;">https://oss.jfrog.org/oss-snapshot-local/&#39;</a><br>        }<br>        jcenter()<br>    }</pre><pre>dependencies {<br>        classpath &#39;me.champeau.gradle:antlr4-gradle-plugin:0.1.1-SNAPSHOT&#39;<br>    }<br>}</pre><pre>repositories {<br>    mavenCentral()<br>    jcenter()<br>}</pre><pre>apply plugin: &#39;java&#39;<br>apply plugin: &#39;me.champeau.gradle.antlr4&#39;</pre><pre>antlr4 {<br>    source = file(&#39;src/main/antlr&#39;)<br>    output = file(&#39;build/generated-src/me/tomassetti/pythonast/parser&#39;)<br>    extraArgs = [&#39;-package&#39;, &#39;me.tomassetti.pythonast.parser&#39;]<br>}</pre><pre>compileJava.dependsOn antlr4</pre><pre>sourceSets.main.java.srcDirs += antlr4.output</pre><pre>configurations {<br>    compile.extendsFrom antlr4<br>}</pre><pre>task fatJar(type: Jar) {<br>    manifest {<br>        attributes &#39;Implementation-Title&#39;: &#39;Python-Parser&#39;,<br>                   &#39;Implementation-Version&#39;: &#39;0.0.1&#39;<br>    }<br>    baseName = project.name + &#39;-all&#39;<br>    from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } }<br>    with jar<br>}</pre><p>I also added a <strong>fatJar</strong> task. That tasks produce a JAR containing all the dependencies. I use it to import the parser into Jetbrains MPS more easily.</p><p>To generate the parser from the grammar you can just run <em>gradle antlr4.</em></p><p>You can then have to explain to your IDE that it should consider the code under <em>build/generated-src.</em></p><h3>How to invoke the parser</h3><p>Now let’s see how we can invoke the parser.</p><pre>public class ParserFacade {</pre><pre>private static String readFile(File file, Charset encoding) throws IOException {<br>        byte[] encoded = Files.readAllBytes(file.toPath());<br>        return new String(encoded, encoding);<br>    }</pre><pre>public Python3Parser.File_inputContext parse(File file) throws IOException {<br>        String code = readFile(file, Charset.forName(&quot;UTF-8&quot;));<br>        Python3Lexer lexer = new Python3Lexer(new ANTLRInputStream(code));<br>        CommonTokenStream tokens = new CommonTokenStream(lexer);<br>        Python3Parser parser = new Python3Parser(tokens);<br>        return parser.file_input();<br>    }<br>}</pre><p>Our <em>ParserFacade </em>has just one public method named <em>parse</em>. It gets a File and it returns an AST. It could hardly be simpler than that.</p><h3>Let’s look at some ASTs</h3><p>Let’s take a simple file:</p><pre>def sum(a, b):<br>    return a + b</pre><pre>print(&quot;The sum of %i and %i is %i&quot; % (5, 3, sum(5, 3)))</pre><p>And now get the AST. We can print it using this code:</p><pre>public class AstPrinter {</pre><pre>public void print(RuleContext ctx) {<br>        explore(ctx, 0);<br>    }</pre><pre>private void explore(RuleContext ctx, int indentation) {<br>        String ruleName = Python3Parser.ruleNames[ctx.getRuleIndex()];<br>        for (int i=0;i&lt;indentation;i++) {<br>            System.out.print(&quot;  &quot;);<br>        }<br>        System.out.println(ruleName);<br>        for (int i=0;i&lt;ctx.getChildCount();i++) {<br>            ParseTree element = ctx.getChild(i);<br>            if (element instanceof RuleContext) {<br>                explore((RuleContext)element, indentation + 1);<br>            }<br>        }<br>    }<br>    <br>}</pre><p>If we parse the simple example and print it with AstPrinter we get a super complex AST. The first lines look like:</p><pre>file_input<br>  stmt<br>    compound_stmt<br>      funcdef<br>        parameters<br>          typedargslist<br>            tfpdef<br>            tfpdef<br>        suite<br>          stmt<br>            simple_stmt<br>              small_stmt<br>                flow_stmt<br>                  return_stmt<br>                    testlist<br>                      ...</pre><p>For the way the parser it is build there are a lot of annidated rules. That makes sense while parsing but it produces a very polluted AST. I think there are two different ASTS: as a <em>parsing AST </em>which is easy to produce, and a <em>logic AST</em> that it is easy to reason about. Luckily we can transform the first one in the latter without too much effort.</p><p>One simple way is to list all the rules that are just wrappers and skip them, taking their only child instead. We could have to refine this but as a first approximation let’s just skip the nodes which have just one children which is another parser rule (no terminals).</p><p>In this way we go from 164 nodes to 28. The resulting <em>logic AST</em> is:</p><pre>file_input<br>  funcdef<br>    parameters<br>      typedargslist<br>        tfpdef<br>        tfpdef<br>    suite<br>      simple_stmt<br>        return_stmt<br>          arith_expr<br>            atom<br>            atom<br>  simple_stmt<br>    power<br>      atom<br>      trailer<br>        term<br>          string<br>          atom<br>            testlist_comp<br>              integer<br>              integer<br>              power<br>                atom<br>                trailer<br>                  arglist<br>                    integer<br>                    integer</pre><p>In this tree everything should we mapped to a concept we understand, with no artificial nodes in the way, nodes just created for parsing reasons.</p><h3>Conclusions</h3><p>Writing parsers is no where we can produce most value. We can easily reusing existing grammars, generate parsers and build our smart applications using those parsers.</p><p>There are several parser-generators out there and most of them are good enough for most goals you can have. Among them I tend to use ANTLR more than others: it is mature, it is supported, it is fast. The ASTs it produces can be navigated both using hereogeneous APIs (we have single classes genered for each kind of node) and homogeneous APIs (we can ask to each node which rule it represents and the list of its children).</p><p>Another great benefit of ANTLR is the presence of grammars ready to be used. Building grammars require experience and some work. Especially for complex GPL like Java or Python. It also requires very extensive testing. We are still finding minor issues with the Java 8 grammars behind JavaParser even if we have parsed literally hundreds of thousands of files using it. This is a very good reason to not write your own grammar if you can avoid that.</p><p>By the way, all the code is available on github: <a href="https://github.com/ftomassetti/python-ast">python-ast</a></p><p><strong>Question: are you interested in parsing? Do you prefer other parser generators to ANTLR?</strong></p><p><em>Originally published at </em><a href="http://tomassetti.me/parsing-any-language-in-java-in-5-minutes-using-antlr-for-example-python/"><em>tomassetti.me</em></a><em> on February 24, 2016.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e4670266274a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The 5 things a developer expects from a Project Manager: how a Project Manager can help developers…]]></title>
            <link>https://medium.com/@ftomassetti/the-5-things-a-developer-expects-from-a-project-manager-how-a-project-manager-can-help-developers-361b6494f010?source=rss-debe1b49278d------2</link>
            <guid isPermaLink="false">https://medium.com/p/361b6494f010</guid>
            <dc:creator><![CDATA[Federico Tomassetti]]></dc:creator>
            <pubDate>Wed, 04 Nov 2015 08:17:00 GMT</pubDate>
            <atom:updated>2016-11-16T20:20:21.191Z</atom:updated>
            <content:encoded><![CDATA[<p>The 5 things a developer expects from a Project Manager: how a Project Manager can help developers becoming much more productive</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/965/0*Kmkz3xTEJJhoprsr." /></figure><p>How project managers can help developers being productive</p><p>To be effective as a software developer technical excellence is not enough. On top of that there are several other aspects on which a great professional should focus. Near the top of my list there is the ability to interact with other people involved in the project. Whatever is the nature of your project you will need to interact to get things done:</p><ul><li>as an open-source contributor you have to collaborate reviewing patches or having your patches reviewed, you have to address the issues brought up by users, you want to communicate your features to new users, planning with other committers or co-maintainers</li><li>as a freelance you have to interact with the current customers and with potential ones. You have also to interact with other developers or designers or testers involved in the projects and you need to communicate clearly who is responsible for what</li><li>when working in a company you have to coordinate with the other developers, in your team and in other teams, to communicate with your manager and most importantly <strong><em>to interact with project managers</em></strong></li></ul><h3>Developers and PMs… not always love at first sight</h3><p>Relationships with Project Managers can be a bit controversial from time to time: we as developers are very prone to complain about them. After all they are the ones bothering us about some change that need to go in at 6 PM on a Friday. Or the ones who keep pushing for features which do not make sense to us.</p><p>However I think that Project Managers play a fundamental role in successful team. And as a developer I can be successful only if the team is successful. For this reason I think that having a great relationship with Project Managers is they key for delivering results. I was lucky enough to work with amazing Project Managers that really helped me. This was especially true when I was at TripAdvisor: there I met PMs who were absolutely great. That does not mean I did not complain about them with my fellow developers from time to time 🙂</p><p>It does mean however that I understood that if we were going in the right direction, if we were delivering features at an high speed and if we were able to coordinate our projects with the rest of the company it was because of their work.</p><p>So I am very convinced that PMs can make a difference. But <strong><em>this difference can be either atrociously negative or wonderfully positive</em></strong>. I am not saying I understand all the responsibilities of a PM and I am sure there is a lot of things they do without interacting with developers like me. I am just thinking specifically to how PMs interact with developers and what kind of expectations I have as a developer towards PMs.</p><p>From my point of view a Project manager could make the life of developers easier by doing these five things.</p><h3>1) Communicate business priorities and consider technical priorities</h3><p>We are all over-worked and we all have crazy piles of work who someone expects us to do within the week. As a developer I need to be able to evaluate the effort needed to complete each task and the relations between tasks. Perhaps a certain refactoring will simplify developing a certain feature, so it makes sense to order those tasks accordingly. A certain task could take two weeks, while implementing these three other features could require just half a day for each of them. So I could prefer to work on them first.</p><p>But technical aspects are just half of the picture: ordering tasks require us to be aware of the priorities business have. What is most important for the customer? Which features can have an immediate impact on our revenues? That is very important for us to know to decide where to spend our energy and what to deliver first. I think PMs need to discuss frequently with developers what are the priorities and understand that business priorities and technical priorities need both to be considered to decide on what we are going to work next.</p><p>And sometimes PMs technical priorities are important too: they cannot always be ignored to consider only business priorities, because doing so it is going to affect our ability to deliver software and consequently the business.</p><h3>2) Let the developers know about deadlines well in advance</h3><p>Did ever happen to you to find out that something is needed… later today? Or that the customer was promised to receive a new build yesterday? Those are not nice surprises. Let’s be clear: <strong><em>sh*t happens</em></strong> and require us to deal with it. The application goes down and the company loses money <em>every minute</em>: you stop whatever you are doing and you fix it. A new bug is found and it is bad, a security concern needs to addressed as soon as possible. In real life there are things that we cannot plan for. We can just react to them.</p><p>However this cannot be the case for everything, we cannot always be doing <em>Emergency-driven development</em>. This is just bad practice. Deadlines need to be agreed upon and communicated, so that they can be planned for. As developers we frequently do not see the whole picture but the same goes for PMs: there are technical aspects that they could be ignoring and that could make not possible to meet a deadline, without knowing far in advance. So please dear PMs: let us know about deadlines as soon as you know them.</p><p><strong>Caveat:</strong> when I say “let the developers know about deadlines” I mean the <em>real</em> deadlines. One of the worst thing a PM could ever do is to present some fake, self-imposed deadline. Some PMs came up with the idea of assuring themselves a time buffer by telling the devleopers that the customer expects the delivery to happen on the 1st when they actually agreed on the 15th. Maybe they do that because we frequently deliver things late but… guess what? Sooner or later developers will find out and think about all the useless stress or the long hours they went through because of your lies. How do you think they are going to react?</p><h3>3) Manage communications</h3><p>I know it could sound not plausible but developers tend to have a few defects. One of them is that developers tend to communicate… differently. They tend to be direct and blunt. And that works great with machines, a bit less with customers. Yes, it is possible that the SDK the customer given to you seems… suboptimal. Yes, it could seems a perfect example of what a bunch of monkeys could do if we would give them a box of cheap whiskey and a manual of all the possible bad practices in software engineering. Well, still it is not a good idea to let the customer knows about that. It is much better to let the PM rephrase that for us.</p><p>And I love when the PM chase the customer about that decision we need him to make. Or talk with the PM of another team to convince them to answer to the requests from our team. <em>Yes, we really needed an answer, all the three times we forwarded again the same requests that was ignored for the last couple of weeks.</em></p><p>A fantastic PM will provide us with all the information we need to do a our job and ensure communication run smoothly with all the parties involved. We probably will not even realize the effort he had to put on this.</p><h3>4) Shield developers from issues</h3><p>Life in a company is stressful. Stress comes from different sources and it needs to be managed. As Software Developers we deal with a lot of stress coming from technical challenges: a bug which is very difficult to reproduce, an intermittent issue depending on threads synchronization, the new release of a framework which silently breaks everything, an unreliable piece of infrastructure that makes some integration test to fail. We have plenty of reasons to be stressed.</p><p>And PMs have too: they deal more than us with the internal politics, they take part in discussions about what features to develop and which not, they could fight to get resources for the team. They could compete with other teams, they have customers screaming at them. Well I am sorry for that but if PMs start to push all their stress on the developers, then we, the developers, will end up being between the anvil and the hammer. We would be both constrained by the reality of technical difficulties and the pink-pony-crazyness of customers and politics. That is just too much so let’s have a deal: we stress with technical stuff and that is our burden. All the rest, I am sorry for you PMs, but this is for you to deal with.</p><h3>5) Make sure we work on relevant projects</h3><p>It could happen to work very, very hard on building something that has a very limited impact on the company product. While it could be a satisfaction anyway for someone who loves to build cool stuff in the long run this is going to be detrimental for your career. It does not help you getting a promotion if you keep working only on irrelevant stuff. Working on something that can really make a difference for the business have several advantages instead: it gives you additional motivation, the possibility to get noticed and possibly access to more resources and more support from the organization.</p><p>Working on irrelevant features is not yet the worst thing that could happen. It could happen to work on projects that are discarded before or immeditaly after they are completed. Imagine put your passion and sweat on something that is just thrown away. Not a nice feeling, eh? So it is better to work with PMs who do not put you in such situations.</p><h3>In the end</h3><p>I think Project Managers intercept many problems before we can even see them. They are our interface with the customers and with the rest of the organization. They ensure we are working on something which will provide value to our users. They keep track of the whole picture so that we can have the luxury to just focus on the next feature to implement, test it and ship it.</p><p>And it is a fact that developers take PMs for granted most of the time. I think it is pretty common for developers to underestimate PMs. We often just do not understand all their responsibilities. But trust me, if you work with a great PM and a… not so great one, you will definitely notice the difference.</p><p>I hope both us developers and project managers can benefit from a better relationship. I can tell you it is possible to do so: I live with a PM who happens to be my girlfriend 🙂</p><p><em>Originally published at </em><a href="http://tomassetti.me/the-5-things-a-developer-expects-from-a-project-manager-how-a-project-manager-can-help-developers-becoming-much-more-productive/"><em>tomassetti.me</em></a><em> on November 4, 2015.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=361b6494f010" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started with Spark: it is possible to create lightweight RESTful application also in Java]]></title>
            <link>https://medium.com/@ftomassetti/getting-started-with-spark-it-is-possible-to-create-lightweight-restful-application-also-in-java-9f214a7ac32b?source=rss-debe1b49278d------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f214a7ac32b</guid>
            <category><![CDATA[android]]></category>
            <category><![CDATA[android-app-development]]></category>
            <dc:creator><![CDATA[Federico Tomassetti]]></dc:creator>
            <pubDate>Wed, 01 Apr 2015 08:30:07 GMT</pubDate>
            <atom:updated>2016-11-16T20:27:45.828Z</atom:updated>
            <content:encoded><![CDATA[<p>Getting started with Spark: it is possible to create lightweight RESTful application also in Java</p><p>Recently I have been writing a RESTful service using <a href="http://sparkjava.com/">Spark</a>, a web framework for Java (which is <em>not</em> related to Apache Spark). When we planned to write this I was ready to the unavoidable <em>Javaesque</em> avalanche of interfaces, boilerplate code and deep hierarchies. I was very surprised to find out that an alternative world exists also for the developers confined to Java.</p><p>In this post we are going to see how to build a RESTful application for a blog, using JSON to transfer data. We will see:</p><ul><li>how to create a simple Hello world in Spark</li><li>how to specify the layout of the JSON object expected in the request</li><li>how to send a post request to create a new post</li><li>how to send a get request to retrieve the list of posts</li></ul><p>We are not going to see how to insert this data in a DB. We will just keep the list in memory (in my real service I have been using <a href="http://www.sql2o.org/">sql2o</a>).</p><p><strong><em>Note:</em></strong><em> I wrote a bunch of other tutorials on Spark. Take a look at </em><a href="https://sparktutorials.github.io/"><em>Spark tutorials website</em></a><em>.</em></p><h3>A few dependencies</h3><p>We will be using Maven so I will start by creating a new pom.xml throwing in a few things. Basically:</p><ul><li>Spark</li><li>Jackson</li><li>Lombok</li><li>Guava</li><li>Easymock (used only in tests, not presented in this post)</li><li>Gson</li></ul><pre>&lt;dependencies&gt;<br>    &lt;dependency&gt;<br>      &lt;groupId&gt;junit&lt;/groupId&gt;<br>      &lt;artifactId&gt;junit&lt;/artifactId&gt;<br>      &lt;version&gt;4.12&lt;/version&gt;<br>      &lt;scope&gt;test&lt;/scope&gt;<br>    &lt;/dependency&gt;<br>    &lt;dependency&gt;<br>      &lt;groupId&gt;com.sparkjava&lt;/groupId&gt;<br>      &lt;artifactId&gt;spark-core&lt;/artifactId&gt;<br>      &lt;version&gt;2.1&lt;/version&gt;<br>    &lt;/dependency&gt;<br>    &lt;dependency&gt;<br>      &lt;groupId&gt;com.fasterxml.jackson.core&lt;/groupId&gt;<br>      &lt;artifactId&gt;jackson-core&lt;/artifactId&gt;<br>      &lt;version&gt;2.5.1&lt;/version&gt;<br>    &lt;/dependency&gt;<br>    &lt;dependency&gt;<br>      &lt;groupId&gt;com.fasterxml.jackson.core&lt;/groupId&gt;<br>      &lt;artifactId&gt;jackson-databind&lt;/artifactId&gt;<br>      &lt;version&gt;2.5.1&lt;/version&gt;<br>    &lt;/dependency&gt;<br>    &lt;dependency&gt;<br>      &lt;groupId&gt;org.projectlombok&lt;/groupId&gt;<br>      &lt;artifactId&gt;lombok&lt;/artifactId&gt;<br>      &lt;version&gt;1.16.2&lt;/version&gt;<br>      &lt;scope&gt;provided&lt;/scope&gt;<br>    &lt;/dependency&gt;<br>      &lt;dependency&gt;<br>        &lt;groupId&gt;org.sql2o&lt;/groupId&gt;<br>        &lt;artifactId&gt;sql2o&lt;/artifactId&gt;<br>        &lt;version&gt;1.5.4&lt;/version&gt;<br>      &lt;/dependency&gt;<br>    &lt;dependency&gt;<br>      &lt;groupId&gt;org.postgresql&lt;/groupId&gt;<br>      &lt;artifactId&gt;postgresql&lt;/artifactId&gt;<br>      &lt;version&gt;9.4-1201-jdbc41&lt;/version&gt;<br>    &lt;/dependency&gt;<br>    &lt;dependency&gt;<br>      &lt;groupId&gt;com.google.guava&lt;/groupId&gt;<br>      &lt;artifactId&gt;guava&lt;/artifactId&gt;<br>      &lt;version&gt;18.0&lt;/version&gt;<br>    &lt;/dependency&gt;<br>    &lt;dependency&gt;<br>      &lt;groupId&gt;org.easymock&lt;/groupId&gt;<br>      &lt;artifactId&gt;easymock&lt;/artifactId&gt;<br>      &lt;version&gt;3.3.1&lt;/version&gt;<br>      &lt;scope&gt;test&lt;/scope&gt;<br>    &lt;/dependency&gt;<br>    &lt;dependency&gt;<br>      &lt;groupId&gt;com.google.code.gson&lt;/groupId&gt;<br>      &lt;artifactId&gt;gson&lt;/artifactId&gt;<br>      &lt;version&gt;2.3.1&lt;/version&gt;<br>      &lt;scope&gt;test&lt;/scope&gt;<br>    &lt;/dependency&gt;<br>  &lt;/dependencies&gt;<br>  <br>  &lt;build&gt;<br>    &lt;plugins&gt;<br>      &lt;plugin&gt;<br>        &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;<br>        &lt;artifactId&gt;maven-compiler-plugin&lt;/artifactId&gt;<br>        &lt;version&gt;3.2&lt;/version&gt;<br>        &lt;configuration&gt;<br>          &lt;source&gt;1.8&lt;/source&gt;<br>          &lt;target&gt;1.8&lt;/target&gt;<br>        &lt;/configuration&gt;<br>      &lt;/plugin&gt;<br>      &lt;plugin&gt;<br>        &lt;groupId&gt;org.codehaus.mojo&lt;/groupId&gt;<br>        &lt;artifactId&gt;exec-maven-plugin&lt;/artifactId&gt;<br>        &lt;version&gt;1.2.1&lt;/version&gt;<br>        &lt;configuration&gt;<br>          &lt;mainClass&gt;me.tomassetti.BlogService&lt;/mainClass&gt;<br>          &lt;arguments&gt;<br>          &lt;/arguments&gt;<br>        &lt;/configuration&gt;<br>      &lt;/plugin&gt;<br>    &lt;/plugins&gt;<br>  &lt;/build&gt;</pre><h3>Spark hello world</h3><p>Do you have all of this? Cool let’s write some code then.</p><pre>package me.tomassetti;<br> <br>import static spark.Spark.get;<br>import static spark.Spark.post;<br>import spark.Request;<br>import spark.Response;<br>import spark.Route;</pre><pre>public class BlogService <br>{<br>    public static void main( String[] args) {<br>        get(&quot;/posts&quot;, (req, res) -&gt; {<br>            return &quot;Hello Sparkingly World!&quot;;<br>        });<br>    }<br>}</pre><p>And now we can run it with something like:</p><pre>mvn compile &amp;&amp; mvn exec:java</pre><p>Let’s open a browser and visit localhost <a href="http://localhost:4567/posts."><em>http://localhost:4567/posts</em>.</a> Here we want to do a simple get. For performing posts you could want to use the Postman plugin for your browser or just run <a href="http://en.wikipedia.org/wiki/CURL">curl</a>. Whatever works for you.</p><h3>Using Jackson and Lombok for awesome descriptive exchange objects</h3><p>In a typical RESTful application we expect to receive POST requests with json objects as part of the payload. Our job will be to check the code is well-formed JSON, that it corresponds to the expected structure, that the values are in the valid ranges, etc. Kind of boring and repetitive. We could do that in different ways. The most basic one is to use <a href="https://code.google.com/p/google-gson/">gson</a>:</p><pre>JsonParser parser = new JsonParser();<br>JsonElement responseData = parser.parse(response);<br>if (!responseData.isJsonObject()){<br>  // send an error like: &quot;Hey, you did not pass an Object!<br>}<br>JsonObject obj = responseData.getAsJsonObject();<br>if (!obj.hasField(&quot;title&quot;)){<br>  // send an error like: &quot;Hey, we were expecting a field name title!<br>}<br>JsonElement titleAsElem = obj.get(&quot;title&quot;);<br>if (!titleAsElem.isString()){<br>  // send an error like: &quot;Hey, title is not an string!<br>}<br>// etc, etc, etc</pre><p>We probably do not want to do that.</p><p>A more declarative way to specify what structure we expect is creating a specific class.</p><pre>class NewPostPayload {<br>   private String title;<br>   private List&lt;String&gt; categories;<br>   private String content;<br>   <br>   public String getTitle() { ... }<br>   public void setTitle(String title) { ... }<br>   public List&lt;String&gt; getCategories() { ... }<br>   public void setCategories(List&lt;String&gt; categories){ ... }<br>   public String getContent() { ... }<br>   public void setContent(String content) { ... }<br>}</pre><p>And then we could use Jackson:</p><pre>try {<br>   ObjectMapper mapper = new ObjectMapper();<br>   NewPostPayload newPost = mapper.readValue(request.body(), NewPostPayload.class);<br>} catch (JsonParseException e){<br>   // Hey, you did not send a valid request!<br>}</pre><p>In this way Jackson check automatically for us if the payload has the expected structure. We could want to verify if additional constraints are respected. For example we could want to check if the title is not empty and at least one category is specified. We could create an interface just for validation:</p><pre>interface Validable {<br>   boolean isValid();<br>}</pre><pre>class NewPostPayload implements Validable {<br>   private String title;<br>   private List&lt;String&gt; categories;<br>   private String content;<br> <br>   public String getTitle() { ... }<br>   public void setTitle(String title) { ... }<br>   public List&lt;String&gt; getCategories() { ... }<br>   public void setCategories(List&lt;String&gt; categories){ ... }<br>   public String getContent() { ... }<br>   public void setContent(String content) { ... }</pre><pre>public boolean isValid() {<br>      return title != null &amp;&amp; !title.isEmpty() &amp;&amp; !categories.isEmpty();<br>   }<br>}</pre><p>Still we have a bunch of boring getters and setters. They are not very informative and just pollute the code. We can get rid of them using <a href="http://projectlombok.org/">Lombok</a>. Lombok is an annotation processor that add repetitive methods for you (getters, setters, equals, hashCode, etc.). You can think of it as a plugin for your compiler that looks for annotations (like <em>@Data</em>) and generates methods based on them. If you add it to your dependencies maven will be fine but your IDE could not give you auto-completion for the methods that Lombok adds. You may want to install a plugin. For Intellij Idea I am using <em>Lombok Plugin</em> version 0.9.1 and it works great.</p><p>Now you can revise the class <em>NewPostPayload</em> as:</p><pre><a href="http://twitter.com/Data">@Data</a><br>class NewPostPayload {<br>   private String title;<br>   private List&lt;String&gt; categories;<br>   private String content;<br>   <br>   public boolean isValid() {<br>       return title != null &amp;&amp; !title.isEmpty() &amp;&amp; !categories.isEmpty();<br>   }<br>}</pre><p>Much nicer, eh?</p><h3>A complete example</h3><p>We need to do basically two things:</p><ol><li>insert a new post</li><li>retrieve the whole list of posts</li></ol><p>The first operation should be implemented as a POST (it has side effects), while the second one as a GET. Both of them are operation on the <em>posts</em> collection so we will use the endpoint <em>/posts</em> .</p><p>Let’s start by inserting post. First of all we will parse</p><pre>        // insert a post (using HTTP post method)<br>        post(&quot;/posts&quot;, (request, response) -&gt; {<br>            try {<br>                ObjectMapper mapper = new ObjectMapper();<br>                NewPostPayload creation = mapper.readValue(request.body(), NewPostPayload.class);<br>                if (!creation.isValid()) {<br>                    response.status(HTTP_BAD_REQUEST);<br>                    return &quot;&quot;;<br>                }<br>                int id = model.createPost(creation.getTitle(), creation.getContent(), creation.getCategories());<br>                response.status(200);<br>                response.type(&quot;application/json&quot;);<br>                return id;<br>            } catch (JsonParseException jpe) {<br>                response.status(HTTP_BAD_REQUEST);<br>                return &quot;&quot;;<br>            }<br>        });</pre><p>And then see how to retrieve all the posts:</p><pre>        // get all post (using HTTP get method)<br>        get(&quot;/posts&quot;, (request, response) -&gt; {<br>            response.status(200);<br>            response.type(&quot;application/json&quot;);<br>            return dataToJson(model.getAllPosts());<br>        });</pre><p>And the final code is:</p><pre>package me.tomassetti;<br> <br>import static spark.Spark.get;<br>import static spark.Spark.post;</pre><pre>import com.fasterxml.jackson.core.JsonParseException;<br>import com.fasterxml.jackson.databind.ObjectMapper;<br>import com.fasterxml.jackson.databind.SerializationFeature;<br>import lombok.Data;<br>import spark.Request;<br>import spark.Response;<br>import spark.Route;</pre><pre>import java.io.IOException;<br>import java.io.StringWriter;<br>import java.util.HashMap;<br>import java.util.LinkedList;<br>import java.util.List;<br>import java.util.Map;<br>import java.util.stream.Collector;<br>import java.util.stream.Collectors;</pre><pre>public class BlogService <br>{</pre><pre>private static final int HTTP_BAD_REQUEST = 400;<br>    <br>    interface Validable {<br>        boolean isValid();<br>    }</pre><pre><a href="http://twitter.com/Data">@Data</a><br>    static class NewPostPayload {<br>        private String title;<br>        private List&lt;String&gt; categories = new LinkedList&lt;&gt;();<br>        private String content;</pre><pre>public boolean isValid() {<br>            return title != null &amp;&amp; !title.isEmpty() &amp;&amp; !categories.isEmpty();<br>        }<br>    }<br>    <br>    // In a real application you may want to use a DB, for this example we just store the posts in memory<br>    public static class Model {<br>        private int nextId = 1;<br>        private Map&lt;Integer, Post&gt; posts = new HashMap&lt;&gt;();<br>        <br>        <a href="http://twitter.com/Data">@Data</a><br>        class Post {<br>            private int id;<br>            private String title;<br>            private List&lt;String&gt; categories;<br>            private String content;<br>        }<br>        <br>        public int createPost(String title, String content, List&lt;String&gt; categories){<br>            int id = nextId++;<br>            Post post = new Post();<br>            post.setId(id);<br>            post.setTitle(title);<br>            post.setContent(content);<br>            post.setCategories(categories);<br>            posts.put(id, post);<br>            return id;<br>        }<br>        <br>        public List&lt;Post&gt; getAllPosts(){<br>            return posts.keySet().stream().sorted().map((id) -&gt; posts.get(id)).collect(Collectors.toList());<br>        }<br>    }</pre><pre>public static String dataToJson(Object data) {<br>        try {<br>            ObjectMapper mapper = new ObjectMapper();<br>            mapper.enable(SerializationFeature.INDENT_OUTPUT);<br>            StringWriter sw = new StringWriter();<br>            mapper.writeValue(sw, data);<br>            return sw.toString();<br>        } catch (IOException e){<br>            throw new RuntimeException(&quot;IOException from a StringWriter?&quot;);<br>        }<br>    }<br>    <br>    public static void main( String[] args) {<br>        Model model = new Model();<br>        <br>        // insert a post (using HTTP post method)<br>        post(&quot;/posts&quot;, (request, response) -&gt; {<br>            try {<br>                ObjectMapper mapper = new ObjectMapper();<br>                NewPostPayload creation = mapper.readValue(request.body(), NewPostPayload.class);<br>                if (!creation.isValid()) {<br>                    response.status(HTTP_BAD_REQUEST);<br>                    return &quot;&quot;;<br>                }<br>                int id = model.createPost(creation.getTitle(), creation.getContent(), creation.getCategories());<br>                response.status(200);<br>                response.type(&quot;application/json&quot;);<br>                return id;<br>            } catch (JsonParseException jpe) {<br>                response.status(HTTP_BAD_REQUEST);<br>                return &quot;&quot;;<br>            }<br>        });<br>        <br>        // get all post (using HTTP get method)<br>        get(&quot;/posts&quot;, (request, response) -&gt; {<br>            response.status(200);<br>            response.type(&quot;application/json&quot;);<br>            return dataToJson(model.getAllPosts());<br>        });<br>    }<br>}</pre><h3>Using PostMan to try the application</h3><p>You may want to use curl instead, if you prefer the command line. I like not having to escape my JSON and having a basic editor so I use PostMan (a Chrome plugin).</p><p>Let’s insert a post. We specify all the fields as part of a Json object inserted in the body of the request. We get back the ID of the post created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*r96DlJfyyOTB3M33." /></figure><p>Then we can get the list of the posts. In this case we use a GET (no body in the request) and we get the data of all the posts (just the one we inserted above).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CuGD5FNIv0laO4dT." /></figure><h3>Conclusions</h3><p>I have to say that I was positively surprised by this project. I was ready for the worse: this is the kind of application that requires a basic logic and a lot of plumbing. I found out that Python, Clojure and Ruby do all a great jobs for this kinds of problems, while the times I wrote simple web applications in Java the logic was drown in boilerplate code. Well, things can be different. The combination of Spark, Lombok, Jackson and Java 8 is really tempting. I am very grateful to the authors of these pieces of software, they are really improving the life of Java developers. I consider it also a lesson: great frameworks can frequently improves things much more than we think.</p><p><strong>Edit:</strong> I received a suggestion to improve one of the example from the good folks on reddit. Thanks! Please keep the good suggestions coming!</p><p><em>Originally published at </em><a href="http://tomassetti.me/getting-started-with-spark-it-is-possible-to-create-lightweight-restful-application-also-in-java/"><em>tomassetti.me</em></a><em> on April 1, 2015.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f214a7ac32b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>