<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pavel Potapenkov on Medium]]></title>
        <description><![CDATA[Stories by Pavel Potapenkov on Medium]]></description>
        <link>https://medium.com/@pavelpotapenkov?source=rss-eeb4dbe37152------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*Gp4y7PpdlVb7E4anBscVmA.jpeg</url>
            <title>Stories by Pavel Potapenkov on Medium</title>
            <link>https://medium.com/@pavelpotapenkov?source=rss-eeb4dbe37152------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 11:15:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pavelpotapenkov/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Advocating YAML as DSL]]></title>
            <link>https://medium.com/@pavelpotapenkov/advocating-yaml-as-dsl-7f5fe695fba9?source=rss-eeb4dbe37152------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f5fe695fba9</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Pavel Potapenkov]]></dc:creator>
            <pubDate>Thu, 23 Feb 2017 12:01:52 GMT</pubDate>
            <atom:updated>2017-02-23T12:01:52.462Z</atom:updated>
            <content:encoded><![CDATA[<p>Domain specific languages are everywhere. Some scripting language embedded into some system — is a DSL. Formatting syntax in the online text editor — is a DSL. SQL — is a DSL. Even syntax of config files — is a DSL (rather limited most of the times).</p><p>Everything is quite simple when we talk about simple config files. That ones, where you configure on what port to listen to new connections and at what host some useful API is living on. Generally you may use whatever syntax you want — it won’t make great difference, especially if you don’t have enormous amount of different properties. Things are a bit trickier with embedded scripting languages. If you need something turing-complete, you’ll have to choose wisely. But that’s not the case I want to discuss here.</p><h3>A tiny bit of context</h3><p>To describe what kind of DSL I’m talking about, I’ll give two examples of systems, that are nearly useless without good language, that can describe their configuration. But first, couple of words about environment. Imagine rather big infrastructure. Let’s say you have several thousands of servers. And this is not some monolithic cluster — all these servers are divided into couple of hundreds of different groups, each of which doing some work. Some of them are frontends of service A, some — are backends of service B, and some — are a part of computing cluster C. And so on, and so on, and so on.</p><p>So, the first example is torrent-based system for deployment of some static data. For example, you may need to generate something like search index and distribute it over all servers, that will serve users’ search requests. But you don’t want to allow just anybody to upload new data to the torrent-tracker. So, you have at least two requirements: to describe by whom (or from what clusters) the particular dataset is allowed to be uploaded; and to describe who should download this particular dataset. Also, there may be some additional options, like TTL or how many older generations you want to store (ability to rollback really quick— is priceless). And since we have rather big infrastructure, let’s assume, that there are several hundreds of different datasets, that are being generated in different places, and are being downloaded by different clusters. This seems like we will have pretty huge configuration file. And once in several days some new datasets, or generator clusters, or client clusters are appeared.</p><p>The second example is graph plotting system. When you have such broad infrastructure, you definitely want to plot lots of graphs. And likely you want not to just collect some generic data, but create reasonably complex rules. For example, you may want a graph, that represents a rate of successful requests on example.com with url, that starts with /some_handler or /blah, but excluding /some_handler_internal, from clients with user-agent, that contains Firefox, that came via IPv6, and who’s request serving duration was greater than 300 milliseconds. Such requirements may sound insane to some people, but believe me — that’s not the most complicated rule, that you may want to use. And my experience says, that on average there will be much more, than one new graph per day. So the configuration format should be a convenient tool, that allows you to describe such graphs without titanic effort.</p><h3>The pain of choice</h3><p>So I hope these two examples will help to understand what kind of DSLs I’m talking about. There’s lots of possible choises, and when deciding what language to use (or create from scratch) for describing such configuration, one should consider several factors:</p><ol><li><strong>Target audience.</strong> Some strict and easily parseable format is good from technical point of view. XML or JSON (or something else, but with similar functionality) are very easy to parse and therefore using them will save development time. And they are relatively convenient if you understand them — so we may assume, that for tech people such formats will be a good solution. Like in the case of data-deployment system. But in the case of graphs not only tech people would like to edit them. Some data may be crucial from business point of view, but not from technical one. Have you ever tried to explain your business-development team, what XML is? It’s insanely hard, and much easier solution will be to create some intermediate tool, or make configuration syntax less strict and more close to the natural language.</li><li><strong>Complexity of the language itself.</strong> The less complex the language is, the more freedom you have in choosing your DSL. In data-deployment you basically have only four different possible expressions: where data is generated, where it is deployed, how long it’s TTL is, and how many copies to store as backup. From this point of view you may choose any language. And what about graphs? You’ll need at least several comparison operators (some of them may use regular expressions) and aggregation operators like “and” and “or”. And if you have some monitoring or analytics based on data from graphs (graphite, for example, has API that will give you raw time-series data), then you likely will want to store some additional parameters, like limits of “normal” request serving duration, or maximum of allowed errors, or something like that.</li><li><strong>Complexity of possible expressions.</strong> Once again: the less complex your language expressions, the more freedom you have. In data-deployer rules are mostly simple and it may be even tempting to use natural-like language — it may save lots of keystrokes, if you have to write something as simplistic, as dataset ds-0001 deploy to group_1, group_2 in your configuration. Similar syntax seems to be a good idea for graphs on the first glance. But let’s write our previous example of graph description in syntax similar to any programming language: $hostname == &quot;example.com&quot; &amp;&amp; ($url ^= &quot;/some_handler&quot; || $url ^= &quot;/blah&quot;) &amp;&amp; !($url ^= &quot;/some_handler_internal) &amp;&amp; $user_agent ~= &quot;Firefox&quot; &amp;&amp; $ip ^= &quot;::ffff:&quot; &amp;&amp; $request_time &gt;= 300ms. Not very readable, isn’t it? And if you change different special characters like “&amp;&amp;” to there textual analogues like “and”, it will still be not so pleasant to read and write. And the main source of errors here will be operations priority and order of evaluation. Just remove round brackets from that expression and try to figure out what it will actually match with. And that’s only “query” without any additional info like name of graph and possible alerting limits! So, obviously we need something a bit more strict.</li><li><strong>Amount of expressions.</strong> The single expression (though rather complex), that describes single graph is a total mess. Should I say, that if you have several thousands of them, they will be completely unmanageable? And that simplistic syntax of data-deployment system may be messy too. If you have long lists of groups, that particular dataset should be deployed to, and/or you wish to tune TTL and amount of older versions on per-group basis, then writing configuration on one line will be too inconvenient to read — it may become wider than your screen. So you’ll have to use some sort of multiline syntax, ant that will ruin your ability to quickly find all datasets, that are deployed to particular group. Or if you choose to describe generators and deployments in different expressions, you may end with situation, when they will eventually find themselves in different ends of configuration file.</li></ol><h3>And the silver bullet is…</h3><p>Actually, there’s no silver bullet when deciding what DSL to choose. But as for me, I really think that YAML — is as close to be that bullet as possible. First of all, it is strict: it describes data structures in the manner similar to JSON. It’s syntax is actually a superset of JSON syntax, but allowing to write much less special characters if there’s no ambiguity. It also allows you to write come expressions on the single line, and stretch other ones across several lines. And if you use it’s syntax well, you’ll get really readable configuration. And there are open-source libraries, that will do parsing of YAML for you — it’s definitely a huge advantage unless you <em>want</em> to write you own parser for your own DSL.</p><p>YAML is definitely good for tech people. It will even be okay for some of your non-tech colleagues, especially if we’re talking about not very complex syntax. And for all other contributors of our configuration files we can write tools convenient for casual people, thus using YAML as intermediate representation. You invoke something like update-config.py &quot;deploy dataset-1234 to group_8&quot; and it loads configuration, parses it, changes, and saves it back. It’s trivial with YAML, but may be insanely tricky with home-brewed syntax.</p><p>And let’s look at our graph example, written in one of possible YAML representations:</p><pre>And:<br>  - hostname: { equals: &quot;example.com&quot; }<br>  - Or: [{url: {starts: &quot;/some_handler&quot;}}, {url: {starts: &quot;/blah&quot;}}]<br>  - Not: {url: {starts: &quot;/some_handler_internal&quot;}}<br>  - user_agent: { contains: &quot;Firefox&quot; }<br>  - ip: { starts: &quot;::ffff:&quot; }<br>  - request_time: { gte: 300 }</pre><p>Or the same configuration without most of special characters:</p><pre>And:<br>  - hostname:<br>      equals: example.com<br>  - Or:<br>      - url:<br>          starts: /some_handler<br>      - url:<br>          starts: /blah<br>  - Not:<br>      url:<br>        starts: /some_handler_internal<br>  - user_agent:<br>      contains: Firefox<br>  - ip:<br>      starts: &quot;::ffff:&quot;<br>  - request_time:<br>      gte: 300</pre><p>Oddly enough it’s very readable and even looks pretty natural if you don’t pay attention to different brackets and colons. Second version is more sparse and looks more convenient to some people. As for me, I prefer more compact notation — it fits better, when you have lots of graphs in one file. Well, editing such configuration is not exactly the easiest thing in the world, but it’s not hard and since modern text-editors highlight syntax, it will be not so easy to forget some brackets or miscalculate indentation.</p><h3>Conclusion</h3><p>So, YAML is a very powerful tool when designing DSLs of particular type. It can be highly readable when formatted right, and writing new configurations (or editing existing ones) is fairly easy too, allowing you to create rather complex configurations, that look neat. And since it is a worldwide standard, it is easily parseable by third-party libraries — that saves lots of time during development and allows other people to easily create tools, that will either use your configuration as an input source for some purpose, or use your configuration as intermediate representation while implementing alternative front-ends for configuring your systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f5fe695fba9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When there’s medieval got on your ass]]></title>
            <link>https://medium.com/@pavelpotapenkov/when-theres-medieval-got-on-your-ass-46ddf8d1d12?source=rss-eeb4dbe37152------2</link>
            <guid isPermaLink="false">https://medium.com/p/46ddf8d1d12</guid>
            <category><![CDATA[system-administration]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Pavel Potapenkov]]></dc:creator>
            <pubDate>Sun, 12 Feb 2017 22:27:59 GMT</pubDate>
            <atom:updated>2017-02-13T10:16:52.892Z</atom:updated>
            <content:encoded><![CDATA[<p>Anyone more or less related to IT operations knows one simple truth: fuckups are inevitable. No matter how good your infrastructure is, no matter how professional your people are. Sooner or later it will happen and you’ll have to deal with it. And some time later it will happen again, and again, and again — and the only thing you can really do is to be prepared. Oh, and also to make sure, that none of later fuckups will be the same as the earlier ones.</p><h3>Definitions.</h3><p>But what fuckup actually is? Generally, it’s some sort of situation where the company is loosing money. The loss may be either direct: if you can’t sell something, or show ads, or make anything else you’re doing for money — that’s definitely a fuckup situation. Or it may be indirect: if there’s one of your internal services is broken in some way so it’s not possible to deploy new features or to efficiently communicate — that’s also a fuckup. Everything is clear with the direct loss — you can always measure it. But the indirect one is a bit tricky sometimes.</p><p>Let’s say, you’re a developer. Regardless of the deadlines you want to write code fast. May be not “fast” in conventional sense, but it can be really frustrating if you have an inspiration, but you can’t use it right now — may be your favourite development server is down or is slow, may be your build system is broken or may be your deploy system became insane. Either way, by the time the infrastructure is fixed, you may become out of the right mood. Yes, of course, if you’re a professional with years of experience something like mood may not have crucial impact on your work, but still you’re likely to code faster and better when you are in a right condition. And faster coding means delivering features faster, which means (not always, though) more money. And better code means less technical debt. And therefore coding slower and worse — is obviously an indirect money loss for company.</p><p>Now put yourself to the operations camp. It’s really tempting to focus 99% of your attention on production systems. They are fun! And all that development stuff? It’s boring and unimportant. And it’s impact on the overall health of the company is barely measurable (in comparison with the money per request in production systems). So what do you do while recovering your systems after some major outage? Let’s assume, that it is something like one of your datacenters accidentally was shut off and that affected almost all of your systems in some way. Chances are, you’ll start from your production systems and won’t pay attention to anything else until they are in their normal state again. And that’s assuming you’re postponing fixing of various development and testing environments — and that may leave developers with total inability to do something, that in normal situation takes few seconds.</p><h3>Classification</h3><p>So, that gives us the main rule of fuckup mitigation: minimize the loss of money. That assumes, that you need to order different systems by their priority. Generally, it’s good idea to first make sure, that all of your production systems are working and your clients are satisfied. After that it will be nice of you to fix all essential internal services — therefore giving other departments opportunity to continue their work. And only after that you should return to extinguishing remaining fire, restoring redundancy, reverting manual changes (they are truly evil by nature, but when every single minute of outage costs you [tens of] thousands of dollars — even darkest of magic is okay up to some extent), writing postmortems, brainstorming about causes and all that stuff.</p><p>So priorities are: 1) health of the services from clients’ point of view; 2) health of the services from the neighbourhood departments’ point of view; 3) health of the services from operations department point of view.</p><p>But of course not all fuckups are born equal. Roughly we can group all possible fuckups into three types. First and mildest one is invisible to external clients, but affects internal ones in some way. Second — is visible to external clients, but does not affect crucial functionality, or may be if affects very small group of clients. And the last and most scary type: it is visible from the outside and significant part of clients is affected.</p><h3>Mitigation</h3><p>So, you got the alert. It may be notification from your monitoring system, or may be report from your colleague, or may be you noticed something wrong by yourself — that really does not matter. At the moment of getting the alert you’re likely to know nothing about the scale of the problem. So where should you start?</p><p>The graphs are always a good start. Chances are, during the outage you will see the growth of errors of different kind. But not necessarily. So the very first graph you should be staring on is rate of successful requests. This will most likely give you understanding how bad things are — just compare current rate with yesterday (or week ago if it significantly varies depending on weekday) and see how many of requests are lost. And if none — that’s a possibly good sign of not-so-big outage. Though you should check other graphs anyway, because for example if the fuckup is caused by increased load, the seemingly normal requests rate may be just a fraction of the actual one. And of course it may be purely functional fuckup: when your service replies with good response codes, but with absolutely wrong data.</p><p>So, you understood the scale. And you see the problem. Now you have basically two options: to understand and eliminate the root cause or to apply some quick hack to make service work. The right answer is definitely “apply quick fix”. But don’t forget to leave some part of the affected service broken (and make sure it will not receive any production requests while in that state) so you can still investigate the root cause. Deep understanding of the reasons is crucial to avoid similar situation in the future, but not by the cost of prolonging outage. Well, sometimes you do not have opportunity or technical means for leaving part of the infrastructure broken — and in this situation you should collect as many fuckup-related diagnostics as possible and do it as quick as it is reasonable. Prolonging one outage for the sake of eliminating all the future ones — may be a noble and totally reasonable sacrifice.</p><h3>Posmortem</h3><p>You are writing postmortems, aren’t you? Good. So you know how tempting is it to add another monitoring check to catch problems like described in the outage timeline you’re writing right now. But actually that’s not such a good idea sometimes. It’s always better to think what you can do to prevent this from happening again instead of how to detect the problem when it strikes back (though, these two ways of thinking are definitely NOT mutually exclusive).</p><p>We’re not talking here about really complex fuckups. Your servers in one of the datacenters lost network connectivity with everything but load balancers and LBs continue to send requests, that are doomed to be responded with errors? You may ensure that LBs will not do that again, you may add more redundancy to your network… And your milage may vary, bringing lots of other possible solutions.</p><p>So let’s talk about more simple situations. Say, you’ve commited something terribly wrong to your configuration management system and if caused some problems visible to clients. What’s the best way to prevent this in future? One of possible solutions is to get rid of pushing changes directly to the repository, but to write a tool (may be in a form of pre-commit hook), that will first validate your changes and only then push them. Or may be you’re experiencing outage because of memory leaks leading to OOM kills. It’s definitely good idea to be alerted about such stuff. But even better solution is to make some watchdog, that will kill leaking software before it cause damage to server. And it would be great, if this watchdog is made only as a temporary solution, while the leak is being debugged and fixed. And you know what would be excellent in this kind of situations? The system, that will track memory consumption, alert you when it reaches uncommon levels and acting as a watchdog when the situation is almost fucked up, but we still have time to kill the nasty piece of software leaving important system stuff intact.</p><p>So, try not to rely on monitoring too much. Automate everything that possible, so you can spend more time doing really interesting stuff, while automation mitigates microfuckups, prefuckup situations and all other kinds of events, that may lead to full-sized outage if not treated timely.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46ddf8d1d12" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>