<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by darkliquid on Medium]]></title>
        <description><![CDATA[Stories by darkliquid on Medium]]></description>
        <link>https://medium.com/@darkliquid?source=rss-ceb2afd921f2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*CtxtRUkWg2P25_I7.jpg</url>
            <title>Stories by darkliquid on Medium</title>
            <link>https://medium.com/@darkliquid?source=rss-ceb2afd921f2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 05:50:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@darkliquid/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[RCON clients in Go]]></title>
            <link>https://medium.com/multiplay/rcon-clients-in-go-47140310c822?source=rss-ceb2afd921f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/47140310c822</guid>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[darkliquid]]></dc:creator>
            <pubDate>Tue, 28 Nov 2017 18:11:28 GMT</pubDate>
            <atom:updated>2018-04-19T10:12:39.471Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bGrB8DuFhB4B6A4ZQJZ_Zw.jpeg" /></figure><p>As part of our continued efforts to make game server hosting easier, we recently built and released into the wild a series of RCON clients for popular game RCON protocols as open source on Github.</p><p>Go makes writing these things particularly satisfying as its standard library includes a whole bunch of networking code that is nice and simple to use. We built these clients because we want to have our hosting management software able to communicate with the actual games it manages so it can both monitor them for information (like player counts) and also ask them to shutdown cleanly (for games that don’t cleanly shutdown when they receive the TERM signal and instead potentially corrupt any open files they are writing to). While there are many, many different game protocols out there, many games tend to use one of the following:</p><ul><li><a href="https://github.com/multiplay/go-source">A2S</a> (<a href="https://goo.gl/STu1Vs">Source RCON Protocol</a>)</li><li><a href="https://github.com/multiplay/go-battleye">BattlEye</a> (<a href="https://www.battleye.com/downloads/BERConProtocol.txt">BattlEye RCON Protocol</a>)</li></ul><p>We also implemented a specific client just for TeamSpeak, since it is very popular.</p><ul><li><a href="https://github.com/multiplay/go-ts3">TS3</a> (<a href="http://media.teamspeak.com/ts3_literature/TeamSpeak%203%20Server%20Query%20Manual.pdf">TeamSpeak 3 Server Query Protocol</a>)</li></ul><h3>How do I use them?</h3><p>Examples of basic usage can be found on the Github pages linked above. The clients for A2S and BattlEye are designed to be as basic as possible and know basically nothing about what they might be talking to. There are no wrappers and convenience methods for known commands for game X or server Y, instead the clients provide a simple interface to talking directly at the protocol level to allow you to use them for any game or server that speaks that language. To copy the example code from the BattlEye client, it’s quite simple:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e79c9d6af07b1b61e1832e906dd14a3/href">https://medium.com/media/5e79c9d6af07b1b61e1832e906dd14a3/href</a></iframe><p>The code above is near identical for the A2S client and you can see it is very simple to just send a command to the server and receive a response.</p><p>The TS3 client works a little differently because it is a specific client for a specific server. In this case, we provide methods for each command rather than just a raw protocol client. The full documentation is available via the Github page, but a short example demonstrates just how simple it is to use:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3ba24ae18da17eb79c133a6cd8b4a810/href">https://medium.com/media/3ba24ae18da17eb79c133a6cd8b4a810/href</a></iframe><h3>How do YOU use them?</h3><p>Well now, that would be telling…</p><p>Oh okay then, I’ll give you a brief overview. As mentioned above, being able to nicely shutdown servers that maintain a lot of writable open files so we don’t cause corruption on process termination is pretty important. No-one likes to log into their Minecraft server after restarting it and find that their entire world is just gone.</p><p>To facilitate this, our ClanForge server management platform does a number of checks and controls around how it communicates with the servers it is managing and how it shuts them down. Here’s a snippet from one of our core management services on exactly how it does that:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f3a1604850abf77ae6e66d49cefdb29/href">https://medium.com/media/4f3a1604850abf77ae6e66d49cefdb29/href</a></iframe><p>The example above is taken from a function that gets executed whenever a stop request is actioned (i.e. shutting down the game server) on a managed server.</p><p>On the first line, we grab a client using the identifier for the game we’re trying to operate on. We’ve written a number of game-specific clients that build on top of the basic protocol libraries we’ve open sourced that are all registered at compile time against their game identifiers. If we don’t have a client for that game, then we return early. Not every game needs a client and if it doesn’t have one, we just carry on as normal.</p><p>However, if we <em>do</em> have a client, <em>and</em> that client supports the PreStop hook — a hook we define as meaning “things that need to be done before the process is forcibly terminated” — then we let our other management processes know that a stop is expected so they don’t think the game just crashed and try to restart it. Assuming that all goes well, we execute our clients’ PreStop hook to run through all the things we need to ensure your server doesn’t break anything when/if we force it to shutdown later, then we wait for it to exit gracefully and resort to hard terminating it if it takes too long.</p><p>PreStop for most servers involves the basic steps of:</p><ol><li>Tell the players they are going to be disconnected</li><li>Wait for a bit to give them a chance to finish whatever they are doing in game</li><li>Kick all the players from the server</li><li>Tell the game to save any data to disk it might have in memory</li><li>Tell the game to shutdown</li></ol><p>Simples!</p><p>We hope you find our RCON clients useful and we look forward to hearing about how you’re using them and seeing contributions from the open source community.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47140310c822" width="1" height="1" alt=""><hr><p><a href="https://medium.com/multiplay/rcon-clients-in-go-47140310c822">RCON clients in Go</a> was originally published in <a href="https://medium.com/multiplay">Silicon Barn | The Official Blog From Multiplay’s Game Server Hosting Specialists</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Upgrading Golang 1.9 for great justice]]></title>
            <link>https://medium.com/multiplay/upgrading-golang-1-9-for-great-justice-39fe31261cb7?source=rss-ceb2afd921f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/39fe31261cb7</guid>
            <category><![CDATA[bugs]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[tcp]]></category>
            <category><![CDATA[time]]></category>
            <category><![CDATA[upgrade]]></category>
            <dc:creator><![CDATA[darkliquid]]></dc:creator>
            <pubDate>Wed, 18 Oct 2017 12:51:36 GMT</pubDate>
            <atom:updated>2017-10-18T12:51:36.026Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BP8LTIpqrrxD7VXgsPR5bQ.jpeg" /></figure><p>Golang 1.9 fixed a few of issues for us that were ‘interesting’ to say the least; in particular, an issue resulting from the tickers in the golang time package not being monotonic, and one in the runtime netpoller itself that caused us some major pain at scale.</p><h3>Let’s do the time warp again</h3><p>When taking actions periodically, especially at fine granularity, you generally care about doing them in order, maintaining cause and effect. For our stats logging, we were using the golang time package Tickers to take snapshots of data and send it periodically. This all seems great until you remember we live in a world where time isn’t quite as fixed as it seems.</p><p>Time doesn’t always move forward in computers. Sometimes, it stops, or goes backwards, or jumps into the future. Clocks go out of sync, time servers become unavailable, clock skew happens due to hardware or software issues, timezones change and daylight savings happen. All of these things can affect what time your computer thinks it is, <em>while it is running.</em> What happens when you are sending data every minute and suddenly the next minute is happening in 2 seconds instead of another 59 because the clock has changed?</p><p>Well, this is where monotonic time sources come in. Monotonic clocks are not subject to clock adjustment so they reliably increase as time goes on.</p><p>In some of our applications, we have things happening at regular intervals and upstream these get aggregated into measurement periods (for example, sending stuff at 1 minute intervals and aggregating things into 1 minute chunks). For these things, it doesn’t matter <em>where</em> in the aggregation period an event occurs, just that no more than one occurs in the same period — events must be guaranteed to happen exactly one interval length apart.</p><p>When time is flexible, this can no longer be true — which is where monotonic time saves the day. Upgrading to 1.9 resolved an issue where clock skew would sometimes have us seeing double entries in one period and no entries in the next one, and now this no longer occurs. Thanks 1.9!</p><h3>Wake me up, before you go go</h3><p>We discovered we were victims of a peculiar issue in the Go runtime that caused our TCP connections to error when talking to GCE. As it turns out, this was down to spurious wakeups happening that meant the connections would think they were established when they actually weren’t. Consequently, we’d be grabbing a connection and using it, but it would actually be doomed to fail from the start. Some excellent analysis of the issue can be seen here: <a href="https://github.com/google/google-api-go-client/issues/220">https://github.com/google/google-api-go-client/issues/220</a> and here: <a href="https://github.com/golang/go/issues/19289">https://github.com/golang/go/issues/19289</a>. Ultimately, the fix got merged into 1.9 here: <a href="https://github.com/golang/go/commit/bf0f69220255941196c684f235727fd6dc747b5c">https://github.com/golang/go/commit/bf0f69220255941196c684f235727fd6dc747b5c</a>. Apparently the issue was quite rare, but for our use cases we were seeing it multiple times a day and it made a couple of things quite unreliable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39fe31261cb7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/multiplay/upgrading-golang-1-9-for-great-justice-39fe31261cb7">Upgrading Golang 1.9 for great justice</a> was originally published in <a href="https://medium.com/multiplay">Silicon Barn | The Official Blog From Multiplay’s Game Server Hosting Specialists</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fluxgate for Grafana]]></title>
            <link>https://medium.com/multiplay/fluxgate-for-grafana-38833869aa88?source=rss-ceb2afd921f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/38833869aa88</guid>
            <category><![CDATA[game-server]]></category>
            <category><![CDATA[influxdb]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[proxy]]></category>
            <dc:creator><![CDATA[darkliquid]]></dc:creator>
            <pubDate>Thu, 12 Oct 2017 15:16:17 GMT</pubDate>
            <atom:updated>2017-10-12T15:16:17.148Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*16fVpvfljtXh2m8HQNLtKw.jpeg" /></figure><p>Grafana is awesome and it’s a super-useful tool for exploring your data — we use it heavily for internal graphs and for exposing graphing capabilities to customers. However, one area Grafana falls down on is isolating specific things from specific users. It’s difficult if not impossible to provide access to only specific tables or measurements, and furthermore even harder to decide how to enforce those limitations based what user they log into Grafana as when everything is backed by the same InfluxDB cluster.</p><p>To solve some of these problems, we developed an internal InfluxDB proxy server called <strong>Fluxgate</strong>.</p><p>InfluxDB’s query interface is HTTP, so it is easy to make a little HTTP server which can proxy requests to a second upstream server (in this case, the real InfluxDB cluster). Fluxgate takes in these requests and looks at the HTTP Basic auth to determine the user from our own accounts systems. Naturally, since it’s basic auth, you’ll want to use HTTPS, but this user/password combo doesn’t have to be the same as anything that provides authorisation for anything other than Fluxgate. Equally, if you are using auth on InfluxDB itself as well, you can specify both sets of auth by using query parameters too.</p><p>From the user, Fluxgate can then determine exactly what services and features the user has available to them and can map those directly to prefixes which we use to shard our InfluxDB measurements by service. With all of this information and the magic of open-source, we can use the <a href="https://github.com/influxdata/influxdb/tree/master/influxql">query parser from InfluxDB</a> itself to pull apart the queries being made, check them against basic whitelists (no writes allowed, etc) and rewrite them to explicitly limit themselves to only measurements the user has access to. Since this is happening at the AST level, we can do a number of quite fine-grained checks and edits without hacky and error-prone solutions (such as string replacement with regexes), then reconstruct the query and pass it down the chain to InfluxDB proper.</p><p>At present, Fluxgate limits access not only to queries against measurements, but also to any of the more informational queries such as SHOW MEASUREMENTS, to prevent discovery and enumeration of other customer IDs. All non-whitelisted queries are rejected by returning a standard InfluxQL result structure with an error message, as are most queries that use regexes or other query methods that attempt to work around access restrictions. This does limit queries to more exact searches, but for most usages this has not proven to be an issue.</p><p>In this manner, fluxgate allows us to filter exactly what databases, measurements, and types of query a user is allowed to use. To use it, all you need to do in Grafana is setup an InfluxDB datasource using the InfluxDB URL set to Fluxgate with correct basic auth details and everything transparently works. The cool thing about this is that we don’t even need to provide Grafana ourselves to customers — if they have their own Grafana system they want to integrate with, they can just add Fluxgate as a datasource themselves into their own monitoring and graphing systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38833869aa88" width="1" height="1" alt=""><hr><p><a href="https://medium.com/multiplay/fluxgate-for-grafana-38833869aa88">Fluxgate for Grafana</a> was originally published in <a href="https://medium.com/multiplay">Silicon Barn | The Official Blog From Multiplay’s Game Server Hosting Specialists</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dealing with process termination and shutdowns in Windows]]></title>
            <link>https://medium.com/multiplay/dealing-with-process-termination-and-shutdowns-in-windows-8c9d45a3e4f8?source=rss-ceb2afd921f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c9d45a3e4f8</guid>
            <category><![CDATA[servers]]></category>
            <category><![CDATA[windows]]></category>
            <category><![CDATA[gaming]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[darkliquid]]></dc:creator>
            <pubDate>Wed, 23 Aug 2017 14:26:37 GMT</pubDate>
            <atom:updated>2017-08-23T14:26:37.885Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y6ylaxFE8yqZg6PxnvRDUw.jpeg" /></figure><p>It is unfortunately still a fact of life that our gamehosting machines sometimes need rebooting, and this is more true for Windows machines than any other. Windows updates often require that the machine gets restarted and there are only so many things you can do to mitigate this. Ultimately, restarts are inevitable and it is better to get them over and done with sooner rather than later, instead of putting it off and leaving a system unpatched and at risk.</p><p>So when a Windows update wants to shutdown the system automatically, what do you do? Just letting it happen isn’t an option — what if players are playing games on the servers running on that machine? Just having the servers disappear would be a terrible user experience and that’s without taking into account the issue of server-side save files. For games that persist data about players or the game world, if these servers just terminate, what happens to that data? Does it get corrupted? If not cleanly terminated, do players risk losing everything they’ve worked for? In some cases, the answer is yes, they do — and that is unacceptable.</p><p>So what can you do about it?</p><p>In an ideal world, all server code would catch the termination signals sent by Windows when a process is asked to exit, cleanly write everything important to disk and disconnect any players with a nice apology for the inconvenience. However, many, if not most, servers do not do this, and so we need to do this for them. This is where Clanforge comes in.</p><p>Clanforge manages every game server process on a machine, and this management system knows details about every game and whether or not it needs special handling. The management layer stops each server, using the relevant remote console protocols to send messages to players, kick them and trigger server saves before terminating the processes. But none of that is useful unless we can do all this before Windows decides to just kill all the processes itself and shutdown. Thus begins our journey down the rabbit hole of the Windows shutdown process…</p><h3>What Happens When Windows Shuts Down</h3><p>The Windows shutdown process isn’t well documented publicly, but helpfully there is a lot of information dotted around, including the lovely Windows Internals book. One of the main things you’ll see said about it is that Windows is going to terminate all your processes and shutdown — there is almost no avoiding it once that process has begun, as Windows makes it very difficult to circumvent.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FPAswVl_KZVU%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DPAswVl_KZVU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FPAswVl_KZVU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/bf370f2750d9823379dc069b389717e6/href">https://medium.com/media/bf370f2750d9823379dc069b389717e6/href</a></iframe><p>The gist of the process is that first, all processes which are running in an interactive session are terminated in reverse order of their shutdown priority (highest priority number first). The termination inspects what kind of process it is (console app, GUI with a top-level window, service) and sends it an event, waiting on the response to see whether it can continue the shutdown process. If it gets no response, or a response asking Windows to wait, it waits for a period of time specified by a registry key (different keys depending on process type). After everything in this list has successfully terminated, we move onto processes in <em>Session 0</em> — this is where system services live.</p><p>Services are controlled by the SCM (Service Control Manager), and the SCM first sends pre-shutdown notifications which allow services to block the shutdown process almost indefinitely, and then stop requests to the services in dependency order. It waits for services to report they stopped, or just terminates them if they exceed a timeout.</p><p>So what is the problem here?</p><p>Well, the services that run and manage the game servers are all <em>services.</em> This means they run in Session 0 and get shutdown last — sounds good, right?</p><p>Wrong. Because our management services run the actual games in the interactive session. Why? Because many game servers assume that they are being run on a normal interactive desktop and break in terrible, eldritch ways when you do otherwise. What this means is that Windows pulls out the rug from under the management services before it can do anything — the servers get killed before the service managing them even knows what hit it. A bit of a pain, to say the least.</p><p>So how do you work around problems like this? Well, there are a number of options, all with their various upsides and downsides:</p><ol><li>Prevent shutdown happening at all — basically disable the services that cause shutdowns to happen and manually do Windows updates with your own scheduled task.</li><li>Detect when a shutdown is requested, cancel it, shutdown the servers cleanly and then trigger a new shutdown</li><li>Block shutdown using the various methods available for doing that, and have a method that detects the shutdown request and shuts down the servers cleanly, then allow the shutdown process to continue.</li><li>Don’t run the management stuff as a Session 0 service, give it a huge shutdown priority and have it handle its own termination by cleanly shutting down the servers it is managing.</li></ol><p>Option 1 is a little drastic and personally, I don’t like it a whole lot. Disabling automatic updates feels both dirty and dangerous — which for many people is right up their street, but for game server hosting machines, not so much. If your own scheduler fails for some reason (for example, a rogue Windows update decides to trash/reset your scheduled tasks) or the Windows update service gets re-enabled, then you are in for a bad time. Additionally, manually triggering Windows updates just <em>feels</em> like it’s going to be less safe than letting the service manage it, due to the possibility of being late on applying critical or emergency patches. It also means eliminating the possibility of doing something nice for your users, like only applying updates and rebooting when there are no players online, since it’s just a dumb scheduled task — unless you build a separate system to detect these events and schedule things accordingly.</p><p>Option 2 involves having <em>something</em> detect that the system enters a shutdown state (for example, a process running in the interactive session with a high shutdown priority). When you notice this state, you can cancel it, trigger the safe shutdown procedure for all running game servers and then trigger a new shutdown at your leisure. This isn’t actually too bad, though one of the downsides is the possibility that you may <em>never</em> shutdown, because you don’t reschedule one, or new updates appear, or some other thing that it is now your responsibility to do doesn’t get done.</p><p>Option 3 is similar in some ways to Option 2 but instead of reacting to a shutdown in progress and cancelling it, we instead use the ShutdownBlockReasonCreate Windows call to register a block on the shutdown procedure as soon as our service starts. However, we <em>can’t</em> do that in the service, we need a process with a hidden window running in the interactive session to receive the WM_QUERYENDSESSION event and so the block should go there. When we receive this event, we can then trigger the service to shutdown all servers cleanly and when it finishes, wait for a message from it to tell our dummy blocking process to cancel the block using ShutdownBlockReasonDestroy and exit.</p><p>Option 4 loses us any benefits of using the Windows service management system (dependency management, remote control and event logging, to name a few). However, in many ways it does solve a bunch of issues that need to be worked around (like having to impersonate the current user in order to spawn server processes into the interactive session). It means having to change how the services are started, by using scheduled tasks or simply the autostart/startup functionality in Windows so the services get started at login, but those are trivial to handle. Not running as a service means we can do all the shutdown blocking stuff of options 2 and 3, but we don’t need an extra helper process, as our management processes can use them directly.</p><p>As you can see, each of these methods comes with their own benefits and caveats. Depending on exactly how you need to manage your systems and what kinds of complexity you want to deal with, any of the options may be appropriate, so it’s up to you to decide as there isn’t really a correct answer for all occasions. At the moment, our current solutions involve some manual intervention or are worked around in other ways that mitigate the issue of unexpected shutdowns without having to deal with them directly.</p><p>We’re still in the process of evaluating, experimenting with a few of the options until we settle on one that fits best into a long-term management and automation plan. With all the pros and cons of each, we have a lot of testing ahead of us! Game server hosting in particular is a complex beast; you don’t have any control over the way the server code you are running actually works and have a number of tight constraints (performance, OS, uptime, etc). For your own use-case, the answers might be more clear cut.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c9d45a3e4f8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/multiplay/dealing-with-process-termination-and-shutdowns-in-windows-8c9d45a3e4f8">Dealing with process termination and shutdowns in Windows</a> was originally published in <a href="https://medium.com/multiplay">Silicon Barn | The Official Blog From Multiplay’s Game Server Hosting Specialists</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>