Skip to content

Commit 61f70f4

Browse files
jim80netrtyler
authored andcommitted
some edits for your consideration
1 parent e1c858d commit 61f70f4

File tree

1 file changed

+25
-27
lines changed

1 file changed

+25
-27
lines changed

_posts/2020-06-24-shipping-rust-to-production.md

Lines changed: 25 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -10,18 +10,20 @@ team: Core Platform
1010
author: rtyler
1111
---
1212

13-
When we set our goals at the beginning of the year "deploy Rust to production"
14-
was not among them, yet here we are. Our pattern of deploying small services in containers
15-
allowed us to easily bring Rust into production, replacing a difficult to
16-
manage service in the process. In January, the Core Platform team started working on a
17-
project called "View Analytics". The effort was primarily to replace an aging
18-
system which was literally referred to as "old analytics." As part of the View
19-
Analytics design we needed to provide an entry point for [Fastly](https://fastly.com) to relay access logs as syslog
20-
formatted messages which could then be written into [Apache Kafka](https://kafka.apache.org), driving the entire
21-
View Analytics data pipeline. Our initial rollout shipped an [rsyslog](https://www.rsyslog.com)-based solution
22-
for the “rsyslog-kafka” service.. Using rsyslogd worked fairly well, but had a
23-
couple of significant downsides. Last month, we deployed its replacement: a
24-
custom open source daemon written in Rust: [hotdog](https://github.com/reiseburo/hotdog) 🌭.
13+
Deploying small services in containers allowed us to easily replace a production service
14+
with a weekend project written in Rust.
15+
16+
The Core Platform team maintains a project to provide an entry point for
17+
[Fastly](https://fastly.com) access logs to be written into
18+
[Apache Kafka](https://kafka.apache.org).
19+
20+
Our initial rollout of "View Analytics" used [rsyslogd](https://www.rsyslog.com) for the ingestion service,
21+
“rsyslog-kafka.” Using rsyslogd worked fairly well, but had a couple of significant
22+
downsides.
23+
24+
Last month, we open sourced and deployed "rsyslog-kafka"'s replacement: a daemon
25+
that provides better metrics, JMESPath rules, and RFC compliant message parsing:
26+
[hotdog](https://github.com/reiseburo/hotdog) 🌭.
2527

2628

2729
(**Note:** _This specific use-case was well suited to Rust. That does not mean
@@ -38,16 +40,13 @@ aforementioned `rsyslog-kafka` service relied on it because of its ubiquity. We
3840
had a problem that looked like routing logs from one thing (Fastly) to another
3941
thing (Kafka), and that's basically what `rsyslogd` does!
4042

41-
This flexibility comes at a price. When explaining to colleagues what rsyslog
43+
However, when explaining to colleagues what rsyslog
4244
_really_ is, I would describe it as "an old C-based scripting engine that just
4345
happens to forward logs." If they didn't believe me, I would send them the
4446
documentation to
4547
[Rainerscript](https://rsyslog.readthedocs.io/en/latest/rainerscript/), named
46-
after [Rainer Gerhards](https://en.wikipedia.org/wiki/Rainer_Gerhards) the
47-
author of `rsyslog`. Accepting this "zen of rsyslog" makes it slightly
48-
easier to understand and work with the rsyslog configuration syntax.
49-
That said, I still find it incredibly difficult to work with, and even harder
50-
to test.
48+
after [Rainer Gerhards](https://en.wikipedia.org/wiki/Rainer_Gerhards), the
49+
author of `rsyslog`. I find it incredibly difficult to work with, and even harder to test.
5150

5251
In our pipeline, we needed to bring JSON formatted messages from Fastly and
5352
route them to the appropriate topics, using the approximate format of:
@@ -63,28 +62,27 @@ route them to the appropriate topics, using the approximate format of:
6362
}
6463
```
6564

66-
JSON parsing in rsyslog is feasible, but not easy. To make matters worse, there
67-
was no way to handle JSON keys which use the dollar-sign `$`, because the
65+
JSON parsing in rsyslog is feasible, but not easy. For example, there
66+
is no way to handle JSON keys which use the dollar-sign `$`, because the
6867
scripting interpreter treats `$` characters as variable references. The
69-
original version of our rsyslog-kafka gateway that went into production ended
70-
up using regular expressions to fish out the topic we needed for routing!
68+
original version of our rsyslog-kafka gateway that went into production
69+
uses regular expressions to fish out the topic!
7170

7271
Unfortunately, the daemon also does not emit metrics or statistics natively in
7372
a format we could easily get into Datadog. The only way to get the statistics
74-
we needed out of the daemon would be to ingest statistics written out to a file through a sidecar
73+
we needed would be to ingest statistics written out to a file through a sidecar
7574
container and report those into Datadog. This would have required building a
7675
custom daemon to parse the rsyslogd stats output which seemed like a lot of
77-
work without a lot of benefit.
76+
work for a little bit of benefit.
7877

79-
This all left us with very little understanding of how a service which was
80-
difficult to configure and test would actually run in production.
78+
We didn't know how this difficult and untestable service would actually run in production.
8179

8280

8381
## Makin' hotdogs
8482

8583
Bored one weekend with nothing to do, I asked myself “how hard could getting syslog into Kafka be?” As it turned out: _not that hard_.
8684

87-
I continued to improve the [daemon](https://github.com/reiseburo/hotdog) over a number of
85+
I continued to improve [hotdog](https://github.com/reiseburo/hotdog) over a number of
8886
weeks until I had feature parity with our rsyslogd use-case, and then some!
8987

9088
* RFC 5424/3164 syslog-formatted message parsing

0 commit comments

Comments
 (0)