From a software provider’s point of view, fixing issues in an On-Prem solution is inherently problematic, and so we have strived to make the solution simple. To realize this goal, a Humio installation consists only of a single process per node running Humio itself, being dependent on Kafka running nearby. (We recommend deploying one Humio node per physical CPU so a dual-socket machine typically runs two Humio nodes.)

We use Kafka for two things: buffering ingest and as a sequencer of events among the nodes of a Humio cluster.

Ingest Queue/Commit Log

The ingest queue is used for buffering ingest, to take peak loads, or deal with similar situations. But the ingest queue also plays the role of commit log. That is, whenever we finish a block and append that to one of Humio’s segment files, we also write the offset of the last message batch received into the on-disk block. That way we can restart with no loss by reading the last block of the Humio segment under construction, and restart ingest from the Kafka offset listed in that block.

Event Sourcing

The primary coordination mechanism of our cluster state also goes via Kafka. The ‘shared cluster state’ is maintained using a straightforward event sourcing model, where we push updates through a single-partition/multiple-replica topic called global-events. All nodes dump a snapshot of the internal state periodically to local disk, and when a node starts up, it will read it’s own snapshot and re-load updates starting at the offset written in the snapshot. This has proven a very simple mechanism to deal with distributed state.

To be on the safe side there are some more details in this, such as also recording the Kafka-cluster-id as the epoch of Kafka offsets. So if you wipe your Kafka or redirect to a different Kafka cluster some manual intervention is required.

Humio/Kafka configurations

As for ways to run Kafka, Humio supports three different configurations:

1. Single node: We provide one Docker container with Zookeeper, Kafka, and Humio in it. This makes it easy to try it out; just say docker run humio/humio and go.
2. Run a dedicated Kafka/Zookeeper to service your Humio cluster. We provide specialized containers for this scenario that deal with some management issues.
3. Bring-your-own Kafka. We try to stay out of the way of managing it.

The first option is great for a medium-small instance running on a single server. Also having both option 2 and 3 do complicate things somewhat.

On the other hand, many users want to run both Humio and Kafka as a cluster for scale and resilience. Unless the customer already has a dedicated team to manage Kafka, they’ll typically be using Humio with our humio/zookeeper and humio/kafka containers. In this scenario, we need to help users balance the Kafka cluster and configure replication etc. Humio uses a handful of topics that need varying configurations, and if the customer isn’t knowledgable with Kafka, it can easily be configured wrong (too small number of replicas, all replicas end up on a single node in the Kafka cluster etc.). So we ship the entire Kafka admin client with Humio and perform these re-configurations for the user. This mode is configured in Humio by setting KAFKA_MANAGED_BY_HUMIO=true.

On the other hand, for customers that are already managing a Kafka solution, we need to stay out of their way. A customer who ‘knows what they are doing’ will just be annoyed if we try to reconfigure their Kafka. To make sure our topic names don’t conflict with the existing ones, you can configure HUMIO_KAFKA_TOPIC_PREFIX to name space the topics we need.

Some details on our Kafka usage

We’ve found some issues that might be interesting for a Kafka’esque audience.

• We don’t use Kafka’s built-in compression, but provide our own. The issue is that in a high-performance scenario, the stock LZ4 compression uses JNI that can cause the garbage collector to lock. With many consumers running in the same JVM, we can easily get into a situation where there’s always a process that prevents GC from happening, leading to long doing-nothing pauses. We did run into the same issue with our web framework’s use of gzip and replaced that with com.jcraft.jzlib.
• We started using the deleteRecords functionality in Kafka (when ingested data is safely replicated to storage nodes, we can delete old entries in the ingest topic), but have since regretted that somehow. It took some versions of Kafka to make it work properly, and at the end of the day, the user needs the assigned retention space in case Humio has issues keeping up for some reason. It does save on typical disk space usage, but not worst case disk space needs. Should we decide today, we probably wouldn’t have used it; but now we may have customers depending on the disk space that’s released this way.
• We use the endOffsets / beginningOffsets methods in the consumer to determine the size of the inbound queue and to determine if ‘we’re running behind’ in ingest processing. These calls can sometimes take a very long time, and so it’s hard to say if we’re behind because the queue is long, or because it takes a long time to determine if the queue is long.

Anyhow, we love Kafka. It has made implementing a distributed system significantly easier than if we had to do the distributed queue from scratch. Using Kafka as a building block has allowed us to focusing our attention on building a fast and pleasant search experience.

How we use Kafka was originally published in Humio on Medium, where people are continuing the conversation by highlighting and responding to this story.

As a Humio user you already know our concept of a ‘dataspace’. With this release, we’re abolishing that notion of dataspaces and replacing it with two separate entities: Repositories (which contain data) and Views (which only show data). Don’t worry, with the upgrade all your dataspaces have been converted to repositories and you can still use all the same APIs and UI interactions as before.

About the Names

There are a few really hard problems in computer science, and one of them is naming things. The names View and Repo has been the subject of much debate here at Humio, and a good way to think of these two words is to compare them with the names Table and View as used in context of a relational database. In a database tables contain the data and views contain a read-only result of a query potentially joining multiple tables. Likewise in Humio: a repository contains data, and a view is a filtered aggregate across repositories. Data contained in Humio is not as structured as it would be in a relational database, so it doesn’t make sense to call it a table. Thus, the more generic concept of a repository.

Play with Views!

To get started, you can just click the ‘Add’ button on the front page, which will bring you to this choice between a green pill, and a purple pill:

Choosing the green pill lets you create a view. Views and repos have separate access control (list of users who can see it), and you need to be admin in a repository to be able to create a view that makes reference to that repository.

Example: Restricting Access

Say you have a repository that contains all your AD logs from your entire organization, but you have decentralized help-desk operations in different parts of the company. Maybe it is a problem that the various help-desks can see everyone else’s AD activity.

Now you can create a view of just the AD logs that pertain to part of your organization, and only assign access rights to personel in the local helpdesk team so that that they only have access to the AD logs for their own users. It might look like this:

Then you can go to the settings of the east-AD-helpdesk view, and add just the users who should be allowed to see event data from AD4.prod and AD5.prod . Those users do not need to have access to the underlying repository to be able to access the view.

Mixed Retention

Previously in Humio, it was an issue that you were not able to mix data with different retention. Now you can create a view that merges data from two separate repositories, each with their own retention setting. Beware though, that when you get into the ‘tail’ of the view, not all repositories contribute to the data content.

More about Views

The expressions that limit which part of a repo that is visible to a view can contain filter expressions, which also includes rewriting events. For instance, you can hide information such as social security numbers or credit card information in the view by using the replace function in the filter expression, or you can rewrite field names using rename or simply reassign then usin := to be similar across different data sources.

Many elements are local to views: files, saved queries, dashboards. These are not shared across views or repositories, so they can also be used as a place to make such changes local.

Views do not make queries more expensive or slow. In general they should reduce the workload for on-prem customers because less data ends up being searched. As always however, the best way to make queries run fast is to use tags #tag=value, and preferrably as early as possible in the query expression. Since the filter expressions used in defining a view are essentially prepended on the queries, it is particularly beneficial if you put the tag filters in the view definition.

So What About Repositories?

As noted, all existing data spaces have been converted to repositories, which contain all the same saved queries, files and dashboard that you already had. And for simple use cases, you can just keep doing everything in a repository like you used to.

Once you start creating views however, you’ll notice that some things are missing in views settings menu: Parsers, Ingest Tokens, Retention Settings, i.e., everything having to do with data ingest, parsing and storage. The distinctive feature with repos is that they contain the data, and so this is also where you have to go to configure these aspects.

Other Things in Humio 1.1.0

We’ve also done a wealth of other improvements that most people probably have missed. Maybe you’ll notice that it runs up to 30% faster on long searches. We’ve removed a lot of small bottlenecks in the core of the search engine, and all together we find that it runs much more smooth.

One final note for on-prem installations: Updating to 1.1 is not reversible, so you cannot roll back safely. The release notes have some provisions for how to backup your pre-upgrade state to be able to do a lossy rollback; i.e. you can do a partial roll back loosing certain changes that happened since the update.

Humio 1.1: Introducing Views & Repos was originally published in Humio on Medium, where people are continuing the conversation by highlighting and responding to this story.

The cluster configuration allows:

• scaling out to deal with high ingest
• scaling out to improve query performance

In a Humio cluster, some nodes are arrival nodes (they actually receive the data from the real world), and you can specify which nodes should deal with ingest processing, and which nodes to use for search. The nodes responsible for search also store the underlying segment files. In the default setup all nodes are equal — they all play the roles of arrival, ingest and search nodes, so the load gets evenly distributed across your nodes, which is a good starting point.

We discussed part of the ingest pipeline in a previous post, but left out how we scale out ingest to a cluster.

Data Arrives

Any node in the cluster can act as point of arrival, as all nodes expose the Humio HTTP API. You’ll have to control which nodes are arrival nodes in a load balancer or using DNS, or using some other means outside of Humio.

When data arrives, the request is validated: the ingest token is validated, and the request is parsed. If, for instance, you send nonconforming JSON, then the request is rejected.

After parsing, we know the destination dataspace and the full set of tags, and thus we’ll know which data source the event belongs to. The first time an arrival node observes a new data source, this information is hashed to decide which ingest partition to send the data to; each ingest partition has it’s own Kafka partition used to deliver the log entry to an ingest node.

In larger setups, we recommend having a designated ‘arrival node’, which is neither resposible for ingest processing nor search, only arrival processing.

Arrival nodes are also responsible for coordinating, gathering, and merging the final result of queries that span multiple nodes.

Scaling Out Data Ingest

The next link in the chain is an ingest node which reads data off Kafka, as it is sent there from an arrival node. An ingest node is responsible for two things: assembing segment files, and processing real-time queries.

You can control the assignment of ingest-partitions to nodes using the ingest-partitions HTTP api. To get the current ingest partition assignment you can run something like this on any one of the nodes in the cluster:

> export TOKEN=`cat humio-data/local-admin-token.txt`
> curl -H "Authorization: Bearer $TOKEN" http://localhost:8080/api/v1/clusterconfig/ingestpartitions > ingest-partitions.json

The task of an ingest node is then to receive logs on it’s designated ingest partitions and create segments out of the data, and process live queries.

The assembly of segment files involves maintaining a 1MB work-in-progress buffer, which is flushed to the segment when the buffer is full or when 10 minutes has passed. As you can imagine, this work is a lot less expensive than having to build a full free-text index.

Whenever a 1MB block is done and flushed, the ingest queue may be pruned to the index of the oldest entry in that block. The Kafka ofsset of the last event to go into a daa block is simply stored in the block. In case of a crash-restart, the ingest node will re-read enqueued log entries up from the last indes safely stored in the last block of a incomplete segment.

Once an entire segment is full (which means that it has a configured number of blocks, defaulting to 1000), the segment is passed on to a storage node. Storage nodes are responsible for processing interactive searches of ‘old’ data, whereas ingest nodes participate in interactive search of data segments under assembly.

In addition to building segments, ingest nodes also deal with real-time queries. This involves pushing all events through the pipelines corresponding to current real-time queries.

Since a real-time query may involve data being processed on a number of nodes (it may involve multiple data sources that end up being ingested on distinct nodes), the ingest nodes keep just the part of the query that relates to the data arriving on that node. These partial results are sent to the coordinating arrival node on a periodic basis, where they are merged to form a full result. The coordinating arrival node is typically the node that received the HTTP request to initiate the real-time query, but may be some other node in case of e.g. alerts.

Scaling Out Search

The last element of scaling out is search. This is both ‘historic’ searches, and the loading of events needed to load data for the initial time window of real-time search.

As described above — when a segment file is “full” it is copied to one or more storage nodes, from which it will also eventually be searched. Thus, the more storage nodes, the faster search.

To place a segment, it is hashed and using the storage-partitioning scheme, placed on one or more nodes. The current allocation of storage partitions can be queried using the segment-partitioning HTTP API:

> export TOKEN=`cat humio-data/local-admin-token.txt`
> curl -H "Authorization: Bearer $TOKEN" "http://localhost:8080/api/v1/clusterconfig/segments/partitions" > segments-partitions.json
> cat segments-partitions.json | json_pp
[{ "hosts" : [1,2], "partition" : 0 },
 { "hosts" : [2,3], "partition" : 1 },
 { "hosts" : [3,1], "partition" : 2 }]

The current placement of segments, the relevant time interval for data in a segment, as well as the data sources those segments belong to are known by all nodes in the system.

When an interactive search is received by an ‘API node’, it determines which segments are relevant for the search, and select which hosts should be used for this search. This is chosen so that the search is spread across as many hosts as possible so as to increase the processing power.

As a consequence of this, you will observe that large queries run faster in terms of GB/sec, because they end up being assigned to more hosts/CPUs, and thus exhibiting more parallelism in the search. We only parallelize based on whole segments, so if you’re only searching through ~1GB of data that happens entirely sequential. But that’s also something a single core can do in a few hundred milliseconds, so it’s not an issue.

In this blog post you can read about how we make search processing the individual segments fast.

How Humio Does Scale-Out Clustering was originally published in Humio on Medium, where people are continuing the conversation by highlighting and responding to this story.

When data arrives in Humio, it comes associated with a number of tags. These are distinguished values #tag=value, which are used to guide and optimize how data is stored and searched, and as such your use and management of tags impacts both ingest speed and query performance.

You can specify tags e.g. in your filebeat configuration, when you create an endpoint to receive data on, tags can be created based on the contents of the log line.

Ingest & Search

As data arrives at a Humio, it is internaly split into ‘data sources’. A data source is identified by the unique combination of tags. So e.g.,#type=accesslog,#host=prod01 is a seperate data source from #type=accesslog,#host=prod02.

When issuing a query that contains tags, say #type=accesslog, Humio will only search the data sources that match this tag. As such, tags are paramount for query perforance because they let you limit what data to scan through.

Each data source’s ingest is processed sequentially, on one machine in your Humio cluster. For each data source, Humio collects logs in a work-in-progress buffer of ~1MB. Whenever that WIP buffer is full, we compress it and flush to disk, appending the block to a segment file. Each data source has its own directory, containing the segment files that make up its log data.

All of this is nice and simple and contributed to making the implementaion of Humio relatively simple and efficient, which is good for many reasons.

However, there are three situations that pose challenges:

• If a data source is receiving too much data to be able to process it sequentially (typically around 10.000 events/sec). If we didn’t do anything about this, then Humio would not be able to follow the ingest load.
• If a data source is too slow, then it creates very small blocks which are not efficient for I/O purposes.
• If there are too many distinct combinations of tags, we get too many data sources, using too many 1MB WIP-buffers, and too many directories which may push the limits of the file system.

The following describe how we deal with these situations.

High Velocity Data Sources

Dealing with high-velocity data sources is fortunately relatively simple: create more. We do this by monitoring the ingest load for each data source, and if it reaches a high water mark, then we simply split it by adding a random tag to it such as#humioAutoShard=0 or #humioAutoShard=1 thereby artifically splitting the data source in two, and increasing the ingest capacity by parallelizing the ingest processing. This automatically doubles up to 128 distinct auto shards, currently limiting us to ~1M events/sec on a single data source.

You can review the currently assighed autos harding by issuing a GET to /api/v1/dataspaces/$DATASPACE/datasources/$DATASOURCE/autosharding. You can see the data source ID’s in the UI under data space settings, it’s a long UUID.

Edit: Since writing this we have implemented automatic reduction of the autosharding level as ingest velocity becomes lower. See release notes for version 1.1.7.

Low Velocity Data Sources

If data sources are slow, they also create inefficiency. Every 10 minutes we flush WIP buffers, and if this is significantly less than 1MB, then the searching and I/O buffers will operate at sub-par performance. We flush buffers periodically, because we want to avoid loosing that WIP in case of a crash, which would mean we have to read it again from the Kafka that we use to buffer ingest. So, in worst-case we have to re-read 10 minutes of ingest from Kafka on restart.

In order to deal with small block sizes we merge small segment files into larger ones periodically. This is relatively simple: just reread the events, artifically ingest them again, and once done we can discard the old segment files.

High Variability Tags

Another problem is caused by having too many data sources, which happens if some of the tags have high variability. Say, you have chosen to make the IP address of the source host a tag, such as #ip=123.456.78.90 . This can easily overload Humio with too many WIP buffers (memory hungry), too many directories (limitations in the operating system) and put an uncomfortable load on the servers. For these reason, we currently reject ingest that would create more than 10.000 data sources per data space.

But there is also a remedy for this, you can set up rules to take certain tags and hash them into a lower variability value domain. We call this tag grouping as documented here.

What tag grouping does is that it takes the hash value of the incoming tags modulus a size that you decide. Say you want the #ip tag to be bucketed into just 15 groups, you would specify {“field”:”ip”, ”modulus”: 15} in your grouping rule. Given such a rule, the #ip tag would only contribute to 15 distinct data sources (but other tags may still make this number grow).

The value of your tag would then be stored in each individual event rather than just in the data source configuration itself, thus increasing storage requirements a bit, but reducing resource consumption on WIP buffers and filesystem load. When a search happens, say #ip=123.456.78.90, you would be scanning through all the data for which the hash of the tag #ip has the same value, selecting just those where the value actually matches. The user making the query doesn’t see the difference, except that the query optimizer now cannot be smart with searches like #ip=123.456.* because we cannot say which hash values would be produced by values that match that. So if you issue such a query, all data with an #ip tag would be searched.

Edit: As of release 1.1.7, we have implemented automatic enabling of tag groups for tags where Humio observes more than 100 distinct values, they will be set to 32 groups.

I hope this shed some light on how we work hard to make high performance ingest and search work for you.

Understanding Humio’s Data Sources was originally published in Humio on Medium, where people are continuing the conversation by highlighting and responding to this story.

A typical SSD lets you read on the order of 1GB/s, which means that you can copy the file contents from disk into memory at that speed.

Next, you will then need to scan through that 1GB of memory using some string search algorithm.

If you try to run a plain string search (memmem) on 1GB, you realize that it also comes at a cost. A decent implementation of memmem will do ~10GB/s, so it adds another 1/10th of a second to your result to search through 1GB of data. Total time: 1.1 second (or 0.9GB/s).

Now, what if we compress the input first?

Imagine for simplicity that the input compresses 10x using lz4 to 0.1GB (on most workloads we see 5–10x compression). It takes just 0.1 second to read in 0.1GB at 1GB/s from disk into main memory. lz4 decompresses at ~2GB/s on a stock Intel i7, or 0.5 second for 1GB. Add search time of 0.1 second to a total of 0.6s for reading from disk and decompressing, and we can now search through 1GB in just 0.7s (or 1.4GB/s). And all of the above is on a single machine. Who needs clusters?

Compressing the input has the obvious additional advantage that the data takes up less disk space, so you can keep more data around and/or keep it for a longer period of time. If, on the other hand, you use a search system that builds an index, then you’re likely to bloat your storage requirements by 5–10x. This is why Humio lets you store 25–100x the data of systems that use indexing.

Assuming we’re on a 4-core i7 machine, we can split the compressed data it into four units of work that are individually decompressed and searched on each core for an easy 4x speed up; 1/4th of 0.6 seconds on each core is 0.125s. This gives us a total search time of 0.225 seconds, or 4.4GB/s on a single 4-core machine.

But we can do better.

All of the above assumes that we work in main memory, which is limited by a theoretical ~50GB/s bandwith on a modern CPU, in practice we see ~25GB/s.

Once data is on the CPU’s caches it can be accessed even faster. The downside is that the caches are rather small. The level-2 cache for instance is 256kbytes. In the previous example, by the time the decompression of 1/4 of 1GB is done, the beginning of those 256MB have long been evicted from the cache.

So what if we move the data onto the level-2 cache in little compressed chunks, so that their decompression also fits in the same cache, and then search in an incremental way? Memory-accesses on the level-2 cache are ~10x faster than main memory, so this would let us speed up the decompress-and-search phase by an order of magnitude.

To achieve this, we preprocess the input by splitting the 1GB into up to 128k chunks that are individually compressed.

Adding all this up for a search of 1GB to 0.1s for read-from-disk, 0.004s main-to-core 0.1GB @ 25GB/s, and blazing 10x at 0.0125s to decompress-and-search, for a total of 0.1265 seconds reaching 7.9GB/s.

But what if the 1GB file contents is already in the operating system’s file system cache? If it was recently written, or if this is the second time around doing a similar search.The loading the file contents would be instantaneous, and the entire processing would be just 0.0265 seconds, or 37GB/s.

Loading data from disk can be done concurrently with processing data, so the loading and processing can overlap in time. Notice that we’re now again dominated by I/O (the blue bar above is wider than the other ones combined), which is why Humio searches faster the better the input compresses. If you search more than a few GBs, then processing is essentially limited by the speed at which we can load the compressed data from disk.

To enable even faster searches you simply employ multiple machines. The problem is trivially parallelizable, so to be searching at 100GB/s would just need 3 machines the likes of a desktop i7.

The beauty is that this generalizes not just to search, but many other data processing problems which can be expressed in Humio’s query language. Whatever processing is presented the entire input; which makes it easy to extract data and do aggregations such as averages, percentiles, count distinct, etc.

But in the Real World…

Many interesting aggregate computations require non-trivial state (probabilistic percentiles need a sample pool, the hyper-log-log we use for count distinct needs some fancy bitmaps), and these ruin the on-CPU caching somewhat, thereby reducing the performance. Even something as simple as keeping the most recent 200 entries around slows down things.

In all honesty, most of the above is more or less wishful thinking. It’s the theoretical limits of an optimal program. For several reasons, we really only get around 6GB/s* or 1/6th of the theoretical speed, not ~37GB/s per node that I tallied up above. Trouble is that our system does many other things that end up influencing the outcome, and it is really hard to measure exactly where the problem is at the appropriate level of detail without influencing the outcome. But performance is still decent — and (unfortunately) our customers are asking for more features, not more performance, at present. *Edit: performance has since been improved to ~12GB/sec on a 4core i7, or twice the speed — compared to when this blog post was originally written.

The system really lends itself to a data processing problem where lots of data is ingested but queries are relatively rare. So it’s a good match for a logging tool: logs arrive continually, they are relatively fast to compress, and few people such as sysops and developers initiate queries. Humio easily sustains a large volume of ingest, we have seen successful single-node deployments taking in +1TB/day; when someone comes around to ask a question, it will use all available processing power (for a short while) for just that single query.

In a later post, I’ll get back to how we improve these tradeoffs using stream processing to maintain ‘views’ that are readily available for retrieval.

How Fast Can You Grep? was originally published in Humio on Medium, where people are continuing the conversation by highlighting and responding to this story.

Here Comes Alerts!

Probably our most requested feature is now here. With alerts you can be notified immediately whenever something goes wrong in your system. Alerts currently support being sent with Slack, Email, and generic Web Hook notifiers. Alerts is a premium feature not available on free accounts.

In order to create an alert, you just write a humio query. Whenever that query provides a result, the query triggers. To enable alerts based on a threshold level such as free disk space or memory usage, you simply embed the threshold in the query. Alerts can be throttled, so they will only trigger once every so often; and they can be used to send periodic reports.

Dashboards are Better Now

Dashboards have been improved in a number of ways. Most importantly, they’re faster and consume less CPU in the browser. A dashboard on a hidden browser tab will only update minimally to keep the live queries from timing out on the server.

If it has been a while since you worked with dashboards, you will also notice that we now support a free-form layout of dashboards letting you drag to move and resize widgets.

We also added a nifty pulsing ‘alive’ indicator to dashboards that tells you that the browser’s connection to the server is doing well, and makes sure to reload dashboards upon changes or server upgrades.

On premium and on-premise accounts, users can share a read-only link to dashboards which makes it accessible without a humio login.

Better Search Experience

On the search tab, we’ve improved the way auto-suggest works (it activates on ⎇+space) and it is less in-your-face when you don’t need it. Use arrow keys up/down to access available tags, functions and saved queries.

We also added a feature for use with aggregate queries (such as count, groupby or timechart) to see the events that make up the aggregation. Clicking the Events list button takes you to a view that provides the events that come out of the filter part of the query that makes up the shown result.

We’ve also improved the graphing options for time charts with various display modes, log scale graphs, step-line or connected graphs, labels, and more. Click the little gear icon in the corner of the chart to access these options.

New On-Premise Features

For our on-premise customers, there is a long list of new features that might be interesting

Distributed Version

We now support running Humio clustered, providing a solution that lets you scale up both ingest capacity and query speed with the number of nodes. Our own stress tests indicate that ingest capacity is 1TB/day per node, and query performance is 6GB/sec per node. Your milage may vary, we run on stock 4-core i7 nodes with 64GB of RAM. You can read more about how to set it up in the online documentation.

TCP/UDP ingest

We’ve added raw TCP and UDP based ingest, which makes it easy to integrate directly with e.g. syslog or other custom logging providers. At present this feature is only available for on-premise customers. Use the HTTP API to activate this feature as described here.

Improved LDAP authentication

Finally we have improved the authentication options for integrating with an ldap server, so that you’re not limited to anonymous bind. You still need to manage role assignment in the humio settings tab for each dataspace.

Humio Product Update was originally published in Humio on Medium, where people are continuing the conversation by highlighting and responding to this story.

With modern multi-host software systems, we are missing this feeling for normal (or lack thereof) coming from subconscious sensing — we are missing a way to touch our systems. Modern software runs in datacenters that are far away and disconnected from the developers who are creating applications. We have lost that connection.

Metrics dashboards provide visibility, and it is a great place to start, but they are limited as they only provide a view. They provide means to know that something is wrong, but usually not what is wrong. And dashboards are only really good for known problems, new unknown issues are harder. To do that, you need to interact with your system — touch it in real-time and do ad-hoc analytics to see what is going on. Logs are a great place to start because they are already there but in many cases it is just too difficult to get useful information quickly out of large volumes of logs.

Touch your logs

We learned the value of these matters being part of building a large highly available system for the Danish health care system. And we have realized that a lot can be gained by enabling developers and operations teams to easily interact with the logs. Enabling people to explore, iterate and understand their logs — and thereby applications, datacenters, and devices — makes a huge difference. We built Humio from a wish list of things we wanted to be able to do with logs:

Observe and Monitor

To begin, we want to make it simple to look at logs — and just see how many there are? Then begin to ask questions and dig deeper — search for errors or filter by certain parameters. This provides a super low friction way to get a feel for the health of a system because the logs are already there. Real-time live queries and monitoring are extremely critical and we want to be able keep an eye on a particular area of our system in an up-to the second view.

Understand and Debug

Most of you have used tail -f and grep, but when you have a lot of logs, it is difficult to search and find out what is happening quickly. We wanted to be able to have an easy way to get through massive amounts of logs and be able to troubleshoot with a easy to use query language.

Make it possible to have lots of logs

We have experienced log systems with a strict quota on logs, with organization policies prohibiting developers to log, and in turn logging systems breaking down when you need them the most — in the middle of some system capacity situation. Text logs by nature are easy to compress and store, and going far back in time to understand history is extremely useful. So, we wanted to build a system that would make it cost-effective to keep lots of log around for future reference, and at the same time be able to absorb large spikes in ingest.

Design Goals

When developing Humio, we wanted to create a product that does things differently. The user experience and performance of the product goes hand in hand to deliver a very different feel.

Interactive Experience

We wanted interactions with the logs to feel live, it should feel interactive like Excel and you should be able to play with and explore your data. You should be able to try different ad-hoc queries and then quickly be able to view them in a graph, timechart, or in a dashboard.

Real-time

We wanted to be able to do stream processing as data comes in, not just looking at old data. Avoid heavy up-front indexing as that will leave users with a sytem where they can only look into the past.

Schema on Read

How many times have you wished you had indexed or saved something in your database after it is too late. Humio is text based you can use Ad-Hoc Schema on Read and you can just go back in time and ask questions when they come up without worry.

Keep it simple

The last thing you want is that the logging systems ends up being a major headache. So, we stretched ourselves to make a system that is simple and robust.

You can touch your logs too!

We have a Free Trial, both On-Prem and Hosted. Sign up and let us know what you think.

Humio — Logging for Systems That You Care About was originally published in Humio on Medium, where people are continuing the conversation by highlighting and responding to this story.