Types of Anomalies in Data (Part 4): Binary/Categorical-Based Detection
1. Introduction
What is Binary/Boolean Data?
Binary (boolean) data takes on just two values: typically 0/1 or True/False. In operational systems, this could represent:
- Device status: On/Off
- Event outcome: Success/Failure
- User activity: Active/Inactive
- Flag presence: Error/No error
Why is Anomaly Detection Important Here?
Sudden changes in these states — especially in aggregate — can signal underlying issues:
- A spike in device “off” states could indicate outages.
- An unusual drop in “success” events might reveal a system bug.
- An uptick in “error” flags could point to network or configuration problems.
To make sense of all these on/off signals and spot problems as they arise, we first need to turn the raw binary data into something we can actually analyze.
2. Data Transformations: Rate/Proportion Monitoring
Let’s assume we have a time series of binary device status data, where 1 means “on” and 0 means “off.” Visually, it’s clear that during periods 125–150, there are significantly more “off” statuses, which may indicate anomalous behavior. But how can we detect such events quantitatively?
Get Zhe Sun, Phd’s stories in your inbox
Join Medium for free to get updates from this writer.
To answer this, let’s explore systematic methods for anomaly detection in binary time series data.
Binary event data (such as device status) can be transformed from individual 1/0 values into a proportion or rate of “True” events over time. This aggregation converts many raw binary events into a meaningful time series, where sudden changes in the overall proportion can indicate underlying anomalies.
Methodology:
- Aggregate Binary Data:
For each time window (e.g., minute, hour, or day), calculate the proportion of “True” (1) events.
Example: Out of 10,000 devices, if 9,900 are “on,” the on-rate for that period is 99%. - Establish Baseline and Thresholds:
Use historical data to determine the typical proportion (baseline) and set thresholds for what constitutes an anomaly.
This can be a fixed limit (e.g., <98%) or statistically derived (e.g., mean ± 3 standard deviations).
Statistical tools for thresholds:
The most challenging and important aspect is Step 2: setting appropriate thresholds. Once the binary data is aggregated into rates or counts for each period, the problem essentially becomes a standard numerical time series anomaly detection task.
- Control charts (Shewhart, CUSUM, EWMA)
- Z-score or moving average filters
- Proportion tests (binomial, chi-square for group comparisons)
