Grepr Blog

Grepr is a tool that sits between log shippers and aggregation backends to intelligently manage and reduce log data volume using machine learning. It analyzes the semantics of each incoming log message in real time, parsing them into a structured format with fields like ID, timestamps, tags, severity, and message content. By automatically masking frequently changing values such as numbers and IP addresses, Grepr improves its ability to recognize patterns and clusters similar messages together using advanced similarity metrics. When a pattern becomes repetitive, it applies a logarithmic sampling algorithm to limit the number of forwarded messages, significantly reducing unnecessary data flow. At regular intervals, Grepr generates concise summaries for each pattern, providing details like a unique pattern ID, a link to raw logs, and a count of aggregated messages. A built-in rules engine allows users to fine-tune which messages are always passed through, ensuring important data is never lost. While only select messages are sent to the aggregation backend, all raw logs are saved in low-cost storage and can be easily queried or backfilled during incidents, giving teams access to comprehensive data without the cost of storing everything in the main backend.

Grepr’s Dynamic Backfill feature lets teams retain all log data at low cost while only sending essential logs to their main logging backend, cutting processing and storage costs by around 90%. Unlike traditional logging that risks missing key data before an incident is detected, Grepr stores everything in affordable storage and allows engineers to selectively backfill detailed logs when issues arise—like turning up log detail after the fact. This ensures engineers have full context for debugging, with no disruption to existing workflows, balancing deep visibility with major cost savings.

Grepr helps companies lower observability costs while keeping engineers' workflows unchanged. A key feature is its ability to ensure full logs for a sampled set of traces, even with log aggregation and sampling in place. Users can specify how much of their data to sample and provide trace ID paths to identify relevant logs. Grepr's system uses intelligent sampling, backfilling, and a rule engine to collect and process trace logs efficiently. To optimize performance and memory, it evolved from a basic set-based approach to a scalable design using Bloom filters and a custom resizing Bloom filter. This allows Grepr to maintain high throughput while managing memory use effectively, ensuring reliable trace log visibility.

Discover how Grepr's intelligent log management solution can reduce your logging costs by 90% without sacrificing visibility. Our two-tier storage system uses machine learning to identify patterns and store less critical logs in low-cost storage, while maintaining immediate access to important data. When incidents occur, Grepr's dynamic backfill feature automatically retrieves relevant logs to your existing tools. Implement smarter logging today without changing your workflows or compromising on troubleshooting capabilities.

Storing logs long-term doesn't have to be super expensive. Using a data lake can reduce storage costs by more than 90% while still keeping the logs queryable and immediately accessible.

This post describes advanced techniques to reduce log volume: automatic sampling by pattern, logarithmic sampling by pattern, and sampling with automatic backfilling.

This blog discusses 6 ways that the Grepr Data Lake is optimized for logs.

This blog reviews basic techniques to reduce logging volumes. These are available in most log aggregation systems.

Announcing our raise from Andreessen Horowitz and boldstart ventures to tackle the exponentially growing cost of observability without forcing migrations. Grepr combines machine learning with an observability data lake to reduce costs by 90% with minimal effort.