Grepr Blog

This blog post shows how to reduce log volume by up to 90% by integrating New Relic with Grepr. Using a simple Docker-based microservices demo, we walk through configuring Fluent Bit to ship logs to New Relic, then show how easily Grepr can be inserted into the pipeline to intelligently filter out noise. The result is cleaner, more actionable log data, reduced observability spend, and no disruption to existing workflows. All raw data is retained in low-cost storage and can be backfilled on demand—helping teams stay in control of both their visibility and their budget.

Goldsky, a Web3 realtime data platform, partnered with Grepr to significantly reduce their log management costs while maintaining observability performance. Initially facing misalignment between logging spend and value, Goldsky deployed Grepr, led by Lead Engineer Paymahn Moghadasian, who quickly integrated it using Terraform and Datadog's dual-shipping feature. Over four weeks, they successfully filtered noisy logs and transitioned their production environment with zero disruption. The result: a 96% reduction in indexed logs, 93% less data ingested, and over 85% savings in Datadog costs—without any negative impact on Mean Time to Resolution (MTTR). Additional benefits included improved log readability, faster searches without rehydration, and white-glove support from Grepr.

Grepr and Cribl both offer data pipelines for observability, but they differ in complexity and approach. Cribl is a powerful, flexible platform requiring significant setup, ongoing management, and learning its custom query language. Grepr is the newer, simpler option, using AI to automate data filtering and reduce manual configuration by 90%. While Cribl offers more integrations, Grepr supports common sources, uses familiar query languages, and enables faster, lower-maintenance deployment. Cribl suits large enterprises with dedicated teams, while Grepr is ideal for organizations seeking a faster, more automated solution.

Grepr reduces observability costs by storing all data in low-cost storage and using machine learning to forward only unique or summarized insights to platforms like DataDog, Splunk, or New Relic. Engineers can query retained data, generate reports, power AI, or trigger dynamic backfill during incidents—automatically via webhooks or manually through the Grepr interface. To learn more or request a demo, visit grepr.ai.

Grepr is an intelligent observability pipeline that sits between your agents and observability platform to optimize, analyze, and route data in real time. By using machine learning and a rules engine, Grepr detects patterns in data streams, holds back noisy or repetitive information, and forwards only essential summaries or unique messages. With a simple configuration change, it integrates seamlessly into existing systems—helping teams cut observability costs by up to 90%, retain data long-term, and make observability insights available for business reporting and AI.

Everything we do at Grepr is around making sure we reduce costs with minimal impact to existing workflows. Grepr can automatically parse existing alerts in Datadog (Splunk and New Relic coming up in the next few weeks) and avoid modifying logs that power them. This way, you can roll out Grepr to prod without worrying about having to rewrite all your alerts.

In this video we highlight Grepr's ability to work with Splunk. We have Grepr receiving data from Splunk Heavy Forwarders using S2S. We configure Splunk to reduce the data and forward it to Splunk. Grepr massively compresses the logs passing through, but the logs are still in the Grepr data lake. They can be queried using SPL, and sent back to Splunk with a manual backfill if needed. You can also see this compressed data stream in Splunk, and if you want to see the raw data that corresponds to a summary message, you can use the embedded link in summary messages to quickly get to it.

Grepr uses machine learning to reduce log volume without losing visibility. It parses and structures logs in real time, groups similar messages, and applies smart sampling to cut noise. Critical logs still get through, and full raw data is stored separately for easy access during incidents—keeping your backend lean and your team in control.

Grepr’s Dynamic Backfill feature lets teams retain all log data at low cost while only sending essential logs to their main logging backend, cutting processing and storage costs by around 90%. Unlike traditional logging that risks missing key data before an incident is detected, Grepr stores everything in affordable storage and allows engineers to selectively backfill detailed logs when issues arise—like turning up log detail after the fact. This ensures engineers have full context for debugging, with no disruption to existing workflows, balancing deep visibility with major cost savings.

Grepr helps companies lower observability costs while keeping engineers' workflows unchanged. A key feature is its ability to ensure full logs for a sampled set of traces, even with log aggregation and sampling in place. Users can specify how much of their data to sample and provide trace ID paths to identify relevant logs. Grepr's system uses intelligent sampling, backfilling, and a rule engine to collect and process trace logs efficiently. To optimize performance and memory, it evolved from a basic set-based approach to a scalable design using Bloom filters and a custom resizing Bloom filter. This allows Grepr to maintain high throughput while managing memory use effectively, ensuring reliable trace log visibility.

Discover how Grepr's intelligent log management solution can reduce your logging costs by 90% without sacrificing visibility. Our two-tier storage system uses machine learning to identify patterns and store less critical logs in low-cost storage, while maintaining immediate access to important data. When incidents occur, Grepr's dynamic backfill feature automatically retrieves relevant logs to your existing tools. Implement smarter logging today without changing your workflows or compromising on troubleshooting capabilities.

Storing logs long-term doesn't have to be super expensive. Using a data lake can reduce storage costs by more than 90% while still keeping the logs queryable and immediately accessible.