Scanner

How It Works

Scanner Organizes Your Data for Fast Search

Scanner is a tool that turns raw log data in S3 into an easily searchable resource. By organizing your logs and indexing them in a highly optimized way, Scanner makes it fast to find what you need, whether it's a critical security event or insights hidden deep in your logs.

Here's how it works.

Your Data, Optimized for Search

Scanner creates a dedicated AWS account for each customer. This AWS account runs Scanner's indexers and queriers. The indexers organize and make the data searchable, and the queriers perform search and analysis.

To connect with Scanner, customers set up AWS resources, either with infrastructure-as-code tools like CloudWatch, Terraform, or Pulumi, or through a manual setup walk through. Specifically, customers create a new S3 bucket to store Scanner's index files, and an IAM role that allows Scanner to read raw log data and organize it into the Scanner index files bucket.

Customers maintain custody of the data, so there is no vendor lock-in.

Once the setup is complete, Scanner starts monitoring the raw log data in your S3 bucket. When new log data files are uploaded, Scanner is notified via via SNS messages, which enables the indexers to begin their work immediately. The Scanner indexers are optimized to run in the same AWS region as your S3 buckets, which means that there are no S3 data transfer costs while minimizing latency.

Fast, Flexible Indexing

Scanner's indexers transform your raw log data into a custom format designed for high compression and rapid parsing, with a compression rate of 85% on average.

The indexers also generate text and numerical indexes, along with other files that store statistics used for autocomplete and query planning.

These indexes allow for lightning-fast, full-text searches without the need to decide upfront which fields should be indexed—all fields are indexed and searchable by default. Over time, the indexers merge smaller index files into larger ones, which further optimizes query speed.

The metadata about these indexes is kept in a SQL database that helps determine the optimal way to execute a query, considering the query's time range and selected indexes. This ensures search is both fast and efficient.

High-Speed Querying

When you submit a search query, Scanner's API server consults the metadata database to identify the relevant index files. Scanner's queriers, powered by AWS Lambda, analyze the required indexes rapidly using S3 byte-range reads. By leveraging SIMD instructions, Scanner achieves high-speed text matching, which makes it easy to search massive volumes of logs.

The combination of Scanner's indexes and queriers reduces the search space dramatically, making it possible to search petabytes of log data in just seconds. For example, a search for an IP address in a petabyte of log data can be completed in tens of seconds—a feat that traditional tools like Amazon Athena could take hours, even up to 12+ hours, to accomplish.

Scanner's queries can return raw log matches or aggregated statistics, and the results are forwarded to the user interface for easy access.

Real-Time Detection Rules

Beyond search, Scanner supports powerful detection rules to help security teams stay on top of potential threats. As new log data comes in, Scanner's indexers apply detection rule queries, aggregating the results into a rollup tree structure stored in S3.

This rollup is continuously analyzed to determine if a detection rule has been triggered. Alerts are then sent to Slack or Webhook endpoints, ensuring you stay informed in real-time.

Built for Security Insights

Scanner is designed for flexibility and speed. It allows for full-text searches on even the most complex and deeply nested JSON log data, ensuring you can find the proverbial "needle in a haystack" when it matters most.

Whether you're looking for specific indicators of compromise or running detection rules across petabytes of logs, Scanner provides the performance and flexibility your security operations need.

Experience Scanner Today