Ingest Optimization Guide

If your daily data ingest volume is consistently high, the most common causes are configuration issues, many of which can be resolved without reducing your security coverage.

This guide provides tuning recommendations in priority order. Where a recommendation involves a deliberate coverage trade-off, that trade-off is called out explicitly so you can make an informed decision before applying it.

Before applying any recommendation, check your current usage in Platform Monitoring | Log Ingest. Your daily allocation and a per-log-type breakdown are shown there and will help you identify which recommendations will have the most impact.

The recommendations below can be applied in combination. If applying only a subset, the earlier recommendations tend to have the greatest impact.

1. Verify packet broker configuration

The number one culprit for above-average log volume is a misconfigured packet broker. Correct configuration avoids common issues including:

• Duplicate logs: The same packets delivered to the sensor multiple times, inflating all downstream log volumes.

• High volume of one-way connections: Asymmetric traffic delivery that generates thousands of unproductive connection entries in conn.log with zero application-layer data.

Refer to your packet broker documentation for validated deployment patterns.

2. Filter logs that don’t support your use case

Corelight sensors generate a wide range of log types, from core network transaction logs that power detections, to internal diagnostic telemetry that exists solely for sensor health monitoring. By default, the Dynamic Exporter may forward log types to Investigator that consume ingest allocation without contributing to security outcomes. These logs do not contribute to Investigator detections or investigations and can be safely excluded from Investigator ingestion.

Review the categories below and exclude any log types that do not support your operational use cases. Exclusions are applied via the Dynamic Exporter configuration and take effect immediately.

Sensor operational and diagnostic logs

These logs report on internal sensor state — parser performance, cluster coordination, packet capture health, and configuration metadata. They are valuable for sensor troubleshooting but provide no security detection value in Investigator.

broker · capture_loss · cluster · config · corelight_burst-log · corelight_license_capacity · corelight_metrics_zeek_doctor · corelight_overall_capture_loss · corelight_pipeline_test · corelight_profiling · dpd · loaded_scripts · local_subnets · local_subnets_dj · local_subnets_neighbor · logschema · netcontrol · netcontrol_drop · netcontrol_shunt · openflow · packet_filter · print · reporter · rfb · stats · suricata_stats · suricata_zeek_stats · unknown_smartpcap · weird

Enrichment summary logs

These Corelight-generated logs aggregate or reformat data already captured in primary transaction logs (such as conn, ssh, http, and ntlm). The underlying security telemetry remains available through the source logs; these summaries do not drive additional detections in Investigator.

corelight_ad_admin_shares · corelight_admin_shares · corelight_ew_servers_services · corelight_exe_files · corelight_ftp_connections · corelight_http_user_agents · corelight_ns_servers_services · corelight_ntlm_auth · corelight_rdp · corelight_ssh_clients · corelight_ssh_connections · dhcp_fp

Unsupported reduced and aggregated formats

Investigator’s data pipelines do not process reduced (_red) or aggregated (_agg) log formats. Exporting them consumes your ingest allocation but the data is neither queryable nor available for detections.

conn_red · dns_red · files_red · http_red · ssl_red · weird_red · x509_red · all *_agg variants

3. Remove redundant tap locations

Corelight sensors can be deployed at multiple points within an enterprise network. However, when multiple sensors observe the same session as it traverses different network segments, each sensor logs that session independently. Investigator does not deduplicate these redundant connection and transaction logs, which can lead to significantly inflated ingest rates.

To address this:

• Audit your sensor placement to identify locations where the same traffic is visible to more than one sensor.

• Prioritize maximum network coverage with minimum overlap. The goal is to see every session once, not to see the same session from multiple vantage points.

• If overlapping coverage is architecturally necessary, consider applying BPF filters or exporter-level log exclusions on the redundant sensors to limit what they export to Investigator.

4. Exclude known scanners and heavy infrastructure traffic with BPF filters

Berkeley Packet Filters (BPF) are the most efficient filtering mechanism available on Corelight sensors. Because BPF filtering is applied at the packet capture level, before the Zeek parser processes the data stream, it preserves sensor CPU cycles in addition to reducing log volume.

Use BPF exclusions for:

• Vulnerability assessment scanners: A single scan can overwhelm a sensor and saturate the export channel. Because this traffic originates from known, trusted internal assets and is highly repetitive, capturing it creates noise that obscures actual post-breach behavior.

• Trusted high-volume database and backup traffic: Internal database replication (e.g., MySQL, PostgreSQL) and storage backup paths represent massive data flows. When these communication paths are statically defined and well-secured, continuous logging produces minimal detection value relative to the log volume generated.

• Known-good monitoring and administrative systems: Internal health checks, SNMP polling, and other infrastructure management traffic that generates predictable, high-volume patterns.

Configuration

• Fleet Manager: Navigate to Policies, select the active policy, open the Localize tab, and update the Traffic exclusion filter field.

• Standalone Sensor: Use corelight-client configuration update --zeek.site.ignore_bpf "<filter expression>"

Example filter expression

To exclude traffic involving a primary database server (10.100.1.50) communicating with a backup replica (10.200.1.50) over port 3306, as well as a local vulnerability scanner (10.10.10.99):

not ( (host 10.100.1.50 and host 10.200.1.50 and tcp port 3306) or host 10.10.10.99 )

Security trade-off

BPF exclusions create complete blindness to the excluded IP addresses or host-port combinations. Ensure robust access controls exist on any systems you exclude, and document your exclusions for periodic review.

5. Use Global Log Filter Presets

Global Log Filter Presets provide pre-built filters for common high-volume, low-value log patterns. These presets cover three areas:

• External Scans: Removes external scans from conn.log. External scans are connections where local_orig is False and history is S or Sr.

• Simple DNS: Removes simple DNS entries from conn.log. Simple DNS entries are connections where service is dns and orig_pkts is less than the simple DNS threshold (5 packets by default).

• Reverse DNS: Removes reverse DNS entries from dns.log. Reverse DNS entries are queries where the query string ends with in-addr.arpa.

These presets reduce log volume without requiring custom filter development and can be enabled directly in your sensor or Fleet Manager policy.

6. Eliminate internal DNS duplication

A common source of hidden log volume occurs when a sensor captures both legs of an internal DNS resolution: the client-to-internal-resolver query and the internal-resolver-to-external-resolver query for the same lookup. This doubles your DNS log volume for every external query without providing additional security value.

To resolve this:

• Identify whether your sensor placement captures traffic on both sides of your internal DNS resolvers.

• If so, filter one leg, typically the internal-resolver-to-external leg, since the client-to-resolver query preserves the attribution to the originating endpoint that matters for investigation.

• Alternatively, position sensors so they only see one side of the DNS resolution path.

Additionally, locally answered DNS PTR records and legacy NetBIOS wildcard queries (*) drive high log volumes. Automated infrastructure, application routing, and asset management tools constantly perform reverse lookups that are locally resolved and represent standard administrative behavior. Global Log Filter Presets suppress these automatically.

7. Filter outside-the-firewall taps to relevant data

Tapping traffic outside the firewall provides visibility into inbound threats before they reach your perimeter. However, this tap point is uniquely noisy because it captures all traffic destined for your public IP space, including internet background radiation, mass scanning, DDoS traffic, and connection attempts to unused ports and services.

Unlike internal taps where most traffic represents legitimate business activity, outside-the-firewall taps are dominated by unsolicited traffic that generates enormous log volumes with minimal investigative value.

To make an outside-the-firewall tap sustainable:

• Filter to relevant inbound traffic: Limit collection to traffic destined for your actual public-facing services (web servers, mail servers, VPN endpoints). Traffic hitting unused IP space or closed ports generates logs with minimal investigative value.

• Filter to relevant protocols: Focus on the protocols that matter for your security use cases (HTTP, TLS, DNS, SMTP). High-volume, low-signal traffic such as UDP floods or connectionless scanning can typically be dropped without impacting detection coverage.

If you cannot apply granular filters, consider whether the outside-the-firewall tap is providing security value that your internal sensors do not already cover.

8. Turn off or tune low-value alerts

Corelight can alert on activities such as network scanning or certificate hygiene issues, but in some environments these can generate millions of alerts per hour. Review your alert volume in Investigator and tune out the noisiest scenarios on the sensor; reducing just the top few offenders can yield significant returns.

Common candidates for tuning:

• Internal vulnerability scanner detections (already excluded via BPF if recommendation 4 is applied)

• Certificate validation failures from known legacy systems

• Repeated protocol anomalies triggered by specific infrastructure

9. Enrich logs in Investigator, not at the sensor

Log enrichment integrations (such as EDR context, asset inventory, or identity data) can be enabled either at the sensor or within Investigator. Enabling enrichment at the sensor appends the enrichment data to every individual log entry, multiplying it across millions of records per day.

Enabling the same integration in Investigator is significantly more efficient: the enrichment data is ingested and stored once as a reference table, then joined at query time when viewing Detections, entities, or individual events. The security value is identical, but the ingest cost is dramatically lower.

If you currently have enrichment integrations enabled at the sensor level, consider migrating them to Investigator instead.

10. Filter syslog traffic at the packet level

Syslog can generate many different detections based on the content of the log itself. If your sensor is positioned where it can observe syslog traffic, adding a BPF filter to prevent syslog from reaching the Zeek and Suricata parsers avoids both unnecessary log generation and wasted CPU cycles.

Example filter to exclude syslog collector traffic and sensor management interface syslog:

not (host <syslog_collector_ip> and udp port 514)

Replace <syslog_collector_ip> with your syslog collector addresses. If the sensor’s management interface also carries syslog traffic, include it in the exclusion.

11. Consider a split-logging architecture for full-fidelity retention

If your organization requires long-term forensic retention of full-fidelity logs but is constrained by Investigator ingest limits, consider a split-logging architecture:

• Stream optimized logs to Investigator: Apply tuning for the real-time detection, triage, and investigation pipeline.

• Route full-fidelity logs to low-cost storage: Export complete, unreduced raw JSON logs to inexpensive object storage (Amazon S3, GCP GCS, or local SFTP) in compressed format for retroactive threat intelligence matching, compliance archival, and long-horizon forensic investigations.

This approach allows you to maintain full network visibility for forensic purposes while keeping your Investigator ingest focused on the high-fidelity, security-relevant telemetry that powers Agentic Triage and automated detection workflows.

Quick reference: Expected impact

Recommendation	Typical volume reduction	Security trade-off
1. Packet broker configuration	Varies widely; resolves duplicate logs entirely	None: corrects misconfiguration
2. Filter logs that don’t support your use case	Varies by prior customization	None: removes zero-security-value system logs
3. Remove redundant tap locations	Proportional to overlap (2x or more per overlapping session)	None: eliminates pure duplication
4a. BPF filters: Vulnerability scanners	Up to 80% reduction during scan windows	Complete blindness to excluded hosts
4b. BPF filters: Database/backup traffic	Up to 50% of conn volume depending on environment	No detection on excluded host-port pairs
5. Global Log Filter Presets	20-30% reduction in conn and DNS logs	Omits unproductive connections and local DNS
6. Internal DNS deduplication	Up to 50% reduction in DNS log volume	None: removes pure duplication
7. Outside-the-firewall filtering	Varies; often the highest-volume single tap	Reduced visibility into internet noise
8. Alert tuning	Varies by environment	Reduced alerting on tuned scenarios
9. Enrich in Investigator	Proportional to enrichment field size x log count	None: identical security value
10. Syslog BPF filter	Varies by syslog volume	No network-level syslog visibility
11. Split-logging architecture	N/A (manages retention, not reduction)	None: full fidelity preserved in storage