Destinations

Overview

In the context of Query Security Data Pipelines, Destinations are any object storage, data lake, data warehouse, SIEM, or log aggregation tool that Query supports sending data to.

Query writes data the right way and abides by best practices for writing to these sources, typically with the mind of using the data for second-order tasks beyond archival and storage. We write data to performantly support use cases such as analytics, AI, BI, machine learning, or ad-hoc search using your own query engine (or Query Federated Search Connectors).

These best-practices include using search-optimized compression codecs, using Apache Parquet (where possible), partitioning the files as well as setting up Hive-like partitions in object storage, compacting Parquet, using GZIP-compression over the wire, and monitoring request limits of downstream sources, and many other optimizations per-platform such as exponential backoff, minimum necessary permissions, multi-threaded writes, and otherwise.