FSQL Technical Reference
About Query's Search Capabilities
Query’s search contract is the interface into its query engine and Query's federated search capabilities. It is the combination of the data model and filters allowed on that data.
Query views data through an OCSF lens: all of your data, regardless of source, are represented as OCSF events. Query Federated Search provides a view of these events narrowed based on your search criteria and selected connectors.
When you build a search request for Query, you select events and, optionally, attributes from the data model to be included in results. Filters can be specified for each desired event type to narrow the result set, and connections can be specified. Query requires that a time range be specified to avoid large scanning queries to data sources, and all events have a time property.
Filters describe characteristics of records to be included in the result set.
The most basic form of filters – predicates – describe an attribute (e.g. dns_activity.src_endpoint.ip), an operator (=, contains, etc.), and a value for comparison. The allowed filter operators for an attribute are determined by its data type, which is defined in the QDM.
Filters on lists or arrays – more formally known as quantifiers – can be thought of as a predicate that must be true for ANY or ALL elements in the list. We believe that ANY is the right default choice but support ALL for all lists, with some connector-specific exceptions. There are roughly 16,000 lists in OCSF 1.3, so this comes up more often than you might think.
The search contract also allows grouping filters with AND (all filters must match) or OR (at least one filter must match). It also supports negating filters, so that entire groups can be negated. This is different from using negated forms of operators, but can be used to the same effect.
Attribute Selectors
Attribute selectors are how attributes (fields) are described in FSQL. They are heavily influenced by selectors in CSS and XPath, and are novel in the world of SIEMs and OCSF. An FSQL attribute selector may describe one or many attributes in the schema.
As a refresher, attribute paths always have the form event[.attribute[.attribute[...]]]. Examples include authentication.user.email_addr and dns_activity.src_endpoint.ip.
FSQL attribute selectors use a dot (.) to separate parts of the expression, letting simple direct paths read just like the dot notation names commonly found in other interfaces. A few other rules: FSQL attribute selector commands begin with a colon (:), and strings with spaces must be surrounded by single quotes (’). Command parameters are listed inside of parentheses, and empty parentheses are optional when parameters are not used. Commands and command aliases (like *) do not require a dot in front of them. Command names are case insensitive.
Direct Paths
The simplest paths go directly through the schema to a single attribute by using event and attribute names – the internal OCSF/QDM name with underscores instead of space – separated by dots.
FSQL also allows captions – the label we show to humans – to be used in place of machine friendly names. Captions must be surrounded by single quotes (’). Captions and names can be mixed. Example: authentication.user.’email address’.
Path Expansions
Path expansions add attributes to the selection. Names and captions are the simplest form of expansions, but a path expression can describe more than one attribute in the schema. Other expansions describe multiple paths through the schema.
| Expansion | Aliases | Description | Examples |
|---|---|---|---|
:all | * | Expands to everything at the current path level. If no events have been specified in a path, the :all command will expand to all events. Otherwise, it will expand to all attributes of the selected events and any attributes specified so far. | *.time *.* authentication.actor.* network_activity.*.ip |
** | Like :all, but always expands all paths to non-recursive ends. | ** #network.** | |
:category | # | Expands to all events in a given category. | :category(network) #discovery #network.*.ip #findings.evidence |
Expansion Modifiers
The behavior of the :all expansion can be altered with modifiers. Modifiers change the behavior of all expansions following them in the path. The short form aliases of expansion modifiers can also be substituted for the dot operator for more concise syntax.
The :!recursive modifier is enabled by default.
| Modifier | Aliases | Description | Examples |
|---|---|---|---|
:deep | // | Causes the :all expansion to expand to include all attributes until either the maximum depth or a recursive relationship is reached. | :deep :deep.#network.* //#network.* #network//* |
:shallow | / | The opposite of :deep, the :shallow modifier restricts expansions to just one level of the path. | |
:recursive | + | Causes expansions to follow recursive relationships. A depth limit is required. To disable recursion, use :!recursive. | |
:depth | Sets a maximum depth limit. This is required when traversing recursive relationships. | :depth(4) |
Path Filters
Filters narrow the list of selected paths in an attribute selector expression..
All filters can be negated with an exclamation point. For example, :!array will select all paths that are not arrays and is the logical equivalent of :scalar.
Be advised that the short form aliases will be preceded by :all with the :deep modifier temporarily enabled. This means searches like @ip_t will match every IP field in the schema.
| Filter | Aliases | Description | Examples |
|---|---|---|---|
:type | @ | Removes all paths that do not end in an attribute of type. The type may be a primitive type or an object type, and the trailing _t may be omitted for primitive types. | *.*.*.:type(ip_t) @ip_t @ip |
:observable | %, :entity | Removes all paths that do not end in a given observable. The observable can be specified by its caption (in quotes) or its type_id. There are also several aliases like ip that match the Query Splunk App. | //*.*.:observable(2) #network//*.:observable(‘IP’) %ip |
:primitive | Removes all paths that do not end in an attribute of a primitive data type. | ||
:object | Removes all paths that do not end in an attribute of an object data type. | ||
:scalar | Removes all paths that are arrays. | ||
:array | Removes all paths that are scalars. | ||
:distinct | Removes all duplicate paths. | ||
:min_depth | :minDepth | Removes all paths that are not at least n levels deep. | @ip_t.:min_depth(3) |
:max_depth | :maxDepth | Removes all paths that are over n levels deep. | @ip_t.:max_depth(7) |
:deprecated | Removes all paths to deprecated attributes. | ||
:enum | Removes all paths that are not enumerations. | ||
:sibling | Removes all paths that are not sibling fields. For instance, :sibling would select activity_name but not activity_id. | ||
:group | # | Removes all paths to attributes that don’t match a specific group. | ssh_activity.*:group(primary) ssh_activity.#primary |
Transformers
Transformers modify the attribute paths in the selection.
| Filter | Aliases | Description | Examples |
|---|---|---|---|
:root | Truncates all paths to the root event. | ||
:truncate | Truncates all paths to n levels deep. This is similar to max_depth, but the number of matching paths will remain unchanged after applying truncate. | @hostname_t.:truncate(5) **.:truncate(5).:distinct | |
:parent | Replaces all paths with their parent path. Similar to .. in filesystem paths. Paths to events will not be changed. | @ip_t.:parent | |
:primitive_parent | :primitiveParent | Replaces all paths to primitive type attributes with their parent object or event. For instance, network_activity.dst_endpoint.ip will be replaced with network_activity.dst_endpoint, but authentication.actor will not be changed. |
Set Operations
Partial attribute selectors can be combined with set operations and grouped with parentheses.
Some examples:
network_activity.(* - (start_time | end_time))
(network_activity.* | dns_activity.*)
#network - dns_activity
| Set Operator | Name | Description | ||
|---|---|---|---|---|
| ` | ` | Union (or) | `A | B` selects all paths from both A and B. |
& | Intersection (and) | A & B selects all paths from A that are also in B. | ||
- | Difference (minus) | A - B selects all paths from A that are not in B. |
Search Filter Operators
Search filters describe predicates to narrow your search results.
| Operator | Description |
|---|---|
= | Field equals <value>. |
== | Like =, but case insensitive. |
!= | Field does not equal <value>. |
!== | Like !=, but case insensitive. |
~, contains | String field contains <value>. |
~~, icontains | Like ~ but case insensitive. |
^=, startswith | String field starts with <value>. |
^==, istartswith | Like ^= but case insensitive. |
$=, endswith | String field ends with <value>. |
$==, iendswith | Like $= but case insensitive. |
in | Enum field is one of <values>, where values is a comma-separated list. |
iin | Case insensitive in. |
<, >, <=, >= | Numeric field is less than, greater than, less than or equal to, or greater than or equal to <value> |
empty | Field is empty (ex: null or None) |
Search Filter Operators by Attribute Data Type
| Operator | number | timestamp | string | boolean | enum | hash | IP | JSON | arrays |
|---|---|---|---|---|---|---|---|---|---|
| Equality | |||||||||
=, == | x | x | x | x | x | x | x | ||
!=, !== | x | x | x | x | x | x | x | ||
in, iin | x | x | x | x | x | x | x | ||
| Sorting | |||||||||
< | x | x | x | ||||||
<= | x | x | x | ||||||
>= | x | x | x | ||||||
> | x | x | x | ||||||
| Emptiness | |||||||||
empty | x | x | x | x | x | x | x | x | x |
| Strings | |||||||||
^=, ^== | x | ||||||||
$=,$== | x | ||||||||
~, ~~ | x | ||||||||
| Quantifiers | |||||||||
ANY | x | ||||||||
ALL | x |
Date Units
Date units are case-insensitive.
- Hours:
h,hr,hour,hrs - Days:
d,day,days - Weeks:
w,week,weeks - Months:
m,month,months
Updated 2 days ago