VRL function reference
Here you’ll find a comprehensive list of all built-in VRL functions. Functions are categorized by their purpose and sorted alphabetically for easy discovery. To use these functions in Vector, see the documentation on function call expressions and Vector’s remap
transform.
Array functions
append
infallibleitems
array to the end of the value
array.chunks
falliblevalue
into slices of length chunk_size
bytesFunction spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | array string | The array of bytes to split. | yes | |
chunk_size | integer | The desired length of each chunk in bytes. This may be constrained by the host platform architecture. | yes |
Errors
Thechunks
function is fallible, which means that
error handling is required for these errors:chunk_size
must be at least 1 bytechunk_size
is too largepush
infallibleitem
to the end of the value
array.Codec functions
decode_base16
falliblevalue
(a Base16 string) into its original string.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The Base16 data to decode. | yes |
Errors
Thedecode_base16
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid encoded Base16 string.decode_base64
falliblevalue
(a Base64 string) into its original string.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The Base64 data to decode. | yes | |
charset | string | The character set to use when decoding the data. | standard | no |
Errors
Thedecode_base64
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid encoded Base64 string.decode_gzip
falliblevalue
(a Gzip string) into its original string.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The Gzip data to decode. | yes |
Errors
Thedecode_gzip
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid encoded Gzip string.decode_mime_q
falliblevalue
with their original string.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string with encoded-words to decode. | yes |
Errors
Thedecode_mime_q
function is fallible, which means that
error handling is required for these errors:value
has invalid encoded encoded-word string.decode_percent
infalliblevalue
like a URL.decode_zlib
falliblevalue
(a Zlib string) into its original string.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The Zlib data to decode. | yes |
Errors
Thedecode_zlib
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid encoded Zlib string.decode_zstd
falliblevalue
(a Zstandard string) into its original string.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The Zstandard data to decode. | yes |
Errors
Thedecode_zstd
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid encoded Zstd string.encode_base16
infalliblevalue
to Base16.encode_base64
infalliblevalue
to Base64.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to encode. | yes | |
padding | boolean | Whether the Base64 output is padded. | true | no |
charset | string | The character set to use when encoding the data. | standard | no |
encode_gzip
infalliblevalue
to Gzip.encode_json
infalliblevalue
to JSON.encode_key_value
falliblevalue
to in key/value format with customizable delimiters. Default delimiters match
the logfmt format.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | object | The value to convert to a string. | yes | |
fields_ordering | array | The ordering of fields to preserve. Any fields not in this list will appear unordered, after any ordered fields. | no | |
key_value_delimiter | string | The string that separates the key from the value. | = | no |
field_delimiter | string | The string that separates each key/value pair. |
| no |
flatten_boolean | boolean | Whether to encode key/value with a boolean value as a standalone key if true and nothing if false . | no |
Notices
This function has special behavior that you should be aware of.fields_ordering
is specified then the function is fallible else it is infallible.Errors
Theencode_key_value
function is fallible, which means that
error handling is required for these errors:fields_ordering
contains a non-string elementExamples
Encode with default delimiters (no ordering)
encode_key_value({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})
lvl=info msg="This is a message" ts=2021-06-05T17:20:00Z
Encode with default delimiters (fields ordering)
encode_key_value!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])
ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345
Encode with default delimiters (nested fields)
encode_key_value({"agent": {"name": "vector"}, "log": {"file": {"path": "my.log"}}, "event": "log"})
agent.name=vector event=log log.file.path=my.log
Encode with default delimiters (nested fields ordering)
encode_key_value!({"agent": {"name": "vector"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])
event=log log.file.path=my.log agent.name=vector
Encode with custom delimiters (no ordering)
encode_key_value(
{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"},
field_delimiter: ",",
key_value_delimiter: ":"
)
lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00Z
Encode with custom delimiters and flatten boolean
encode_key_value(
{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "beta": true, "dropped": false},
field_delimiter: ",",
key_value_delimiter: ":",
flatten_boolean: true
)
beta,lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00Z
encode_logfmt
falliblevalue
to logfmt.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | object | The value to convert to a logfmt string. | yes | |
fields_ordering | array | The ordering of fields to preserve. Any fields not in this list will appear unordered, after any ordered fields. | no |
Notices
This function has special behavior that you should be aware of.fields_ordering
is specified then the function is fallible else it is infallible.Errors
Theencode_logfmt
function is fallible, which means that
error handling is required for these errors:fields_ordering
contains a non-string elementExamples
Encode to logfmt (no ordering)
encode_logfmt({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})
lvl=info msg="This is a message" ts=2021-06-05T17:20:00Z
Encode to logfmt (fields ordering)
encode_logfmt!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])
ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345
encode_percent
infalliblevalue
with percent encoding to safely be used in URLs.encode_zlib
infalliblevalue
to Zlib.encode_zstd
infalliblevalue
to Zstandard.Coerce functions
to_bool
falliblevalue
into a boolean.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | boolean integer float null string | The value to convert to a Boolean. | yes |
Errors
Theto_bool
function is fallible, which means that
error handling is required for these errors:value
isn’t a supported boolean representationto_float
falliblevalue
into a float.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | integer float boolean string timestamp | The value to convert to a float. Must be convertible to a float, otherwise an error is raised. | yes |
Errors
Theto_float
function is fallible, which means that
error handling is required for these errors:value
isn’t a supported float representationto_int
falliblevalue
into an integer.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | integer float boolean string timestamp | The value to convert to an integer. | yes |
Errors
Theto_int
function is fallible, which means that
error handling is required for these errors:value
is a string but the text is not an integervalue
is not a string, int, or timestampto_regex
falliblevalue
into a regex.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The value to convert to a regex. | yes |
Notices
This function has special behavior that you should be aware of.Errors
Theto_regex
function is fallible, which means that
error handling is required for these errors:value
is not a string.to_string
falliblevalue
into a string.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | integer float boolean string timestamp null | The value to convert to a string. | yes |
Errors
Theto_string
function is fallible, which means that
error handling is required for these errors:value
is not an integer, float, boolean, string, timestamp, or nullto_timestamp
falliblevalue
into a timestamp.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string float integer timestamp | The value that is to be converted to a timestamp. If a string, must be a valid representation of a timestamp otherwise an ArgumentError will be raised. | yes | |
unit | string | The time unit. | seconds | no |
Notices
This function has special behavior that you should be aware of.Errors
Theto_timestamp
function is fallible, which means that
error handling is required for these errors:value
is a string
, it is not a valid timestamp formatvalue
is an int
, it is not within the Unix timestamp rangevalue
is a float
, it is not within the Unix timestamp rangeExamples
Coerce a string to a timestamp
to_timestamp!("2020-10-21T16:00:00Z")
2020-10-21T16:00:00Z
Coerce a unix timestamp (integer) to a timestamp
to_timestamp!(1675968923)
2023-02-09T18:55:23Z
Coerce a unix timestamp (float) to a timestamp
to_timestamp!(1675968923.567)
2023-02-09T18:55:23.566999912Z
Convert functions
to_syslog_facility
falliblevalue
, a Syslog facility code, into its corresponding
Syslog keyword. i.e. 0 into "kern"
, 1 into "user"
, etc.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | integer | The facility code. | yes |
Errors
Theto_syslog_facility
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid Syslog facility code.to_syslog_level
falliblevalue
, a Syslog severity level, into its corresponding keyword,
i.e. 0 into "emerg"
, 1 into "alert"
, etc.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | integer | The severity level. | yes |
Errors
Theto_syslog_level
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid Syslog severity level.to_syslog_severity
fallibleFunction spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The Syslog level keyword to convert. | yes |
Errors
Theto_syslog_severity
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid Syslog level keywordto_unix_timestamp
infallibleConverts the value
timestamp into a Unix timestamp.
Returns the number of seconds since the Unix epoch by default, but milliseconds or nanoseconds can also be
specified by unit
.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | timestamp | The timestamp to convert to Unix. | yes | |
unit | string | The time unit. | seconds | no |
Debug functions
assert
falliblecondition
, which must be a Boolean expression. The program is aborted with
message
if the condition evaluates to false
.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
condition | boolean | The condition to check. | yes | |
message | string | An optional custom error message. If the equality assertion fails, message is
appended to the default message prefix. See the examples below
for a sample fully formed log message. | no |
Notices
This function has special behavior that you should be aware of.assert
function should be used in a standalone fashion and only when you want to abort the program. You
should avoid it in logical expressions and other situations in which you want the program to continue if the
condition evaluates to false
.Errors
Theassert
function is fallible, which means that
error handling is required for these errors:condition
evaluates to false
assert_eq
infallibleleft
and right
, have the same value. The program is
aborted with the message
if they are unequal.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
left | any | The value to check for equality against right . | yes | |
right | any | The value to check for equality against left . | yes | |
message | string | An optional custom error message. If the equality assertion fails, message is
appended to the default message prefix. See the examples
below for a sample fully formed log message. | no |
Notices
This function has special behavior that you should be aware of.assert_eq
function should be used in a standalone fashion and only when you want to
abort the program. You should avoid it in logical expressions and other situations in which
you want the program to continue if the condition evaluates to false
.log
infallibleFunction spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | any | The value to log. | yes | |
level | string | The log level. | info | no |
rate_limit_secs | integer | Specifies that the log message is output no more than once per the given number of seconds.
Use a value of 0 to turn rate limiting off. | 1 | no |
Enrichment functions
find_enrichment_table_records
infallibleSearches an enrichment table for rows that match the provided condition.
For file
enrichment tables this condition needs to be a VRL object in which
the key-value pairs indicate a field to search mapped to a value to search in that field.
This function returns the rows that match the provided condition(s). All fields need to
match for rows to be returned; if any fields don’t match, no rows are returned.
There are currently two forms of search criteria:
Exact match search. The given field must match the value exactly. Case sensitivity can be specified using the
case_sensitive
argument. An exact match search can use an index directly into the dataset, which should make this search fairly “cheap” from a performance perspective.Date range search. The given field must be greater than or equal to the
from
date and less than or equal to theto
date. Note that a date range search involves sequentially scanning through the rows that have been located via any exact match criteria. This can be an expensive operation if there are many rows returned by any exact match criteria. We recommend using date ranges as the only criteria when the enrichment data set is very small.
For geoip
enrichment tables this condition needs to be a VRL object with a single key-value pair
whose value needs to be a valid IP address. Example: {"ip": .ip }
. If a return field is expected
and without a value, null
will be used. This table can return following fields:
ISP databases:
autonomous_system_number
autonomous_system_organization
isp
organization
City databases:
city_name
continent_code
country_code
country_name
region_code
region_name
metro_code
latitude
longitude
postal_code
timezone
Connection-Type databases:
connection_type
To use this function, you need to update your Vector configuration to
include an
enrichment_tables
parameter.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
table | string | The enrichment table to search. | yes | |
condition | object | The condition to search on. Since the condition is used at boot time to create indices into the data, these conditions must be statically defined. | yes | |
select | array | A subset of fields from the enrichment table to return. If not specified, all fields are returned. | no | |
case_sensitive | boolean | Whether text fields need to match cases exactly. | true | no |
get_enrichment_table_record
fallibleSearches an enrichment table for a row that matches the provided condition. A single row must be matched. If either no rows or more than one row is found, an error is returned.
For file
enrichment tables this condition needs to be a VRL object in which
the key-value pairs indicate a field to search mapped to a value to search in that field.
This function returns the rows that match the provided condition(s). All fields need to
match for rows to be returned; if any fields don’t match, no rows are returned.
There are currently two forms of search criteria:
Exact match search. The given field must match the value exactly. Case sensitivity can be specified using the
case_sensitive
argument. An exact match search can use an index directly into the dataset, which should make this search fairly “cheap” from a performance perspective.Date range search. The given field must be greater than or equal to the
from
date and less than or equal to theto
date. Note that a date range search involves sequentially scanning through the rows that have been located via any exact match criteria. This can be an expensive operation if there are many rows returned by any exact match criteria. We recommend using date ranges as the only criteria when the enrichment data set is very small.
For geoip
enrichment tables this condition needs to be a VRL object with a single key-value pair
whose value needs to be a valid IP address. Example: {"ip": .ip }
. If a return field is expected
and without a value, null
will be used. This table can return following fields:
ISP databases:
autonomous_system_number
autonomous_system_organization
isp
organization
City databases:
city_name
continent_code
country_code
country_name
region_code
region_name
metro_code
latitude
longitude
postal_code
timezone
Connection-Type databases:
connection_type
To use this function, you need to update your Vector configuration to
include an
enrichment_tables
parameter.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
table | string | The enrichment table to search. | yes | |
condition | object | The condition to search on. Since the condition is used at boot time to create indices into the data, these conditions must be statically defined. | yes | |
select | array | A subset of fields from the enrichment table to return. If not specified, all fields are returned. | no | |
case_sensitive | boolean | Should text fields match case exactly. | true | no |
Errors
Theget_enrichment_table_record
function is fallible, which means that
error handling is required for these errors:Enumerate functions
compact
infalliblevalue
by removing “empty” values, where emptiness is defined using the
available parameters.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | array object | The object or array to compact. | yes | |
recursive | boolean | Whether the compaction be recursive. | true | no |
null | boolean | Whether null should be treated as an empty value. | true | no |
string | boolean | Whether an empty string should be treated as an empty value. | true | no |
object | boolean | Whether an empty object should be treated as an empty value. | true | no |
array | boolean | Whether an empty array should be treated as an empty value. | true | no |
nullish | boolean | Tests whether the value is “nullish” as defined by the is_nullish function. | no |
filter
infallibleFilter elements from a collection.
This function currently does not support recursive iteration.
If you have a need for recursive iteration using filter
, then
please let us know!
The function uses the “function closure syntax” to allow reading the key/value or index/value combination for each item in the collection.
The same scoping rules apply to closure blocks as they do for regular blocks, meaning, any variable defined in parent scopes are accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.
Check out the examples below to learn about the closure syntax.
flatten
infalliblevalue
into a single-level representation.for_each
infallibleIterate over a collection.
This function currently does not support recursive iteration.
If you have a need for recursive iteration using for_each
,
which can’t be solved using the map_keys
or map_values
enumeration functions (which do support recursion), then
please let us know!
The function uses the “function closure syntax” to allow reading the key/value or index/value combination for each item in the collection.
The same scoping rules apply to closure blocks as they do for regular blocks, meaning, any variable defined in parent scopes are accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.
Check out the examples below to learn about the closure syntax.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | array object | The array or object to iterate. | yes |
Examples
Tally elements
tally = {}
for_each(array!(.tags)) -> |_index, value| {
# Get the current tally for the `value`, or
# set to `0`.
count = int(get!(tally, [value])) ?? 0
# Increment the tally for the value by `1`.
tally = set!(tally, [value], count + 1)
}
tally
{
"bar": 1,
"baz": 1,
"foo": 2
}
includes
infalliblevalue
array includes the specified item
.keys
infalliblelength
infallibleReturns the length of the value
.
- If
value
is an array, returns the number of elements. - If
value
is an object, returns the number of top-level keys. - If
value
is a string, returns the number of bytes in the string. If you want the number of characters, seestrlen
.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | array object string | The array or object | yes |
map_keys
infallibleMap the keys within an object.
If recursive
is enabled, the function iterates into nested
objects, using the following rules:
- Iteration starts at the root.
- For every nested object type:
- First return the key of the object type itself.
- Then recurse into the object, and loop back to item (1) in this list.
- Any mutation done on a nested object before recursing into it, are preserved.
- For every nested array type:
- First return the key of the array type itself
- Then find all objects within the array, and apply item (2) to each individual object.
Practically speaking, the above rules mean that map_keys
with
recursive
enabled will find all keys in the target,
regardless of whether nested objects are nested inside arrays.
The function uses the “function closure syntax” to allow reading the key for each item in the object.
The same scoping rules apply to closure blocks as they do for regular blocks, meaning, any variable defined in parent scopes are accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.
Check out the examples below to learn about the closure syntax.
map_values
infallibleMap the values within a collection.
If recursive
is enabled, the function iterates into nested
collections, using the following rules:
- Iteration starts at the root.
- For every nested collection type:
- First return the collection type itself.
- Then recurse into the collection, and loop back to item (1) in the list
- Any mutation done on a collection before recursing into it, are preserved.
The function uses the “function closure syntax” to allow mutating the value for each item in the collection.
The same scoping rules apply to closure blocks as they do for regular blocks, meaning, any variable defined in parent scopes are accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.
Check out the examples below to learn about the closure syntax.
match_array
infalliblevalue
array matches the pattern
- by default it checks at least one element matches, but can be set to determine if all the elements match.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | array | The array. | yes | |
pattern | regex | The regular expression pattern to match against. | yes | |
all | boolean | Whether to match on all elements of value . | no |
strlen
infallibleReturns the number of UTF-8 characters in value
. This differs from
length
which would count the number of bytes of a string.
Note that this is the count of unicode scalar values which can sometimes differ from unicode code points.
unique
infallibleReturns unique values for an array.
The first occurrence of each element is kept.
values
infallibleEvent functions
get_secret
infallibleremove_secret
infallibleset_secret
infalliblePath functions
del
infallibleRemoves the field specified by the static path
from the target.
For dynamic path deletion, see the remove
function.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
path | path | The path of the field to delete. | yes | |
compact | boolean | If compact is true, after deletion, if an empty object or array is left behind, it should be removed as well, cascading up to the root. This only applies to the path being deleted, and any parent paths. | no |
exists
infallibleChecks whether the path
exists for the target.
This function allows you to distinguish between a missing path,
or a path with a null
value, something a regular path lookup
such as .foo
would not allow, since that always returns null
if the path doesn’t exist.
get
fallibleDynamically get the value of a given path.
When you know the path you want to look up, you should use
static paths such as .foo.bar[1]
to get the value of that
path. However, when you don’t know the path names in advance,
you can use this dynamic get function to get at the requested
value.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | object array | The object or array to query. | yes | |
path | array | An array of path segments to look up the value for. | yes |
Errors
Theget
function is fallible, which means that
error handling is required for these errors:remove
fallibleDynamically remove the value for a given path.
When you know the path you want to remove, you should use
the del
function and static paths such as del(.foo.bar[1])
to remove the value at that path. The del
function returns the
deleted value, and is more performant than this function.
However, when you don’t know the path names in advance, you can
use this dynamic remove function to remove the value at the
provided path.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | object array | The object or array to remove data from. | yes | |
path | array | An array of path segments to remove the value at. | yes | |
compact | boolean | Whether — after deletion — empty objects or arrays should be removed. | no |
Errors
Theremove
function is fallible, which means that
error handling is required for these errors:set
fallibleDynamically insert data into the path of a given object or array.
When you know the path you want to assign a value to, you should
use static path assignments such as .foo.bar[1] = true
for
improved performance and readability. However, when you don’t
know the path names in advance, you can use this dynamic
insertion function to insert the data into the object or array.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | object array | The object or array to insert data into. | yes | |
path | array | An array of path segments to insert the value to. | yes | |
data | any | The data to be inserted. | yes |
Errors
Theset
function is fallible, which means that
error handling is required for these errors:Cryptography functions
decrypt
fallibleDecrypts a string with a symmetric encryption algorithm.
Supported Algorithms:
- AES-256-CFB (key = 32 bytes, iv = 16 bytes)
- AES-192-CFB (key = 24 bytes, iv = 16 bytes)
- AES-128-CFB (key = 16 bytes, iv = 16 bytes)
- AES-256-OFB (key = 32 bytes, iv = 16 bytes)
- AES-192-OFB (key = 24 bytes, iv = 16 bytes)
- AES-128-OFB (key = 16 bytes, iv = 16 bytes)
- AES-256-CTR (key = 32 bytes, iv = 16 bytes)
- AES-192-CTR (key = 24 bytes, iv = 16 bytes)
- AES-128-CTR (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
ciphertext | string | The string to decrypt. The should be raw bytes (not encoded). | yes | |
algorithm | string | The algorithm to use. | yes | |
key | string | The key for decryption. The should be raw bytes of the key (not encoded). The length must match the algorithm requested. | yes | |
iv | string | The IV for decryption. The should be raw bytes of the IV (not encoded). The length must match the algorithm requested.
A new IV should be generated for every message. You can use random_bytes to generate a cryptographically secure random value.
The value should match the one used during encryption. | yes |
Errors
Thedecrypt
function is fallible, which means that
error handling is required for these errors:algorithm
isn’t a supported algorithmkey
length doesn’t match the key size required for the algorithm specifiediv
length doesn’t match the iv size required for the algorithm specifiedencrypt
fallibleEncrypts a string with a symmetric encryption algorithm.
Supported Algorithms:
- AES-256-CFB (key = 32 bytes, iv = 16 bytes)
- AES-192-CFB (key = 24 bytes, iv = 16 bytes)
- AES-128-CFB (key = 16 bytes, iv = 16 bytes)
- AES-256-OFB (key = 32 bytes, iv = 16 bytes)
- AES-192-OFB (key = 24 bytes, iv = 16 bytes)
- AES-128-OFB (key = 16 bytes, iv = 16 bytes)
- AES-256-CTR (key = 32 bytes, iv = 16 bytes)
- AES-192-CTR (key = 24 bytes, iv = 16 bytes)
- AES-128-CTR (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
plaintext | string | The string to encrypt. | yes | |
algorithm | string | The algorithm to use. | yes | |
key | string | The key for encryption. The should be raw bytes of the key (not encoded). The length must match the algorithm requested. | yes | |
iv | string | The IV for encryption. The should be raw bytes of the IV (not encoded). The length must match the algorithm requested.
A new IV should be generated for every message. You can use random_bytes to generate a cryptographically secure random value. | yes |
Errors
Theencrypt
function is fallible, which means that
error handling is required for these errors:algorithm
isn’t a supported algorithmkey
length doesn’t match the key size required for the algorithm specifiediv
length doesn’t match the iv size required for the algorithm specifiedExamples
Encrypt value
plaintext = "super secret message"
iv = "1234567890123456" # typically you would call random_bytes(16)
key = "16_byte_keyxxxxx"
encrypted_message = encrypt!(plaintext, "AES-128-CBC-PKCS7", key, iv: iv)
encode_base64(encrypted_message)
GBw8Mu00v0Kc38+/PvsVtGgWuUJ+ZNLgF8Opy8ohIYE=
hmac
infallibleCalculates a HMAC of the value
using the given key
.
The hashing algorithm
used can be optionally specified.
For most use cases, the resulting bytestream should be encoded into a hex or base64 string using either encode_base16 or encode_base64.
This function is infallible if either the default algorithm
value, or a recognized-valid compile-time
algorithm
string literal is used. Otherwise it is fallible.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to calculate the HMAC for. | yes | |
key | string | The string to use as the cryptographic key. | yes | |
algorithm | string | The hashing algorithm to use. | SHA-256 | no |
Examples
Calculate message HMAC (defaults: SHA-256), encoding to a base64 string
encode_base64(hmac("Hello there", "super-secret-key"))
eLGE8YMviv85NPXgISRUZxstBNSU47JQdcXkUWcClmI=
md5
infalliblevalue
.seahash
infalliblevalue
.
Note: Due to limitations in the underlying VRL data types, this function converts the unsigned 64-bit integer seahash result to a signed 64-bit integer. Results higher than the signed 64-bit integer maximum value wrap around to negative values.sha1
infalliblevalue
.sha2
infalliblevalue
.sha3
infalliblevalue
.IP functions
ip_aton
fallibleConverts IPv4 address in numbers-and-dots notation into network-order bytes represented as an integer.
This behavior mimics inet_aton.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The IP address to convert to binary. | yes |
Errors
Theip_aton
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid IPv4 addressip_cidr_contains
fallibleip
is contained in the block referenced by the cidr
.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
cidr | string | The CIDR mask (v4 or v6). | yes | |
ip | string | The IP address (v4 or v6). | yes |
Errors
Theip_cidr_contains
function is fallible, which means that
error handling is required for these errors:cidr
isn’t a valid CIDRip
isn’t a valid IP addressip_ntoa
fallibleConverts numeric representation of IPv4 address in network-order bytes to numbers-and-dots notation..
This behavior mimics inet_ntoa.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The integer representation of an IPv4 address. | yes |
Errors
Theip_ntoa
function is fallible, which means that
error handling is required for these errors:value
cannot fit in u32ip_ntop
fallibleConverts IPv4 and IPv6 addresses from binary to text form.
This behavior mimics inet_ntop.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The binary data to convert from. For IPv4 addresses, it must be 4 bytes (32 bits) long. For IPv6 addresses, it must be 16 bytes (128 bits) long. | yes |
Notices
This function has special behavior that you should be aware of.decode_base64
or
decode_percent
can still be used correctly.Errors
Theip_ntop
function is fallible, which means that
error handling is required for these errors:value
must be of length 4 or 16 bytesip_pton
fallibleConverts IPv4 and IPv6 addresses from text to binary form.
- The binary form of IPv4 addresses is 4 bytes (32 bits) long.
- The binary form of IPv6 addresses is 16 bytes (128 bits) long.
This behavior mimics inet_pton.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The IP address (v4 or v6) to convert to binary form. | yes |
Notices
This function has special behavior that you should be aware of.encode_base64
or encode_percent
can
still process it correctly.Errors
Theip_pton
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid IP (v4 or v6) address in text formip_subnet
fallibleip
using the supplied subnet
.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
ip | string | The IP address (v4 or v6). | yes | |
subnet | string | The subnet to extract from the IP address. This can be either a prefix length like /8 or a net mask
like 255.255.0.0 . The net mask can be either an IPv4 or IPv6 address. | yes |
Notices
This function has special behavior that you should be aware of.Errors
Theip_subnet
function is fallible, which means that
error handling is required for these errors:ip
isn’t a valid IP addresssubnet
isn’t a valid subnet.ip_to_ipv6
fallibleip
to an IPv6 address.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
ip | string | The IP address to convert to IPv6. | yes |
Errors
Theip_to_ipv6
function is fallible, which means that
error handling is required for these errors:ip
isn’t a valid IP addressipv6_to_ipv4
fallibleip
to an IPv4 address. ip
is returned unchanged if it’s already an IPv4 address. If ip
is
currently an IPv6 address then it needs to be IPv4 compatible, otherwise an error is thrown.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
ip | string | The IPv4-mapped IPv6 address to convert. | yes |
Errors
Theipv6_to_ipv4
function is fallible, which means that
error handling is required for these errors:ip
isn’t a valid IP addressip
is an IPv6 address that isn’t compatible with IPv4is_ipv4
infallibleCheck if the string is a valid IPv4 address or not.
An [IPv4-mapped][https://datatracker.ietf.org/doc/html/rfc6890] or [IPv4-compatible][https://datatracker.ietf.org/doc/html/rfc6890] IPv6 address is not considered valid for the purpose of this function.
is_ipv6
infallibleNumber functions
abs
infalliblevalue
.ceil
infalliblevalue
up to the specified precision
.floor
infalliblevalue
down to the specified precision
.format_int
falliblevalue
into a string representation using the given base/radix.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | integer | The number to format. | yes | |
base | integer | The base to format the number in. Must be between 2 and 36 (inclusive). | 10 | no |
Errors
Theformat_int
function is fallible, which means that
error handling is required for these errors:format_number
infalliblevalue
into a string representation of the number.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | integer float | The number to format as a string. | yes | |
scale | integer | The number of decimal places to display. | no | |
decimal_separator | string | The character to use between the whole and decimal parts of the number. | . | no |
grouping_separator | string | The character to use between each thousands part of the number. | , | no |
mod
falliblevalue
divided by modulus
.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | integer float | The value the modulus is applied to. | yes | |
modulus | integer float | The modulus value. | yes |
Errors
Themod
function is fallible, which means that
error handling is required for these errors:value
isn’t an integer or floatmodulus
isn’t an integer or floatmodulus
is equal to 0round
infalliblevalue
to the specified precision
.Object functions
match_datadog_query
infalliblemerge
infalliblefrom
object into the to
object.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
to | object | The object to merge into. | yes | |
from | object | The object to merge from. | yes | |
deep | boolean | A deep merge is performed if true , otherwise only top-level fields are merged. | no |
unnest
fallibleUnnest an array field from an object to create an array of objects using that field; keeping all other fields.
Assigning the array result of this to .
will result in multiple events being emitted from remap
. See the
remap
transform docs for more details.
This is also referred to as explode
ing in some languages.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
path | path | The path of the field to unnest. | yes |
Errors
Theunnest
function is fallible, which means that
error handling is required for these errors:Parse functions
parse_apache_log
falliblecommon
,
combined
, or default error
format.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes | |
format | string | The format to use for parsing the log. | yes | |
timestamp_format | string | The date/time format to use for encoding the timestamp. The time is parsed in local time if the timestamp doesn’t specify a timezone. | %d/%b/%Y:%T %z | no |
Notices
This function has special behavior that you should be aware of.-
. These fields are omitted in the result.Errors
Theparse_apache_log
function is fallible, which means that
error handling is required for these errors:value
doesn’t match the specified formattimestamp_format
isn’t a valid format stringvalue
fails to parse using the provided timestamp_format
Examples
Parse via Apache log format (common)
parse_apache_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326", format: "common")
{
"host": "127.0.0.1",
"identity": "bob",
"message": "GET /apache_pb.gif HTTP/1.0",
"method": "GET",
"path": "/apache_pb.gif",
"protocol": "HTTP/1.0",
"size": 2326,
"status": 200,
"timestamp": "2000-10-10T20:55:36Z",
"user": "frank"
}
Parse via Apache log format (combined)
parse_apache_log!(
s'127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth" "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"',
"combined",
)
{
"agent": "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0",
"host": "127.0.0.1",
"identity": "bob",
"message": "GET /apache_pb.gif HTTP/1.0",
"method": "GET",
"path": "/apache_pb.gif",
"protocol": "HTTP/1.0",
"referrer": "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth",
"size": 2326,
"status": 200,
"timestamp": "2000-10-10T20:55:36Z",
"user": "frank"
}
Parse via Apache log format (error)
parse_apache_log!(
s'[01/Mar/2021:12:00:19 +0000] [ab:alert] [pid 4803:tid 3814] [client 147.159.108.175:24259] I will bypass the haptic COM bandwidth, that should matrix the CSS driver!',
"error"
)
{
"client": "147.159.108.175",
"message": "I will bypass the haptic COM bandwidth, that should matrix the CSS driver!",
"module": "ab",
"pid": 4803,
"port": 24259,
"severity": "alert",
"thread": "3814",
"timestamp": "2021-03-01T12:00:19Z"
}
parse_aws_alb_log
falliblevalue
in the Elastic Load Balancer Access format.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | Access log of the Application Load Balancer. | yes |
Errors
Theparse_aws_alb_log
function is fallible, which means that
error handling is required for these errors:value
isn’t a properly formatted AWS ALB logExamples
Parse AWS ALB log
parse_aws_alb_log!(
"http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\""
)
{
"actions_executed": "forward",
"chosen_cert_arn": null,
"classification": null,
"classification_reason": null,
"client_host": "192.168.131.39:2817",
"domain_name": null,
"elb": "app/my-loadbalancer/50dc6c495c0c9188",
"elb_status_code": "200",
"error_reason": null,
"matched_rule_priority": "0",
"received_bytes": 34,
"redirect_url": null,
"request_creation_time": "2018-11-30T22:22:48.364000Z",
"request_method": "GET",
"request_processing_time": 0,
"request_protocol": "HTTP/1.1",
"request_url": "http://www.example.com:80/",
"response_processing_time": 0,
"sent_bytes": 366,
"ssl_cipher": null,
"ssl_protocol": null,
"target_group_arn": "arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067",
"target_host": null,
"target_port_list": [],
"target_processing_time": 0.001,
"target_status_code": "200",
"target_status_code_list": [],
"timestamp": "2018-11-30T22:23:00.186641Z",
"trace_id": "Root=1-58337364-23a8c76965a2ef7629b185e3",
"type": "http",
"user_agent": "curl/7.46.0"
}
parse_aws_cloudwatch_log_subscription_message
fallibleaws_kinesis_firehose
source.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string representation of the message to parse. | yes |
Errors
Theparse_aws_cloudwatch_log_subscription_message
function is fallible, which means that
error handling is required for these errors:value
isn’t a properly formatted AWS Cloudwatch Log subscription messageExamples
Parse AWS Cloudwatch Log subscription message
parse_aws_cloudwatch_log_subscription_message!(.message)
{
"log_events": [
{
"id": "35683658089614582423604394983260738922885519999578275840",
"message": "{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}",
"timestamp": "2020-09-14T19:09:29.039Z"
}
],
"log_group": "test",
"log_stream": "test",
"message_type": "DATA_MESSAGE",
"owner": "111111111111",
"subscription_filters": [
"Destination"
]
}
parse_aws_vpc_flow_log
falliblevalue
in the VPC Flow Logs format.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | VPC Flow Log. | yes | |
format | string | VPC Flow Log format. | no |
Errors
Theparse_aws_vpc_flow_log
function is fallible, which means that
error handling is required for these errors:value
isn’t a properly formatted AWS VPC Flow logExamples
Parse AWS VPC Flow log (default format)
parse_aws_vpc_flow_log!("2 123456789010 eni-1235b8ca123456789 - - - - - - - 1431280876 1431280934 - NODATA")
{
"account_id": 123456789010,
"action": null,
"bytes": null,
"dstaddr": null,
"dstport": null,
"end": 1431280934,
"interface_id": "eni-1235b8ca123456789",
"log_status": "NODATA",
"packets": null,
"protocol": null,
"srcaddr": null,
"srcport": null,
"start": 1431280876,
"version": 2
}
Parse AWS VPC Flow log (custom format)
parse_aws_vpc_flow_log!(
"- eni-1235b8ca123456789 10.0.1.5 10.0.0.220 10.0.1.5 203.0.113.5",
"instance_id interface_id srcaddr dstaddr pkt_srcaddr pkt_dstaddr"
)
{
"dstaddr": "10.0.0.220",
"instance_id": null,
"interface_id": "eni-1235b8ca123456789",
"pkt_dstaddr": "203.0.113.5",
"pkt_srcaddr": "10.0.1.5",
"srcaddr": "10.0.1.5"
}
parse_cef
falliblevalue
in CEF (Common Event Format) format. Ignores everything up to CEF header. Empty values are returned as empty strings. Surrounding quotes are removed from values.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes | |
translate_custom_fields | boolean | Toggles translation of custom field pairs to key: value . | no |
Notices
This function has special behavior that you should be aware of.Errors
Theparse_cef
function is fallible, which means that
error handling is required for these errors:value
isn’t a properly formatted CEF stringExamples
Parse output generated by PTA
parse_cef!(
"CEF:0|CyberArk|PTA|12.6|1|Suspected credentials theft|8|suser=mike2@prod1.domain.com shost=prod1.domain.com src=1.1.1.1 duser=andy@dev1.domain.com dhost=dev1.domain.com dst=2.2.2.2 cs1Label=ExtraData cs1=None cs2Label=EventID cs2=52b06812ec3500ed864c461e deviceCustomDate1Label=detectionDate deviceCustomDate1=1388577900000 cs3Label=PTAlink cs3=https://1.1.1.1/incidents/52b06812ec3500ed864c461e cs4Label=ExternalLink cs4=None"
)
{
"cefVersion": "0",
"cs1": "None",
"cs1Label": "ExtraData",
"cs2": "52b06812ec3500ed864c461e",
"cs2Label": "EventID",
"cs3": "https://1.1.1.1/incidents/52b06812ec3500ed864c461e",
"cs3Label": "PTAlink",
"cs4": "None",
"cs4Label": "ExternalLink",
"deviceCustomDate1": "1388577900000",
"deviceCustomDate1Label": "detectionDate",
"deviceEventClassId": "1",
"deviceProduct": "PTA",
"deviceVendor": "CyberArk",
"deviceVersion": "12.6",
"dhost": "dev1.domain.com",
"dst": "2.2.2.2",
"duser": "andy@dev1.domain.com",
"name": "Suspected credentials theft",
"severity": "8",
"shost": "prod1.domain.com",
"src": "1.1.1.1",
"suser": "mike2@prod1.domain.com"
}
Ignore syslog header
parse_cef!(
"Sep 29 08:26:10 host CEF:1|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232"
)
{
"cefVersion": "1",
"deviceEventClassId": "100",
"deviceProduct": "threatmanager",
"deviceVendor": "Security",
"deviceVersion": "1.0",
"dst": "2.1.2.2",
"name": "worm successfully stopped",
"severity": "10",
"spt": "1232",
"src": "10.0.0.1"
}
Translate custom fields
parse_cef!(
"CEF:0|Dev|firewall|2.2|1|Connection denied|5|c6a1=2345:0425:2CA1:0000:0000:0567:5673:23b5 c6a1Label=Device IPv6 Address",
translate_custom_fields: true
)
{
"Device IPv6 Address": "2345:0425:2CA1:0000:0000:0567:5673:23b5",
"cefVersion": "0",
"deviceEventClassId": "1",
"deviceProduct": "firewall",
"deviceVendor": "Dev",
"deviceVersion": "2.2",
"name": "Connection denied",
"severity": "5"
}
parse_common_log
falliblevalue
using the Common Log Format (CLF).Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes | |
timestamp_format | string | The date/time format to use for encoding the timestamp. | %d/%b/%Y:%T %z | no |
Notices
This function has special behavior that you should be aware of.-
. These fields are omitted in the result.Errors
Theparse_common_log
function is fallible, which means that
error handling is required for these errors:value
doesn’t match the Common Log Formattimestamp_format
isn’t a valid format stringvalue
fails to parse using the provided timestamp_format
Examples
Parse via Common Log Format (with default timestamp format)
parse_common_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326")
{
"host": "127.0.0.1",
"identity": "bob",
"message": "GET /apache_pb.gif HTTP/1.0",
"method": "GET",
"path": "/apache_pb.gif",
"protocol": "HTTP/1.0",
"size": 2326,
"status": 200,
"timestamp": "2000-10-10T20:55:36Z",
"user": "frank"
}
Parse via Common Log Format (with custom timestamp format)
parse_common_log!(
"127.0.0.1 bob frank [2000-10-10T20:55:36Z] \"GET /apache_pb.gif HTTP/1.0\" 200 2326",
"%+"
)
{
"host": "127.0.0.1",
"identity": "bob",
"message": "GET /apache_pb.gif HTTP/1.0",
"method": "GET",
"path": "/apache_pb.gif",
"protocol": "HTTP/1.0",
"size": 2326,
"status": 200,
"timestamp": "2000-10-10T20:55:36Z",
"user": "frank"
}
parse_csv
fallibleFunction spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes | |
delimiter | string | The field delimiter to use when parsing. Must be a single-byte utf8 character. | , | no |
Notices
This function has special behavior that you should be aware of.Errors
Theparse_csv
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid CSV stringparse_duration
falliblevalue
into a human-readable duration format specified by unit
.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string of the duration. | yes | |
unit | string | The output units for the duration. | yes |
Errors
Theparse_duration
function is fallible, which means that
error handling is required for these errors:value
isn’t a properly formatted durationparse_glog
falliblevalue
using the glog (Google Logging Library) format.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes |
Errors
Theparse_glog
function is fallible, which means that
error handling is required for these errors:value
doesn’t match the glog
formatparse_grok
fallibleFunction spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes | |
pattern | string | The Grok pattern. | yes |
Notices
This function has special behavior that you should be aware of.Errors
Theparse_grok
function is fallible, which means that
error handling is required for these errors:value
fails to parse using the provided pattern
parse_groks
fallibleFunction spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes | |
patterns | array | The Grok patterns, which are tried in order until the first match. | yes | |
aliases | object | The shared set of grok aliases that can be referenced in the patterns to simplify them. | true | no |
Notices
This function has special behavior that you should be aware of.Errors
Theparse_groks
function is fallible, which means that
error handling is required for these errors:value
fails to parse using the provided pattern
Examples
Parse using multiple Grok patterns
parse_groks!(
"2020-10-02T23:22:12.223222Z info Hello world",
patterns: [
"%{common_prefix} %{_status} %{_message}",
"%{common_prefix} %{_message}",
],
aliases: {
"common_prefix": "%{_timestamp} %{_loglevel}",
"_timestamp": "%{TIMESTAMP_ISO8601:timestamp}",
"_loglevel": "%{LOGLEVEL:level}",
"_status": "%{POSINT:status}",
"_message": "%{GREEDYDATA:message}"
}
)
{
"level": "info",
"message": "Hello world",
"timestamp": "2020-10-02T23:22:12.223222Z"
}
parse_int
falliblevalue
representing a number in an optional base/radix to an integer.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes | |
base | integer | The base the number is in. Must be between 2 and 36 (inclusive). If unspecified, will use the string prefix to try to determine the base: “0b”, 8 for “0” or “0o”, 16 for “0x”, and 10 otherwise | no |
Errors
Theparse_int
function is fallible, which means that
error handling is required for these errors:parse_json
falliblevalue
as JSON.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string representation of the JSON to parse. | yes | |
max_depth | integer | Number of layers to parse for nested JSON-formatted documents. The value must be in range 1..128. | no |
Notices
This function has special behavior that you should be aware of.string
into a timestamp
, consider the
parse_timestamp
function.Errors
Theparse_json
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid JSON-formatted payloadparse_key_value
fallibleParses the value
in key/value format. Also known as logfmt.
- Keys and values can be wrapped with
"
. "
characters can be escaped using\
.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes | |
key_value_delimiter | string | The string that separates the key from the value. | = | no |
field_delimiter | string | The string that separates each key/value pair. |
| no |
whitespace | string | Defines the acceptance of unnecessary whitespace surrounding the configured key_value_delimiter . | lenient | no |
accept_standalone_key | boolean | Whether a standalone key should be accepted, the resulting object will associate such keys with boolean value true | true | no |
Notices
This function has special behavior that you should be aware of.Errors
Theparse_key_value
function is fallible, which means that
error handling is required for these errors:value
isn’t a properly formatted key/value stringExamples
Parse logfmt log
parse_key_value!(
"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
{
"@timestamp": "Sun Jan 10 16:47:39 EST 2021",
"id": "ConsumerFetcherManager-1382721708341",
"level": "info",
"module": "kafka.consumer.ConsumerFetcherManager",
"msg": "Stopping all fetchers",
"tag#production": "stopping_fetchers"
}
Parse comma delimited log
parse_key_value!(
"path:\"/cart_link\", host:store.app.com, fwd: \"102.30.171.16\", dyno: web.1, connect:0ms, service:87ms, status:304, bytes:632, protocol:https",
field_delimiter: ",",
key_value_delimiter: ":"
)
{
"bytes": "632",
"connect": "0ms",
"dyno": "web.1",
"fwd": "102.30.171.16",
"host": "store.app.com",
"path": "/cart_link",
"protocol": "https",
"service": "87ms",
"status": "304"
}
parse_klog
falliblevalue
using the klog format used by Kubernetes components.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes |
Errors
Theparse_klog
function is fallible, which means that
error handling is required for these errors:value
doesn’t match the klog
formatparse_linux_authorization
fallible/var/log/auth.log
(for Debian-based systems) or
/var/log/secure
(for RedHat-based systems) according to Syslog format.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The text containing the message to parse. | yes |
Notices
This function has special behavior that you should be aware of.Errors
Theparse_linux_authorization
function is fallible, which means that
error handling is required for these errors:value
isn’t a properly formatted Syslog messageExamples
Parse Linux authorization event
parse_linux_authorization!(
s'Mar 23 01:49:58 localhost sshd[1111]: Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar'
)
{
"appname": "sshd",
"hostname": "localhost",
"message": "Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar",
"procid": 1111,
"timestamp": "2023-03-23T01:49:58Z"
}
parse_logfmt
fallibleParses the value
in logfmt.
- Keys and values can be wrapped using the
"
character. "
characters can be escaped by the\
character.- As per this logfmt specification, the
parse_logfmt
function accepts standalone keys and assigns them a Boolean value oftrue
.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes |
Errors
Theparse_logfmt
function is fallible, which means that
error handling is required for these errors:value
isn’t a properly formatted key/value stringExamples
Parse logfmt log
parse_logfmt!(
"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
{
"@timestamp": "Sun Jan 10 16:47:39 EST 2021",
"id": "ConsumerFetcherManager-1382721708341",
"level": "info",
"module": "kafka.consumer.ConsumerFetcherManager",
"msg": "Stopping all fetchers",
"tag#production": "stopping_fetchers"
}
parse_nginx_log
fallibleFunction spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes | |
format | string | The format to use for parsing the log. | yes | |
timestamp_format | string | The date/time format to use for encoding the timestamp. The time is parsed
in local time if the timestamp doesn’t specify a timezone. The default format is %d/%b/%Y:%T %z for
combined logs and %Y/%m/%d %H:%M:%S for error logs. | %d/%b/%Y:%T %z | no |
Notices
This function has special behavior that you should be aware of.-
. These fields are omitted in the result.Errors
Theparse_nginx_log
function is fallible, which means that
error handling is required for these errors:value
doesn’t match the specified formattimestamp_format
isn’t a valid format stringvalue
fails to parse using the provided timestamp_format
Examples
Parse via Nginx log format (combined)
parse_nginx_log!(
s'172.17.0.1 - alice [01/Apr/2021:12:02:31 +0000] "POST /not-found HTTP/1.1" 404 153 "http://localhost/somewhere" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"',
"combined",
)
{
"agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36",
"client": "172.17.0.1",
"compression": "2.75",
"method": "POST",
"path": "/not-found",
"protocol": "HTTP/1.1",
"referer": "http://localhost/somewhere",
"request": "POST /not-found HTTP/1.1",
"size": 153,
"status": 404,
"timestamp": "2021-04-01T12:02:31Z",
"user": "alice"
}
Parse via Nginx log format (error)
parse_nginx_log!(
s'2021/04/01 13:02:31 [error] 31#31: *1 open() "/usr/share/nginx/html/not-found" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "POST /not-found HTTP/1.1", host: "localhost:8081"',
"error"
)
{
"cid": 1,
"client": "172.17.0.1",
"host": "localhost:8081",
"message": "open() \"/usr/share/nginx/html/not-found\" failed (2: No such file or directory)",
"pid": 31,
"request": "POST /not-found HTTP/1.1",
"server": "localhost",
"severity": "error",
"tid": 31,
"timestamp": "2021-04-01T13:02:31Z"
}
parse_query_string
infalliblevalue
as a query string.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes |
parse_regex
fallibleParses the value
via the provided Regex pattern
.
This function differs from the parse_regex_all
function in that it returns only the first match.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to search. | yes | |
pattern | regex | The regular expression pattern to search against. | yes | |
numeric_groups | regex | If true, the index of each group in the regular expression is also captured. The 0th index will contain the whole match. | no |
Notices
This function has special behavior that you should be aware of.parse_regex
function, see if a VRL parse_*
function
already exists for your format. If not, we recommend opening an issue to request
support for the desired format.Errors
Theparse_regex
function is fallible, which means that
error handling is required for these errors:value
fails to parse using the provided pattern
parse_regex_all
fallibleParses the value
via the provided Regex pattern
.
This function differs from the parse_regex
function in that it returns all matches, not just the first.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to search. | yes | |
pattern | regex | The regular expression pattern to search against. | yes | |
numeric_groups | regex | If true , the index of each group in the regular expression is also captured. The 0th index
contains the whole match. | no |
Notices
This function has special behavior that you should be aware of.parse_regex
function, see if a VRL parse_*
function
already exists for your format. If not, we recommend opening an issue to request
support for the desired format.Errors
Theparse_regex_all
function is fallible, which means that
error handling is required for these errors:value
fails to parse via the provided pattern
parse_ruby_hash
falliblevalue
as ruby hash.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string representation of the ruby hash to parse. | yes |
Notices
This function has special behavior that you should be aware of.string
into a timestamp
, consider the
parse_timestamp
function.Errors
Theparse_ruby_hash
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid ruby hash formatted payloadparse_syslog
falliblevalue
in Syslog format.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The text containing the Syslog message to parse. | yes |
Notices
This function has special behavior that you should be aware of.Errors
Theparse_syslog
function is fallible, which means that
error handling is required for these errors:value
isn’t a properly formatted Syslog messageExamples
Parse Syslog log (5424)
parse_syslog!(
s'<13>1 2020-03-13T20:45:38.119Z dynamicwireless.name non 2426 ID931 [exampleSDID@32473 iut="3" eventSource= "Application" eventID="1011"] Try to override the THX port, maybe it will reboot the neural interface!'
)
{
"appname": "non",
"exampleSDID@32473": {
"eventID": "1011",
"eventSource": "Application",
"iut": "3"
},
"facility": "user",
"hostname": "dynamicwireless.name",
"message": "Try to override the THX port, maybe it will reboot the neural interface!",
"msgid": "ID931",
"procid": 2426,
"severity": "notice",
"timestamp": "2020-03-13T20:45:38.119Z",
"version": 1
}
parse_timestamp
fallibleFunction spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The text of the timestamp. | yes | |
format | string | The strptime format. | yes |
Errors
Theparse_timestamp
function is fallible, which means that
error handling is required for these errors:value
fails to parse using the provided format
parse_tokens
fallibleParses the value
in “token” format. A token is considered to be one of the following:
- A word surrounded by whitespace.
- Text delimited by double quotes:
".."
. Quotes can be included in the token if they are escaped by a backslash (\
). - Text delimited by square brackets:
[..]
. Closing square brackets can be included in the token if they are escaped by a backslash (\
).
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to tokenize. | yes |
Notices
This function has special behavior that you should be aware of.Errors
Theparse_tokens
function is fallible, which means that
error handling is required for these errors:value
isn’t a properly formatted tokenized stringparse_url
falliblevalue
in URL format.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The text of the URL. | yes | |
default_known_ports | boolean | If true and the port number is not specified in the input URL
string (or matches the default port for the scheme), it will be
populated from well-known ports for the following schemes:
http , https , ws , wss , and ftp . | no |
Errors
Theparse_url
function is fallible, which means that
error handling is required for these errors:value
isn’t a properly formatted URLparse_user_agent
infalliblevalue
as a user agent string. Which has a loosely defined format
so this parser only provides best effort guarantee.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes | |
mode | string | Determines performance and reliability characteristics. | fast | no |
Notices
This function has special behavior that you should be aware of.null
.Examples
Fast mode
parse_user_agent(
"Mozilla Firefox 1.0.1 Mozilla/5.0 (X11; U; Linux i686; de-DE; rv:1.7.6) Gecko/20050223 Firefox/1.0.1"
)
{
"browser": {
"family": "Firefox",
"version": "1.0.1"
},
"device": {
"category": "pc"
},
"os": {
"family": "Linux",
"version": null
}
}
Reliable mode
parse_user_agent(
"Mozilla/4.0 (compatible; MSIE 7.66; Windows NT 5.1; SV1; .NET CLR 1.1.4322)",
mode: "reliable"
)
{
"browser": {
"family": "Internet Explorer",
"version": "7.66"
},
"device": {
"category": "pc"
},
"os": {
"family": "Windows XP",
"version": "NT 5.1"
}
}
Enriched mode
parse_user_agent(
"Opera/9.80 (J2ME/MIDP; Opera Mini/4.3.24214; iPhone; CPU iPhone OS 4_2_1 like Mac OS X; AppleWebKit/24.783; U; en) Presto/2.5.25 Version/10.54",
mode: "enriched"
)
{
"browser": {
"family": "Opera Mini",
"major": "4",
"minor": "3",
"patch": "24214",
"version": "10.54"
},
"device": {
"brand": "Apple",
"category": "smartphone",
"family": "iPhone",
"model": "iPhone"
},
"os": {
"family": "iOS",
"major": "4",
"minor": "2",
"patch": "1",
"patch_minor": null,
"version": "4.2.1"
}
}
parse_xml
falliblevalue
as XML.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string representation of the XML document to parse. | yes | |
include_attr | boolean | Include XML tag attributes in the returned object. | true | no |
attr_prefix | string | String prefix to use for XML tag attribute keys. | @ | no |
text_key | string | Key name to use for expanded text nodes. | text | no |
always_use_text_key | boolean | Always return text nodes as {"<text_key>": "value"}. | no | |
parse_bool | boolean | Parse “true” and “false” as boolean. | true | no |
parse_null | boolean | Parse “null” as null. | true | no |
parse_number | boolean | Parse numbers as integers/floats. | true | no |
Notices
This function has special behavior that you should be aware of.Errors
Theparse_xml
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid XML documentExamples
Parse XML
value = s'<book category="CHILDREN"><title lang="en">Harry Potter</title><author>J K. Rowling</author><year>2005</year></book>';
parse_xml!(value, text_key: "value", parse_number: false)
{
"book": {
"@category": "CHILDREN",
"author": "J K. Rowling",
"title": {
"@lang": "en",
"value": "Harry Potter"
},
"year": "2005"
}
}