VRL function reference

Here you’ll find a comprehensive list of all built-in VRL functions. Functions are categorized by their purpose and sorted alphabetically for easy discovery. To use these functions in Vector, see the documentation on function call expressions and Vector’s remap transform.

Array functions

append

infallible
Appends each item in the items array to the end of the value array.

Function spec

append(value: <array>, items: <array>)
:: <array>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
The initial arrayyes
itemsarray
The items to appendyes

Examples

Append to an array
Source
append([1, 2], [3, 4])
Return
[1,2,3,4]

push

infallible
Adds the item to the end of the value array.

Function spec

push(value: <array>, item: <any>)
:: <array>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
The target array.yes
itemany
The item to push.yes

Examples

Push an item onto an array
Source
push([1, 2], 3)
Return
[1,2,3]

Codec functions

decode_base64

fallible
Decodes the value (a Base64 string) into its original string.

Function spec

decode_base64(value: <string>, [charset: <string>])
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The Base64 data to decode.yes
charsetstring
The character set to use when decoding the data.standardno

Errors

The decode_base64 function is fallible, which means that error handling is required for these errors:
value isn’t a valid encoded Base64 string.

Examples

Decode Base64 data (default)
Source
decode_base64!("eW91IGhhdmUgc3VjY2Vzc2Z1bGx5IGRlY29kZWQgbWU=")
Return
you have successfully decoded me
Decode Base64 data (URL safe)
Source
decode_base64!("eW91IGNhbid0IG1ha2UgeW91ciBoZWFydCBmZWVsIHNvbWV0aGluZyBpdCB3b24ndA==", charset: "url_safe")
Return
you can't make your heart feel something it won't

decode_percent

infallible
Decodes a percent-encoded value like a URL.

Function spec

decode_percent(value: <string>)
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to decode.yes

Examples

Percent decode a value
Source
decode_percent("foo%20bar%3F")
Return
foo bar?

encode_base64

infallible
Encodes the value to Base64.

Function spec

encode_base64(value: <string>, [padding: <boolean>, charset: <string>])
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to encode.yes
paddingboolean
Whether the Base64 output is padded.trueno
charsetstring
The character set to use when encoding the data.standardno

Examples

Encode to Base64 (default)
Source
encode_base64("please encode me")
Return
cGxlYXNlIGVuY29kZSBtZQ==
Encode to Base64 (without padding)
Source
encode_base64("please encode me, no padding though", padding: false)
Return
cGxlYXNlIGVuY29kZSBtZSwgbm8gcGFkZGluZyB0aG91Z2g
Encode to Base64 (URL safe)
Source
encode_base64("please encode me, but safe for URLs", charset: "url_safe")
Return
cGxlYXNlIGVuY29kZSBtZSwgYnV0IHNhZmUgZm9yIFVSTHM=

encode_json

infallible
Encodes the value to JSON.

Function spec

encode_json(value: <any>)
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value to convert to a JSON string.yes

Examples

Encode to JSON
Source
.payload = encode_json({"hello": "world"})
Return
{"hello":"world"}

encode_key_value

fallible
Encodes the value to in key/value format with customizable delimiters. Default delimiters match the logfmt format.

Function spec

encode_key_value(value: <object>, [fields_ordering: <array>, key_value_delimiter: <string>, field_delimiter: <string>, flatten_boolean: <boolean>])
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueobject
The value to convert to a string.yes
fields_orderingarray
The ordering of fields to preserve. Any fields not in this list will appear unordered, after any ordered fields.no
key_value_delimiterstring
The string that separates the key from the value.=no
field_delimiterstring
The string that separates each key/value pair.no
flatten_booleanboolean
Whether to encode key/value with a boolean value as a standalone key if true and nothing if false.no

Notices

This function has special behavior that you should be aware of.
If fields_ordering is specified then the function is fallible else it is infallible.

Errors

The encode_key_value function is fallible, which means that error handling is required for these errors:
fields_ordering contains a non-string element

Examples

Encode with default delimiters (no ordering)
Source
encode_key_value({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})
Return
lvl=info msg="This is a message" ts=2021-06-05T17:20:00Z
Encode with default delimiters (fields ordering)
Source
encode_key_value!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])
Return
ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345
Encode with default delimiters (nested fields)
Source
encode_key_value({"agent": {"name": "vector"}, "log": {"file": {"path": "my.log"}}, "event": "log"})
Return
agent.name=vector event=log log.file.path=my.log
Encode with default delimiters (nested fields ordering)
Source
encode_key_value!({"agent": {"name": "vector"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])
Return
event=log log.file.path=my.log agent.name=vector
Encode with custom delimiters (no ordering)
Source
encode_key_value(
	{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"},
	field_delimiter: ",",
	key_value_delimiter: ":"
)
Return
lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00Z
Encode with custom delimiters and flatten boolean
Source
encode_key_value(
	{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "beta": true, "dropped": false},
	field_delimiter: ",",
	key_value_delimiter: ":",
	flatten_boolean: true
)
Return
beta,lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00Z

encode_logfmt

fallible
Encodes the value to logfmt.

Function spec

encode_logfmt(value: <object>, [fields_ordering: <array>])
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueobject
The value to convert to a logfmt string.yes
fields_orderingarray
The ordering of fields to preserve. Any fields not in this list will appear unordered, after any ordered fields.no

Notices

This function has special behavior that you should be aware of.
If fields_ordering is specified then the function is fallible else it is infallible.

Errors

The encode_logfmt function is fallible, which means that error handling is required for these errors:
fields_ordering contains a non-string element

Examples

Encode to logfmt (no ordering)
Source
encode_logfmt({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})
Return
lvl=info msg="This is a message" ts=2021-06-05T17:20:00Z
Encode to logfmt (fields ordering)
Source
encode_logfmt!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])
Return
ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345
Encode to logfmt (nested fields)
Source
encode_logfmt({"agent": {"name": "vector"}, "log": {"file": {"path": "my.log"}}, "event": "log"})
Return
agent.name=vector event=log log.file.path=my.log
Encode to logfmt (nested fields ordering)
Source
encode_logfmt!({"agent": {"name": "vector"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])
Return
event=log log.file.path=my.log agent.name=vector

encode_percent

infallible
Encodes a value with percent encoding to safely be used in URLs.

Function spec

encode_percent(value: <string>, [ascii_set: <string>])
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to encode.yes
ascii_setstring
The ascii set to use when encoding the data.NON_ALPHANUMERICno

Examples

Percent encode all non-alphanumeric characters (default)
Source
encode_percent("foo bar?")
Return
foo%20bar%3F
Percent encode only control characters
Source
encode_percent("foo 	bar", ascii_set: "CONTROLS")
Return
foo %09bar

Coerce functions

to_bool

fallible
Coerces the value into a boolean.

Function spec

to_bool(value: <boolean | integer | float | null | string>)
:: <boolean> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueboolean
integer
float
null
string
The value to convert to a Boolean.yes

Errors

The to_bool function is fallible, which means that error handling is required for these errors:
value isn’t a supported boolean representation

Examples

Coerce to a Boolean (string)
Source
to_bool!("yes")
Return
true
Coerce to a Boolean (float)
Source
to_bool(0.0)
Coerce to a Boolean (int)
Source
to_bool(0)
Coerce to a Boolean (null)
Source
to_bool(null)
Coerce to a Boolean (Boolean)
Source
to_bool(true)
Return
true

to_float

fallible
Coerces the value into a float.

Function spec

to_float(value: <integer | float | boolean | string | timestamp>)
:: <float> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
float
boolean
string
timestamp
The value to convert to a float. Must be convertible to a float, otherwise an error is raised.yes

Errors

The to_float function is fallible, which means that error handling is required for these errors:
value isn’t a supported float representation

Examples

Coerce to a float
Source
to_float!("3.145")
Return
3.145
Coerce to a float (timestamp)
Source
to_float(t'2020-12-30T22:20:53.824727Z')
Return
1609366853.824727

to_int

fallible
Coerces the value into an integer.

Function spec

to_int(value: <integer | float | boolean | string | timestamp>)
:: <integer> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
float
boolean
string
timestamp
The value to convert to an integer.yes

Errors

The to_int function is fallible, which means that error handling is required for these errors:
value is a string but the text is not an integer
value is not a string, int, or timestamp

Examples

Coerce to an int (string)
Source
to_int!("2")
Return
2
Coerce to an int (timestamp)
Source
to_int(t'2020-12-30T22:20:53.824727Z')
Return
1609366853

to_regex

fallible
Coerces the value into a regex.

Function spec

to_regex(value: <string>)
:: <regex> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The value to convert to a regex.yes

Notices

This function has special behavior that you should be aware of.
Compiling a regular expression is an expensive operation and can limit Vector throughput. Don’t use this function unless you are absolutely sure there is no other way!

Errors

The to_regex function is fallible, which means that error handling is required for these errors:
value is not a string.

Examples

Coerce to a regex
Source
to_regex("^foo$") ?? r''
Return
^foo$

to_string

fallible
Coerces the value into a string.

Function spec

to_string(value: <integer | float | boolean | string | timestamp | null>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
float
boolean
string
timestamp
null
The value to convert to a string.yes

Errors

The to_string function is fallible, which means that error handling is required for these errors:
value is not an integer, float, boolean, string, timestamp, or null

Examples

Coerce to a string (Boolean)
Source
to_string(true)
Return
true
Coerce to a string (int)
Source
to_string(52)
Return
52
Coerce to a string (float)
Source
to_string(52.2)
Return
52.2

to_timestamp

fallible
Coerces the value into a timestamp.

Function spec

to_timestamp(value: <string | float | integer | timestamp>)
:: <timestamp> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
float
integer
timestamp
The value that is to be converted to a timestamp. If a string, must be a valid representation of a timestamp, and no default exists, an ArgumentError will be raised.yes

Errors

The to_timestamp function is fallible, which means that error handling is required for these errors:
When value is a string, it is not a valid timestamp format
When value is an int, it is not within the Unix timestamp range
When value is a float, it is not within the Unix timestamp range

Examples

Coerce to a timestamp
Source
to_timestamp!("2020-10-21T16:00:00Z")
Return
2020-10-21T16:00:00Z

Convert functions

to_syslog_facility

fallible
Converts the value, a Syslog facility code, into its corresponding Syslog keyword. i.e. 0 into "kern", 1 into "user", etc.

Function spec

to_syslog_facility(value: <integer>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
The facility code.yes

Errors

The to_syslog_facility function is fallible, which means that error handling is required for these errors:
value isn’t a valid Syslog facility code.

Examples

Coerce to a Syslog facility
Source
to_syslog_facility!(4)
Return
auth

to_syslog_level

fallible
Converts the value, a Syslog severity level, into its corresponding keyword, i.e. 0 into "emerg", 1 into "alert", etc.

Function spec

to_syslog_level(value: <integer>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
The severity level.yes

Errors

The to_syslog_level function is fallible, which means that error handling is required for these errors:
value isn’t a valid Syslog severity level.

Examples

Coerce to a Syslog level
Source
to_syslog_level!(5)
Return
notice

to_syslog_severity

fallible
Converts the value, a Syslog log level keyword, into a Syslog integer severity level (0 to 7).

Function spec

to_syslog_severity(value: <string>)
:: <integer> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The Syslog level keyword to convert.yes

Errors

The to_syslog_severity function is fallible, which means that error handling is required for these errors:
value isn’t a valid Syslog level keyword

Examples

Coerce to Syslog severity
Source
to_syslog_severity!("alert")
Return
1

to_unix_timestamp

infallible

Converts the value timestamp into a Unix timestamp.

Returns the number of seconds since the Unix epoch by default, but milliseconds or nanoseconds can also be specified by unit.

Function spec

to_unix_timestamp(value: <timestamp>, [unit: <string>])
:: <integer>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuetimestamp
The timestamp to convert to Unix.yes
unitstring
The time unit.secondsno

Examples

Convert to a Unix timestamp (seconds)
Source
to_unix_timestamp(t'2021-01-01T00:00:00+00:00')
Return
1609459200
Convert to a Unix timestamp (milliseconds)
Source
to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "milliseconds")
Return
1609459200000
Convert to a Unix timestamp (nanoseconds)
Source
to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "nanoseconds")
Return
1609459200000000000

Debug functions

assert

fallible
Asserts the condition, which must be a Boolean expression. The program is aborted with message if the condition evaluates to false.

Function spec

assert(condition: <boolean>, [message: <string>])
:: <null> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
conditionboolean
The condition to check.yes
messagestring
An optional custom error message. If the equality assertion fails, message is appended to the default message prefix. See the examples below for a sample fully formed log message.no

Notices

This function has special behavior that you should be aware of.
The assert function should be used in a standalone fashion and only when you want to abort the program. You should avoid it in logical expressions and other situations in which you want the program to continue if the condition evaluates to false.

Errors

The assert function is fallible, which means that error handling is required for these errors:
condition evaluates to false

Examples

Assertion (true)
Source
ok, err = assert("foo" == "foo", message: "\"foo\" must be \"foo\"!")
Return
true
Assertion (false)
Source
assert!("foo" == "bar", message: "\"foo\" must be \"foo\"!")

assert_eq

infallible
Asserts that two expressions, left and right, have the same value. The program is aborted with the message if they are unequal.

Function spec

assert_eq(left: <any>, right: <any>, [message: <string>])
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
leftany
The value to check for equality against right.yes
rightany
The value to check for equality against left.yes
messagestring
An optional custom error message. If the equality assertion fails, message is appended to the default message prefix. See the examples below for a sample fully formed log message.no

Notices

This function has special behavior that you should be aware of.
The assert_eq function should be used in a standalone fashion and only when you want to abort the program. You should avoid it in logical expressions and other situations in which you want the program to continue if the condition evaluates to false.

Examples

Successful assertion
Source
assert_eq!(1, 1)
Return
true
Unsuccessful assertion
Source
assert_eq!(127, [1, 2, 3])
Unsuccessful assertion with custom log message
Source
 assert_eq!(1, 0, message: "Unequal integers")

log

infallible
Logs the value to Vector’s stdout at the specified level.

Function spec

log(value: <any>, [level: <string>, rate_limit_secs: <integer>])
:: <null>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value to log.yes
levelstring
The log level.infono
rate_limit_secsinteger
Specifies that the log message is output no more than once per the given number of seconds. Use a value of 0 to turn rate limiting off.1no

Examples

Log a message
Source
log("Hello, World!", level: "info", rate_limit_secs: 60)
Log an error
Source
ts, err = to_int(.field)
if err != null {
	log(err, level: "error")
}

Enumerate functions

compact

infallible
Compacts the value by removing “empty” values, where emptiness is defined using the available parameters.

Function spec

compact(value: <array | object>, [recursive: <boolean>, null: <boolean>, string: <boolean>, object: <boolean>, array: <boolean>, nullish: <boolean>])
:: <array | object>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
object
The object or array to compact.yes
recursiveboolean
Whether the compaction be recursive.trueno
nullboolean
Whether null should be treated as an empty value.trueno
stringboolean
Whether an empty string should be treated as an empty value.trueno
objectboolean
Whether an empty object should be treated as an empty value.trueno
arrayboolean
Whether an empty array should be treated as an empty value.trueno
nullishboolean
Tests whether the value is “nullish” as defined by the is_nullish function.no

Examples

Compact an array
Source
compact(["foo", "bar", "", null, [], "buzz"], string: true, array: true, null: true)
Return
["foo","bar","buzz"]
Compact an object
Source
compact({"field1": 1, "field2": "", "field3": [], "field4": null}, string: true, array: true, null: true)
Return
{
  "field1": 1
}

flatten

infallible
Flattens the value into a single-level representation.

Function spec

flatten(value: <array | object>)
:: <array | object>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
object
The array or object to flatten.yes

Examples

Flatten array
Source
flatten([1, [2, 3, 4], [5, [6, 7], 8], 9])
Return
[1,2,3,4,5,6,7,8,9]
Flatten object
Source
flatten({
	"parent1": {
		"child1": 1,
		"child2": 2
	},
	"parent2": {
		"child3": 3
	}
})
Return
{
  "parent1.child1": 1,
  "parent1.child2": 2,
  "parent2.child3": 3
}

includes

infallible
Determines whether the value array includes the specified item.

Function spec

includes(value: <array>, item: <any>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
The array.yes
itemany
The item to check.yes

Examples

Array includes
Source
includes(["apple", "orange", "banana"], "banana")
Return
true

length

infallible
Returns the length of the value.

Function spec

length(value: <array | object | string>)
:: <integer>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
object
string
The array or objectyes

Examples

Length (object)
Source
length({
	"portland": "Trail Blazers",
	"seattle":  "Supersonics"
})
Return
2
Length (nested object)
Source
length({
	"home": {
		"city":  "Portland",
		"state": "Oregon"
	},
	"name": "Trail Blazers",
	"mascot": {
		"name": "Blaze the Trail Cat"
	}
})
Return
3
Length (array)
Source
length(["Trail Blazers", "Supersonics", "Grizzlies"])
Return
3
Length (string)
Source
length("The Planet of the Apes Musical")
Return
30

match_array

infallible
Determines whether the elements in the value array matches the pattern - by default it checks at least one element matches, but can be set to determine if all the elements match.

Function spec

match_array(value: <array>, pattern: <regex>, [all: <boolean>])
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
The array.yes
patternregex
The regular expression pattern to match against.yes
allboolean
Whether to match on all elements of value.no

Examples

Match at least one element
Source
	match_array(["foobar", "bazqux"], r'foo')
Return
true
Match all elements
Source
	match_array(["foo", "foobar", "barfoo"], r'foo', all: true)
Return
true
No matches
Source
	match_array(["bazqux", "xyz"], r'foo')
Not all elements match
Source
	match_array(["foo", "foobar", "baz"], r'foo', all: true)

unique

infallible

Returns unique values for an array.

The first occurrence of each element is kept.

Function spec

unique(value: <array>)
:: <array>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
The array to return unique elements from.yes

Examples

Unique
Source
unique(["foo", "bar", "foo", "baz"])
Return
["foo","bar","baz"]

Path functions

del

infallible

Removes the field specified by the static path from the target.

For dynamic path deletion, see the remove function.

Function spec

del(path: <path>)
:: <any>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
pathpath
The path of the field to delete.yes

Notices

This function has special behavior that you should be aware of.
The del function modifies the current event in place and returns the value of the deleted field.

Examples

Delete a field
Source
del(.field1)
Rename a field
Source
.new_field = del(.old_field)

exists

infallible

Checks whether the path exists for the target.

This function allows you to distinguish between a missing path, or a path with a null value, something a regular path lookup such as .foo would not allow, since that always returns null if the path doesn’t exist.

Function spec

exists(path: <path>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
pathpath
The path of the field to check.yes

Examples

Exists (field)
Source
exists(.field)
Return
true
Exists (array element)
Source
exists(.array[2])
Return
true

get

fallible

Dynamically get the value of a given path.

When you know the path you want to look up, you should use static paths such as .foo.bar[1] to get the value of that path. However, when you don’t know the path names in advance, you can use this dynamic get function to get at the requested value.

Function spec

get(value: <object | array>, path: <array>)
:: <any> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueobject
array
The object or array to query.yes
patharray
An array of path segments to look up the value for.yes

Errors

The get function is fallible, which means that error handling is required for these errors:
path segment must be either “string” or “integer”

Examples

single-segment top-level field
Source
get!(value: { "foo": "bar" }, path: ["foo"])
Return
bar
multi-segment nested field
Source
get!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"])
Return
baz
array indexing
Source
get!(value: ["foo", "bar", "baz"], path: [-2])
Return
bar

remove

fallible

Dynamically remove the value for a given path.

When you know the path you want to remove, you should use the del function and static paths such as del(.foo.bar[1]) to remove the value at that path. The del function returns the deleted value, and is more performant than this function. However, when you don’t know the path names in advance, you can use this dynamic remove function to remove the value at the provided path.

Function spec

remove(value: <object | array>, path: <array>, [compact: <boolean>])
:: <object | array> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueobject
array
The object or array to remove data from.yes
patharray
An array of path segments to remove the value at.yes
compactboolean
Whether — after deletion — empty objects or arrays should be removed.no

Errors

The remove function is fallible, which means that error handling is required for these errors:
path segment must be either “string” or “integer”

Examples

single-segment top-level field
Source
remove!(value: { "foo": "bar" }, path: ["foo"])
multi-segment nested field
Source
remove!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"])
Return
{
  "foo": {}
}
array indexing
Source
remove!(value: ["foo", "bar", "baz"], path: [-2])
Return
["foo","baz"]
compaction
Source
remove!(value: { "foo": { "bar": [42], "baz": true } }, path: ["foo", "bar", 0], compact: true)
Return
{
  "foo": {
    "baz": true
  }
}

set

fallible

Dynamically insert data into the path of a given object or array.

When you know the path you want to assign a value to, you should use static path assignments such as .foo.bar[1] = true for improved performance and readability. However, when you don’t know the path names in advance, you can use this dynamic insertion function to insert the data into the object or array.

Function spec

set(value: <object | array>, path: <array>, data: <any>)
:: <object | array> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueobject
array
The object or array to insert data into.yes
patharray
An array of path segments to insert the value to.yes
dataany
The data to be inserted.yes

Errors

The set function is fallible, which means that error handling is required for these errors:
path segment must be either “string” or “integer”

Examples

single-segment top-level field
Source
set!(value: { "foo": "bar" }, path: ["foo"], data: "baz")
Return
{
  "foo": "baz"
}
multi-segment nested field
Source
set!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"], data: "qux")
Return
{
  "foo": {
    "bar": "qux"
  }
}
array
Source
set!(value: ["foo", "bar", "baz"], path: [-2], data: 42)
Return
["foo",42,"baz"]

Hash functions

md5

infallible
Calculates an md5 hash of the value.

Function spec

md5(value: <string>)
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to calculate the hash for.yes

Examples

Create md5 hash
Source
md5("foo")
Return
acbd18db4cc2f85cedef654fccc4a4d8

sha1

infallible
Calculates a SHA-1 hash of the value.

Function spec

sha1(value: <string>)
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to calculate the hash for.yes

Examples

Calculate sha1 hash
Source
sha1("foo")
Return
0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33

sha2

infallible
Calculates a SHA-2 hash of the value.

Function spec

sha2(value: <string>, [variant: <string>])
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to calculate the hash for.yes
variantstring
The variant of the algorithm to use.SHA-512/256no

Examples

Calculate sha2 hash
Source
sha2("foo", variant: "SHA-512/224")
Return
d68f258d37d670cfc1ec1001a0394784233f88f056994f9a7e5e99be

sha3

infallible
Calculates a SHA-3 hash of the value.

Function spec

sha3(value: <string>, [variant: <string>])
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to calculate the hash for.yes
variantstring
The variant of the algorithm to use.SHA3-512no

Examples

Calculate sha3 hash
Source
sha3("foo", variant: "SHA3-224")
Return
f4f6779e153c391bbd29c95e72b0708e39d9166c7cea51d1f10ef58a

IP functions

ip_aton

fallible

Converts IPv4 address in numbers-and-dots notation into network-order bytes represented as an integer.

This behavior mimics inet_aton.

Function spec

ip_aton(value: <string>)
:: <integer> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The IP address to convert to binary.yes

Errors

The ip_aton function is fallible, which means that error handling is required for these errors:
value isn’t a valid IPv4 address

Examples

IPv4 to integer
Source
ip_aton!("1.2.3.4")
Return
16909060

ip_cidr_contains

fallible
Determines whether the ip is contained in the block referenced by the cidr.

Function spec

ip_cidr_contains(cidr: <string>, ip: <string>)
:: <boolean> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
cidrstring
The CIDR mask (v4 or v6).yes
ipstring
The IP address (v4 or v6).yes

Errors

The ip_cidr_contains function is fallible, which means that error handling is required for these errors:
cidr isn’t a valid CIDR
ip isn’t a valid IP address

Examples

IPv4 contains CIDR
Source
ip_cidr_contains!("192.168.0.0/16", "192.168.10.32")
Return
true
IPv6 contains CIDR
Source
ip_cidr_contains!("2001:4f8:4:ba::/64", "2001:4f8:4:ba:2e0:81ff:fe22:d1f1")
Return
true

ip_ntoa

fallible

Converts numeric representation of IPv4 address in network-order bytes to numbers-and-dots notation..

This behavior mimics inet_ntoa.

Function spec

ip_ntoa(value: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The integer representation of an IPv4 address.yes

Errors

The ip_ntoa function is fallible, which means that error handling is required for these errors:
value cannot fit in u32

Examples

Integer to IPv4
Source
ip_ntoa!(16909060)
Return
1.2.3.4

ip_subnet

fallible
Extracts the subnet address from the ip using the supplied subnet.

Function spec

ip_subnet(ip: <string>, subnet: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
ipstring
The IP address (v4 or v6).yes
subnetstring
The subnet to extract from the IP address. This can be either a prefix length like /8 or a net mask like 255.255.0.0. The net mask can be either an IPv4 or IPv6 address.yes

Notices

This function has special behavior that you should be aware of.
Works with both IPv4 and IPv6 addresses. The IP version for the mask must be the same as the supplied address.

Errors

The ip_subnet function is fallible, which means that error handling is required for these errors:
ip isn’t a valid IP address
subnet isn’t a valid subnet.

Examples

IPv4 subnet
Source
ip_subnet!("192.168.10.32", "255.255.255.0")
Return
192.168.10.0
IPv6 subnet
Source
ip_subnet!("2404:6800:4003:c02::64", "/32")
Return
2404:6800::

ip_to_ipv6

fallible
Converts the ip to an IPv6 address.

Function spec

ip_to_ipv6(ip: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
ipstring
The IP address to convert to IPv6.yes

Errors

The ip_to_ipv6 function is fallible, which means that error handling is required for these errors:
ip isn’t a valid IP address

Examples

IPv4 to IPv6
Source
ip_to_ipv6!("192.168.10.32")
Return
::ffff:192.168.10.32

ipv6_to_ipv4

fallible
Converts the ip to an IPv4 address. ip is returned unchanged if it’s already an IPv4 address. If ip is currently an IPv6 address then it needs to be IPv4 compatible, otherwise an error is thrown.

Function spec

ipv6_to_ipv4(ip: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
ipstring
The IPv4-mapped IPv6 address to convert.yes

Errors

The ipv6_to_ipv4 function is fallible, which means that error handling is required for these errors:
ip isn’t a valid IP address
ip is an IPv6 address that isn’t compatible with IPv4

Examples

IPv6 to IPv4
Source
ipv6_to_ipv4!("::ffff:192.168.0.1")
Return
192.168.0.1

Number functions

ceil

infallible
Rounds the value up to the specified precision.

Function spec

ceil(value: <integer | float>, [precision: <integer>])
:: <integer | float>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
float
The number to round up.yes
precisioninteger
The number of decimal places to round to.no

Examples

Round a number up (without precision)
Source
ceil(4.345)
Return
5
Round a number up (with precision)
Source
ceil(4.345, precision: 2)
Return
4.35

floor

infallible
Rounds the value down to the specified precision.

Function spec

floor(value: <integer | float>, [precision: <integer>])
:: <integer | float>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
float
The number to round down.yes
precisioninteger
The number of decimal places to round to.no

Examples

Round a number down (without precision)
Source
floor(4.345)
Return
4
Round a number down (with precision)
Source
floor(4.345, precision: 2)
Return
4.34

format_int

fallible
Formats the integer value into a string representation using the given base/radix.

Function spec

format_int(value: <integer>, [base: <integer>])
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
The number to format.yes
baseinteger
The base to format the number in. Must be between 2 and 36 (inclusive).10no

Errors

The format_int function is fallible, which means that error handling is required for these errors:
base is not between 2 and 36

Examples

Format as a hexadecimal integer
Source
format_int!(42, 16)
Return
2a
Format as a negative hexadecimal integer
Source
format_int!(-42, 16)
Return
-2a

format_number

infallible
Formats the value into a string representation of the number.

Function spec

format_number(value: <integer | float>, [scale: <integer>, decimal_separator: <string>, grouping_separator: <string>])
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
float
The number to format as a string.yes
scaleinteger
The number of decimal places to display.no
decimal_separatorstring
The character to use between the whole and decimal parts of the number..no
grouping_separatorstring
The character to use between each thousands part of the number.,no

Examples

Format a number (3 decimals)
Source
format_number(1234567.89, 3, decimal_separator: ".", grouping_separator: ",")
Return
1,234,567.890

round

infallible
Rounds the value to the specified precision.

Function spec

round(value: <integer | float>, [precision: <integer>])
:: <integer | float>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
float
The number to round.yes
precisioninteger
The number of decimal places to round to.no

Examples

Round a number (without precision)
Source
round(4.345)
Return
4
Round a number (with precision)
Source
round(4.345, precision: 2)
Return
4.35

Object functions

match_datadog_query

infallible
Matches an object against a Datadog Search Syntax query.

Function spec

match_datadog_query(value: <object>, query: <string>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueobject
The object.yes
querystring
The Datadog Search Syntax query.yes

Examples

OR query
Source
match_datadog_query({"message": "contains this and that"}, "this OR that")
Return
true
AND query
Source
match_datadog_query({"message": "contains only this"}, "this AND that")
Facet wildcard
Source
match_datadog_query({"custom": {"name": "vector"}}, "@name:vec*")
Return
true
Tag range
Source
match_datadog_query({"tags": ["a:x", "b:y", "c:z"]}, s'b:["x" TO "z"]')
Return
true

merge

infallible
Merges the from object into the to object.

Function spec

merge(to: <string>, from: <object>, [deep: <boolean>])
:: <object>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
tostring
The object to merge into.yes
fromobject
The object to merge from.yes
deepboolean
A deep merge is performed if true, otherwise only top-level fields are merged.no

Examples

Object merge (shallow)
Source
merge(
	{
		"parent1": {
			"child1": 1,
			"child2": 2
		},
		"parent2": {
			"child3": 3
		}
	},
	{
		"parent1": {
			"child2": 4,
			"child5": 5
		}
	}
)
Return
{
  "parent1": {
    "child2": 4,
    "child5": 5
  },
  "parent2": {
    "child3": 3
  }
}
Object merge (deep)
Source
merge(
	{
		"parent1": {
			"child1": 1,
			"child2": 2
		},
		"parent2": {
			"child3": 3
		}
	},
	{
		"parent1": {
			"child2": 4,
			"child5": 5
		}
	},
	deep: true
)
Return
{
  "parent1": {
    "child1": 1,
    "child2": 4,
    "child5": 5
  },
  "parent2": {
    "child3": 3
  }
}

unnest

fallible

Unnest an array field from an object to create an array of objects using that field; keeping all other fields.

Assigning the array result of this to . will result in multiple events being emitted from remap. See the remap transform docs for more details.

This is also referred to as explodeing in some languages.

Function spec

unnest(path: <path>)
:: <array> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
pathpath
The path of the field to unnest.yes

Errors

The unnest function is fallible, which means that error handling is required for these errors:
Field path refers to is not an array

Examples

Unnest an array field
Source
. = unnest!(.messages)
Unnest nested an array field
Source
. = unnest!(.event.messages)

Parse functions

parse_apache_log

fallible
Parses Apache access and error log lines. Lines can be in common, combined, or default error format.

Function spec

parse_apache_log(value: <string>, format: <string>, [timestamp_format: <string>])
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes
formatstring
The format to use for parsing the log.yes
timestamp_formatstring
The date/time format to use for encoding the timestamp. The time is parsed in local time if the timestamp doesn’t specify a timezone.%d/%b/%Y:%T %zno

Notices

This function has special behavior that you should be aware of.
Missing information in the log message may be indicated by -. These fields are omitted in the result.

Errors

The parse_apache_log function is fallible, which means that error handling is required for these errors:
value doesn’t match the specified format
timestamp_format isn’t a valid format string
The timestamp in value fails to parse using the provided timestamp_format

Examples

Parse via Apache log format (common)
Source
parse_apache_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326", format: "common")
Return
{
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}
Parse via Apache log format (combined)
Source
parse_apache_log!(
	s'127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth" "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"',
	"combined",
)
Return
{
  "agent": "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0",
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "referrer": "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}
Parse via Apache log format (error)
Source
parse_apache_log!(
	s'[01/Mar/2021:12:00:19 +0000] [ab:alert] [pid 4803:tid 3814] [client 147.159.108.175:24259] I will bypass the haptic COM bandwidth, that should matrix the CSS driver!',
	"error"
)
Return
{
  "client": "147.159.108.175",
  "message": "I will bypass the haptic COM bandwidth, that should matrix the CSS driver!",
  "module": "ab",
  "pid": 4803,
  "port": 24259,
  "severity": "alert",
  "thread": "3814",
  "timestamp": "2021-03-01T12:00:19Z"
}

parse_aws_alb_log

fallible

Function spec

parse_aws_alb_log(value: <string>)
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
Access log of the Application Load Balancer.yes

Errors

The parse_aws_alb_log function is fallible, which means that error handling is required for these errors:
value isn’t a properly formatted AWS ALB log

Examples

Parse AWS ALB log
Source
parse_aws_alb_log!(
	"http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\""
)
Return
{
  "actions_executed": "forward",
  "chosen_cert_arn": null,
  "classification": null,
  "classification_reason": null,
  "client_host": "192.168.131.39:2817",
  "domain_name": null,
  "elb": "app/my-loadbalancer/50dc6c495c0c9188",
  "elb_status_code": "200",
  "error_reason": null,
  "matched_rule_priority": "0",
  "received_bytes": 34,
  "redirect_url": null,
  "request_creation_time": "2018-11-30T22:22:48.364000Z",
  "request_method": "GET",
  "request_processing_time": 0,
  "request_protocol": "HTTP/1.1",
  "request_url": "http://www.example.com:80/",
  "response_processing_time": 0,
  "sent_bytes": 366,
  "ssl_cipher": null,
  "ssl_protocol": null,
  "target_group_arn": "arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067",
  "target_host": null,
  "target_port_list": [],
  "target_processing_time": 0.001,
  "target_status_code": "200",
  "target_status_code_list": [],
  "timestamp": "2018-11-30T22:23:00.186641Z",
  "trace_id": "Root=1-58337364-23a8c76965a2ef7629b185e3",
  "type": "http",
  "user_agent": "curl/7.46.0"
}

parse_aws_cloudwatch_log_subscription_message

fallible
Parses AWS CloudWatch Logs events (configured through AWS Cloudwatch subscriptions) from the aws_kinesis_firehose source.

Function spec

parse_aws_cloudwatch_log_subscription_message(value: <string>)
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string representation of the message to parse.yes

Errors

The parse_aws_cloudwatch_log_subscription_message function is fallible, which means that error handling is required for these errors:
value isn’t a properly formatted AWS Cloudwatch Log subscription message

Examples

Parse AWS Cloudwatch Log subscription message
Source
parse_aws_cloudwatch_log_subscription_message!(.message)
Return
{
  "log_events": [
    {
      "id": "35683658089614582423604394983260738922885519999578275840",
      "message": "{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}",
      "timestamp": "2020-09-14T19:09:29.039Z"
    }
  ],
  "log_group": "test",
  "log_stream": "test",
  "message_type": "DATA_MESSAGE",
  "owner": "111111111111",
  "subscription_filters": [
    "Destination"
  ]
}

parse_aws_vpc_flow_log

fallible
Parses value in the VPC Flow Logs format.

Function spec

parse_aws_vpc_flow_log(value: <string>, [format: <string>])
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
VPC Flow Log.yes
formatstring
VPC Flow Log format.no

Errors

The parse_aws_vpc_flow_log function is fallible, which means that error handling is required for these errors:
value isn’t a properly formatted AWS VPC Flow log

Examples

Parse AWS VPC Flow log (default format)
Source
parse_aws_vpc_flow_log!("2 123456789010 eni-1235b8ca123456789 - - - - - - - 1431280876 1431280934 - NODATA")
Return
{
  "account_id": 123456789010,
  "action": null,
  "bytes": null,
  "dstaddr": null,
  "dstport": null,
  "end": 1431280934,
  "interface_id": "eni-1235b8ca123456789",
  "log_status": "NODATA",
  "packets": null,
  "protocol": null,
  "srcaddr": null,
  "srcport": null,
  "start": 1431280876,
  "version": 2
}
Parse AWS VPC Flow log (custom format)
Source
parse_aws_vpc_flow_log!(
	"- eni-1235b8ca123456789 10.0.1.5 10.0.0.220 10.0.1.5 203.0.113.5",
	"instance_id interface_id srcaddr dstaddr pkt_srcaddr pkt_dstaddr"
)
Return
{
  "dstaddr": "10.0.0.220",
  "instance_id": null,
  "interface_id": "eni-1235b8ca123456789",
  "pkt_dstaddr": "203.0.113.5",
  "pkt_srcaddr": "10.0.1.5",
  "srcaddr": "10.0.1.5"
}

parse_common_log

fallible
Parses the value using the Common Log Format (CLF).

Function spec

parse_common_log(value: <string>, [timestamp_format: <string>])
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes
timestamp_formatstring
The date/time format to use for encoding the timestamp.%d/%b/%Y:%T %zno

Notices

This function has special behavior that you should be aware of.
Missing information in the log message may be indicated by -. These fields are omitted in the result.

Errors

The parse_common_log function is fallible, which means that error handling is required for these errors:
value doesn’t match the Common Log Format
timestamp_format isn’t a valid format string
The timestamp in value fails to parse using the provided timestamp_format

Examples

Parse via Common Log Format (with default timestamp format)
Source
parse_common_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326")
Return
{
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}
Parse via Common Log Format (with custom timestamp format)
Source
parse_common_log!(
	"127.0.0.1 bob frank [2000-10-10T20:55:36Z] \"GET /apache_pb.gif HTTP/1.0\" 200 2326",
	"%+"
)
Return
{
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}

parse_csv

fallible
Parses a single CSV formatted row. Only the first row is parsed in case of multiline input value.

Function spec

parse_csv(value: <string>, [delimiter: <string>])
:: <array> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes
delimiterstring
The field delimiter to use when parsing. Must be a single-byte utf8 character.,no

Notices

This function has special behavior that you should be aware of.
All values are returned as strings. We recommend manually coercing values to desired types as you see fit.

Errors

The parse_csv function is fallible, which means that error handling is required for these errors:
delimiter must be a single-byte utf8 character
value isn’t a valid CSV string

Examples

Parse a single CSV formatted row
Source
parse_csv!("foo,bar,\"foo \"\", bar\"")
Return
["foo","bar","foo \", bar"]
Parse a single CSV formatted row with custom delimiter
Source
parse_csv!("foo bar", delimiter: " ")
Return
["foo","bar"]

parse_duration

fallible
Parses the value into a human-readable duration format specified by unit.

Function spec

parse_duration(value: <string>, unit: <string>)
:: <float> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string of the duration.yes
unitstring
The output units for the duration.yes

Errors

The parse_duration function is fallible, which means that error handling is required for these errors:
value isn’t a properly formatted duration

Examples

Parse duration (milliseconds)
Source
parse_duration!("1005ms", unit: "s")
Return
1.005

parse_glog

fallible
Parses the value using the glog (Google Logging Library) format.

Function spec

parse_glog(value: <string>)
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes

Errors

The parse_glog function is fallible, which means that error handling is required for these errors:
value doesn’t match the glog format

Examples

Parse via glog
Source
parse_glog!("I20210131 14:48:54.411655 15520 main.c++:9] Hello world!")
Return
{
  "file": "main.c++",
  "id": 15520,
  "level": "info",
  "line": 9,
  "message": "Hello world!",
  "timestamp": "2021-01-31T14:48:54.411655Z"
}

parse_grok

fallible
Parses the value using the grok format. All patterns listed here are supported.

Function spec

parse_grok(value: <string>, pattern: <string>, [remove_empty: <boolean>])
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes
patternstring
The Grok pattern.yes
remove_emptyboolean
If set to true, any patterns that resolve to an empty value are removed from the result.trueno

Notices

This function has special behavior that you should be aware of.
We recommend using community-maintained Grok patterns when possible, as they’re more likely to be properly vetted and improved over time than bespoke patterns.

Errors

The parse_grok function is fallible, which means that error handling is required for these errors:
value fails to parse using the provided pattern

Examples

Parse using Grok
Source
parse_grok!(
	"2020-10-02T23:22:12.223222Z info Hello world",
	"%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"
)
Return
{
  "level": "info",
  "message": "Hello world",
  "timestamp": "2020-10-02T23:22:12.223222Z"
}

parse_int

fallible
Parses the string value representing a number in an optional base/radix to an integer.

Function spec

parse_int(value: <string>, [base: <integer>])
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes
baseinteger

The base the number is in. Must be between 2 and 36 (inclusive).

If unspecified, will use the string prefix to try to determine the base: “0b”, 8 for “0” or “0o”, 16 for “0x”, and 10 otherwise

no

Errors

The parse_int function is fallible, which means that error handling is required for these errors:
base is not between 2 and 36
number cannot be parsed in the base

Examples

Parse decimal
Source
parse_int!("-42")
Return
-42
Parse binary
Source
parse_int!("0b1001")
Return
9
Parse octal
Source
parse_int!("0o42")
Return
34
Parse hexadecimal
Source
parse_int!("0x2a")
Return
42
Parse explicit base
Source
parse_int!("2a", 17)
Return
44

parse_json

fallible
Parses the value as JSON.

Function spec

parse_json(value: <string>)
:: <boolean | integer | float | string | object | array | null> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string representation of the JSON to parse.yes

Notices

This function has special behavior that you should be aware of.
Only JSON types are returned. If you need to convert a string into a timestamp, consider the parse_timestamp function.

Errors

The parse_json function is fallible, which means that error handling is required for these errors:
value isn’t a valid JSON-formatted payload

Examples

Parse JSON
Source
parse_json!("{\"key\": \"val\"}")
Return
{
  "key": "val"
}

parse_key_value

fallible

Parses the value in key/value format. Also known as logfmt.

  • Keys and values can be wrapped with ".
  • " characters can be escaped using \.

Function spec

parse_key_value(value: <string>, [key_value_delimiter: <string>, field_delimiter: <string>, whitespace: <string>, accept_standalone_key: <boolean>])
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes
key_value_delimiterstring
The string that separates the key from the value.=no
field_delimiterstring
The string that separates each key/value pair.no
whitespacestring
Defines the acceptance of unnecessary whitespace surrounding the configured key_value_delimiter.lenientno
accept_standalone_keyboolean
Whether a standalone key should be accepted, the resulting object will associate such keys with boolean value truetrueno

Notices

This function has special behavior that you should be aware of.
All values are returned as strings. We recommend manually coercing values to desired types as you see fit.

Errors

The parse_key_value function is fallible, which means that error handling is required for these errors:
value isn’t a properly formatted key/value string

Examples

Parse logfmt log
Source
parse_key_value!(
	"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
Return
{
  "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
  "id": "ConsumerFetcherManager-1382721708341",
  "level": "info",
  "module": "kafka.consumer.ConsumerFetcherManager",
  "msg": "Stopping all fetchers",
  "tag#production": "stopping_fetchers"
}
Parse comma delimited log
Source
parse_key_value!(
	"path:\"/cart_link\", host:store.app.com, fwd: \"102.30.171.16\", dyno: web.1, connect:0ms, service:87ms, status:304, bytes:632, protocol:https",
	field_delimiter: ",",
	key_value_delimiter: ":"
)
Return
{
  "bytes": "632",
  "connect": "0ms",
  "dyno": "web.1",
  "fwd": "102.30.171.16",
  "host": "store.app.com",
  "path": "/cart_link",
  "protocol": "https",
  "service": "87ms",
  "status": "304"
}
Parse comma delimited log with standalone keys
Source
parse_key_value!(
	"env:prod,service:backend,region:eu-east1,beta",
	field_delimiter: ",",
	key_value_delimiter: ":",
)
Return
{
  "beta": true,
  "env": "prod",
  "region": "eu-east1",
  "service": "backend"
}

parse_klog

fallible
Parses the value using the klog format used by Kubernetes components.

Function spec

parse_klog(value: <string>)
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes

Errors

The parse_klog function is fallible, which means that error handling is required for these errors:
value doesn’t match the klog format

Examples

Parse via klog
Source
parse_klog!("I0505 17:59:40.692994   28133 klog.go:70] hello from klog")
Return
{
  "file": "klog.go",
  "id": 28133,
  "level": "info",
  "line": 70,
  "message": "hello from klog",
  "timestamp": "2021-05-05T17:59:40.692994Z"
}

parse_linux_authorization

fallible
Parses Linux authorization logs usually found under either /var/log/auth.log (for Debian-based systems) or /var/log/secure (for RedHat-based systems) according to Syslog format.

Function spec

parse_linux_authorization(value: <string>)
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The text containing the message to parse.yes

Notices

This function has special behavior that you should be aware of.
The function resolves the year for messages that don’t include it. If the current month is January, and the message is for December, it will take the previous year. Otherwise, take the current year.

Errors

The parse_linux_authorization function is fallible, which means that error handling is required for these errors:
value isn’t a properly formatted Syslog message

Examples

Parse Linux authorization event
Source
parse_linux_authorization!(
	s'Mar 23 01:49:58 localhost sshd[1111]: Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar'
)
Return
{
  "appname": "sshd",
  "hostname": "localhost",
  "message": "Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar",
  "procid": 1111,
  "timestamp": "2021-03-23T01:49:58Z"
}

parse_logfmt

fallible

Parses the value in logfmt.

  • Keys and values can be wrapped using the " character.
  • " characters can be escaped by the \ character.
  • As per this logfmt specification, the parse_logfmt function accepts standalone keys and assigns them a Boolean value of true.

Function spec

parse_logfmt(value: <string>)
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes

Notices

This function has special behavior that you should be aware of.
If fields_ordering is specified then the function is fallible else it is infallible.

Errors

The parse_logfmt function is fallible, which means that error handling is required for these errors:
value isn’t a properly formatted key/value string

Examples

Parse logfmt log
Source
parse_logfmt!(
	"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
Return
{
  "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
  "id": "ConsumerFetcherManager-1382721708341",
  "level": "info",
  "module": "kafka.consumer.ConsumerFetcherManager",
  "msg": "Stopping all fetchers",
  "tag#production": "stopping_fetchers"
}

parse_nginx_log

fallible
Parses Nginx access and error log lines. Lines can be in combined, or error format.

Function spec

parse_nginx_log(value: <string>, format: <string>, [timestamp_format: <string>])
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes
formatstring
The format to use for parsing the log.yes
timestamp_formatstring
The date/time format to use for encoding the timestamp. The time is parsed in local time if the timestamp doesn’t specify a timezone. The default format is %d/%b/%Y:%T %z for combined logs and %Y/%m/%d %H:%M:%S for error logs.%d/%b/%Y:%T %zno

Notices

This function has special behavior that you should be aware of.
Missing information in the log message may be indicated by -. These fields are omitted in the result.

Errors

The parse_nginx_log function is fallible, which means that error handling is required for these errors:
value doesn’t match the specified format
timestamp_format isn’t a valid format string
The timestamp in value fails to parse using the provided timestamp_format

Examples

Parse via Nginx log format (combined)
Source
parse_nginx_log!(
    s'172.17.0.1 alice - [01/Apr/2021:12:02:31 +0000] "POST /not-found HTTP/1.1" 404 153 "http://localhost/somewhere" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"',
    "combined",
)
Return
{
  "agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36",
  "client": "172.17.0.1",
  "compression": "2.75",
  "method": "POST",
  "path": "/not-found",
  "protocol": "HTTP/1.1",
  "referer": "http://localhost/somewhere",
  "request": "POST /not-found HTTP/1.1",
  "size": 153,
  "status": 404,
  "timestamp": "2021-04-01T12:02:31Z",
  "user": "alice"
}
Parse via Nginx log format (error)
Source
parse_nginx_log!(
    s'2021/04/01 13:02:31 [error] 31#31: *1 open() "/usr/share/nginx/html/not-found" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "POST /not-found HTTP/1.1", host: "localhost:8081"',
    "error"
)
Return
{
  "cid": 1,
  "client": "172.17.0.1",
  "host": "localhost:8081",
  "message": "open() \"/usr/share/nginx/html/not-found\" failed (2: No such file or directory)",
  "pid": 31,
  "request": "POST /not-found HTTP/1.1",
  "server": "localhost",
  "severity": "error",
  "tid": 31,
  "timestamp": "2021-04-01T13:02:31Z"
}

parse_query_string

infallible
Parses the value as a query string.

Function spec

parse_query_string(value: <string>)
:: <object>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes

Notices

This function has special behavior that you should be aware of.
All values are returned as strings. We recommend manually coercing values to desired types as you see fit. Empty keys and values are allowed.

Examples

Parse query string
Source
parse_query_string("foo=%2B1&bar=2&bar=3&xyz")
Return
{
  "bar": [
    "2",
    "3"
  ],
  "foo": "+1",
  "xyz": ""
}
Parse Ruby on Rails' query string
Source
parse_query_string("?foo%5b%5d=1&foo%5b%5d=2")
Return
{
  "foo[]": [
    "1",
    "2"
  ]
}

parse_regex

fallible

Parses the value via the provided Regex pattern.

This function differs from the parse_regex_all function in that it returns only the first match.

Function spec

parse_regex(value: <string>, pattern: <regex>, [numeric_groups: <regex>])
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to search.yes
patternregex
The regular expression pattern to search against.yes
numeric_groupsregex
If true, the index of each group in the regular expression is also captured. The 0th index will contain the whole match.no

Notices

This function has special behavior that you should be aware of.
VRL aims to provide purpose-specific parsing functions for common log formats. Before reaching for the parse_regex function, see if a VRL parse_* function already exists for your format. If not, we recommend opening an issue to request support for the desired format.
All values are returned as strings. We recommend manually coercing values to desired types as you see fit.

Errors

The parse_regex function is fallible, which means that error handling is required for these errors:
value fails to parse using the provided pattern

Examples

Parse using Regex (with capture groups)
Source
parse_regex!("first group and second group.", r'(?P<number>.*?) group')
Return
{
  "number": "first"
}
Parse using Regex (without capture groups)
Source
parse_regex!("first group and second group.", r'(\w+) group', numeric_groups: true)
Return
{
  "0": "first group",
  "1": "first"
}

parse_regex_all

fallible

Parses the value via the provided Regex pattern.

This function differs from the parse_regex function in that it returns all matches, not just the first.

Function spec

parse_regex_all(value: <string>, pattern: <regex>, [numeric_groups: <regex>])
:: <array> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to search.yes
patternregex
The regular expression pattern to search against.yes
numeric_groupsregex
If true, the index of each group in the regular expression is also captured. The 0th index contains the whole match.no

Notices

This function has special behavior that you should be aware of.
VRL aims to provide purpose-specific parsing functions for common log formats. Before reaching for the parse_regex function, see if a VRL parse_* function already exists for your format. If not, we recommend opening an issue to request support for the desired format.
All values are returned as strings. We recommend manually coercing values to desired types as you see fit.

Errors

The parse_regex_all function is fallible, which means that error handling is required for these errors:
value fails to parse via the provided pattern

Examples

Parse using Regex (all matches)
Source
parse_regex_all!("first group and second group.", r'(?P<number>\w+) group', numeric_groups: true)
Return
[{"0":"first group","1":"first","number":"first"},{"0":"second group","1":"second","number":"second"}]

parse_ruby_hash

fallible
Parses the value as ruby hash.

Function spec

parse_ruby_hash(value: <string>)
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string representation of the ruby hash to parse.yes

Notices

This function has special behavior that you should be aware of.
Only ruby types are returned. If you need to convert a string into a timestamp, consider the parse_timestamp function.

Errors

The parse_ruby_hash function is fallible, which means that error handling is required for these errors:
value isn’t a valid ruby hash formatted payload

Examples

Parse ruby hash
Source
parse_ruby_hash!(s'{ "test" => "value", "testNum" => 0.2, "testObj" => { "testBool" => true, "testNull" => nil } }')
Return
{
  "test": "value",
  "testNum": 0.2,
  "testObj": {
    "testBool": true,
    "testNull": null
  }
}

parse_syslog

fallible
Parses the value in Syslog format.

Function spec

parse_syslog(value: <string>)
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The text containing the Syslog message to parse.yes

Notices

This function has special behavior that you should be aware of.
The function makes a best effort to parse the various Syslog formats that exists out in the wild. This includes RFC 6587, RFC 5424, RFC 3164, and other common variations (such as the Nginx Syslog style).
All values are returned as strings. We recommend manually coercing values to desired types as you see fit.

Errors

The parse_syslog function is fallible, which means that error handling is required for these errors:
value isn’t a properly formatted Syslog message

Examples

Parse Syslog log (5424)
Source
parse_syslog!(
	s'<13>1 2020-03-13T20:45:38.119Z dynamicwireless.name non 2426 ID931 [exampleSDID@32473 iut="3" eventSource= "Application" eventID="1011"] Try to override the THX port, maybe it will reboot the neural interface!'
)
Return
{
  "appname": "non",
  "exampleSDID@32473.eventID": "1011",
  "exampleSDID@32473.eventSource": "Application",
  "exampleSDID@32473.iut": "3",
  "facility": "user",
  "hostname": "dynamicwireless.name",
  "message": "Try to override the THX port, maybe it will reboot the neural interface!",
  "msgid": "ID931",
  "procid": 2426,
  "severity": "notice",
  "timestamp": "2020-03-13T20:45:38.119Z",
  "version": 1
}

parse_timestamp

fallible
Parses the value in strptime format.

Function spec

parse_timestamp(value: <string>, format: <string>)
:: <timestamp> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The text of the timestamp.yes
formatstring
The strptime format.yes

Errors

The parse_timestamp function is fallible, which means that error handling is required for these errors:
value fails to parse using the provided format

Examples

Parse timestamp
Source
parse_timestamp!("10-Oct-2020 16:00+00:00", format: "%v %R %:z")
Return
2020-10-10T16:00:00Z

parse_tokens

fallible

Parses the value in “token” format. A token is considered to be one of the following:

  • A word surrounded by whitespace.
  • Text delimited by double quotes: "..". Quotes can be included in the token if they are escaped by a backslash (\).
  • Text delimited by square brackets: [..]. Closing square brackets can be included in the token if they are escaped by a backslash (\).

Function spec

parse_tokens(value: <string>)
:: <array> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to tokenize.yes

Notices

This function has special behavior that you should be aware of.
All token values are returned as strings. We recommend manually coercing values to desired types as you see fit.

Errors

The parse_tokens function is fallible, which means that error handling is required for these errors:
value isn’t a properly formatted tokenized string

Examples

Parse tokens
Source
parse_tokens(
	"A sentence \"with \\\"a\\\" sentence inside\" and [some brackets]"
)
Return
["A","sentence","with \\\"a\\\" sentence inside","and","some brackets"]

parse_url

fallible
Parses the value in URL format.

Function spec

parse_url(value: <string>, [default_known_ports: <boolean>])
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The text of the URL.yes
default_known_portsboolean
If true and the port number is not specified in the input URL string (or matches the default port for the scheme), it will be populated from well-known ports for the following schemes: http, https, ws, wss, and ftp.no

Errors

The parse_url function is fallible, which means that error handling is required for these errors:
value isn’t a properly formatted URL

Examples

Parse URL
Source
parse_url!("ftp://foo:bar@vector.dev:4343/foobar?hello=world#123")
Return
{
  "fragment": "123",
  "host": "vector.dev",
  "password": "bar",
  "path": "/foobar",
  "port": 4343,
  "query": {
    "hello": "world"
  },
  "scheme": "ftp",
  "username": "foo"
}
Parse URL with default port
Source
parse_url!("https://vector.dev", default_known_ports: true)
Return
{
  "fragment": null,
  "host": "vector.dev",
  "password": "",
  "path": "/",
  "port": 443,
  "query": {},
  "scheme": "https",
  "username": ""
}

parse_user_agent

infallible
Parses the value as a user agent string. Which has a loosely defined format so this parser only provides best effort guarantee.

Function spec

parse_user_agent(value: <string>, [mode: <string>])
:: <object>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes
modestring
Determines performance and reliability characteristics.fastno

Notices

This function has special behavior that you should be aware of.
All values are returned as strings or as null. We recommend manually coercing values to desired types as you see fit.
Different modes return different schema.
Field which were not parsed out are set as null.

Examples

Fast mode
Source
parse_user_agent(
	"Mozilla Firefox 1.0.1 Mozilla/5.0 (X11; U; Linux i686; de-DE; rv:1.7.6) Gecko/20050223 Firefox/1.0.1"
)
Return
{
  "browser": {
    "family": "Firefox",
    "version": "1.0.1"
  },
  "device": {
    "category": "pc"
  },
  "os": {
    "family": "Linux",
    "version": null
  }
}
Reliable mode
Source
parse_user_agent(
	"Mozilla/4.0 (compatible; MSIE 7.66; Windows NT 5.1; SV1; .NET CLR 1.1.4322)",
	mode: "reliable"
)
Return
{
  "browser": {
    "family": "Internet Explorer",
    "version": "7.66"
  },
  "device": {
    "category": "pc"
  },
  "os": {
    "family": "Windows XP",
    "version": "NT 5.1"
  }
}
Enriched mode
Source
parse_user_agent(
	"Opera/9.80 (J2ME/MIDP; Opera Mini/4.3.24214; iPhone; CPU iPhone OS 4_2_1 like Mac OS X; AppleWebKit/24.783; U; en) Presto/2.5.25 Version/10.54",
	mode: "enriched"
)
Return
{
  "browser": {
    "family": "Opera Mini",
    "major": "4",
    "minor": "3",
    "patch": "24214",
    "version": "10.54"
  },
  "device": {
    "brand": "Apple",
    "category": "smartphone",
    "family": "iPhone",
    "model": "iPhone"
  },
  "os": {
    "family": "iOS",
    "major": "4",
    "minor": "2",
    "patch": "1",
    "patch_minor": null,
    "version": "4.2.1"
  }
}

parse_xml

fallible
Parses the value as XML.

Function spec

parse_xml(value: <string>, [include_attr: <boolean>, attr_prefix: <string>, text_key: <string>, always_use_text_key: <boolean>, parse_bool: <boolean>, parse_null: <boolean>, parse_number: <boolean>])
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string representation of the XML document to parse.yes
include_attrboolean
Include XML tag attributes in the returned object.trueno
attr_prefixstring
String prefix to use for XML tag attribute keys.@no
text_keystring
Key name to use for expanded text nodes.textno
always_use_text_keyboolean
Always return text nodes as {"<text_key>": "value"}.no
parse_boolboolean
Parse “true” and “false” as boolean.trueno
parse_nullboolean
Parse “null” as null.trueno
parse_numberboolean
Parse numbers as integers/floats.trueno

Notices

This function has special behavior that you should be aware of.
Valid XML must contain exactly one root node. Always returns an object.

Errors

The parse_xml function is fallible, which means that error handling is required for these errors:
value isn’t a valid XML document

Examples

Parse XML
Source
value = s'<book category="CHILDREN"><title lang="en">Harry Potter</title><author>J K. Rowling</author><year>2005</year></book>';

parse_xml!(value, text_key: "value", parse_number: false)
Return
{
  "book": {
    "@category": "CHILDREN",
    "author": "J K. Rowling",
    "title": {
      "@lang": "en",
      "value": "Harry Potter"
    },
    "year": "2005"
  }
}

Random functions

uuid_v4

infallible
Generates a random UUIDv4 string.

Function spec

uuid_v4()
:: <string>
required optional <types | ...>

Examples

Create a UUIDv4
Source
uuid_v4()
Return
1d262f4f-199b-458d-879f-05fd0a5f0683

String functions

contains

infallible
Determines whether the value string contains the specified substring.

Function spec

contains(value: <string>, substring: <string>, [case_sensitive: <boolean>])
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The text to search.yes
substringstring
The substring to search for in value.yes
case_sensitiveboolean
Whether the match should be case sensitive.trueno

Examples

String contains (case sensitive)
Source
contains("The Needle In The Haystack", "Needle")
Return
true
String contains (case insensitive)
Source
contains("The Needle In The Haystack", "needle", case_sensitive: false)
Return
true

downcase

infallible
Downcases the value string, where “downcase” is defined according to the terms of the Unicode Derived Core Property Lowercase.

Function spec

downcase(value: <string>)
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to convert to lowercase.yes

Examples

Downcase a string
Source
downcase("Hello, World!")
Return
hello, world!

ends_with

infallible
Determines whether the value string ends with the specified substring.

Function spec

ends_with(value: <string>, substring: <string>, [case_sensitive: <boolean>])
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to search.yes
substringstring
The substring with which value must end.yes
case_sensitiveboolean
Whether the match should be case sensitive.trueno

Examples

String ends with (case sensitive)
Source
ends_with("The Needle In The Haystack", "The Haystack")
Return
true
String ends with (case insensitive)
Source
ends_with("The Needle In The Haystack", "the haystack", case_sensitive: false)
Return
true

find

infallible
Determines the start position of the first found element in value, from left to right, that matches the pattern or returns -1 if not found.

Function spec

find(value: <string>, pattern: <regex | string>, [from: <integer>])
:: <integer>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to find the pattern in.yes
patternregex
string
The regular expression or string pattern to match against.yes
frominteger
Offset to start searching.no

Examples

Match text
Source
find("foobar", "foo")
Match regex
Source
find("foobar", r'b.r')
Return
3
No matches
Source
find("foobar", "baz")
Return
-1
With an offset
Source
find("foobarfoobarfoo", "bar", 4)
Return
9

join

infallible
Joins each string in the value array into a single string, with items optionally separated from one another by a separator.

Function spec

join(value: <array>, [separator: <string>])
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
The array of strings to join together.yes
separatorstring
The string separating each original element when joined.no

Examples

Join array (no separator)
Source
join!(["bring", "us", "together"])
Return
bringustogether
Join array (comma separator)
Source
join!(["sources", "transforms", "sinks"], separator: ", ")
Return
sources, transforms, sinks

match

infallible
Determines whether the value matches the pattern.

Function spec

match(value: <string>, pattern: <regex>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The value to match.yes
patternregex
The regular expression pattern to match against.yes

Examples

Regex match on a string
Source
match("I'm a little teapot", r'teapot')
Return
true

match_any

infallible
Determines whether the value matches any the given patterns. All patterns are checked in a single pass over the target string, giving this function a potentially significant performance advantage over multiple calls to match.

Function spec

match_any(value: <string>, patterns: <array>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The value to match.yes
patternsarray
The array of regular expression patterns to match against.yes

Examples

Regex match on a string
Source
match_any("I'm a little teapot", [r'frying pan', r'teapot'])
Return
true

redact

infallible

Redact sensitive data in value such as:

This can help achieve compliance by ensuring sensitive data never leaves your network.

Function spec

redact(value: <string | object | array>, [filters: <array>])
:: <string | object | array>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
object
array

The value to redact sensitive data from.

Its behavior differs depending on the type of value:

  • For strings, it simply redacts the sensitive data and returns a new string
  • For arrays, it redacts the sensitive data in each string element
  • For objects, it masks the sensitive data in each string value, but not keys

For arrays and objects it will recurse into any nested arrays or objects. Any non-string elements will be skipped.

Any redacted text will be replaced with [REDACTED].

yes
filtersarray

List of filters to be applied to the value.

Each filter can be specified in one of three ways:

  • As a regular expression directly, which will be used to redact text matching it
  • As an object with a type key that corresponds to a named filter and additional keys for customizing that filter
  • As a named filter, if it has no required parameters

Named filters are:

  • pattern: Redact text matching any regular expressions specified in the, required, patterns key. This is the expanded form of just passing a regular expression as a filter.
  • us_social_security_number: Redact US social security card numbers.

See examples for more details.

This parameter must be a static expression. You cannot use variables or other dynamic expressions with it. This allows us to validate the argument at compile-time to avoid runtime errors.

no

Examples

Replace text using a regex
Source
redact("my id is 123456", filters: [r'\d+'])
Return
my id is [REDACTED]
Replace us social security numbers in any field
Source
redact({ "name": "John Doe", "ssn": "123-12-1234"}, filters: ["us_social_security_number"])
Return
{
  "name": "John Doe",
  "ssn": "[REDACTED]"
}

replace

infallible

Replaces all matching instances of pattern in the value.

The pattern argument accepts regular expression capture groups. Note that $foo is interpreted in a Vector configuration file, instead use $$foo.

Function spec

replace(value: <string>, pattern: <regex | string>, with: <string>, [count: <integer>])
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The original string.yes
patternregex
string
Replace all matches of this pattern. Can be a static string or a regular expression.yes
withstring
The string that the matches are replaced with.yes
countinteger
The maximum number of replacements to perform. -1 means replace all matches.-1no

Examples

Replace literal text
Source
replace("Apples and Bananas", "and", "not")
Return
Apples not Bananas
Replace via regular expression
Source
replace("Apples and Bananas", r'(?i)bananas', "Pineapples")
Return
Apples and Pineapples
Replace first instance
Source
replace("Bananas and Bananas", "Bananas", "Pineapples", count: 1)
Return
Pineapples and Bananas
Replace with capture groups
Source
replace("foo123bar", r'foo(?P<num>\d+)bar', "$num")
Return
123

slice

infallible

Returns a slice of the value between the start and end positions.

If the start and end parameters are negative, they refer to positions counting from the right of the string or array. If end refers to a position that is greater than the length of the string or array a slice up to the end of the string or array is returned.

Function spec

slice(value: <array | string>, start: <integer>, [end: <integer>])
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
string
The string or array to slice.yes
startinteger
The inclusive start position. A zero-based index that can be negative.yes
endinteger
The inclusive end position. A zero-based index that can be negative.String lengthno

Examples

Slice a string (positive index)
Source
slice!("Supercalifragilisticexpialidocious", start: 5, end: 13)
Return
califrag
Slice a string (negative index)
Source
slice!("Supercalifragilisticexpialidocious", start: 5, end: -14)
Return
califragilistic

split

infallible
Splits the value string via the pattern.

Function spec

split(value: <string>, pattern: <string | regex>, [limit: <integer>])
:: <array>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to split.yes
patternstring
regex
The string is split whenever this pattern is matched.yes
limitinteger
The maximum number of substrings to return.no

Examples

Split a string (no limit)
Source
split("apples and pears and bananas", " and ")
Return
["apples","pears","bananas"]
Split a string (with a limit)
Source
split("apples and pears and bananas", " and ", limit: 2)
Return
["apples","pears and bananas"]

starts_with

infallible
Determines whether the value begins with the substring.

Function spec

starts_with(value: <string>, substring: <string>, [case_sensitive: <boolean>])
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to search.yes
substringstring
The substring that the value must start with.yes
case_sensitiveboolean
Whether the match should be case sensitive.trueno

Examples

String starts with (case sensitive)
Source
starts_with("The Needle In The Haystack", "The Needle")
Return
true
String starts with (case insensitive)
Source
starts_with("The Needle In The Haystack", "the needle", case_sensitive: false)
Return
true

strip_ansi_escape_codes

infallible
Strips ANSI escape codes from the value.

Function spec

strip_ansi_escape_codes(value: <string>)
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to strip.yes

Examples

Strip ANSI escape codes
Source
strip_ansi_escape_codes("\e[46mfoo\e[0m bar")
Return
foo bar

strip_whitespace

infallible
Strips whitespace from the start and end of the value, where whitespace is defined by the Unicode White_Space property

Function spec

strip_whitespace(value: <string>)
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to trim.yes

Examples

Strip whitespace
Source
strip_whitespace("  A sentence.  ")
Return
A sentence.

truncate

infallible
Truncates the value string up to the limit number of characters.

Function spec

truncate(value: <string>, limit: <integer | float>, ellipsis: <boolean>)
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to truncate.yes
limitinteger
float
The number of characters to truncate the string after.yes
ellipsisboolean
An ellipsis (...) is appended if this is set to true and the value string ends up being truncated because it’s exceeded the limit.yes

Examples

Truncate a string
Source
truncate("A rather long sentence.", limit: 11, ellipsis: true)
Return
A rather lo...

upcase

infallible
Upcases the value, where “upcase” is defined according to the terms of the Unicode Derived Core Property Uppercase.

Function spec

upcase(value: <string>)
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to convert to uppercase.yes

Examples

Upcase a string
Source
upcase("Hello, World!")
Return
HELLO, WORLD!

System functions

get_env_var

fallible
Returns the value of the environment variable specified by name.

Function spec

get_env_var(name: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
namestring
The name of the environment variable.yes

Errors

The get_env_var function is fallible, which means that error handling is required for these errors:
Environment variable name doesn’t exist
Value of environment variable name isn’t valid Unicode

Examples

Get an environment variable
Source
get_env_var!("HOME")
Return
/root

get_hostname

fallible
Returns the local system’s hostname.

Function spec

get_hostname()
:: <string> , <error>
required optional <types | ...>

Errors

The get_hostname function is fallible, which means that error handling is required for these errors:
Internal hostname resolution failed.

Examples

Get hostname
Source
.hostname = get_hostname!()

Timestamp functions

format_timestamp

infallible
Formats the value into a string representation of the timestamp.

Function spec

format_timestamp(value: <timestamp>, format: <string>)
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuetimestamp
The timestamp to format as text.yes
formatstring
The format string as described by the Chrono library.yes

Examples

Format a timestamp (ISO8601/RFC 3339)
Source
format_timestamp!(t'2020-10-21T16:00:00Z', format: "%+")
Return
2020-10-21T16:00:00+00:00
Format a timestamp (custom)
Source
format_timestamp!(t'2020-10-21T16:00:00Z', format: "%v %R")
Return
21-Oct-2020 16:00

now

infallible
Returns the current timestamp in the UTC timezone with nanosecond precision.

Function spec

now()
:: <timestamp>
required optional <types | ...>

Examples

Generate a current timestamp
Source
now()
Return
2021-03-04T10:51:15.928937Z

Type functions

array

fallible
Returns the value if it’s an array and errors otherwise. This enables the type checker to guarantee that the returned value is an array and can be used in any function that expects one.

Function spec

array(value: <any>)
:: <array> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value that you need to ensure is an array.yes

Errors

The array function is fallible, which means that error handling is required for these errors:
value isn’t an array.

Examples

Declare an array type
Source
array!(.value)
Return
[1,2,3]

bool

fallible
Returns the value if it’s a Boolean and errors otherwise. This enables the type checker to guarantee that the returned value is a Boolean and can be used in any function that expects one.

Function spec

bool(value: <any>)
:: <boolean> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value that you need to ensure is a Boolean.yes

Errors

The bool function is fallible, which means that error handling is required for these errors:
value isn’t a Boolean.

Examples

Declare a Boolean type
Source
bool!(.value)

float

fallible
Returns the value if it’s a float and errors otherwise. This enables the type checker to guarantee that the returned value is a float and can be used in any function that expects one.

Function spec

float(value: <any>)
:: <float> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value that you need to ensure is a float.yes

Errors

The float function is fallible, which means that error handling is required for these errors:
value isn’t a float.

Examples

Declare a float type
Source
float!(.value)
Return
42

int

fallible
Returns the value if it’s an integer and errors otherwise. This enables the type checker to guarantee that the returned value is an integer and can be used in any function that expects one.

Function spec

int(value: <any>)
:: <integer> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value that you need to ensure is an integer.yes

Errors

The int function is fallible, which means that error handling is required for these errors:
value is not an integer.

Examples

Declare an integer type
Source
int!(.value)
Return
42

is_array

infallible
Check if the type of a value is an array or not.

Function spec

is_array(value: <any>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value to checkyes

Examples

Valid array
Source
is_array([1, 2, 3])
Return
true
Non-matching type
Source
is_array("a string")

is_boolean

infallible
Check if the type of a value is a boolean or not.

Function spec

is_boolean(value: <any>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value to checkyes

Examples

Valid boolean
Source
is_boolean(false)
Return
true
Non-matching type
Source
is_boolean("a string")

is_float

infallible
Check if the type of a value is a float or not.

Function spec

is_float(value: <any>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value to checkyes

Examples

Valid float
Source
is_float(0.577)
Return
true
Non-matching type
Source
is_float("a string")

is_integer

infallible
Check if the type of a value is an integer or not.

Function spec

is_integer(value: <any>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value to checkyes

Examples

Valid integer
Source
is_integer(1)
Return
true
Non-matching type
Source
is_integer("a string")

is_null

infallible
Check if the type of a value is null or not. For a more relaxed function please check is_nullish.

Function spec

is_null(value: <any>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value to checkyes

Examples

Null value
Source
is_null(null)
Return
true
Non-matching type
Source
is_null("a string")

is_nullish

infallible
Determines whether the value is “nullish,” where nullish denotes the absence of a meaningful value.

Function spec

is_nullish(value: <any>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value to check for “nullishness,” i.e. a useless value.yes

Examples

Null detection (blank string)
Source
is_nullish("")
Return
true
Null detection (dash string)
Source
is_nullish("-")
Return
true
Null detection (whitespace)
Source
is_nullish("
  
")
Return
true

is_object

infallible
Check if the type of a value is an object or not.

Function spec

is_object(value: <any>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value to checkyes

Examples

Valid object
Source
is_object({"foo": "bar"})
Return
true
Non-matching type
Source
is_object("a string")

is_regex

infallible
Check if the type of a value is a regex or not.

Function spec

is_regex(value: <any>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value to checkyes

Examples

Valid regex
Source
is_regex(r'pattern')
Return
true
Non-matching type
Source
is_regex("a string")

is_string

infallible
Check if the type of a value is a string or not.

Function spec

is_string(value: <any>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value to checkyes

Examples

Valid string
Source
is_string("a string")
Return
true
Non-matching type
Source
is_string([1, 2, 3])

is_timestamp

infallible
Check if the type of a value is a timestamp or not.

Function spec

is_timestamp(value: <any>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value to checkyes

Examples

Valid timestamp
Source
is_timestamp(t'2021-03-26T16:00:00Z')
Return
true
Non-matching type
Source
is_timestamp("a string")

object

fallible
Returns the value if it’s an object and errors otherwise. This enables the type checker to guarantee that the returned value is an object and can be used in any function that expects one.

Function spec

object(value: <any>)
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value that you need to ensure is an object.yes

Errors

The object function is fallible, which means that error handling is required for these errors:
value isn’t an object.

Examples

Declare an object type
Source
object!(.value)
Return
{
  "field1": "value1",
  "field2": "value2"
}

string

fallible
Returns the value if it’s a string and errors otherwise. This enables the type checker to guarantee that the returned value is a string and can be used in any function that expects one.

Function spec

string(value: <any>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value that you need to ensure is a string.yes

Errors

The string function is fallible, which means that error handling is required for these errors:
value isn’t a string.

Examples

Declare a string type
Source
string!(.message)
Return
{"field": "value"}

tag_types_externally

infallible

Adds type information to all (nested) scalar values in the provided value.

The type information is added externally, meaning that value has the shape of "type": value after this transformation.

Function spec

tag_types_externally(value: <any>)
:: <object | array | null>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value to tag with types.yes

Examples

Tag types externally (scalar)
Source
tag_types_externally(123)
Return
{
  "integer": 123
}
Tag types externally (object)
Source
tag_types_externally({
	"message": "Hello world",
	"request": {
		"duration_ms": 67.9
	}
})
Return
{
  "message": {
    "string": "Hello world"
  },
  "request": {
    "duration_ms": {
      "float": 67.9
    }
  }
}
Tag types externally (array)
Source
tag_types_externally(["foo", "bar"])
Return
[{"string":"foo"},{"string":"bar"}]
Tag types externally (null)
Source
tag_types_externally(null)

timestamp

fallible
Returns the value if it’s a timestamp and errors otherwise. This enables the type checker to guarantee that the returned value is a timestamp and can be used in any function that expects one.

Function spec

timestamp(value: <any>)
:: <timestamp> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value that you need to ensure is a timestamp.yes

Errors

The timestamp function is fallible, which means that error handling is required for these errors:
value isn’t a timestamp.

Examples

Declare a timestamp type
Source
ok, err = timestamp(.timestamp)
Return
function call error for "timestamp" at (10:31): expected "timestamp", got "string"