AWS SQS
Collect logs from AWS SQS
Configuration
Example configurations
{
"sources": {
"my_source_id": {
"type": "aws_sqs",
"queue_url": "https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue"
}
}
}
[sources.my_source_id]
type = "aws_sqs"
queue_url = "https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue"
---
sources:
my_source_id:
type: aws_sqs
queue_url: https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue
{
"sources": {
"my_source_id": {
"type": "aws_sqs",
"delete_message": true,
"endpoint": "http://127.0.0.0:5000/path/to/service",
"poll_secs": 15,
"queue_url": "https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue",
"region": "us-east-1",
"visibility_timeout_secs": 300
}
}
}
[sources.my_source_id]
type = "aws_sqs"
delete_message = true
endpoint = "http://127.0.0.0:5000/path/to/service"
poll_secs = 15
queue_url = "https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue"
region = "us-east-1"
visibility_timeout_secs = 300
---
sources:
my_source_id:
type: aws_sqs
delete_message: true
endpoint: http://127.0.0.0:5000/path/to/service
poll_secs: 15
queue_url: https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue
region: us-east-1
visibility_timeout_secs: 300
acknowledgements
optional objectControls how acknowledgements are handled by this source.
This setting is deprecated in favor of enabling acknowledgements
at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
acknowledgements.enabled
optional boolauth
optional objectauth.assume_role
required string literalauth.credentials_file
required string literalauth.imds
optional objectauth.imds.max_attempts
optional uint4
auth.load_timeout_secs
optional uintTimeout for successfully loading any credentials, in seconds.
Relevant when the default credentials chain is used or assume_role
.
auth.profile
optional string literalThe credentials profile to use.
Used to select AWS credentials from a provided credentials file.
default
auth.region
optional string literalThe AWS region to send STS requests to.
If not set, this will default to the configured region for the service itself.
auth.secret_access_key
required string literalclient_concurrency
optional uintNumber of concurrent tasks to create for polling the queue for messages.
Defaults to the number of available CPUs on the system.
Should not typically need to be changed, but it can sometimes be beneficial to raise this value when there is a high rate of messages being pushed into the queue and the messages being fetched are small. In these cases, system resources may not be fully utilized without fetching more messages per second, as it spends more time fetching the messages than processing them.
decoding
optional objectdecoding.codec
optional string literal enumOption | Description |
---|---|
bytes | Uses the raw bytes as-is. |
gelf | Decodes the raw bytes as a GELF message. |
json | Decodes the raw bytes as JSON. |
native | Decodes the raw bytes as Vector’s native Protocol Buffers format. This codec is experimental. |
native_json | Decodes the raw bytes as Vector’s native JSON format. This codec is experimental. |
syslog | Decodes the raw bytes as a Syslog message. Will decode either as the RFC 3164-style format (“old” style) or the more modern RFC 5424-style format (“new” style, includes structured data). |
bytes
delete_message
optional boolWhether to delete the message once it is processed.
It can be useful to set this to false
for debugging or during the initial setup.
true
endpoint
optional string literalframing
optional objectFraming configuration.
Framing deals with how events are separated when encoded in a raw byte form, where each event is a “frame” that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.
framing.character_delimited
required objectmethod = "character_delimited"
framing.character_delimited.delimiter
required uintframing.character_delimited.max_length
optional uintThe maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This will ensure that processing is not truly unbounded.
framing.method
optional string literal enumOption | Description |
---|---|
bytes | Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments). |
character_delimited | Byte frames which are delimited by a chosen character. |
length_delimited | Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length. |
newline_delimited | Byte frames which are delimited by a newline character. |
octet_counting | Byte frames according to the octet counting format. |
bytes
framing.newline_delimited
optional objectmethod = "newline_delimited"
framing.newline_delimited.max_length
optional uintThe maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This will ensure that processing is not truly unbounded.
framing.octet_counting
optional objectmethod = "octet_counting"
framing.octet_counting.max_length
optional uintpoll_secs
optional uintHow long to wait while polling the queue for new messages, in seconds.
Generally should not be changed unless instructed to do so, as if messages are available,
they will always be consumed, regardless of the value of poll_secs
.
15
(seconds)proxy
optional objectProxy configuration.
Configure to proxy traffic through an HTTP(S) proxy when making external requests.
Similar to common proxy configuration convention, users can set different proxies to use based on the type of traffic being proxied, as well as set specific hosts that should not be proxied.
proxy.http
optional string literalProxy endpoint to use when proxying HTTP traffic.
Must be a valid URI string.
proxy.https
optional string literalProxy endpoint to use when proxying HTTPS traffic.
Must be a valid URI string.
proxy.no_proxy
optional [string]A list of hosts to avoid proxying.
Multiple patterns are allowed:
Pattern | Example match |
---|---|
Domain names | example.com matches requests to example.com |
Wildcard domains | .example.com matches requests to example.com and its subdomains |
IP addresses | 127.0.0.1 matches requests to 127.0.0.1 |
CIDR blocks | 192.168.0.0/16 matches requests to any IP addresses in this range |
Splat | * matches all hosts |
queue_url
required string literaltls
optional objecttls.alpn_protocols
optional [string]Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.
tls.ca_file
optional string literalAbsolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
tls.crt_file
optional string literalAbsolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
tls.key_file
optional string literalAbsolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
tls.key_pass
optional string literalPassphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
tls.verify_certificate
optional boolEnables certificate verification.
If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
tls.verify_hostname
optional boolEnables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
visibility_timeout_secs
optional uintThe visibility timeout to use for messages, in seconds.
This controls how long a message is left unavailable after it is received. If a message is received, and
takes longer than visibility_timeout_secs
to process and delete the message from the queue, it is made available again for another consumer.
This can happen if there is an issue between consuming a message and deleting it.
300
(seconds)Environment variables
AWS_ACCESS_KEY_ID
common optional string literalAWS_CONFIG_FILE
common optional string literal~/.aws/config
AWS_DEFAULT_REGION
common optional string literalAWS_PROFILE
common optional string literaldefault
AWS_ROLE_SESSION_NAME
common optional string literalAWS_SECRET_ACCESS_KEY
common optional string literalAWS_SESSION_TOKEN
common optional string literalAWS_SHARED_CREDENTIALS_FILE
common optional string literal~/.aws/credentials
HTTPS_PROXY
common optional string literalThe global URL to proxy HTTPS requests through.
If another HTTPS proxy is set in the configuration file or at a component level, this one will be overridden.
The lowercase variant has priority over the uppercase one.
HTTP_PROXY
common optional string literalThe global URL to proxy HTTP requests through.
If another HTTP proxy is set in the configuration file or at a component level, this one will be overridden.
The lowercase variant has priority over the uppercase one.
NO_PROXY
common optional string literalList of hosts to avoid proxying globally.
Allowed patterns here include:
Pattern | Example match |
---|---|
Domain names | example.com matches requests to example.com |
Wildcard domains | .example.come matches requests to example.com and its subdomains |
IP addresses | 127.0.0.1 matches requests to 127.0.0.1 |
CIDR blocks | 192.168.0.0./16 matches requests to any IP addresses in this range |
Splat | * matches all hosts |
If another no_proxy
value is set in the configuration file or at a component level, this
one is overridden.
The lowercase variant has priority over the uppercase one.
http_proxy
common optional string literalThe global URL to proxy HTTP requests through.
If another HTTP proxy is set in the configuration file or at a component level, this one will be overridden.
The lowercase variant has priority over the uppercase one.
https_proxy
common optional string literalThe global URL to proxy HTTPS requests through.
If another HTTPS proxy is set in the configuration file or at a component level, this one will be overridden.
The lowercase variant has priority over the uppercase one.
no_proxy
common optional string literalList of hosts to avoid proxying globally.
Allowed patterns here include:
Pattern | Example match |
---|---|
Domain names | example.com matches requests to example.com |
Wildcard domains | .example.come matches requests to example.com and its subdomains |
IP addresses | 127.0.0.1 matches requests to 127.0.0.1 |
CIDR blocks | 192.168.0.0./16 matches requests to any IP addresses in this range |
Splat | * matches all hosts |
If another no_proxy
value is set in the configuration file or at a component level, this
one is overridden.
The lowercase variant has priority over the uppercase one.
Outputs
<component_id>
Output Data
Logs
Record
53.126.150.246 - - [01/Oct/2020:11:25:58 -0400] "GET /disintermediate HTTP/2.0" 401 20308
aws_sqs
2020-10-10T17:07:36.452332Z
Telemetry
Metrics
linkcomponent_received_bytes_total
countercomponent_id
instead. The value is the same as component_id
.component_received_event_bytes_total
countercomponent_id
instead. The value is the same as component_id
.component_received_events_total
countercomponent_id
instead. The value is the same as component_id
.component_sent_event_bytes_total
countercomponent_id
instead. The value is the same as component_id
.component_sent_events_total
countercomponent_id
instead. The value is the same as component_id
.events_out_total
countercomponent_sent_events_total
instead.component_id
instead. The value is the same as component_id
.source_lag_time_seconds
histogramcomponent_id
instead. The value is the same as component_id
.sqs_message_delete_failed_total
countercomponent_id
instead. The value is the same as component_id
.How it works
AWS authentication
Vector checks for AWS credentials in the following order:
- The
auth.access_key_id
andauth.secret_access_key
options. - The
AWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
environment variables. - The AWS credentials file (usually located at
~/.aws/credentials
). - The IAM instance profile (only works if running on an EC2 instance with an instance profile/role). Requires IMDSv2 to be enabled. For EKS, you may need to increase the metadata token response hop limit to 2.
If no credentials are found, Vector’s health check fails and an error is logged. If your AWS credentials expire, Vector will automatically search for up-to-date credentials in the places (and order) described above.
Obtaining an access key
auth.access_key_id
and auth.secret_access_key
options.Assuming roles
auth.assume_role
option. This is an
optional setting that is helpful for a variety of use cases, such as cross
account access.AWS SQS
aws_sqs
source receives messages from AWS SQS
(Simple Queue Service). This is a highly scalable / durable queueing system with
at-least-once queuing semantics. Messages are received in batches (up to 10 at a time),
and then deleted in batches (again up to 10). Messages are either deleted immediately
after receiving, or after it has been fully processed by the sinks, depending on the
acknowledgements
setting.Transport Layer Security (TLS)
tls.*
options.