Querying

Apex Logs provides a powerful query language supporting both plain-text and structured logs. This section explains the advantage that structured logging can provide over plain-text logs, as well as the query language reference.

Structured log events

To make the most out of structured logging it’s recommended to provide actionable, low cardinality message values to represent actions, while providing additional context information in fields rather than “baking” it right into the message.

For example using the message File Upload instead of tj@example.com uploaded some.png, as the former contains no dynamic information it’s easy to filter on and perform analytical queries against. Here’s what an example log Event might look like:

{
  "level": "info",
  "message": "Upload",
  "fields": {
    "file": "sloth.png",
    "type": "image/png",
    "duration": 2502,
    "size": 43008,
    "user": {
      "name": "Tobi",
      "email": "tobi@example.com"
    },
    "source": {
      "host": "api-01"
    }
  }
}

Structured logs enable precise queries such as the following:

message = "Upload"
message = "Upload" and user.email = "tobi@example.com" and type = "image/*"
message = "Upload" and duration > 2s
message = "Upload" and duration between 1s and 1m
message = "Upload" and size >= 100kb

Simple search terms can also work in simple cases, as they search the event’s message and fields. For example instead of message = "Upload" you could search:

Upload

And instead of message = "Upload" and user.email = "tobi@example.com" and type = "image/png" a similar, but less precise search query would be:

Upload tobi@example.com image/png

Plain-text log events

Plain-text logs are essentially arbitrary lines of text, lacking any real structure. Here are a few examples:

user tobi@example.com signed in
user tobi@example.com uploaded sloth.png (took 1500ms) (103kb)
user tobi@example.com upload failed: Operation not permitted

A plain-text Event essentially bakes all of the information into the message, however, you may optionally fields to provide additional context such as the source host:

{
  "level": "info",
  "message": "user tobi@example.com uploaded sloth.png (image/png) (took 2.5s) (43kb)",
  "fields": {
    "source": {
      "host": "api-01"
    }
  }
}

To search against these logs you could use queries similar to:

tobi@example.com ("signed in" or "signed out")
tobi@example.com uploaded
"Operation not permitted"

With plain-text logs it’s not possible to perform complex queries against the the duration (took 1500ms) or size (103kb) values, as they’re part of the message string.

With Apex Logs there’s no need to choose between the two techniques, you can mix and match the two as necessary.

Terms

Searches are case sensitive, with terms broken up by whitespace. A term is a single word, such as user or login as shown below, which will match the text “user” or “login” anywhere in the event.

user login

This query would match both of the following, as the two terms are joined with the AND operator:

user tj@apex.sh login
user login with email tj@apex.sh

If the term contains whitespace you can use single or double qoutes for an exact match:

"user login"

Only the following line is now matched:

user login with email tj@apex.sh

Terms search the entire event, so it also works well with structured log events, for example:

login tj@apex.sh

Will match the event:

{
  "level": "info",
  "message": "login",
  "fields": {
    "name": "Tobi",
    "email": "tj@apex.sh"
  }
}

In this case better performance can be achieved with a more precise query:

message = "login" email = "tj@apex.sh"

An unquoted search term can contain any character except the following: (, ), <, >, =, ,, !, ", ', |. For example the following are valid unquoted search terms:

~/path/to/image.png
tobi@example.com
64812AD3-46FD-4991-AE52-4D5514D8E655

Severity

A log event has an associated severity level, indicating if the log is purely informational, or represents a problem with your application. The following severity levels are supported:

  • debug
  • info
  • notice
  • warning
  • error
  • critical
  • alert
  • emergency

Severity levels are treated as integers internally, ranging from 0 to 7, allowing for use with relational operators:

level >= info
level = error

When used alone, the severity level keywords behave as an >= expression. For example these expressions are identical, allowing you quickly show all categories of error:

error
level >= error

If you want to search for the literal text “info” or “error” anywhere in your event you can quote them to treat it as a term:

"info"
"error"

Literals

This section covers the literals available.

Booleans

Booleans use the true and false keywords:

user.is_admin = true
user.is_admin = false

Integers

Integers use a sequence of digits.

cart.total > 15

You may use _ as a thousands separator, for example:

1_000_000

Floats

Floats use a sequence of digits followed by a ., another sequence of digits, and optional exponent.

cart.total > 15.99

You may use _ as a thousands separator, for example:

1_000_000.99

Duration units

The numeric duration units use a millisecond resolution, using the suffixes ms, s, m, h, and d.

message = "uploaded file" and duration > 1500
message = "uploaded file" and duration > 1500ms
message = "uploaded file" and duration > 1s
message = "uploaded file" and duration > 1.5s
message = "uploaded file" and duration > 5m
message = "uploaded file" and duration > 3h
message = "uploaded file" and duration > 1d

Byte size units

The numeric byte size units use the suffixes b, kb, mb, gb, and tb for querying multiples of 1024.

message = "uploaded file" and file.size > 1024
message = "uploaded file" and file.size > 1024b
message = "uploaded file" and file.size > 1kb
message = "uploaded file" and file.size > 2.5mb
message = "uploaded file" and file.size > 1gb

Regular expressions

The backtick is used to denote a regular expression. By default the event’s message and fields are matched against the pattern, for example:

"Cart Checkout" `(tobi|jane)@example.com`

Matches the following event:

{
  "level": "info",
  "message": "Cart Checkout",
  "fields": {
    "user": {
      "name": "Jane",
      "email": "jane@example.com"
    }
  }
}

Logical operators

Logical operators evaluate to a boolean, allowing you to define more complex filter operations.

AND

The and operator evalutes to true if both conditions are truthy:

error and message = "Upload Failed"

The and operator is implied by default:

message = "Upload Failed" host = "api-01"

Is equivalent to:

message = "Upload Failed" and host = "api-01"

OR

The or operator evalutes to true if either condition is truthy:

message = "Uploaded File" and (file.type = "image/png" or file.name = "*.png")

NOT

The ! operator negates an expression:

!(url.path = "/admin" and user.is_admin = true)

EXISTS

The exists operator evalutes to true if the field is defined.

user.cart exists

Equality operators

Equality operators perform comparisons between fields and values.

=

The = operator evalutes to true if both operands are the same.

region = "us-west2"

The wildcard character * can be used to match zero or more characters:

user.email = "*@example.com"
user.email = "tobi@*"
user.email = "*@example.*"

The right-hand operand can be a regular expression with backticks:

user.email = `^(tobi|jane)@example\.com$`

This operator is aliased as ==.

!=

The != operator evalutes to true if both operands are not the same.

region != "us-west2"

The wildcard character * can be used to match zero or more characters:

host != "api-*"

The right-hand operand can be a regular expression with backticks:

user.email != `^(tobi|jane)@example\.com$`

CONTAINS

The contains operator evalutes to true if the left operand contains the string on the right.

product.name contains "Logitec"
product.name not contains "Logitec"

This is an alias of:

product.name = "*Logitec*"
product.name != "*Logitec*"

STARTS WITH

The starts with operator evalutes to true if the left operand starts with the string on the right.

region starts with "us-"
region not starts with "us-"

This is an alias of:

region = "us-*"
region != "us-*"

ENDS WITH

The ends with operator evalutes to true if the left operand ends with the string on the right.

region ends with "-west2"
region not ends with "-west2"

This is an alias of:

region ends with "*-west2"
region not ends with "*-west2"

BETWEEN

The between operator evalutes to true if the left operand is between the range (inclusive).

response.status between 200 and 299
response.status not between 200 and 299

Relational operators

>

The > operator evalutes to true if the left operand is greater than the right.

cart.total > 15

>=

The >= operator evalutes to true if the left operand is greater than or equal to the right.

cart.total >= 15

<

The < operator evalutes to true if the left operand is less than the right.

cart.total < 15

<=

The <= operator evalutes to true if the left operand is less than or equal to the right.

cart.total <= 15

Fields

Event fields are accessed using the . operator. For example the following:

user.name = "Tobi"
user.email = "*@example.com"
cart.total > 15.99

Would match the event:

{
  "level": "info",
  "message": "Cart Checkout",
  "fields": {
    "user": {
      "name": "Tobi",
      "email": "tobi@example.com"
    },
    "cart": {
      "items": [
        {
          "name": "Ferret Food",
          "cost": 15.99
        }
      ],
      "total": 15.99
    },
    "source": {
      "host": "api-01"
    }
  }
}

Array elements are accessed using the [] operator with an index:

cart.items[0].name = "Ferret Food"
cart.items[0].cost > 10

Grouping

You may group expressions using the ( and ) characters:

message = "Uploaded File" and (region = "us-west-2" or region = "us-east-1")

Examples

Show logs with an error severity or greater:

error
{
  "level": "error",
  "message": "Upload Failed",
  "fields": {
    "error": "Operation not permitted",
    "path": "/tmp/image.png"
  }
}

Show missing file errors in plain-text Apache error logs from the US regions:

"File does not exist" region = "us-*"
{
  "level": "info",
  "message": "[Thu Mar 13 19:04:13 2014] [error] [client 50.0.134.125]
File does not exist: /var/www/favicon.ico
",
  "fields": {
    "host": "api-01",
    "region": "us-west-2",
    "source": "apache"
  }
}

Show a specific Lambda request by ID:

REPORT 4b3ab4c4-75e4-431f-81dc-783f4b74a243
{
  "level": "info",
  "message": "REPORT RequestId: 4b3ab4c4-75e4-431f-81dc-783f4b74a243
Duration: 4687.99ms	Billed Duration: 4700 ms	Memory Size: 128 MB
Max Memory Used: 51 MB",
  "fields": {
    "aws": {
      "region": "us-west-2",
      "lambda": {
        "function": "api",
        "version": 9
      }
    }
  }
}

Errors caused by a particular Lambda function and version:

error and function.name = "alert_processor" and function.version = "v1.5.0"
{
  "level": "error",
  "message": "Processing alert",
  "fields": {
    "error": "Error connecting to database: no such host",
    "function": {
      "name": "alert_processor",
      "version": "v1.5.0"
    }
  }
}

Slow responses caused by a particular user.

user.email = "tobi@example.com" and duration >= 1.5s
{
  "level": "info",
  "message": "HTTP response",
  "fields": {
    "duration": 2300,
    "user": {
      "email": "tobi@example.com"
    }
  }
}

Image upload errors:

error and message = "Upload Failed" and file.type = "image/*"
{
  "level": "error",
  "message": "Upload Failed",
  "fields": {
    "file": {
      "name": "sloth.png",
      "type": "image/png",
      "size": 40960
    }
  }
}

Match slow HTTP responses for GET requests to /profile:

GET /profile response duration >= 1.5s
{
  "level": "info",
  "message": "response",
  "fields": {
    "method": "GET",
    "path": "/profile",
    "duration": 1800,
    "user": {
      "email": "tobi@example.com"
    }
  }
}

Match slow HTTP responses for GET requests to /profile using fields:

message = "response"
and method = "GET"
and path = "/profile"
and duration >= 1.5s
{
  "level": "info",
  "message": "response",
  "fields": {
    "method": "GET",
    "path": "/profile",
    "duration": 1800,
    "user": {
      "email": "tobi@example.com"
    }
  }
}