When sending to HEC, you send data to a REST API endpoint. See the sample below and note the JSON payload between the single quotes.
curl -k -H "Authorization: Splunk <YOUR-AUTH-TOKEN>" https://<ILLINOIS-SPLUNK-HEC-FQDN>/services/collector/event -d '{"time": 1616159900, "event": "Replace this sample, lintable json with lintable json based on the samples from the scenarios below."}' |
This document is intended to focus on this JSON payload piece – to help you make choices about it and how to construct it to suit your “field specifying” use cases.
As you’ll see in the samples below, Splunk metadata fields – if/when you need to specify them in your JSON – must always be at the top level of the json element structure.Custom fields can be added within the “fields” top level element (in which case they’ll behave like “metadata” fields – where they don’t show up in your event string in Splunk), or in the “event” element of the JSON, where they will show up in the event string in Splunk.You may notice that we use epoch time in the “time” metadata field in the samples below. While it is ok to use other timestamp formats in this field, epoch (UTC) is preferred/encouraged because it is the most efficient form for Splunk to recognize and to extract. Further… because this is a field that gets assigned directly to the _time metadata field in Splunk, there’s no reason to present it in a human-friendly format because a) it won’t show up in your event string and b) Splunk already automatically presents _time in a human-friendly way for you. You may wish, however, to use more human-friendly time formats (ISO 8601 recommended) if/when you include timestamp values in event strings.
Contents
Scenario 1a: With ‘fixed’ json
When: For when your event payload is json, but you don’t want to change its element structure for whatever reason. (An example might be that your json is based on a standard such as caliper and you do not wish to change the json payload in any way.)
What: Add additional fields as meta fields (not in event element).
Why: Because we don’t want to change or add string content to the json, we should add the fields as meta fields. These fields will not be in event string (_raw) and won’t be searchable w/ lispy. They will only be searchable via key-value pair search, similar to fields host, source, sourcetype, and index. (i.e., container=”my_container”).
How: Use this sample json as a basis to create your own lintable json. (Reference HTTP POST sample at top of page.)
{
"time": 1616159900,
"host": "dataserver992.example.com",
"source": "log_file_name",
"sourcetype": "_json",
"event": {
"message": "Something happened",
"severity": "INFO"
},
"fields": {
"process": "my_app",
"instance_id": "my_container",
"instance_type": "container",
"image_id": "my_container_image",
"cloud_platform": "aws",
"region": "us-east-2"
}
}
Scenario 1b: With custom json
When: For when your event payload is json and there is no reason not to edit/change the json element structure. (E.g., this json is from a local, custom app you’re developing.)
What (A): Send metadata only as sub-elements of event element.
What (B): Conditionally send metadata as either meta fields or sub-elements of event element. (Depending on whether you want them in _raw.)
Why (A): All fields will be in both event string (in _raw) and extracted/indexed.
Why (B): You want to choose which fields will be in event string (and extracted/indexed) and which are only extracted/indexed.
How (A):
{
"time": 1616159900,
"host": "dataserver992.example.com",
"source": "log_file_name",
"sourcetype": "_json",
"event": {
"message": "Something happened",
"severity": "INFO",
"process": "my_app",
"instance_id": "my_container",
"instance_type": "container",
"image_id": "my_container_image",
"cloud_platform": "aws",
"region": "us-east-2"
}
}
How (B):
{
"time": 1616159900,
"host": "dataserver992.example.com",
"source": "log_file_name",
"sourcetype": "_json",
"event": {
"message": "Something happened",
"severity": "INFO",
"process": "my_app",
"instance_id": "my_container",
"instance_type": "container",
"image_id": "my_container_image"
},
"fields: {
"cloud_platform": "aws",
"region": "us-east-2"
}
}
Scenario 2a: With ‘fixed’ syslog-like event string
When: For when your event payload is syslog-like (not well structured like csv, xml, or json) and the event-string pattern follows a well-established, Splunk-pretrained, and/or vendor-supplied pattern.
What: Send all additional fields only as indexed meta fields.
Why: If you want to leave the event string alone, your only option is to add meta fields.
How:
{
"time": 1616159900,
"host": "dataserver992.example.com",
"source": "log_file_name",
"sourcetype": "syslog",
"event": "2021-03-14T13:14:38-05:00 192.7.90.222 other syslog-y stuff",
"fields": {
"process": "my_app",
"instance_id": "my_container",
"instance_type": "container",
"image_id": "my_container_image",
"cloud_platform": "aws",
"region": "us-east-2"
}
}
Scenario 2b: With custom (flexible) syslog-like event string
When: For when your event payload is syslog-like, but there is nothing “sacred” about the event string pattern (sourcetype). We can do what we want with it.
What (A): Send all fields as indexed meta fields. (As with Scenario 2a).
What (B): Add all fields to event string.
What (C): Conditionally send some fields as meta fields and add some to event string.
Why (A): Saves disk not having it in event string. But won’t be searchable in event string. Indexed key/value pair searching is faster anyway.
Why (B): Increases disk consumption, but will be present in event string.
Why (C): You want some fields in your event string, but want some just as indexed fields.
appended text string data (using pattern of syslog event for designating key/value pair) to syslog event string. (Minor consideration: Appending to syslog could negatively impact any pattern-based extractions.)
How (A): Same as Scenario 2a – see Scenario 2a sample.
How (B):
{
"time": 1616159900,
"host": "dataserver992.example.com",
"source": "log_file_name",
"sourcetype": "syslog",
"event": "2021-03-14T13:14:38-05:00 192.7.90.222 other syslog-y stuff fieldname1=value1 fieldname2=value2 process=my_app instance_id=my_container instance_type=container image_id=my_container_image cloud_platform=aws region=us-east-2"
}
How (C):
{
"time": 1616159900,
"host": "dataserver992.example.com",
"source": "log_file_name",
"sourcetype": "syslog",
"event": "2021-03-14T13:14:38-05:00 192.7.90.222 other syslog-y stuff fieldname1=value1 fieldname2=value2 process=my_app instance_id=my_container instance_type="container" image_id=my_container_image",
"fields": {
"cloud_platform": "aws",
"region": "us-east-2"
}
}