The Basics

The HTTP Event Collectors (HEC) are a passive REST endpoint on an array of Splunk Heavy Forwarder instances running in AWS Fargate. These nodes are currently setup in AWS behind an application load balancer across a single availability zone (AZ). It is recommended that you code your application for resiliency in the event of an AWS regional outage impacting an entire AZ. The pool of HEC nodes will automatically expand as use of the HEC method of sending data to Splunk increases.

Numerous guides and walk-throughs exist covering how to send events to the HEC tier. See “More documentation and samples to help you send data to Splunk via HEC” and “Other resources” below.)

Ultimately it comes down to an HTTP request to the Splunk HTTP Event Collector (HEC) with the appropriate token and event data such as this example using curl:

curl -k -H "Authorization: Splunk <YOUR-AUTH-TOKEN>" https://<ILLINOIS-SPLUNK-HEC-FQDN>/services/collector/event -d '{"time": "2021-03-14T13:14:38-05:00", "event": "http auth ftw!"}'

Note the three main components:

1. Authorization token (You must use the token given to you by your Splunk service administrator.)
   Though strongly discouraged, allowQueryStringAuth is available when no other option is available: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf
2. Endpoint URL  (There are more endpoint options, but most use cases are best served with the ‘services/collector/event’ endpoint. Feel free to explore and/or to reach out to us if you think another endpoint is more appropriate for your use case.)
   NOTE: When using /services/collector/event endpoint, know that unless you specify a “time” metadata value in the top level of your JSON, it will scan your JSON payload for a timestamp to “extract”. For that reason, if you aren’t specifying “time” metadata value, please be sure the timestamp you wish to be reflected in Splunk as the “time” value of the event is the first timestamp in the JSON payload and, ideally, as early in the payload as possible. If you don’t want Splunk to attempt to search for a timestamp in your JSON payload, send to /services/collector endpoint instead. (It is the same endpoint functionality but without the timestamp extraction.)
3. JSON payload between single quotes – a payload which, at minimum, must include the “event” element.

More documentation and samples to help you send data to Splunk via HEC

The JSON payload piece can be simple – but we’ve seen from experience that it can get a little complicated and confusing. For situations where you want/need to specify Splunk’s metadata fields (host, source, sourcetype, index, time) or add your own fields, Splunk offers some documentation (event metadata fields and custom fields) – but due to the demand for setting/adding fields and the challenge people have getting it configured correctly – we’ve supplemented Splunk’s documentation with our own:

    HEC: How to format JSON payload to set metadata or custom fields: When, What, Why, How

We’ve also created a page with some sample scripts for you. Please feel free to offer up your own, especially if it’s in a language/flavor we don’t have yet:

    Script samples for sending to HEC

Other resources

Send data to HTTP Event Collector from Splunk
https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector#Send_data_to_HTTP_Event_Collector

Splunk’s doc for “Formatting Events for HEC”:
https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector

• In particular, note this section documenting metadata fields you can set
  https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_metadata

Automate Indexed Field Extraction (i.e., “add custom fields”) with HEC
https://docs.splunk.com/Documentation/Splunk/latest/Data/IFXandHEC

Sending AWS Cloudwatch logs to HEC via a Lambda function
https://www.splunk.com/blog/2017/02/03/how-to-easily-stream-aws-cloudwatch-logs-to-splunk.html

Sending Logs from Docker Containers
https://www.tekstream.com/containerization-and-splunk-how-docker-and-splunk-work-together/

Please contact the splunk-support@illinois.edu team for tokens and endpoint information.