How to optimize your searches

Noticing your searches are slow or getting “canceled” or “skipped”? Has your friendly Splunk administrator reported to you that Splunk is flagging your queries as “inefficient”?

Below are a few tips for improving/optimizing your search(es) that we want to highlight for our environment. The essential principle for all of the below is … “reduce, reduce, reduce”.

Also, please read this very helpful document on this topic from Splunk:

Specify Index(es)

index=apps-myapp_myunit

If you (or anyone else who might be running your search) have access to many indexes, be sure to start your search with an index filter.

INFO: The way Splunk is built, “indexes” are not just metadata values, they’re objects in their own filesystem directories. Failing to specify what index to search forces splunk to plow through the filesystem (the index directories you have access to) for matches to your search terms. If you have access to 21 indexes (typically, if you have access to it, it is included in your “default” set of indexes), specifying an index will help Splunk avoid plowing through every one of those directories. Upshot: It’s *always* good to specify index.

Add Splunk metadata filters early

sourcetype=cisco:asa host=vpn4g*

Splunk has certain “metadata” values that are “indexed”. (These are indexed in files that sit within those ‘index’ directories in filesystem mentioned above.)

• time (_time)
• index
• sourcetype
• host   (here at Illinois, we’ve also added indexed field “forwarder” for intermediary forwarders)
• source

It’s not *required* to use one or more of these metadata fields in your search, but wherever one of these *can* be used, it will help Splunk reduce its effort to find matches to your search terms. I.e., adding sourcetype=juniper to index=network-wired_techsvc at the beginning of your search could significantly reduce the amount of things Splunk has to match against. Upshot: Use metadata restrictions whenever it can help reduce.

To figure out what metadata fields you can add to your search to improve efficiency…

1. Run the more generic search in Verbose mode  (over a short period of time – or hit Stop button after collecting enough events)
2. Use the left pane to browse (click on) metadata fields (listed above) to identify a value or values that would help Splunk reduce your search without being overly restrictive.
3. Click on the item and then on “add to your search” to add it to your search (or add it manually).
4. Save search.

Avoid wildcard prefixes

Splunk will let you be as open-ended as possible with wildcards and wildcard prefixes… But if you’re searching a lot of data, a wildcard by itself or as a prefix forces Splunk to search all possible matches. Upshot: Be as specific as possible, and try to avoid using wildcards at the beginning of a string.

(See TIP in item immediately below for more ways to match against many values.)

Avoid NOT

Similar to the above concept, multiple, positive assertions like…

 index=myweblogs status=50* OR status=404 OR status=403

are far more efficient than …

 index=myweblogs NOT status=202

The latter forces Splunk to evaluate (initially include) every instance of status to make sure it’s not “202”. A positive assertion allows Splunk to evaluate (initially include) only the events with the matching values.

TIP: If you have a lot of positive assertions to match against, there are some tricks to make it easier than a bunch of successive OR statements.

• Use the IN operator (IN is case sensitive!). For example, the above could also be represented with…

      status IN (50*,404)

• Store your (positive) matching values in a lookup (csv) file and reference it in a lookup (subsearch):
  (Where below the lookup file is “acceptablestatuses.csv” and where “status” is the header of the field with the values we want to match on.)

      index=myindex [| inputlookup acceptablestatuses.csv | fields status]

• Very similar to the above, but uses a regular subsearch instead of a lookup file to return the desired set of values.

      index=myindex [search index=someindex action=success status=* | fields status]

Get familiar with Lispy (or “How to think like Splunk”)

One of the almost-magical things about Splunk is how fast it can get results from data stored in a very traditional, (relatively) unstructured way across many “buckets” on the filesystem. To optimize your searches, it helps to know how Splunk’s indexing works.

First, on ingest of data, Splunk evaluates each event string to create a list of “keywords” from it (these keywords, with links back to the original event, Splunk calls “tokens”). To “segment” the string in to keywords, it uses “Major” and “Minor” “breakers”:

Major Breakers:
   [ ] < > ( ) { } | ! ; , ' " * n r s t & ? + %21 %26 %2526 %3B %7C %20 %2B %3D -- %2520 %5D %5B %3A %0A %2C %28 %29

Minor Breakers: 
   / : = @ . -

nbsp; % \ _

In the following example, the green bars denote keywords generated from minor breakers and the blue bars denote keywords from major breakers.

OBSERVE: Notice case of keywords. Notice order of keywords. Notice how both major segments and minor segments are represented in the list of keywords.

Using the event and corresponding list of keywords above as a reference, the sample searches below help describe how Lispy is used for an initial filter (reduction).

How to check base Lispy used in search:

1. Run a query and wait for it to complete (or cancel it).
2. Select “Inspect Job” from “Job” menu under the search bar (to the right).
3. Click on the “search.log” link to open the contents of the search.log file in a browser window.
4. Search for “lispy” (CTRL-f or Command-f)

* Note that because searching for IP addresses or full CIDR representations can be a challenge for the reasons described above, Splunk offers a specific CIDR match function which works for both IP addresses and full CIDR values.

Related articles

Event segmentation and searching

How to optimize your searches

Guidance on implementing timestamps in your events