Contents
Methods
Splunk configurations can be managed remotely or locally at the endpoint.
Deployment Server configured settings would normally be for several systems that have common monitoring needs. Cases such as a production, test, and development server or a web server farm are use cases where the deployment server configurations are usually a good fit.
One off locally configured settings supersede all Deployment Server configurations. These can cause issues when evaluating which configurations are being used and in what cases. A tool for reconciling these is the Splunk btool command.
It is recommended that a system have either locally configured or deployment server configured settings and not to mix the two methodologies.
Best Practice
Things happen. Make sure that you have adequate disk space in the event that a network interruption occurs and the endpoint is unable to send the logs to the Splunk Forwarder tier. Three days should be more than enough.Though very rare, it is also a good idea to keep a few days of logs locally before purging them in the event that they arrive at the Splunk Index Tier corrupted or you need to reprocess them.
Locally Configured
The monitoring of files is completed with stanzas in the splunkforwarder/etc/system/local/inputs.conf file.
These can be manipulated either by directly editing the files or using the Splunk CLI.
Splunk CLI
The Splunk CLI is the team’s least preferred way of adding permanent inputs behind editing the inputs.conf and using the deployment server. Feeling compelled to provide a rounded documentation experience though here is some documentation about this – https://docs.splunk.com/Documentation/Splunk/7.1.2/Data/MonitorfilesanddirectoriesusingtheCLI
This method works well for scripted agent installations where direct editing of the inputs.conf file is not easily performed and you do not want to use the deployment server to centrally manage settings.
$SPLUNK_DIR/bin/splunk add monitor /var/log/
Inputs.conf File
The inputs.conf file is the preferred method for setting up local inputs.
vi $SPLUNK_DIR/etc/system/local/inputs.conf
Please leave the default host information at the top of the file and add stanzas pointing to the individual files that you will want monitored such as the following example blocks.
You can monitor entire directories such as the final example below, though you should only do this when the folder contains the same sourcetype of data. For example, do not put in a single /var/log directory stanza.
The sourcetypes are important from a parsing perspective and we recommend as many sourcetypes as possible at the ingestion of data. Links regarding sourcetype information are found below. Sourcetypes are how the Splunk application will tie together the field extractions and properties and vary wildly in complexity. If you leave out the “sourcetype =” line in the stanza it will automatically assign a sourcetype. That helps a bit for any standard formats of files such as Apache, JSON, or IIS logs, though you want to avoid for anything that is non-standard.
To structure or NOT to structure…
Splunk does very well with structured data such as JSON or XML. This data can be automatically parsed based on the field names and survives well through application upgrades/life-cycles. If you upgrade and a new field is introduced or a subtle format changes, file formats such as CSV or TSV can go awry. JSON is the preferred file format for the Splunk Service team’s own purposes.With that said, JSON and XML both contain lots of information not related to an event entry that will increase your storage size/costs associated with the input in the form of header information, formatting, and version info.
Any changes to inputs.conf will require a Splunk restart to read in the changes.
$SPLUNK_DIR/bin/splunk restart
Settings for the inputs file can be setup for files or directories. The full documentation for the inputs.conf file can be found here – http://docs.splunk.com/Documentation/Splunk/7.0.1/Data/Monitorfilesanddirectorieswithinputs.conf
Comments are supported at the beginning of lines only. They are not supported inline on a stanza.#This is a good commentindex=service_idx #This is a comment that will break your agent
The following code blocks are examples that are useful as a starting point. Make sure to change your sourcetype and index according to use cases.
#monitor single file of the service_access_log sourcetype[monitor:///absolute/path/to/file/tobe/indexed1.log]index = service_idxsourcetype = service_access_log#monitor a second file of the service_app_log sourcetype[monitor:///absolute/path/to/file/indexed2.txt]index = service_idxsourcetype = service_app_log#monitor a sub-folder of an unknown path[monitor:///absolute/path/to/apache/…/logs]index = service_idxsourcetype = access_combined#monitor a sub-folder of files, base the host information in Splunk on the path of the file, do not index any compressed files## think syslog[monitor:///services/syslog-filter/logs/maillogs/relays0[1-3]*/.../local*]disabled = falsesourcetype = sendmailhost_segment = 5index = service_idxfollowSymlink = falseblacklist = \.(gz|bz2|z|zip|tar\.gz|tgz|tbz|tbz2)Sourcetypes
Sourcetypes are a basic way that Splunk recognizes data. These are incredibly important to making sure that timestamps are correct and events are broke along the correct lines.
There are numerous pre-trained sourcetypes available in Splunk plus hundreds more that can be added through apps. The Splunk Service team also has the ability to train custom sourcetypes.
If you are unsure about sourcetypes, go with automatic at first (do not set a sourcetype in inputs.conf), see how it looks in Splunk and we will work from there. This is the primary reason we send to the splunk-on-ramp server first.
After setting up your forwarder and inputs.conf file, we will work on next steps.
If you are brave, you can also check out agent advanced topics.
Deployment Server Configured
Please work with the Splunk Administrator to create the following components
- Deployment Server App
- Add to Server Class
- Inputs.conf
- File stanzas