Install Splunk Agent

When forwarding events from endpoints a common method is to utilize a Splunk Universal Forwarder agent. This agent is installed locally and is configured to monitor log files or directories for changes.

Installing the agent to our knowledge has been non-disruptive, however we recommend that you test on a test machine before making any changes to production systems.

You may want to also consult any change control process before making any changes to production systems. 

LINUX – Installation

There are as many ways to install an application as there are flavors of Linux, so this section will provide an overview of installing a Splunk Universal Forwarder via a tar-ball file. The step-by-step instructions will be specific to CentOS 7/8, so please adapt your strategy accordingly!

Here is a link to the Splunk documenation for installing a Universal Forwarder on Linux:
https://docs.splunk.com/Documentation/Forwarder/8.2.1/Forwarder/Installanixuniversalforwarder

1) The recommendation from both Splunk and the Illinois Splunk team is that you run Splunk as a non-root user. You can use an existing user, however this user will need to be able to read the files and directories that you will want to monitor, it will require access to write to the Universal Forwarder’s directory and execute scripts there, and also bind to network ports that it is listening on. Therefore it is the Illinois Splunk team’s reccomendation that a dedicated user and group are created, named “splunk”. If this is not possible, skip this step and work with the group that manages your infrastructure to ensure the user you do user can accomplish the tasks above.

2) Download the Universal Forwarder tarball from our Splunk Versions and Locations page to a temporary location. The current version will be the most recent and stable release, and the latest version will be the the most recent release. This step will use wget, so please make sure that it is installed on your system prior to this step (in CentOS use $ sudo yum install wget).

3) The recommendation from Splunk is to install the Universal Forwarder in /opt, however this can be changed based on your own system’s configuration. Un-tar the file you just downloaded, either by using the -C flag or by navigating to the directory you want to install.

4) Currently the folder /opt/splunkforwarder and its contents are owned by root (or whatever user you used to install), so change ownership to the user that will be running Splunk.

5) If you are not planning on leveraging the power of the Splunk Deployment server to help you manage apps and connections to Splunk Cloud, skip this step at your own risk! If you do plan on utilizing the Deployment server, you will want to make sure that this app is in place before you start Splunk for the first time. Here you will use wget to get the default Deployment Server app, and then un-tar the .spl file in the appropriate folder. Before moving on to the next step, make sure that directory and file ownership are correct.

6) As the user that owns /opt/splunkforwarder, its finally time start Splunk, accept the license, create a local administrative user & password

7) Finally you will need to enable Splunk to start running at boot time. While there are a number of ways to do this, the method prefered by Splunk has either root or another user that can sudo run the following command

Congrats! You have successfully installed the Splunk Universal Forwarder! Contact splunk-admin@illinois.edu for next steps.

LINUX – Upgrading

The process for upgrading on Linux is straight forward, and we will continue with our CentOS 7/8 example above.

1) Download the Universal Forwarder tarball from our Splunk Versions and Locations page to a temporary location, and for this example we will wget the “latest” version of the Universal Forwarder available on our wiki page.

2) Stop Splunk

3) Un-tar the file you just downloaded, either by using the -C flag or by navigating to the directory you want to install.

5) Start Splunk, if you wish to review the license remove the flags

Congrats! You have successfully updated the Splunk Universal Forwarder!

WINDOWS

1. Download the Universal Forwarder Windows MSI from the Splunk Versions and Locations Wiki page. Consult with the Splunk Administrator as to either the current or latest version is appropriate for your setup. 
2. Run the executable as a user that has Administrator rights
   
3. Check the License Agreement box
4. GOTCHA #1 >You will need to make sure to select the “An on-premise Splunk Enterprise instance” option due to our hybrid configuration.
5. Click Customize Options
   
   
6. The default installation directory, C:Program FilesSplunkUniversalForwarder is preferred, however you are free to install anywhere. Once the preferred location is selected, click Next
   
   
7. SSL Certificates will be provided once the installation process is complete, so for this step click Next
   
   
   
   
   
8. Local System accounts are preferred unless you have a non-user service account to specifically run this as. If you use a Domain Account you will need to make sure you have permissions to the files/eventlogs that will be collected from the machine. Prefix the domain account name with uofiusername to properly create the service. We are still investigating Virtual Account as an option, so for now the typical installation uses the default, so click Next
   
   
9. GOTCHA #2 > Input settings will be managed by a local admin, and some can be setup by Splunk admin team via the deployment server. The Windows Event Logs will be best processed by having the Splunk Add-on for Microsoft Windows installed along with your UF, and by selecting anything in this menu input configurations may be placed in un-ideal locations which may lead to .conf conflicts further down the road. Leave all the checkboxes unmarked and click Next
   
10. Create a Username and Password that will be used to manage the Splunk application on your host. This user/password pair is only used locally, and you will need to provide them in step 17. Please save these values in a password manager as you will need to supply them once in a blue moon and you will never remember them. Once done click Next
    
    
11. Deployment Server settings should remain blank for this step. We will be setting up that connectivity to the Illinois Deployment Server in a later step. For now leave blank and click Next
    
12. Receiving Indexer settings should remain blank. A bundle of forwarding settings and certificates will be pushed from the Deployment Server. For now leave blank and click Next
    
13. You will receive a warning that we left the previous two settings blank. We are fine with this, so click OK
    
    
14. Click Install, watch the bar fill up, then click Finish
15. Download the Deployment Server configuration bundle from https://go.illinois.edu/default-deployment-server
16. Open a command prompt window & navigate to the bin directory of the SplunkUniversalForwarder. For example> cd C:\Program Files\SplunkUniversalForwarder\bin\
17. Run the following commands using the Username & Password you created in step 10 to load the default-deployment-server app and restart
    
    > .\splunk.exe install app  <full path to default-deployment-server.spl> -auth <username>:<password>
    
    
    > .\splunk.exe restart
    
    
    
    

    info

    
    

    AUDIBLE!

    
    

    
    

    Note, you can also extract the .spl bundle using a tool like 7-Zip and then copy the files to the C:Program FilesSplunkUniversalForwarderetcapps folder.

    
    

    
    
    
    
18. Installation is complete! Time to configure inputs.

Mac OSX

1. Download the current supported Universal Forward agent from here – https://go.illinois.edu/splunk-uf-mac-os10-current
2. Launch the .dmg file and Install Splunk Universal Forwarder
   1. 
3. The Installation window will launch and walk you through getting Splunk installed. Accept the License Agreement, select the location you want to install, and click Install.
   1. 
4. During the installation process you will be prompted to create a local administrative user and password. KEEP THESE SOMEPLACE SAFE! They are not recoverable and you will have to uninstall & reinstall Splunk if you forget them
   1. 
5. Installation is then completed.
   1. 
6. Once done, you can start Splunk for the first time a few ways:
   1. At the terminal run/Applications/SplunkForwarder/bin/splunkstart
      1. NOTE the default directory to install Splunk to is different than what the Installer states on the Summary page.
   2. A window will pop-up after the install process Called Splunk’s Little Helper, clicking Start Splunk is the same as running it from the terminal
      1. 
7. Download our Default Deployoment Server configuration from https://go.illinois.edu/default-deployment-server
8. If you installed Splunk and downloaded the configuration to the default directories, use the following command at terminal to configure the Universal Forwarder to communicate with the Deployment Server
   1. /Applications/SplunkForwarder/bin/splunk install app ~/Downloads/9-DEFAULT-splunk-deployment.spl -auth [admin you just created]:[password for admin]
9. Time to configure inputs.
   Depending on the local security settings there may be a few “gotchas!” with the installation. One such instance is when attempting to launch the installation, where it will block you from opening the file because its not an identified developer. Go to System Preferences > Security & Privacy and the blocked application should be shown below, click Open Anyway to continue.

# tar -xvf /tmp/splunkforwarder.tgz -C /opt
or
# cd /opt
# tar -xvf /tmp/splunkforwarder.tgz