Upgrade Splunk Universal Forwarder Agent

There are two main install types of Splunk Forwarders: the Universal Forwarder collects data from endpoints and forwarders to indexers, while the Splunk Heavy Forwarder  parses data first then sends it to the indexing tier. 

We are not going to cover the Splunk Heavy Forwarder as that installation is rare and only required for niche use cases.

Linux

The below example leverages an Amazon Linux 2 AMI instance. Adjustments may need to be made depending on Linux versions and security posture (such as SE Linux)

1. 1. It is recommended that you run Splunk as a non-root user. This user will require access to the files that will be monitored possibly requiring permission adjustments.
      1. It is imperative that once you setup Splunk you do not manipulate any Splunk configuration files (located in /opt/splunkforwarder) under any other user than the user that runs the Splunk agent.
      2. In the following examples we assume that you are running the commands as the account that will be running the agent and you have been given the appropriate permissions to run the binaries.
      3. Please consult your OS specific documentation and your System Administration team for how to create users and adjust permissions so that you can run the agent not as a root user.
   2. Stop the Splunk agent.
      /opt/splunkforwarder/bin/splunk stop
   3. Download the agent from the current supported version located here – http://go.illinois.edu/splunk-uf-linux-current – into a temporary location.wget -O /tmp/universalforwarder-linux.tgz https://go.illinois.edu/splunk-uf-linux-current
   4. Extract the installation and discard of the downloaded file to your Splunk Forwarder directory. In this example we installed to the /opt directory, not all system configurations will allow this.
      tar -xvf /tmp/universalforwarder-linux.tgz -C /opt
      rm /tmp/universalforwarder-linux.tgz
      info

      NOTE – Splunk by default likes to be installed into /opt/splunkforwarder. You can adjust this location based on your system’s configuration.

   5. Run the forwarder for the first time replacing the password if prompted. Not all upgrade scenarios will prompt for a password./opt/splunkforwarder/bin/splunk start –accept-license –no-prompt –answer-yes
   6. To forward to the Splunk environment starting in 2019, you will need to perform the following changes to point to the new Deployment Server.
   7. Verify that there are no Deployment server settings located in $SPLUNK_HOME/etc/system/local/deploymentclient.conf . If they exist, comment them out with a #. The new default deployment-server app will take care of these.
      [target-broker:deploymentServer]
      #targetUri = splunk-deploy.opia.illinois.edu:8089
   8. If the Default Deployment-Server app has not been installed into $SPLUNK_HOME/etc/apps. Download the default app from https://go.illinois.edu/default-deployment-server and configure for the deployment-server. MAKE SURE YOU RESTART IMMEDIATELY as the permissions are initially very permissive. See instructions below if you do not know your local Splunk Universal Forwarder local admin password.
      wget -O /tmp/default-deployment-server.spl https://go.illinois.edu/default-deployment-server
      ### See instructions below if you do not know your local Splunk UF local admin password ###
      /opt/splunkforwarder/bin/splunk install app /tmp/default-deployment-server.spl -auth admin:password_you_entered_in_a_previous_step
      /opt/splunkforwarder/bin/splunk restart
      rm /tmp/default-deployment-server.spl

      If you are unsure what your password is, you can extract the SPL application using tar and put the folder directly into the apps directory located in $SPLUNK_HOME/etc/apps

      ### Stop the agent if it is not already stopped
      /opt/splunkforwarder/bin/splunk stop
      wget -O /tmp/default-deployment-server.spl https://go.illinois.edu/default-deployment-server
      tar xvf /tmp/default-deployment-server.spl -C /opt/splunkforwarder/etc/apps
      /opt/splunkforwarder/bin/splunk start
      rm /tmp/default-deployment-server.spl

Windows

1. Backup Splunk configuration files located in the etc folder. 
   1. Usually they are located in C:Program FilesSplunkUniversalForwarderetc
2. NOTE – If you are running Splunk as an administrator account you may need to perform Run As commands to launch or manipulate the Splunk configuration files. 
3. Download the agent from the current supported version located here and run it – http://go.illinois.edu/splunk-uf-windows-current
   1. Close any applications that Splunk warns are in conflict
4. Wait
   1. 
5. Complete
   1. 
6. Using explorer and your favorite text editor, verify that there are no Deployment server settings located in $SPLUNK_HOMEetcsystemlocaldeploymentclient.conf and comment them out
   1. [target-broker:deploymentServer]
      #targetUri = splunk-deploy.opia.illinois.edu:8089
7. Download the default-deployment-server configuration from https://go.illinois.edu/default-deployment-server
8. Open a a command prompt window
9. Navigate to the bin directory of the SplunkUniversalForwarder. For example cd Program FilesSplunkUniversalForwarderbin
10. Run the following commands to load the default-deployment-server app and restart
    1. splunk stop
       splunk install app  <full path to default-deployment-server.spl> -auth admin:password_you_entered_in_a_previous_step
       splunk start
    2. If you are unsure what your password is, you can extract the application using a tool like 7zip and put the folder directly into the apps directory located in $SPLUNK_HOMEetcapps
11. You are now ready to contact the Splunk Service team to finish whitelisting/configuration.