These steps assume that you already have a Splunk Universal Forwarder installed on your Linux host and that you want to start sending in system events using the Splunk Add-on for Unix and Linux (Splunk_TA_nix).
Frequently Asked Questions will be at the bottom of this page.
The first step will be to engage with the Illinois Splunk team so we can discuss where these events will be indexed. If your team currently uses our Deployment Server to distribute Splunk apps, the Illinois Splunk team will work to get you the correct version of the app in your Server Class. While that engagement is ongoing, you can prepare your host by installing the software dependencies, giving your “splunk” user the proper access to read log files, and adding audit.d rules.
Prep Work for the TA & audit.d # create a group dedicated to syslog groupadd syslog # add your splunk user to the new syslog group # below, splunk = your non-root user that runs splunk usermod -a -G syslog splunk # edit auditd.conf vi /etc/audit/auditd.conf # change the log_group from root to the new group # log_group = root ----> log_group = syslog # change log format to ENRICHED # log_format = RAW ----> log_format = ENRICHED # command to restart auditd to pick up that change service auditd restart # add access control lists for the folders & their sub-folders below for the new group setfacl -Rbkm d:g:syslog:r-X,g:syslog:r-X /var/log setfacl -Rbkm d:g:syslog:r-X,g:syslog:r-X /var/adm setfacl -Rbkm d:g:syslog:r-X,g:syslog:r-X /var/log/audit setfacl -Rbkm d:g:syslog:r-X,g:syslog:r-X /etc # install dependencies that the ta require to capture data yum -y install sysstat yum -y install ntpdate yum -y install lsof yum -y install nfs-utils yum -y install bash yum -y install chrony yum -y install net-tools
Next you will need to update the audit.rules by adding a new rule with everything listed in the following code blocks. The name and placement of the file are important, and the owner permissions need to be root. The name is important because the files take lexicographical precedence. The rules exclude some monitoring of splunkd and the user that runs splunk. You can remove the filters if you prefer, though we promote keeping them in place. Please search for and replace the following with the uid of your user running Splunk: “-F uid!=1001”
Create New audit.rules File vi /etc/audit/rules.d/50-audit.rules
https://github.com/Neo23x0/auditd/blob/master/audit.rules # ___ ___ __ __ # / | __ ______/ (_) /_____/ / # / /| |/ / / / __ / / __/ __ / # / ___ / /_/ / /_/ / / /_/ /_/ / # /_/ |_\__,_/\__,_/_/\__/\__,_/ # # Linux Audit Daemon - Best Practice Configuration # /etc/audit/audit.rules # # Compiled by Florian Roth # # Created : 2017/12/05 # Modified : 2021/09/22 # # Based on rules published here: # Gov.uk auditd rules # https://github.com/gds-operations/puppet-auditd/pull/1 # CentOS 7 hardening # https://highon.coffee/blog/security-harden-centos-7/#auditd---audit-daemon # Linux audit repo # https://github.com/linux-audit/audit-userspace/tree/master/rules # Auditd high performance linux auditing # https://linux-audit.com/tuning-auditd-high-performance-linux-auditing/ # # Further rules # For PCI DSS compliance see: # https://github.com/linux-audit/audit-userspace/blob/master/rules/30-pci-dss-v31.rules # For NISPOM compliance see: # https://github.com/linux-audit/audit-userspace/blob/master/rules/30-nispom.rules # Remove any existing rules -D # Buffer Size ## Feel free to increase this if the machine panic's -b 8192 # Failure Mode ## Possible values: 0 (silent), 1 (printk, print a failure message), 2 (panic, halt the system) -f 1 # Ignore errors ## e.g. caused by users or files not found in the local environment -i # Self Auditing --------------------------------------------------------------- ## Audit the audit logs ### Successful and unsuccessful attempts to read information from the audit records -w /var/log/audit/ -F comm!=splunkd -k auditlog ## Auditd configuration ### Modifications to audit configuration that occur while the audit collection functions are operating -w /etc/audit/ -p wa -k auditconfig -w /etc/libaudit.conf -p wa -k auditconfig -w /etc/audisp/ -p wa -k audispconfig ## Monitor for use of audit management tools -w /sbin/auditctl -p x -k audittools -w /sbin/auditd -p x -k audittools -w /usr/sbin/augenrules -p x -k audittools # Filters --------------------------------------------------------------------- ### We put these early because audit is a first match wins system. ## Ignore SELinux AVC records -a always,exclude -F msgtype=AVC ## Ignore current working directory records -a always,exclude -F msgtype=CWD ## Cron jobs fill the logs with stuff we normally don't want (works with SELinux) -a never,user -F subj_type=crond_t -a never,exit -F subj_type=crond_t ## This prevents chrony from overwhelming the logs -a never,exit -F arch=b64 -S adjtimex -F auid=unset -F uid=chrony -F subj_type=chronyd_t ## This is not very interesting and wastes a lot of space if the server is public facing -a always,exclude -F msgtype=CRYPTO_KEY_USER ## VMWare tools -a never,exit -F arch=b32 -S fork -F success=0 -F path=/usr/lib/vmware-tools -F subj_type=initrc_t -F exit=-2 -a never,exit -F arch=b64 -S fork -F success=0 -F path=/usr/lib/vmware-tools -F subj_type=initrc_t -F exit=-2 ## High Volume Event Filter (especially on Linux Workstations) -a never,exit -F arch=b32 -F dir=/dev/shm -k sharedmemaccess -a never,exit -F arch=b64 -F dir=/dev/shm -k sharedmemaccess -a never,exit -F arch=b32 -F dir=/var/lock/lvm -k locklvm -a never,exit -F arch=b64 -F dir=/var/lock/lvm -k locklvm ## FileBeat -a never,exit -F arch=b32 -F path=/opt/filebeat -k filebeat -a never,exit -F arch=b64 -F path=/opt/filebeat -k filebeat ## More information on how to filter events ### https://access.redhat.com/solutions/2482221 # Rules ----------------------------------------------------------------------- ## Kernel parameters -w /etc/sysctl.conf -p wa -k sysctl -w /etc/sysctl.d -p wa -k sysctl ## Kernel module loading and unloading -a always,exit -F perm=x -F auid!=-1 -F path=/sbin/insmod -k modules -a always,exit -F perm=x -F auid!=-1 -F path=/sbin/modprobe -k modules -a always,exit -F perm=x -F auid!=-1 -F path=/sbin/rmmod -k modules -a always,exit -F arch=b64 -S finit_module -S init_module -S delete_module -F auid!=-1 -k modules -a always,exit -F arch=b32 -S finit_module -S init_module -S delete_module -F auid!=-1 -k modules ## Modprobe configuration -w /etc/modprobe.conf -p wa -k modprobe -w /etc/modprobe.d -p wa -k modprobe ## KExec usage (all actions) -a always,exit -F arch=b64 -S kexec_load -k KEXEC -a always,exit -F arch=b32 -S sys_kexec_load -k KEXEC ## Special files -a always,exit -F arch=b32 -S mknod -S mknodat -k specialfiles -a always,exit -F arch=b64 -S mknod -S mknodat -k specialfiles ## Mount operations (only attributable) -a always,exit -F arch=b64 -S mount -S umount2 -F auid!=-1 -k mount -a always,exit -F arch=b32 -S mount -S umount -S umount2 -F auid!=-1 -k mount ## Change swap (only attributable) -a always,exit -F arch=b64 -S swapon -S swapoff -F auid!=-1 -k swap -a always,exit -F arch=b32 -S swapon -S swapoff -F auid!=-1 -k swap ## Time -a always,exit -F arch=b32 -F uid!=ntp -S adjtimex -S settimeofday -S clock_settime -k time -a always,exit -F arch=b64 -F uid!=ntp -S adjtimex -S settimeofday -S clock_settime -k time ### Local time zone -w /etc/localtime -p wa -k localtime ## Stunnel -w /usr/sbin/stunnel -p x -k stunnel -w /usr/bin/stunnel -p x -k stunnel ## Cron configuration & scheduled jobs -w /etc/cron.allow -p wa -k cron -w /etc/cron.deny -p wa -k cron -w /etc/cron.d/ -p wa -k cron -w /etc/cron.daily/ -p wa -k cron -w /etc/cron.hourly/ -p wa -k cron -w /etc/cron.monthly/ -p wa -k cron -w /etc/cron.weekly/ -p wa -k cron -w /etc/crontab -p wa -k cron -w /var/spool/cron/ -k cron ## User, group, password databases -w /etc/group -p wa -k etcgroup -w /etc/passwd -p wa -k etcpasswd -w /etc/gshadow -k etcgroup -w /etc/shadow -k etcpasswd -w /etc/security/opasswd -k opasswd ## Sudoers file changes -w /etc/sudoers -p wa -k actions -w /etc/sudoers.d/ -p wa -k actions ## Passwd -w /usr/bin/passwd -p x -k passwd_modification ## Tools to change group identifiers -w /usr/sbin/groupadd -p x -k group_modification -w /usr/sbin/groupmod -p x -k group_modification -w /usr/sbin/addgroup -p x -k group_modification -w /usr/sbin/useradd -p x -k user_modification -w /usr/sbin/userdel -p x -k user_modification -w /usr/sbin/usermod -p x -k user_modification -w /usr/sbin/adduser -p x -k user_modification ## Login configuration and information -w /etc/login.defs -p wa -k login -w /etc/securetty -p wa -k login -w /var/log/faillog -p wa -k login -w /var/log/lastlog -p wa -k login -w /var/log/tallylog -p wa -k login ## Network Environment ### Changes to hostname -a always,exit -F arch=b32 -S sethostname -S setdomainname -k network_modifications -a always,exit -F arch=b64 -S sethostname -S setdomainname -k network_modifications ### Successful IPv4 Connections -a always,exit -F arch=b64 -S connect -F a2=16 -F success=1 -F comm!=splunkd -F key=network_connect_4 -a always,exit -F arch=b32 -S connect -F a2=16 -F success=1 -F comm!=splunkd -F key=network_connect_4 ### Successful IPv6 Connections -a always,exit -F arch=b64 -S connect -F a2=28 -F success=1 -F comm!=splunkd -F key=network_connect_6 -a always,exit -F arch=b32 -S connect -F a2=28 -F success=1 -F comm!=splunkd -F key=network_connect_6 ### Changes to other files -w /etc/hosts -p wa -k network_modifications -w /etc/sysconfig/network -p wa -k network_modifications -w /etc/sysconfig/network-scripts -p w -k network_modifications -w /etc/network/ -p wa -k network -a always,exit -F dir=/etc/NetworkManager/ -F perm=wa -k network_modifications ### Changes to issue -w /etc/issue -p wa -k etcissue -w /etc/issue.net -p wa -k etcissue ## System startup scripts -w /etc/inittab -p wa -k init -w /etc/init.d/ -p wa -k init -w /etc/init/ -p wa -k init ## Library search paths -w /etc/ld.so.conf -p wa -k libpath -w /etc/ld.so.conf.d -p wa -k libpath ## Systemwide library preloads (LD_PRELOAD) -w /etc/ld.so.preload -p wa -k systemwide_preloads ## Pam configuration -w /etc/pam.d/ -p wa -k pam -w /etc/security/limits.conf -p wa -k pam -w /etc/security/limits.d -p wa -k pam -w /etc/security/pam_env.conf -p wa -k pam -w /etc/security/namespace.conf -p wa -k pam -w /etc/security/namespace.d -p wa -k pam -w /etc/security/namespace.init -p wa -k pam ## Mail configuration -w /etc/aliases -p wa -k mail -w /etc/postfix/ -p wa -k mail -w /etc/exim4/ -p wa -k mail ## SSH configuration -w /etc/ssh/sshd_config -k sshd -w /etc/ssh/sshd_config.d -k sshd ## root ssh key tampering -w /root/.ssh -p wa -k rootkey # Systemd -w /bin/systemctl -p x -k systemd -w /etc/systemd/ -p wa -k systemd ## SELinux events that modify the system's Mandatory Access Controls (MAC) -w /etc/selinux/ -p wa -k mac_policy ## Critical elements access failures -a always,exit -F arch=b64 -S open -F dir=/etc -F success=0 -k unauthedfileaccess -a always,exit -F arch=b64 -S open -F dir=/bin -F success=0 -k unauthedfileaccess -a always,exit -F arch=b64 -S open -F dir=/sbin -F success=0 -k unauthedfileaccess -a always,exit -F arch=b64 -S open -F dir=/usr/bin -F success=0 -k unauthedfileaccess -a always,exit -F arch=b64 -S open -F dir=/usr/sbin -F success=0 -k unauthedfileaccess -a always,exit -F arch=b64 -S open -F dir=/var -F success=0 -k unauthedfileaccess -a always,exit -F arch=b64 -S open -F dir=/home -F success=0 -k unauthedfileaccess -a always,exit -F arch=b64 -S open -F dir=/srv -F success=0 -k unauthedfileaccess ## Process ID change (switching accounts) applications -w /bin/su -p x -k priv_esc -w /usr/bin/sudo -p x -k priv_esc -w /etc/sudoers -p rw -k priv_esc -w /etc/sudoers.d -p rw -k priv_esc ## Power state -w /sbin/shutdown -p x -k power -w /sbin/poweroff -p x -k power -w /sbin/reboot -p x -k power -w /sbin/halt -p x -k power ## Session initiation information -w /var/run/utmp -p wa -k session -w /var/log/btmp -p wa -k session -w /var/log/wtmp -p wa -k session ## Discretionary Access Control (DAC) modifications -a always,exit -F arch=b32 -S chmod -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b32 -S chown -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b32 -S fchmod -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b32 -S fchmodat -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b32 -S fchown -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b32 -S fchownat -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b32 -S fremovexattr -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b32 -S fsetxattr -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b32 -S lchown -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b32 -S lremovexattr -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b32 -S lsetxattr -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b32 -S removexattr -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b32 -S setxattr -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b64 -S chmod -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b64 -S chown -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b64 -S fchmod -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b64 -S fchmodat -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b64 -S fchown -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b64 -S fchownat -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b64 -S fremovexattr -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b64 -S fsetxattr -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b64 -S lchown -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b64 -S lremovexattr -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b64 -S lsetxattr -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b64 -S removexattr -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod -a always,exit -F arch=b64 -S setxattr -F auid>=1000 -F auid!=-1 comm!=splunkd -k perm_mod # Special Rules --------------------------------------------------------------- ## Reconnaissance ## uid = 1001 = splunk -w /usr/bin/whoami -p x -k recon -w /usr/bin/id -p x -k recon # -w /bin/hostname -p x -k recon -w /bin/hostname -p x -F uid!=1001 -k recon # -w /bin/uname -p x -k recon -w /bin/uname -p x -F uid!=1001 -k recon -w /etc/issue -p r -k recon -w /etc/hostname -p r -k recon ## Suspicious activity -w /usr/bin/wget -p x -k susp_activity -w /usr/bin/curl -p x -k susp_activity -w /usr/bin/base64 -p x -k susp_activity -w /bin/nc -p x -k susp_activity -w /bin/netcat -p x -k susp_activity -w /usr/bin/ncat -p x -k susp_activity -w /usr/bin/ssh -p x -k susp_activity -w /usr/bin/scp -p x -k susp_activity -w /usr/bin/sftp -p x -k susp_activity -w /usr/bin/ftp -p x -k susp_activity -w /usr/bin/socat -p x -k susp_activity -w /usr/bin/wireshark -p x -k susp_activity -w /usr/bin/tshark -p x -k susp_activity -w /usr/bin/rawshark -p x -k susp_activity -w /usr/bin/rdesktop -p x -k susp_activity -w /usr/bin/nmap -p x -k susp_activity ## Added to catch netcat on Ubuntu -w /bin/nc.openbsd -p x -k susp_activity -w /bin/nc.traditional -p x -k susp_activity ## Sbin suspicious activity -w /sbin/iptables -p x -k sbin_susp -w /sbin/ip6tables -p x -k sbin_susp -w /sbin/ifconfig -p x -k sbin_susp -w /usr/sbin/arptables -p x -k sbin_susp -w /usr/sbin/ebtables -p x -k sbin_susp -w /sbin/xtables-nft-multi -p x -k sbin_susp -w /usr/sbin/nft -p x -k sbin_susp -w /usr/sbin/tcpdump -p x -k sbin_susp -w /usr/sbin/traceroute -p x -k sbin_susp -w /usr/sbin/ufw -p x -k sbin_susp ## dbus-send invocation ### may indicate privilege escalation CVE-2021-3560 -w /usr/bin/dbus-send -p x -k dbus_send ## pkexec invocation ### may indicate privilege escalation CVE-2021-4034 -w /usr/bin/pkexec -p x -k pkexec ## Suspicious shells #-w /bin/ash -p x -k susp_shell #-w /bin/bash -p x -k susp_shell #-w /bin/csh -p x -k susp_shell #-w /bin/dash -p x -k susp_shell #-w /bin/busybox -p x -k susp_shell #-w /bin/ksh -p x -k susp_shell #-w /bin/fish -p x -k susp_shell #-w /bin/tcsh -p x -k susp_shell #-w /bin/tclsh -p x -k susp_shell #-w /bin/zsh -p x -k susp_shell ## Shell/profile configurations -w /etc/profile.d/ -p wa -k shell_profiles -w /etc/profile -p wa -k shell_profiles -w /etc/shells -p wa -k shell_profiles -w /etc/bashrc -p wa -k shell_profiles -w /etc/csh.cshrc -p wa -k shell_profiles -w /etc/csh.login -p wa -k shell_profiles -w /etc/fish/ -p wa -k shell_profiles -w /etc/zsh/ -p wa -k shell_profiles ## Injection ### These rules watch for code injection by the ptrace facility. ### This could indicate someone trying to do something bad or just debugging -a always,exit -F arch=b32 -S ptrace -F a0=0x4 -k code_injection -a always,exit -F arch=b64 -S ptrace -F a0=0x4 -k code_injection -a always,exit -F arch=b32 -S ptrace -F a0=0x5 -k data_injection -a always,exit -F arch=b64 -S ptrace -F a0=0x5 -k data_injection -a always,exit -F arch=b32 -S ptrace -F a0=0x6 -k register_injection -a always,exit -F arch=b64 -S ptrace -F a0=0x6 -k register_injection -a always,exit -F arch=b32 -S ptrace -k tracing -a always,exit -F arch=b64 -S ptrace -k tracing ## Privilege Abuse ### The purpose of this rule is to detect when an admin may be abusing power by looking in user's home dir. -a always,exit -F dir=/home -F uid=0 -F auid>=1000 -F auid!=-1 -C auid!=obj_uid -k power_abuse # Software Management --------------------------------------------------------- # RPM (Redhat/CentOS) -w /usr/bin/rpm -p x -k software_mgmt -w /usr/bin/yum -p x -k software_mgmt # DNF (Fedora/RedHat 8/CentOS 8) -w /usr/bin/dnf -p x -k software_mgmt # YAST/Zypper/RPM (SuSE) -w /sbin/yast -p x -k software_mgmt -w /sbin/yast2 -p x -k software_mgmt -w /bin/rpm -p x -k software_mgmt -w /usr/bin/zypper -k software_mgmt # DPKG / APT-GET (Debian/Ubuntu) -w /usr/bin/dpkg -p x -k software_mgmt -w /usr/bin/apt -p x -k software_mgmt -w /usr/bin/apt-add-repository -p x -k software_mgmt -w /usr/bin/apt-get -p x -k software_mgmt -w /usr/bin/aptitude -p x -k software_mgmt -w /usr/bin/wajig -p x -k software_mgmt -w /usr/bin/snap -p x -k software_mgmt # PIP (Python installs) -w /usr/bin/pip -p x -k software_mgmt -w /usr/bin/pip3 -p x -k software_mgmt # Special Software ------------------------------------------------------------ ## GDS specific secrets -w /etc/puppet/ssl -p wa -k puppet_ssl ## IBM Bigfix BESClient -a always,exit -F arch=b64 -S open -F dir=/opt/BESClient -F success=0 -k soft_besclient -w /var/opt/BESClient/ -p wa -k soft_besclient ## CHEF https://www.chef.io/chef/ -w /etc/chef -p wa -k soft_chef ## Docker -w /usr/bin/dockerd -k docker -w /usr/bin/docker -k docker -w /usr/bin/docker-containerd -k docker -w /usr/bin/docker-runc -k docker -w /var/lib/docker -k docker -w /etc/docker -k docker -w /etc/sysconfig/docker -k docker -w /etc/sysconfig/docker-storage -k docker -w /usr/lib/systemd/system/docker.service -k docker ## Kubelet -w /usr/bin/kubelet -k kubelet # High Volume Events ---------------------------------------------------------- ## Remove them if they cause to much volume in your environment ## Root command executions -a always,exit -F arch=b64 -F euid=0 -S execve -k rootcmd -a always,exit -F arch=b32 -F euid=0 -S execve -k rootcmd ## File Deletion Events by User -a always,exit -F arch=b32 -S rmdir -S unlink -S unlinkat -S rename -S renameat -F auid>=1000 -F auid!=-1 -F comm!=splunkd -k delete -a always,exit -F arch=b64 -S rmdir -S unlink -S unlinkat -S rename -S renameat -F auid>=1000 -F auid!=-1 -F comm!=splunkd -k delete ## File Access ### Unauthorized Access (unsuccessful) ## uid = 1001 = splunk -a always,exit -F arch=b32 -S creat -S open -S openat -S open_by_handle_at -S truncate -S ftruncate -F exit=-EACCES -F auid>=1000 -F auid!=-1 -F uid!= 1001 -k file_access -a always,exit -F arch=b32 -S creat -S open -S openat -S open_by_handle_at -S truncate -S ftruncate -F exit=-EPERM -F auid>=1000 -F auid!=-1 -F uid!= 1001 -k file_access -a always,exit -F arch=b64 -S creat -S open -S openat -S open_by_handle_at -S truncate -S ftruncate -F exit=-EACCES -F auid>=1000 -F auid!=-1 -F uid!= 1001 -k file_access -a always,exit -F arch=b64 -S creat -S open -S openat -S open_by_handle_at -S truncate -S ftruncate -F exit=-EPERM -F auid>=1000 -F auid!=-1 -F uid!= 1001 -k file_access ### Unsuccessful Creation -a always,exit -F arch=b32 -S creat,link,mknod,mkdir,symlink,mknodat,linkat,symlinkat -F exit=-EACCES -k file_creation -a always,exit -F arch=b64 -S mkdir,creat,link,symlink,mknod,mknodat,linkat,symlinkat -F exit=-EACCES -k file_creation -a always,exit -F arch=b32 -S link,mkdir,symlink,mkdirat -F exit=-EPERM -k file_creation -a always,exit -F arch=b64 -S mkdir,link,symlink,mkdirat -F exit=-EPERM -k file_creation ### Unsuccessful Modification -a always,exit -F arch=b32 -S rename -S renameat -S truncate -S chmod -S setxattr -S lsetxattr -S removexattr -S lremovexattr -F exit=-EACCES -k file_modification -a always,exit -F arch=b64 -S rename -S renameat -S truncate -S chmod -S setxattr -S lsetxattr -S removexattr -S lremovexattr -F exit=-EACCES -k file_modification -a always,exit -F arch=b32 -S rename -S renameat -S truncate -S chmod -S setxattr -S lsetxattr -S removexattr -S lremovexattr -F exit=-EPERM -k file_modification -a always,exit -F arch=b64 -S rename -S renameat -S truncate -S chmod -S setxattr -S lsetxattr -S removexattr -S lremovexattr -F exit=-EPERM -k file_modification ## 32bit API Exploitation ### If you are on a 64 bit platform, everything _should_ be running ### in 64 bit mode. This rule will detect any use of the 32 bit syscalls ### because this might be a sign of someone exploiting a hole in the 32 ### bit API. -a always,exit -F arch=b32 -S all -k 32bit_api
Restart auditd
# command to restart auditd to start using the newly created rulesservice auditd restart |
The following code block is optional, but encouraged. Adding this line and then rebooting the host forces auditd to not accept any changes to it’s rules without rebooting the host.
CIS Benchmark 4.1.17
echo "-e 2" >> /etc/audit/rules.d/99-finalize.rules# Reboot to take effect |
Once your host is prepped, the Illinois Splunk team will work with you on a time to add the new TA to your required Server Classes, and your UFs will grab the new app and initiate a restart of Splunk. We will then check the destination index to make sure events are flowing in properly.
Attached here is documentation relating to the University of Illinois Systems IT04 Security Standards & the CIS CentOS Linux 7 Benchmark
IT04-Server Security Standard_V1.0
CIS_CentOS_Linux_7_Benchmark_v3.1.2
The following code blocks are the currently enabled inputs for Splunk_TA_nix with adjusted intervals and a transform to correctly extract ps metrics. Please replace the <METRIC INDEX> and <EVENT INDEX> values with the proper index names in Splunk Cloud.
SPLUNK_HOME/etc/apps/Splunk_TA_nix/local/inputs.conf
############################################################## Metric Inputs ###################################################################[script://./bin/vmstat_metric.sh]disabled = falseindex = <METRIC INDEX>interval = 60[script://./bin/iostat_metric.sh]disabled = falseindex = <METRIC INDEX>interval = 60[script://./bin/ps_metric.sh]disabled = falseindex = <METRIC INDEX>interval = 30[script://./bin/df_metric.sh]disabled = falseindex = <METRIC INDEX>interval = 300[script://./bin/interfaces_metric.sh]disabled = falseindex = <METRIC INDEX>interval = 60[script://./bin/cpu_metric.sh]disabled = falseindex = <METRIC INDEX>interval = 30############################################################### Event Inputs #################################################################################### Monitors #####################[monitor:///Library/Logs]disabled = falseindex = <EVENT INDEX>[monitor:///var/log]disabled = falseindex = <EVENT INDEX>[monitor:///var/adm]disabled = falseindex = <EVENT INDEX>[monitor:///var/log/maillog*]sourcetype = postfix_syslogdisabled = falseindex = <EVENT INDEX>[monitor:///var/log/firewalld*]sourcetype = syslogdisabled = falseindex = <EVENT INDEX>################# Scripts ####################### Shows current user sessions[script://./bin/who.sh]sourcetype = whosource = whointerval = 300disabled = falseindex = <EVENT INDEX># Shows last login time for users who have ever logged in[script://./bin/lastlog.sh]sourcetype = lastlogsource = lastloginterval = 3600disabled = falseindex = <EVENT INDEX># Lists users who could login (i.e., they are assigned a login shell)[script://./bin/usersWithLoginPrivs.sh]sourcetype = usersWithLoginPrivssource = usersWithLoginPrivsinterval = 3600disabled = falseindex = <EVENT INDEX># Currently only supports SunOS, Linux, OSX.# May require Splunk forwarder to run as root on some platforms.[script://./bin/service.sh]interval = 3600source = Unix:Servicesourcetype = Unix:Servicedisabled = falseindex = <EVENT INDEX>[script://./bin/time.sh]interval = 3600sourcetype = timesource = timedisabled = falseindex = <EVENT INDEX>[script://./bin/bandwidth.sh]interval = 300sourcetype = bandwidthsource = bandwidthdisabled = falseindex = <EVENT INDEX>[script://./bin/protocol.sh]interval = 300sourcetype = protocolsource = protocoldisabled = falseindex = <EVENT INDEX>[script://./bin/openPorts.sh]interval = 3600sourcetype = openPortssource = openPortsdisabled = falseindex = <EVENT INDEX>################ ES Support ####################[script://./bin/passwd.sh]interval = 3600source = Unix:UserAccountssourcetype = Unix:UserAccountsdisabled = falseindex = <EVENT INDEX>[script://./bin/uptime.sh]interval = 86400source = Unix:Uptimesourcetype = Unix:Uptimedisabled = falseindex = <EVENT INDEX>[script://./bin/version.sh]interval = 86400source = Unix:Versionsourcetype = Unix:Versiondisabled = falseindex = <EVENT INDEX>[script://./bin/hardware.sh]sourcetype = hardwaresource = hardwareinterval = 86400disabled = falseindex = <EVENT INDEX> |
SPLUNK_HOME/etc/apps/Splunk_TA_nix/local/transforms.conf
[metric-schema:extract_metrics_ps]METRIC-SCHEMA-MEASURES= pctCPU,pctMEM,pctNice,RSZ_KB,VSZ_KBMETRIC-SCHEMA-BLACKLIST-DIMS= S,CPUTIME,ELAPSED,TTY,OSName |
Frequently Asked Questions
Q: Where will my events be indexed?
A: System audit logs and metrics will be sent to indexes that have a 90 day retention period and are available and viewable by all employees in the department. For example, since the Splunk hosts are a part of Technology Services, they are indexed to ‘servers-linux_techsvc’ and ‘servers-linux-metrics_techsvc’
Q: The audit.rules file is a HUGE wall of text… how did you decide to implement them?
A: The starting point was the CIS CentOS Linux 7 Benchmark document and we implemented the rules in sections 4.1.3 – 4.1.17. Even after adding filters for Splunk, the number of events being generated was in the thousands per hour and included a number of errors. While investigating options to reduce the errors, I came across the GitHub repo above. It included everything we were looking for plus more, so we decided to implement it and added filters to reduce the amount of logging.
Q: Speaking of events, how many events does this TA generate?
A: I won’t be able to answer to every system, however the Universal Forwarders in our On-Campus tier average approximately 1,250 events/hour and 110,000 metrics/hour.