Splunk Universal Forwarder Troubleshooting

Deployment Client Side

1. Check Service status
   
   • Windows
     
     • Services -> SplunkForwarder [Status:  Running]
     • Else Start Service
       
   • Linux
     
     • /opt/splunkforwarder/bin/splunk status
     • /opt/splunkforwarder/bin/splunk start
     • /opt/splunkforwarder/bin/splunk restart
     • If systemctl configured
       • systemctl status splunkforwader
       • systemctl start splunkforwader
         
   • AIX
     
     • /opt/splunkforwarder/bin/splunk status
     • /opt/splunkforwarder/bin/splunk start
     • /opt/splunkforwarder/bin/splunk restart
     • stopsrc -a splunk
     • startsrc -a splunk
       
2. Check the Deployment Client configuration [verify targetUri Entry]
   
   • Windows
     
     • C:\Program Files\splunkforwarder\etc\system\local\deploymentclient.conf
     • C:\Program Files\splunkforwarder\etc\<deploymentclientapp>\local\deploymentclient.conf
       
   • Linux/AIX
     
     • /opt/splunkforwarder/splunk btool deploymentclient list –debug
     • /opt/splunkforwarder/etc/system/local/deploymentclient.conf                
     • /opt/ splunkforwarder/etc/apps/<deploymentclientapp>/local/ deploymentclient.conf
       
3. Check the ownership [If Splunk is running as non-root user]
   
   • Linux/AIX
     
     • chown -R splunk:splunk /opt/splunkforwarder
       
4. Check mod of the file reading path [If Splunk is running as non-root user]
   
   • Linux
     
     • To set acl to directory recursively:
       • setfacl -R -m u:splunk:r /var/log
         
     • To set acl for individual files:
       • setfacl -m u:splunk:r /var/log/messages ll /var/log/messages
         
     • To view the ACL
       • getfacl -R /var/log
         
   • AIX
     
     • To change a directory’s access:
       • setfacl -m splunk:r /var/log
         
     • To view the ACL
       • getfacl /var/log
         
   • chmod 644 /var/log  [If no other way 😊]
     
5. Network [Ports used in command are default one change as per yours]
   
   • Connectivity
     
     • Windows
       • telnet <deploymentserverIp> 8089
       • Test-Connection <deploymentserverIp> -TCPPort 8089   [Powershell]
         
     • Linux/AIX
       • telnet <deploymentserverIp> 8089
       • nc -nv <deploymentserverIp> 8089
       • nc -s <sourceip> <deploymentserverIp> 8089
         
     • If logs not received, change <deploymentserverIp> with <indexerIP> and 8089 with 9997
       
     • If you couldn’t connect check firewall [Local Firewall also]
       
   • Local Firewall
     
     • Windows
       • netsh advfirewall show allprofiles state
         
     • Linux
       • systemctl status firewalld
         
     • AIX
       • ipfstat
         
   • DNS [If you’re using hostname instead IP]
     
     • Windows/Linux/AIX
       • nslookup <deploymentserverhostname>
         
     • If not resolving IP,  check DNS / Host entry
       
     • Windows
       • ipconfig /displaydns
       • Make Entry in C:\Windows\System32\drivers\etc\hosts
         • 10.10.10.10       xxxxxx.xxxx.xx     xxxxxx
           
     • Linux/AIX
       • cat /etc/resolv.conf
       • edit /etc/hosts
         • 10.10.10.10       xxxxxx.xxxx.xx     xxxxxx
           
   • Note: If you’re using NAT, Use IP address as per that, instead of original IP
     
6.  Check Connection status
   
   • Windows
     • netstat -na | find “8089”
     • netstat -na | Select-String “8089” [Powershell]
       
   • Linux/AIX
     • netstat -an |grep 8089
     • tcpdump -i any host <deploymentserverIp>
       
7. Check the logs
   
   • Windows
     • C:\Program Files\Splunkforwarder\var\logs\splunk\splunkd.log
       
   • Linux/AIX
     • /opt/splunkforwarder/var/logs/splunkd.log
       
       
       
       

Deployment SeRVER Side

1. Check the Connectivity
   
   • Windows
     • Use Wireshark
       
   • Linux
     • tcpdump -i any host <deploymentclient IP>
       
   • If you can see traffic, connectivity is fine.
     
2. Check the Forwarder Management
   
   • In Clients, mention host or IP of the deployment client
     
   • verify the Machine Type and Phone Home
     
   • Check the server class part of the particular deployment client
     • Verify the clients
     • Confirm the apps assigned
       
3. From Search
   
   • index=_internal sourcetype=splunkd component=DC* Handshake | stats count by host
     
   • | rest /services/deployment/server/clients splunk_server=local | table hostname applications.*.serverclasses | untable hostname, applications, serverclass | rex field=applications “applications\.(?.+)\.serverclasses” | stats dc(hostname) as hostname by apps
     
4. From CLI
   
   • /opt/splunk/bin/splunk list deploy-clients

Incorporating best practices like using splunk btool, splunk diag, and monitoring internal logs can significantly improve your ability to troubleshoot efficiently.

Happy troubleshooting!

Deployment Client Side Deployment SeRVER Side Incorporating best practices like using splunk btool, splunk diag, and monitoring internal logs can significantly improve your ability to troubleshoot efficiently. Happy troubleshooting!