By 2017, relying on Splunk’s default self-signed certificates wasn’t sufficient for any serious production environment. Whether you were dealing with regulatory frameworks like FedRAMP or just internal security policy, you needed to replace default certs across every connection surface.

This post outlines how we implemented internal PKI-backed TLS across Universal Forwarders, Deployment Server, Indexers, Search Heads, and Web UI. It covers certificate placement, config examples, validation tips, and failure points.

Why Replace Splunk’s Default Certs

  • Default certs are self-signed and shared across installs
  • No trust chain = warnings and audit failures
  • Can’t support client verification
  • Impossible to manage at scale

What Needs to Be Secured

  • Universal Forwarders ➝ Deployment Server (8089)
  • Universal Forwarders ➝ Indexers (9997)
  • Search Heads ➝ Indexers (management and search)
  • Web interface (8000)
  • Indexer Cluster comms
  • API and REST interfaces

Certificate Types

We used:

  • A single internal CA
  • Unique certs per host role (uf, idx, sh, ds, web)
  • Full PEM chain bundles (leaf, intermediate, root)
  • Encrypted private keys where required

Each cert was stored at:

$SPLUNK_HOME/etc/auth/certs/<role>.pem
$SPLUNK_HOME/etc/auth/certs/internal-rootCA.pem

Config Sections

Universal Forwarder ➝ Deployment Server (deploymentclient.conf)

[deployment-client]
clientName = uf01.company.com

[target-broker:deploymentServer]
targetUri=https://splunk-ds.internal.company.com:8089

[sslConfig]
sslRootCAPath = $SPLUNK_HOME/etc/auth/certs/internal-rootCA.pem
sslVerifyServerCert = true

Universal Forwarder ➝ Indexers (outputs.conf)

[tcpout:indexer_group]
server = idx1.internal.company.com:9997
sslCertPath = $SPLUNK_HOME/etc/auth/certs/uf.pem
sslPassword = $1$YourEncryptedPassword
sslRootCAPath = $SPLUNK_HOME/etc/auth/certs/internal-rootCA.pem
sslVerifyServerCert = true

Indexer Configuration (server.conf and inputs.conf)

[sslConfig]
enableSplunkdSSL = true
serverCert = $SPLUNK_HOME/etc/auth/certs/indexer.pem
sslRootCAPath = $SPLUNK_HOME/etc/auth/certs/internal-rootCA.pem
sslVerifyClient = true

[splunktcp-ssl:9997]
disabled = 0
requireClientCert = true

Web Interface (web.conf)

[settings]
enableSplunkWebSSL = true
serverCert = $SPLUNK_HOME/etc/auth/certs/web.pem
sslRootCAPath = $SPLUNK_HOME/etc/auth/certs/internal-rootCA.pem

Operational Notes

  • Certs must include the full chain if your CA uses intermediates
  • Permissions on cert and key files must allow Splunk user read access
  • Restart Splunk after applying cert changes
  • Validate with:
openssl s_client -connect idx1.internal.company.com:9997

Common Errors

Symptom Likely Cause
“unknown CA” Missing rootCA or broken chain
Forwarder won’t connect Hostname mismatch or sslVerifyServerCert=true
Web UI down Incorrect file path or unreadable key
Internal components fail to handshake Cert mismatch or server restarted without reload

Lessons Learned

  • Secure from day one—retrofits are painful and high risk
  • Use naming conventions for cert files by role and hostname
  • Schedule cert rotation with automation and validation
  • Always test on lower-tier environments before global rollout

Once we implemented this across the environment, our posture improved significantly. It also allowed us to meet internal compliance requirements with minimal recurring maintenance beyond cert renewals.