Splunk got a VM, not a container

The cluster already had a pattern for new services: stand up an LXC, put it behind the reverse proxy, move on. Splunk does not fit that pattern.

Indexes want disk and RAM that survive a container rebuild. The IDS sensor already has a job. Grafana already has a job. The SIEM needed its own guest. This is how that guest was created, how Splunk was installed on it, and what actually sends it data.

Create the VM

Ubuntu 24.04 cloud image on Proxmox. 16 GB RAM, 8 vCPU, 200 GB local disk. Virtio NIC, QEMU guest agent, serial console, start on boot. That sizing is for live syslog plus search, not a default LXC template.

Before qm create, the script pings the reserved address and checks the neighbor table. If something already answers, it aborts. The writeup said the slot was free. The LAN might disagree. A collision means two guests sharing an address. Same failure mode as the Ansible crash.

ping -c 1 -W 2 "$IP" && exit 1

Then:

1. Download the Ubuntu 24.04 cloud image if it is not already on the hypervisor.
2. qm create with host CPU, virtio net, guest agent, onboot 1.
3. Import the disk, attach it as scsi0, add a cloud-init drive, resize to 200 G.
4. Cloud-init: root SSH key, static address, gateway, DNS. Same values the ping just proved were unused.
5. qm start. Wait until SSH answers.

The package is not installed on the hypervisor. The hypervisor only creates the guest.

Install Splunk

SSH in as root on the new VM.

1. Install wget, curl, ca-certificates, libxml2.
2. Download Splunk Enterprise 9.4 (Linux amd64 .deb) from Splunk’s release URL.
3. dpkg -i the package. Own /opt/splunk as user splunk.
4. Generate an admin password. Write it to a root-only file on that guest (/root/splunk-admin.env). It does not go in git.
5. Start Splunk as splunk, accept the license, seed the admin password.
6. splunk enable boot-start -user splunk so a reboot brings the indexer back.

sudo -u splunk /opt/splunk/bin/splunk start \
  --accept-license --answer-yes --no-prompt --seed-passwd "$admin_pass"
/opt/splunk/bin/splunk enable boot-start -user splunk \
  --accept-license --answer-yes --no-prompt

Then create the baseline indexes and the first listener:

for idx in firewall network cloud os app; do
  sudo -u splunk /opt/splunk/bin/splunk add index "$idx" -auth "admin:${admin_pass}"
done

sudo -u splunk /opt/splunk/bin/splunk add udp 1514 \
  -sourcetype syslog -index os -auth "admin:${admin_pass}"

UDP 1514 is the syslog listener Splunk owns. Port 514 needs a privileged bind. Senders point at 1514.

If UFW is on, allow UDP 1514 and 1515, plus TCP 8000 (web), 8088 (HEC), and 9997 (forwarders). Splunk Web stays on the lab network. It is not a public URL.

Listeners and indexes after install

A second pass on the guest adds the IDS listener and retention. UDP 1515 lands Suricata/Zeek in index ids. Sysctl raises UDP buffers. The splunk user gets higher file and process limits. Disk caps keep a 200 GB guest from filling the OS volume.

Listeners:

• UDP 1514, sourcetype syslog. Default index os. Firewall, UniFi, and guest syslog share this port; index routing is per source.
• UDP 1515, sourcetype suricata. Index ids.

Indexes:

• firewall: edge firewall filterlog
• network: UniFi controller, switch, AP
• os: Linux guest syslog
• app: application guests
• ids: Security Onion Suricata and Zeek
• cloud and aws: CloudTrail evidence

ids and aws are created when those feeds are wired, not in the first index loop.

Sources

Feeds were pointed after the box existed. Each source is a role, an index, and a path.

Edge firewall (index firewall)

pfSense remote syslog over the REST API. Filter, DHCP, system, resolver, auth, and VPN logs. Source interface is LAN. Target is the Splunk guest, UDP 1514.

The API rejects a PATCH while extra remote-server fields are empty strings. Disable remote logging first, then apply the full settings blob with enableremotelogging true.

Verify with traffic through the firewall, then search index=firewall OR index=os sourcetype=syslog earliest=-15m.

UniFi (index network)

Controller rsyslog: device, client, admin, security, switches, access points. Same Splunk host, UDP 1514. Adopted switch and AP follow the controller setting.

Linux guests (indexes os and app)

rsyslog omfwd UDP to 1514. Hypervisors and LXCs get a drop-in under /etc/rsyslog.d/. Ubuntu 24 guests that do not load omfwd ship journal lines with a small systemd unit instead of a missing rsyslog module.

*.* action(type="omfwd" target="$SPLUNK_HOST" port="1514" protocol="udp")

IDS sensor (index ids)

Security Onion stays the sensor. A Logstash pipeline on that box forwards Suricata and Zeek datasets to Splunk UDP 1515 as JSON lines. Splunk does not run on the IDS guest.

CloudTrail (index aws)

IAM evidence JSON into index aws with a CloudTrail sourcetype. Same indexer as the firewall logs. Different index.

The Splunk project is the ingest map. This note is the stand-up.

What sits in front of it

Internal only. Wildcard TLS and the reverse proxy in front of Splunk Web (8000). No public Splunk URL. Admin password stays on the guest.

Honest scope

This is not Terraform. Ansible has the host in inventory. The recovery path is still two bash scripts: create the VM on the hypervisor, install the package on the guest, then point each source.

Splunk is on a trial/dev license. Not a 24/7 SOC contract. Searches with real hosts stay off this page.

What it is: a dedicated Ubuntu VM, Splunk Enterprise 9.4, listeners on 1514 and 1515, and five live sources in separate indexes.