Deploying enterprise-grade centralized logging across 13 hosts in 48 hours using open-source tools and Ansible automation. A case study in operational observability at scale.
The Challenge
A 5-node Proxmox cluster running 14 VMs with no centralized log visibility. Troubleshooting required SSH-ing into individual hosts and tailing log files manually , an approach that doesn’t scale and misses cross-system correlation.
The Result
13 hosts shipping logs to a single platform with 8 operational dashboards, security monitoring, and Docker container visibility , deployed in 2 days using Ansible automation.
Why Loki Over the Alternatives
The first decision was tool selection. In enterprise environments, the typical choices are Elastic (ELK), Splunk, Graylog, or Datadog. Each has merits, but for this environment the requirements were clear: lightweight resource footprint, native Grafana integration, and operational simplicity.
Graylog was the initial candidate but requires 4-8GB RAM for the Elasticsearch backend alone. On a cluster where every GB of RAM is allocated to production VMs, that’s a non-starter.
Loki runs on 512MB-1GB RAM because it indexes labels, not full text. For infrastructure logs where you’re querying by host, service, and severity, label-based indexing is exactly the right trade-off. You get fast filtered queries at a fraction of the resource cost.
Architecture
The deployment uses a dual-collection pattern:
- Promtail agents on 13 hosts collecting system logs (syslog, auth.log, systemd journal) plus Proxmox-specific logs (pve-cluster, firewall, daemon)
- Docker Loki driver on 3 Docker hosts (NetBox, MediaWiki, Loki itself) capturing container stdout/stderr with automatic project and service labeling
- Central Loki instance on a dedicated VM (192.168.0.175) with 30-day retention
- Grafana co-located on the same VM for dashboard visualization
This pattern gives comprehensive visibility across three layers: hypervisor (Proxmox nodes), OS (Linux VMs), and application (Docker containers).
The 8 Dashboards
Dashboards were designed around operational use cases, not just log viewing:
Infrastructure Monitoring
- Infrastructure Logs Overview , Log volume by host and job with stream view. The daily homepage for operational awareness.
- Security and Authentication , SSH attempts, failed authentication, sudo activity. Essential for detecting unauthorized access.
- Systemd Services Monitor , Service-level logs with error and warning counts. Catches service degradation before it becomes an outage.
- Host Log Explorer , Filtered text search by host. The troubleshooting workhorse.
- Error and Warning Tracker , Error distribution timeline across all hosts. Identifies systemic issues.
Application Dashboards
- Docker Containers , Container logs by project and service with error tracking. Covers 12 containers across 3 Docker hosts.
- NetBox Application Logs , NetBox, PostgreSQL, and Redis logs in one view. Critical for IPAM/DCIM troubleshooting.
- MediaWiki Application Logs , MediaWiki, MariaDB, and Kroki (diagram renderer) logs. Documentation infrastructure monitoring.
Deployment: Ansible-Automated in 48 Hours
The entire deployment , from empty VM to 13 hosts shipping logs with 8 dashboards , was completed in two working sessions. Ansible playbooks handled the repetitive work:
- Promtail installation and configuration across all Linux VMs
- Specialized Promtail templates for Proxmox nodes (which have different log paths)
- Docker daemon configuration for native Loki driver on container hosts
- Dashboard provisioning via Grafana API
The automation-driven approach meant that configuration templates, playbooks, and dashboard JSON were generated, tested, and deployed in a continuous flow rather than the traditional write-test-debug cycle.
Challenges and Solutions
Proxmox-specific log paths: Proxmox nodes have unique log locations for cluster communication, firewall, and daemon processes. I created a specialized Promtail template that captures these alongside standard system logs.
Docker log driver complexity: Switching Docker hosts to the Loki logging driver requires daemon.json configuration and a Docker restart , which means brief container downtime. Each host was migrated individually with documented rollback procedures.
Resource constraints: Loki’s label-based indexing was the right architectural choice here. Full-text indexing (Elasticsearch) would have consumed 4-8x more RAM for the same log volume, directly impacting VM capacity on the cluster.
The Cost Comparison
What does equivalent functionality cost in commercial tooling?
- Datadog Log Management: ~$0.10/GB ingested/day. At modest enterprise volumes, $12,000-25,000/year.
- Splunk Cloud: $15,000+ annually for comparable ingest volume.
- Elastic Cloud: $8,000-15,000/year for hosted ELK with retention.
- This deployment: $0 in software licensing. VM resource cost only.
The point isn’t that open source is always better than commercial. It’s that understanding the full stack , from log collection agents to query languages to dashboard design , enables strategic decisions about where to spend and where to build.
Key Takeaways
- Label-based indexing is the right trade-off for infrastructure logs. You don’t need full-text search when you’re filtering by host, service, and severity.
- Dual-collection patterns provide layered visibility. Combining agent-based (Promtail) and driver-based (Docker Loki) collection covers all infrastructure layers without gaps.
- Dashboard design should follow operational workflows. Eight focused dashboards beat one sprawling dashboard every time.
- Ansible automation makes multi-host deployment repeatable. Adding the next host takes minutes, not hours.
- Automated deployment compresses timelines dramatically. Two days from zero to full observability is the productivity multiplier in action.
This is the first in a series of case studies from the Innovation Lab. Upcoming: NetBox-to-Naemon automated monitoring pipeline, and the API server architecture that enables automated infrastructure operations.