CASE STUDY / 04
Observability needs its own guardrails.
Deployment configuration and runbooks for a shared analytics and LLM-observability stack, with diagnostic-log retention, worker tuning, resource limits, and backup maintenance.
Deployed baseline · later tuning documented
PostHog / Langfuse / ClickHouse / Docker / Restic · anonymized engagement
The problem
Services shared memory and disk. Unbounded database diagnostic logs could consume shared storage and impair another service’s persistence; worker settings also needed to reflect the actual workload.
My implementation
I contributed Compose deployment configuration, ingestion routing, ClickHouse diagnostic-log caps, Celery worker tuning, and operational documentation. Later authored changes sized resource limits from recorded memory peaks and swap, with the concurrency trade-offs documented.
Backup operations
Transactional PostgreSQL dumps precede file snapshots. Repository-lock retries, retention, and less frequent prune/check operations separate routine backups from expensive maintenance. A daily read of a data subset checks more than the existence of a snapshot.
The trade-offs
Lower worker concurrency saves memory but can delay scheduled tasks. Live analytics-file snapshots remain crash-consistent; partial-read handling requires care. Shared hosting and same-provider backups retain common failure risks.
What is established
The previously reviewed PostHog/Langfuse baseline was confirmed deployed. Later tuning is supported by authored configuration and source-recorded operational observations; its deployment and broader impact are not independently verified. No measured cost savings, uptime improvement, or tested recovery-time claim is made.