Centralized Logging System Based on OpenSearch
Centralized logging platform on OpenSearch, processing 8PB+ daily logs from 300+ global enterprise customers. Delivered ready-to-use dashboards for CloudTrail, CloudFront, RDS, and EKS/EC2 application logs — reducing enterprise observability setup from months to hours, with automated index lifecycle management cutting storage costs by 70%.
System Architecture
Complete centralized logging architecture overview
AWS Service Log Pipeline Architecture
Specialized pipeline for AWS native service logs including CloudTrail, VPC Flow Logs, and CloudFront access logs
EC2/EKS Application Log Pipeline Architecture
Dedicated pipeline for application logs from EC2 instances and EKS clusters using Fluent-bit agents
Project Overview
Addressing the absence of a centralized management service for collecting cloud service logs and user application logs within EKS/EC2 on AWS, this solution is built on OpenSearch (Elasticsearch) to create a centralized logging management platform. It offers many ready-to-use dashboards, significantly reducing the cost for users to create an ELK logging pipeline.
The product has been deployed in production environments by over 300+ global customers, processing petabytes of log data daily with high availability and scalability requirements.
Key Responsibilities
Managed the generation and distribution of Kubernetes Fluent-bit Configuration supporting both DaemonSet and Sidecar deployment modes, ensuring optimal log collection from various container environments.
Designed and developed comprehensive OpenSearch (Elasticsearch) dashboards for various AWS services including CloudTrail, CloudFront, RDS, and application logs, providing actionable insights for operations teams.
Developed automation for OpenSearch Index creation and rollover functionality, ensuring efficient storage management and optimal query performance across millions of daily log entries.
Designed and developed data pipeline insights dashboard, making log processing metrics and system health visible to users on the frontend, enabling proactive monitoring and troubleshooting.
Dashboard Examples
CloudFront Access Logs Dashboard
Real-time monitoring of CloudFront distribution performance, cache hit ratios, and geographic traffic patterns
CloudTrail Security Dashboard
Comprehensive security monitoring including API call analysis, user activity tracking, and anomaly detection
RDS Performance Dashboard
Database performance monitoring with query analysis, connection tracking, and performance bottleneck identification
Technical Challenges & Solutions
Processing 8PB+ of log data daily from 300+ enterprise customers required implementing sophisticated sharding strategies, index optimization, and automated scaling mechanisms to maintain sub-second query response times.
Designed secure multi-tenant architecture ensuring complete data isolation between customers while maintaining operational efficiency and cost-effectiveness through shared infrastructure.
Implemented intelligent data lifecycle management with automated hot-warm-cold tier transitions, reducing storage costs by 70% while maintaining required data accessibility and compliance standards.
Official Resources
Tech Stack
Backend & API
Search & Analytics
AWS Services
All data, metrics, screenshots, and demos shown are based on publicly available information or anonymized data.