How to Set Up Centralized Logging and Monitoring for Your AWS Infrastructure

Why Centralized Logging Matters for Small Businesses

When you're running infrastructure on AWS, logs get scattered everywhere. Your EC2 instances write to one place, Lambda functions to another, and your RDS databases somewhere else entirely. When something breaks at 2 AM, you don't want to hunt through five different consoles to figure out what happened.

Centralized logging pulls all your logs into one place where you can search, analyze, and set up alerts. It's not just about troubleshooting—it's about catching problems before they become emergencies and meeting compliance requirements if you handle sensitive data.

What You'll Need Before Starting

Before diving in, make sure you have:

• AWS account with administrative access
• Basic familiarity with AWS Console
• A list of which AWS services you're currently using
• Budget for CloudWatch Logs storage (typically $10-50/month for small setups)

You don't need to be an AWS expert. This guide walks through the practical steps without assuming you know every AWS service inside and out.

Step 1: Enable AWS CloudTrail

CloudTrail records every API call made in your AWS account. Think of it as a security camera for your infrastructure—it shows who did what and when.

Set up CloudTrail:

1. Open the CloudTrail console
2. Click "Create trail"
3. Give it a name like "company-main-trail"
4. Create a new S3 bucket or select an existing one (CloudTrail will store logs here)
5. Enable log file validation for tamper-proof logs
6. Under "Management events," select "Read and Write events"
7. Click "Create trail"

This captures management activities across all AWS services. For most small businesses, one trail covering all regions is sufficient. The logs go to S3, where storage costs about $0.023 per GB.

Step 2: Set Up CloudWatch Logs

CloudWatch Logs is where your application and system logs live. Unlike CloudTrail (which tracks AWS API calls), CloudWatch captures what's happening inside your applications.

Configure CloudWatch for EC2 instances:

1. Install the CloudWatch agent on your EC2 instances
2. Create an IAM role with CloudWatchAgentServerPolicy
3. Attach the role to your instances
4. Configure which log files to monitor (like /var/log/syslog or application logs)

For Lambda functions, CloudWatch logging happens automatically—you just need to ensure your functions have permission to write logs.

Create log groups strategically:

Organize logs by environment and service:

• /aws/production/webserver
• /aws/production/api
• /aws/staging/webserver

This structure makes it easy to set different retention policies and search specific services.

Step 3: Centralize Logs in S3

While CloudWatch is great for recent logs and real-time monitoring, S3 provides cheaper long-term storage. Export older CloudWatch logs to S3 for cost-effective archival.

Set up log exports:

1. Create a dedicated S3 bucket for log archives
2. Set up lifecycle policies to move logs to S3 Glacier after 90 days
3. Configure CloudWatch log groups to export to S3
4. Enable S3 bucket versioning and encryption

For compliance purposes, you might need to keep logs for years. S3 Glacier costs about $0.004 per GB per month—much cheaper than keeping everything in CloudWatch.

Step 4: Configure Monitoring and Alerts

Logs are useless if nobody looks at them. Set up automated alerts for issues that need immediate attention.

Create CloudWatch alarms:

1. Go to CloudWatch > Alarms
2. Click "Create alarm"
3. Select a metric (CPU utilization, error rates, disk space, etc.)
4. Set threshold values that indicate problems
5. Configure SNS notifications to email or Slack

Practical alarm examples:

• EC2 CPU above 80% for 10 minutes
• Application error rate exceeds 5% of requests
• Disk space below 20%
• Unusual number of failed login attempts (security alert)

Start with 3-5 critical alarms. Too many alerts lead to alarm fatigue where you ignore everything.

Step 5: Set Up Log Analysis with CloudWatch Insights

CloudWatch Logs Insights lets you search and analyze logs using a query language. It's like having grep and awk for all your AWS logs.

Useful queries to save:

Find all errors in the last hour:

fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 100

Track API response times:

fields @timestamp, duration | filter @type = "REPORT" | stats avg(duration), max(duration), min(duration)

Save your common queries so you don't need to rewrite them during an incident.

Step 6: Implement Log Retention Policies

Storing logs forever gets expensive. Set retention policies based on your needs.

Recommended retention schedule:

• CloudWatch Logs: 30 days for most logs, 90 days for security logs
• S3 standard storage: 90 days
• S3 Glacier: 7 years (or whatever compliance requires)

1. Go to CloudWatch > Log groups
2. Select a log group
3. Click "Actions" > "Edit retention setting"
4. Choose your retention period

This simple step can cut your logging costs by 50-70%.

Security Considerations

Your logs contain sensitive information about your infrastructure and potentially customer data. Protect them:

• Enable encryption at rest for S3 buckets and CloudWatch log groups
• Use IAM policies to restrict who can view logs
• Enable MFA delete on your log storage buckets
• Set up CloudTrail log file integrity validation
• Regularly review who has access to logs

For more comprehensive security practices, check out our security services that can help audit your logging setup.

Keeping Costs Under Control

Centralized logging doesn't have to break the bank. Here's how to keep costs reasonable:

• Use log sampling for high-volume, low-value logs
• Filter out noisy logs that don't provide value
• Export to S3 and Glacier aggressively
• Set up billing alerts when logging costs exceed thresholds
• Review and delete unused log groups monthly

Most small businesses spend $50-200/month on logging. If you're spending more, you're probably keeping too much data in CloudWatch.

Automating Your Setup

Once you've tested this manually, automate it. Use Infrastructure as Code tools like Terraform or CloudFormation to:

• Deploy consistent logging across all environments
• Ensure new resources automatically send logs to the right place
• Version control your logging configuration
• Replicate your setup for disaster recovery

Our automation services can help you build Infrastructure as Code templates for your logging infrastructure.

Next Steps

You now have centralized logging that captures what's happening across your AWS infrastructure. Here's what to do next:

1. Spend a week watching your logs to understand normal patterns
2. Refine your alerts to reduce false positives
3. Document your logging architecture for your team
4. Schedule monthly reviews of your logging costs and retention policies
5. Consider adding application performance monitoring (APM) tools for deeper insights

Centralized logging is foundational infrastructure. Combined with proper cloud architecture, it gives you visibility and control over your AWS environment.

Need Help Getting This Set Up?

Setting up logging infrastructure takes time away from building your product. If you'd rather have experts handle it, contact our team to discuss how we can implement and manage centralized logging for your AWS infrastructure.

---

FAQ

How much does centralized logging cost on AWS?

For a typical small business with 5-10 EC2 instances and a few Lambda functions, expect $50-150/month. Costs include CloudWatch Logs ingestion ($0.50/GB), storage ($0.03/GB/month), and S3 storage for archives. The key cost driver is log volume—verbose application logging can significantly increase costs.

Do I need third-party tools like Datadog or Splunk?

Not necessarily. AWS native tools (CloudWatch, CloudTrail, S3) handle most small business needs. Third-party tools offer better visualization, easier querying, and cross-platform monitoring if you use multiple clouds. Start with AWS native tools and add third-party solutions when you hit their limitations.

How long should I keep logs?

It depends on your compliance requirements and use case. For general troubleshooting, 30 days in CloudWatch is sufficient. Security logs should be kept longer—90 days to 1 year in searchable storage, then archived to Glacier. If you're in healthcare, finance, or government sectors, you may have specific retention requirements ranging from 3-7 years.