Hello Every One , In this blog I will explain how to enable S3 logs using CLI to monitor access and activity on your objects in your bucket.
Why Logging is important?
Logging is crucial for several reasons:
Security & Compliance : Helps detect unauthorized access.
Troubleshooting & Debugging : Identifies errors and failed operations in real time.
Auditing & Accountability : Keeps a historical record of all changes and actions.
Monitoring & Performance Optimization : Provides insights into usage patterns and performance.
So to keep track of all operations done not only on bucket and its objects on any AWS service logging is super important. By enabling logging on S3, we can determine who uploaded or modified any settings in the bucket, what actions they performed, where they did it from, and when it happened
Now we understood how important logging is, now it’s time to learn how to enable logging on AWS S3.
Access Logs
Now lets enable access logs for S3 using CLI . After enabling logs for a bucket we have to select a specific bucket to store all these logs. If you try to store the logs in the same bucket, when any object is uploaded to the bucket , log file was uploaded to the bucket , access logs again uploads a log file for this and it continues…
Now lets create 2 buckets .
aws s3 mb s3://korla-goutham
aws s3 mb s3://korla-goutham-log
Now lets try to store, logs of korla-goutham bucket in the korla-goutham-log bucket, to do this on korla-goutham-log bucket, s3 access log must have permissions to put the log files. Using bucket policy we have to give permissions.
// bucket-policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "logging.s3.amazonaws.com"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::korla-goutham-log/*",
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:aws:s3:::korla-goutham"
},
"StringEquals": {
"aws:SourceAccount": "9848022338"
}
}
}
]
}
After the bucket policy has successfully attached to log bucket. We can now enable logging on the source bucket.
logging.json
{
"LoggingEnablepptd": {
"TargetBucket": "korla-goutham-log",
"TargetPrefix": "log/"
}
}
aws s3api put-bucket-logging --bucket korla-goutham --bucket-logging-status file://logging.json
From now when I try to change any setting on korla-goutham bucket logs will be generated in the log path in the korla-goutham-log bucket.
I have uploaded a object using CLI to the korla-goutham bucket
If I check the contents of the bucket
aws s3 ls korla-goutham-log/logs --recursive
Instead of korla-goutham-log I named log bucket as sk-shabeer.123-log .
Using cli I have uploaded an object and this is present in one of the log file . When you enable logging, every action is recorded, even if you haven’t performed any operations. AWS loads the page and retrieve settings like Properties, Permissions etc.. these will also be recorded.
Event Notifications
To get notified via mail on any action happened on S3 we can use these event notifications.
aws s3api put-bucket-notification-configuration \
--bucket my-bucket-name \
--notification-configuration '{
"TopicConfigurations": [
{
"Id": "MySNSTopic",
"TopicArn": "arn:aws:sns:us-east-1:123456789012:MyTopic",
"Events": ["s3:ObjectRemoved:*"]
}
]
}
Now when any object is deleted we get notified.
These logs are for a single account, but if you want to get the S3 logs from all your member accounts to master account in a organization then?.
Now comes cloud trail , as S3 access logs , logs every operation on S3 , Cloud Trail service, logs every operation on all services.
Cloud Trail
Cloud trail is the logging service , that logs every operation that was performed on all services in AWS.
By default cloud trail is enabled which records all the read/list events . How to check ?
aws cloudtrail lookup-events --max-items=10
This command lists all the operations done in that account. The result will be so huge. So minimize the result using —max-items.
Event History in AWS CloudTrail automatically records management events for the last 90 days, even if you haven’t set up a CloudTrail trail. Management events include actions related to bucket configuration changes, such as:
PutBucketPolicy (when a bucket policy is modified)
PutBucketLogging (when logging is enabled/disabled)
CreateBucket/DeleteBucket
IAM role changes
Data events (like uploading, downloading, or deleting objects in S3) are not recorded in Event History—you must create a CloudTrail trail to capture those.
Cloud trail logs who did it , what & when from where .
Cloud trail logs access to API calls and save logs in your S3 bucket , no matter how these calls are made .
Receive notification of log file directory using SNS service.
Aggregate log information from all member accounts into single S3 bucket in master account.
You can integrate cloud trail logs with cloud watch
Now for fun lets create a bucket in member account and this log has to be recorded in the master account.
Before creating cloud trail in the master account and attaching a bucket to store the logs, cloud trail service must have permissions on the bucket to store logs .
Lets create bucket and attach a bucket policy to the bucket.
{ "Version": "2012-10-17", "Statement": [ { "Sid": "AWSCloudTrailAclCheck20150319", "Effect": "Allow", "Principal": { "Service": [ "cloudtrail.amazonaws.com" ] }, "Action": ["s3:GetBucketAcl"], "Resource": "arn:aws:s3:::korla-bucket", "Condition": { "StringEquals": { "aws:SourceArn": "arn:aws:cloudtrail:ap-south-1:9848022338:trail/my-org-trail" } } }, { "Sid": "AWSCloudTrailWrite20150319", "Effect": "Allow", "Principal": { "Service": [ "cloudtrail.amazonaws.com" ] }, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::korla-bucket/AWSLogs/9848022338/*", "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control", "aws:SourceArn": "arn:aws:cloudtrail:ap-south-1:9848022338:trail/my-org-trail" } } }, { "Sid": "AWSCloudTrailOrganizationWrite20150319", "Effect": "Allow", "Principal": { "Service": [ "cloudtrail.amazonaws.com" ] }, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::korla-bucket/AWSLogs/o-bn2gu6z21t/*", "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control", "aws:SourceArn": "arn:aws:cloudtrail:ap-south-1:9848022338:trail/my-org-trail" } } } ] }
There are 3 allow statements in this bucket policy , this policy allows to get bucket acl to cloutrail service and put object to corresponding folders in a korla-bucket to my-org-trail cloud trail service.
2. Lets create my-org-trail cloud trail log in master account and attach korla-bucket to store logs.
aws cloudtrail create-trail \
--name my-org-trail \
--s3-bucket-name korla-bucket \
--s3-key-prefix logs \
--is-multi-region-trail \
--is-organization-trail \
By default when cloud trail created using CLI it is neither a organizational trail , nor a multiregional trail.
After creating cloud trail it is important to enable logging for this trail, without this cloud trail wont be able to log anything.
aws cloudtrail start-logging --name my-org-log-trail
Now logs from all the member accounts from all regions will store in korla-bucket AWSLogs folderx.
Thanks for reading my blog. Have a great day 🎉🎉🎉