Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Close GetObject bodies to avoid leaking memory #70

Closed
wants to merge 1 commit into from

Conversation

mcrute
Copy link

@mcrute mcrute commented Jul 26, 2024

Calls to the GetObject method of the S3 client require that the Body be closed to avoid leaking memory. The returned struct is an http.Response returned by an http.Client (verified by digging into the AWS SDK dependency chain) and is guaranteed to always have a non-nil Body.

The only way I've been able to spot this issue is in prod after a few days of load. I'm testing this patch now and it already appears to be holding Caddy memory usage much more steady.

Related to and should fix #64.

Summary by CodeRabbit

  • Bug Fixes
    • Improved resource management in the S3Proxy class to prevent potential resource leaks by ensuring response bodies are properly closed after use.
  • New Features
    • Enhanced stability and performance of the application when handling multiple S3 object requests.

Copy link

coderabbitai bot commented Jul 26, 2024

Walkthrough

The recent changes to the S3Proxy class in s3proxy.go focus on enhancing resource management by implementing defer statements to ensure that the bodies of S3 object responses are properly closed after use. This aims to prevent resource leaks and improve the application’s stability and performance, particularly in long-running processes.

Changes

Files Change Summary
s3proxy.go Added defer obj.Body.Close() in the serveErrorPage method and multiple instances in the GetHandler method to ensure proper closure of S3 object bodies.

Assessment against linked issues

Objective Addressed Explanation
High CPU usage after 32 days with caddy-s3-proxy (#64)

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 850db19 and b2b8bff.

Files selected for processing (1)
  • s3proxy.go (3 hunks)
Additional comments not posted (3)
s3proxy.go (3)

346-346: Good practice: Close the response body.

The defer obj.Body.Close() statement ensures that the response body is properly closed, preventing memory leaks.


452-452: Good practice: Close the response body.

The defer obj.Body.Close() statement ensures that the response body is properly closed if an index file is found, preventing memory leaks.


510-510: Good practice: Close the response body.

The defer obj.Body.Close() statement ensures that the response body is properly closed, preventing memory leaks.

@mcrute
Copy link
Author

mcrute commented Jul 29, 2024

Eh... this change didn't really fix the issue. Although it also didn't make it worse so it might be worth merging. That being said I've migrated to caddy-fs-s3 instead so I'm going to withdraw this PR.

@mcrute mcrute closed this Jul 29, 2024
@kdm9 kdm9 mentioned this pull request Aug 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

high cpu after 32 days with caddy-s3-proxy
1 participant