Skip to content

crates-io: Add support for db-dump.zip files #421

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 10, 2024
Merged

crates-io: Add support for db-dump.zip files #421

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
688a7d9
crates-io: Add `path` argument to `redirect_to_cloudfront()` fn
Turbo87 Jun 1, 2024
8ae0c44
crates-io: Add support for `db-dump.zip` files
Turbo87 Jun 1, 2024
a0e6a04
crates-io: Use `dynamic` block to reduce duplication
Turbo87 Jun 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions terragrunt/modules/crates-io/compute-static/src/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,9 @@ fn handle_request(config: &Config, mut request: Request) -> Result<Response, Err

// Database dump is too big to cache on Fastly
if request.get_url_str().ends_with("db-dump.tar.gz") {
redirect_db_dump_to_cloudfront(config)
redirect_to_cloudfront(config, "db-dump.tar.gz")
} else if request.get_url_str().ends_with("db-dump.zip") {
redirect_to_cloudfront(config, "db-dump.zip")
} else {
send_request_to_s3(config, &request)
}
Expand Down Expand Up @@ -192,8 +194,8 @@ fn rewrite_download_urls(request: &mut Request) {
///
/// As of early 2023, certain files are too large to be served through Fastly. One of those is the
/// database dump, which gets redirected to CloudFront.
fn redirect_db_dump_to_cloudfront(config: &Config) -> Result<Response, Error> {
let url = format!("https://{}/db-dump.tar.gz", config.cloudfront_url);
fn redirect_to_cloudfront(config: &Config, path: &str) -> Result<Response, Error> {
let url = format!("https://{}/{path}", config.cloudfront_url);
Ok(Response::temporary_redirect(url))
}

Expand Down
20 changes: 12 additions & 8 deletions terragrunt/modules/crates-io/s3-static.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,17 +13,21 @@ resource "aws_s3_bucket" "static" {
max_age_seconds = 3000
}

// Keep only the live db-dump.tar.gz and the previous day's version, removing
// Keep only the live db-dumps and the previous day's versions, removing
// all the other ones. This is needed because we don't want this file to be
// versioned, while all the other ones in the bucket should be versioned.
lifecycle_rule {
id = "purge-db-dump"
enabled = true
prefix = "db-dump.tar.gz"
dynamic "lifecycle_rule" {
for_each = toset(["db-dump.tar.gz", "db-dump.zip"])

content {
id = "purge-${replace(lifecycle_rule.key, ".", "-")}"
enabled = true
prefix = lifecycle_rule.key

abort_incomplete_multipart_upload_days = 1
noncurrent_version_expiration {
days = 1
abort_incomplete_multipart_upload_days = 1
noncurrent_version_expiration {
days = 1
}
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to avoid the duplication here. Can we use a dynamic block for this instead?

dynamic "lifecycle_rule" {
  for_each = toset(["db-dump.tar.gz", "db-dump.zip"])

  content {
    id      = "purge-${replace(lifecycle_rule.key, ".", "-")}"
    enabled = true
    prefix  = lifecycle_rule.key

    abort_incomplete_multipart_upload_days = 1
    noncurrent_version_expiration {
      days = 1
    }
  }
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use a dynamic block for this instead?

you tell me, I have no idea how to check if this works or not 😂

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, let me rephrase the question. Can you please change it to a dynamic block? 😂 I already tested that the code above works.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done :)


Expand Down
Loading