Cannot update crates.io-index properly #76
Comments
|
@onur Thank you for working on this! :-) |
So we can build world in a old fashion way in case of emergency (Ref: #76)
|
This issue still exists and I am not sure what is causing this. This issue is occurring when cargo trying to download a crate. And now it started happening when cargo is trying to update registry index. Main issue is Docs.rs had a huge |
|
Is a file descriptor getting leaked somewhere? Or is directory traversal being done naively somewhere? |
|
@alexcrichton might have ideas |
|
Hm yeah this may just be a standard "some file is being held open for too long" issue perhaps? |
|
Does cargo do any directory traversal? |
|
AFAIK no, but I haven't audited it for something like this. Is this error coming from Cargo? |
|
@alexcrichton last part of the messages are coming from git2 crate: I am not sure what was the cause, this only happened when docs.rs tried to update crates.io-index repository and when cargo tried to update registry. Cleaning |
|
I guess this could be a bug with libgit2 perhaps? Holding open too many file descriptors on large repos? Unsure though. |
So we can build world in a old fashion way in case of emergency (Ref: rust-lang#76)
Docs.rs is currently using crates.io-index git repository to resolve new crates. But method used in docs.rs isn't very stable and doesn't work all the time. Using crates.io web API will solve this issue and bring more stable way to resolve new crates. Fixes: rust-lang#76, rust-lang#60
|
It seems like this may still be an issue. It looks like docs.rs hasn't pulled any crates in 10 hours, including a crate that I released last night. |
|
@shssoichiro docs.rs got stuck while building zxcvbn 0.1.0 |
|
Is there something wrong with that crate that's causing the build to hang? I didn't mean to break docs.rs by publishing a crate 😲 |
|
@shssoichiro no your crate is fine, idk what was the cause but it only got stuck for |
|
Given the latest issues (see #89) I believe it would be best to go back to using the crates index directly, and invest some time into figuring out where the file handle is leaked. If in doubt, one could also set it up as a separate binary that just outputs the required information on stdout. |
|
@alexcrichton I am still having this issue: This error message is coming from git2 crate. Docs.rs is using crates-index-diff to update crates.io-index git repository. When this bug occurs, git2 crate is trying to open bunch of pack files such as: Doing @alexcrichton I believe cargo also have this issue: After it's reaching a certain point, cargo starts failing to update registry. |
|
tl;dr: I would try running A very interesting observation! Initially I thought it might be a resource leak, but that does not seem to be the case as the repository is not persisting in memory. As every fetch seems to (have a chance to) generate a small pack, and as using the repository at all seems to want to open all of them at the same time, we run into issues. Maybe the My intuition would be to initiate a garbage collection (like |
|
@Byron running |
|
@onur I thought cargo has the same issue, and thus it should also be fixable by a |
|
Hm I'm surprised Cargo users haven't run into this yet, are there differences to what's going on? Maybe multiple Perhaps the fd limit could just be raised? (sorry I don't know how to fix this nor why it's happening) |
|
@Byron based on your suggestion I write a simple cron job to run |
|
I haven't ever seen this error, I think |
Docs.rs started failing to update crates.io-index with error messages:
I started working on some workarounds until I investigate this issue. Docs.rs cannot build new crates for now.
The text was updated successfully, but these errors were encountered: