"chown -R sentry /var/lib/sentry/files" from entrypoint.sh takes a long time

[stumbaumr]

Hi,

if there are some more files in /var/lib/sentry/files it really takes a long time to start up.

Maybe better to only change the permissions if necessary instead of just writing hard to the filesystem:

# Set permissions if necessary
find /var/lib/sentry/files ! -user sentry -exec chown -R sentry {} \;

[BYK]

Sounds like a good suggestion, what do you think @mattrobenolt? Also, would you be willing to submit a PR for this @stumbaumr?

[mattrobenolt]

Alternative is to just drop the -R and only make sure the top level directory is sufficient. Lemme see what other images do.

[mattrobenolt]

Ok, so kinda my source of truth for this stuff is Postgres. Postgres manages a ton of files in its data directory and it does exactly this as well. https://github.com/docker-library/postgres/blob/master/12/docker-entrypoint.sh#L34

[mattrobenolt]

MySQL as well. https://github.com/docker-library/mysql/blob/master/8.0/docker-entrypoint.sh#L90

So I’d be curious why this is more problematic for us than a database who most likely has more files than we do.

[stumbaumr]

Use case: We operate data filesystems on NFS mounts on a SSD based appliance. Throughput/Availabilty/Snapshotting is great, but latency is in some cases a problem.

Chowning on a filesystem with many files like this takes time - and since the owner is already sentry it is not necessary. And we are talking currently about 15 Minutes....

[BYK]

Eh, mounting a volume from an NFS mount is not a great idea I guess. I mean if your suggestion makes this better, I'm not opposed to it as long as @mattrobenolt is also okay.

[mattrobenolt]

Ok, so checking through the docker community, it does seem like they are trending towards making this change, it can be seen in here for Postgres rolled up with other changes: docker-library/postgres#496

They're changing to using find "$PGDATA" \! -user postgres -exec chown postgres '{}' which yeah, is exactly what you're proposing.

So I'm ok with this.

I do think NFS is not good, but I don't think it wouldn't work. It's probably fine for most use cases.

[BYK]

@stumbaumr so, open to PR's over at https://github.com/getsentry/docker-sentry :)

[stumbaumr]

Please do not tell NetApp that operating Systems on NFS is a bad idea. I think a major portion of their business model relies on that...

[BYK]

@stumbaumr sorry, didn't want to invalidate or judge anything. NFS itself tends to be slower on some I/O operations, that's why we kind of frowned upon it. Great if it works for you :)

[BYK]

Already fixed in getsentry/docker-sentry#200

[BYK]

Thanks @stumbaumr!

[azhurbilo]

I have 26712 under NFS storage

find /mnt/nfs/sentry/data/ -type f | wc -l
26712

and it's still take ~3 min to start sentry container :(

[BYK]

@azhurbilo sorry to hear that. Which tag are you on for the docker image?

[azhurbilo]

9.1.2 image

but I override entrypoint to this one https://github.com/getsentry/docker-sentry/blob/master/9.1/docker-entrypoint.sh

[BYK]

@azhurbilo and how do you do that override?

[azhurbilo]

@BYK

FROM sentry:9.1.2
...
COPY docker-entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

[azhurbilo]

just for workaround start using -maxdepth 1

find "$SENTRY_FILESTORE_DIR" -maxdepth 1 ! -user sentry -exec chown sentry {} ;

hope, it doesn't broke smth :)

[BYK]

Pretty sure this directory should only have direct descendants so not sure why adding -maxdepth 1 would speed up anything. @mattrobenolt - any theories?

[stumbaumr]

So, for me the startup currently takes about 4 Minutes and I am fine with that.

root@sentry01(dc1.prd):~# find /var/lib/docker/volumes/sentry-data/ -type f | wc -l
1091597
root@sentry01(dc1.prd):~#

This is a SSD-only NetApp attached with 10GBit.