I have a large number (>100k) time series as compressed csv files. I would like to allow users to download each file individually (programmatically) or download all.
I tried to upload all the files to a single fileset, but found a limit of 501 files per fileset. Is this a hard limit, or can it be extended?
If it is a limit, does anyone have a suggestion for allowing programatic access to individual files rather than requiring the user to download the whole zip file and pull out a file locally.
To be clear, do you have 100k files you'd like to share?
Yes. I have ~100k files I would like to share. Each file is pretty small, ~300 KB.
Sorry for dropping off on this, I took a long period of leave. I spoke with the infrastructure team, and whilst it's not a hard limit, we couldn't raise it anywhere close to what you are asking. I don't really have a good solution for you short of zipping stuff as you suggest.