I have 5 million files which take up about 1TB of storage space. I need to transfer these files to a third party.
What's the best way to do this? I have tried reducing the size using .tar.gz, but even though my computer has 8GB RAM, I get an "out of system memory" error.
Is the best solution to snail-mail the files over?
Additional information provided in the comments reveals that the OP is using a GUI method to create the .tar.gz file.
GUI software often includes a lot more bloat than the equivalent command line equivalent software, or performs additional unnecessary tasks for the sake of some "extra" feature such as a progress bar. It wouldn't surprise me if the GUI software is trying to collect a list of all the filenames in memory. It's unnecessary to do that in order to create an archive. The dedicated tools tar and gzip are defintely designed to work with streaming input and output which means that they can deal with input and output a lot bigger than memory.
If you avoid the GUI program, you can most likely generate this archive using a completely normal everyday tar invocation like this:
tar czf foo.tar.gz foo
where foo is the directory that contains all your 5 million files.
The other answers to this question give you a couple of additional alternative tar commands to try in case you want to split the result into multiple pieces, etc...
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments