Transferring Large Files

Very large files can be transferred by splitting them into multiple files and copying them in parallel.

1. Generate Checksum md5sum someLargeFile
2. Split split -b Nm someLargeFile PREFIX where N is the size of each chunk in MB and PREFIX is the prefix for the created chunks.
3. Transfer Run multiple instances of scp to transfer the chunks. At some point adding more scp instances will not increase the total throughput.
4. Join cat PREFIX* > someLargeFile2
5. Check md5sum someLargeFile2 Compare this output against the original files checksum. It should be exactly the same.