Parallel file copy from single source to multiple targets?

"For single files you can use tee to copy to multiple places:

cat <inputfile> | tee <outfile1> <outfile2> > <outfile3>

or if you prefer the demoggified version:

tee <outfile1> <outfile2> > <outfile3> < <inputfile>

Note that as Dennis points out in the comments tee outputs to stdout as well as the listed files, hence using redirect to point to file 3 in the above examples. You could also redirect this to /dev/null as below - this has the advantage of keeping the file list more consistent on the command line (which may make it easier to script up a solution for variable numbers of files) but is a little less efficient (though the efficiency difference is small: about the same as the difference between using the cat version or the version without cat):

cat <inputfile> | tee <outfile1> <outfile2> <outfile3> > /dev/null

You could probably combine one of the above with find quite easily to operate on multiple files in one directory and less easily to operate on files spread over a directory structure. Otherwise you might just have to set the multiple copy operations off in parallel as separate tasks and hope that the OS disk cache is bright and/or big enough that each of the parallel tasks used cached read data from the first instead of causing drive-head thrashing.

AVAILABILITY: tee is commonly available on standard Linux setups and other unix or unix-alike systems, usually as part of the GNU ""coreutils"" package. If you are using Windows (your question doesn't specify) then you should find it in the various Windows ports such as Cygwin.

PROGRESS INFORMATION: As copying a large file off optical media may take some time (or over slow network, or an even larger file from even local fast media), progress information can be useful. On the command line I tend to use pipe viewer (available in most Linux distros & many Windows port collections and easy to compile yourself where not available directly) for this - just replace cat with pv like so:

pv <inputfile> | tee <outfile1> <outfile2> > <outfile3>"

"Based off of the answer given for a similar question Another way is to use GNU Parallel to run multiple cp instances at once:

parallel -j 0 -N 1 cp file1 ::: Destination1 Destination2 Destination3

The above command will copy file1 to all three destination folders in parallel"

"According to this answer: superuser.com/a/1064516/702806

A better solution is to use tar and tee. The command is more complicated but tar seems to be very powerful for transfer AND it needs to read the source just once.

tar -c /source/dirA/ /source/file1 | tee >(cd /foo/destination3/; tar -x) >(cd /bar/destination2/; tar -x) >(cd /foobar/destination1/; tar -x) > /dev/null

To use it in a script, you may need to launch your script with bash -x script.sh"

"In bash:

for x in dest1 dest2 dest3; do cp srcfile $x &>/dev/null &; done; wait;"

"If you want to do this in Windows from PowerShell, it is not possible by default, because unlike the -Path argument, the -Destination doesn't take multiple arguments. However, you can use -Passthrough and daisy-chain the commands. (But that's no fun.)

The best solution is to make your own, as is shown here."