How to download list of files from a file server?

"Ref: http://blog.incognitech.in/download-files-from-apache-server-listing-directory/

You can use following command:

wget --execute=""robots = off"" --mirror --convert-links --no-parent --wait=5 <website-url>

Explanation with each options

wget: Simple Command to make CURL request and download remote files to our local machine.
--execute=""robots = off"": This will ignore robots.txt file while crawling through pages. It is helpful if you're not getting all of the files.
--mirror: This option will basically mirror the directory structure for the given URL. It's a shortcut for -N -r -l inf --no-remove-listing which means:

-N: don't re-retrieve files unless newer than local
-r: specify recursive download
-l inf: maximum recursion depth (inf or 0 for infinite)
--no-remove-listing: don't remove '.listing' files

--convert-links: make links in downloaded HTML or CSS point to local files
--no-parent: don't ascend to the parent directory
--wait=5: wait 5 seconds between retrievals. So that we don't thrash the server.
<website-url>: This is the website url from where to download the files.

Happy Downloading :smiley:"