Rip a website via HTTP to download images, HTML and CSS

"wget -erobots=off --no-parent --wait=3 --limit-rate=20K -r -p -U ""Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)"" -A htm,html,css,js,json,gif,jpeg,jpg,bmp http://example.com

This runs in the console.

this will grab a site, wait 3 seconds between requests, limit how fast it downloads so it doesn't kill the site, and mask itself in a way that makes it appear to just be a browser so the site doesn't cut you off using an anti-leech mechanism.

Note the -A parameter that indicates a list of the file types you want to download.

You can also use another tag, -D domain1.com,domain2.com to indicate a series of domains you want to download if they have another server or whatever for hosting different kinds of files. There's no safe way to automate that for all cases, if you don't get the files.

wget is commonly preinstalled on Linux, but can be trivially compiled for other Unix systems or downloaded easily for Windows: GNUwin32 WGET

Use this for good and not evil."

"On Linux systems, 'wget' does this, pretty much.

Its also been ported to several other platforms, as several of the other answers mention."

"You need to use wget - which is available for most platforms. curl will not request documents recursively, which is one of wget's major strengths.

Linux: (usually included in the distro) http://www.gnudotorg/software/wget/
Windows: http://gnuwin32.sourceforge.net/packages/wget.htm
Mac: http://www.geekology.co.za/blog/2009/02/macports-compile-and-install-open-source-software-on-mac-os-x/

PLEASE make sure you aren't hammering the website - set up suitable delays between requests, and make sure it's within the site's terms of service.

-Adam"

"Actually, following up my comment in GWLlosa's post, I just remembered I have GnuWin32 installed, and sure enough it contains a Windows port of wget.

http://sourceforge.net/projects/gnuwin32/

GnuWin32 provides Win32-versions of GNU tools,
or tools with a similar open source licence.
The ports are native ports, that is they rely
only on libraries provided with any 32-bits
MS-Windows operating system, such as
MS-Windows 95 / 98 / 2000 / NT / XP"

"I used this some years ago and it worked well. Windows only. Used to be adware but no longer, apparently:

http://www.webreaper.net/"