How to use curl to download all files

Super User is a question and answer site for computer enthusiasts and power users. It only takes a minute to sign up. Connect and share knowledge within a single location that is structured and easy to search.

I am trying to download a full website directory using CURL. The following command does not work:. Any idea how to download all files in the specified directory? HTTP doesn't really have a notion of directories.

If you want to download the whole site, your best bet is to traverse all the links in the main page recursively. Curl can't do it, but wget can. This will work if the website is not too dynamic in particular, wget won't see links that are constructed by Javascript code. If the website tries to block automated downloads, you may need to change the user agent string -U Mozilla , and to ignore robots. In this case, curl is NOT the best tool.

You can use wget with the -r argument, like this:. This is the most basic form, and and you can use additional arguments as well.

For more information, see the manpage man wget. This isn't possible. There is no standard, generally implemented, way for a web server to return the contents of a directory to you. Most servers do generate an HTML index of a directory, if configured to do so, but this output isn't standard, nor guaranteed by any means. You could parse this HTML, but keep in mind that the format will change from server to server, and won't always be enabled.

You can use the Firefox extension DownThemAll! It will let you download all the files in a directory in one click.

The output shows that the URL was redirected. The first line of the output tells you that it was moved, and the Location line tells you where:. You could use curl to make another request manually, or you can use the --location or -L argument which tells curl to redo the request to the new location whenever it encounters a redirect. Give it a try:.

You can combine the -L argument with some of the aforementioned arguments to download the file to your local system:. Warning : Many resources online will ask you to use curl to download scripts and execute them.

You can learn more by viewing the manual page for curl by running man curl. Where would you like to share this to? The option -k does not always work. Show 1 more comment. Thank you. This is a nice solution and providing working example is great! Thank you for answering the question that was asked, as I can not install wget on the server I need to use — StevenWernerCS.

It is helpful if you're not getting all of the files. It's a shortcut for -N -r -l inf --no-remove-listing which means: -N : don't re-retrieve files unless newer than local -r : specify recursive download -l inf : maximum recursion depth inf or 0 for infinite --no-remove-listing : don't remove '.

So that we don't thrash the server. Happy Downloading :smiley:. Udit Desai Udit Desai 1 1 silver badge 4 4 bronze badges. The Overflow Blog. Who owns this outage? To demonstrate the usage of the curl program, first, we need a dummy file to download. Any online file will work for this, as long as you have the direct download link.

For this guide, I will use the small file provided by think broadband. This is a very basic way of using curl. We will download the dummy file. With this flag, the file will be downloaded and saved at the current working directory.

Need to download multiple files? Follow the command structure shown below. Curl allows you to limit the download speed. Here, the download speed is limited to 1mb. It is also possible to manage an FTP server using curl.