Example of Using Wget

3 minute read

Published:

This post records some examples of using Wget to download data from FTP sites.

The above figure is generated by DALL·E 3 with prompt: Vector design showcasing a desktop setup with a keyboard, mouse, and a screen displaying the Wget command in action. FTP icons are scattered above the screen, with some of them having downward arrows, signifying the download process.

Introduction

Wget (short for GNU Wget) is a free software that allows you to download files using the HTTP, HTTPS, and FTP protocols. It is a non-interactive command-line tool, making it easy to invoke from scripts or terminals without requiring X-window support. For detailed information, refer to the Wget official website. Wget is particularly useful for batch downloading data.

Installing Wget on Linux

Most Linux distributions come with Wget pre-installed. If it’s not installed, you can use the following commands to install it:

For Ubuntu/Debian:

sudo apt-get install wget

For CentOS/RHEL/Fedora:

sudo yum install wget

Using Wget

The usage of Wget is wget [options]... [URL]...:

Usage: wget [OPTION]... [URL]...

You can use the wget -h command to view the available options.

Examples of Using Wget

Downloading all files from a file list

For example, suppose you have a file list named filelist.txt with the following content:

ftp://example.com/1.jpg
ftp://example.com/2.jpg
ftp://example.com/3.jpg
ftp://example.com/4.jpg
......

To download all the files listed in this file, enter the following command in the terminal:

wget -c -N -i filelist.txt

This command will download all the files listed in the file list. The -c and -N options ensure that if the download is interrupted, you can resume it by running the command again.

Downloading one or multiple directories

For example, if you want to download data from the following FTP address: ftp://n5eil01u.ecs.nsidc.org/SAN/AMSA/AE_L2A.003/, and upon opening it, you find the following directory structure:

......
2006.01.01/     13/6/28 上午12:00:00
2006.01.02/     13/6/28 上午12:00:00
2006.01.03/     13/6/28 上午12:00:00
2006.01.04/     13/6/28 上午12:00:00
2006.01.05/     13/6/28 上午12:00:00
2006.01.06/     13/6/28 上午12:00:00
2006.01.07/     13/6/28 上午12:00:00
2006.01.08/     13/6/28 上午12:00:00
2006.01.09/     13/6/28 上午12:00:00
2006.01.10/     13/6/28 上午12:00:00
2006.01.11/     13/6/28 上午12:00:00
2006.01.12/     13/6/28 上午12:00:00
......

To download all the files in the directories with the format “2006*” and the file format “hdf” for the year 2006, use the following command in the terminal:

wget -c -N -r -nd -np -k -L -p -A hdf ftp://n5eil01u.ecs.nsidc.org/SAN/AMSA/AE_L2A.003/2006*

This command will download all the files in the directories with the format “2006*” and the file format “hdf” for the year 2006, and save them in the current directory.

Here is the explanation of the options used:

OptionDescription
-cResume downloading partially downloaded files
-NOnly download files that are newer than the local copies
-rRecursive download, download all files in the specified webpage directory (including subdirectories)
-ndDo not create a hierarchy of directories when recursively downloading, download all files to the current directory
-npDo not ascend to the parent directory when recursively downloading
-kConvert absolute links to relative links
-LDo not follow links to other hosts when recursively downloading
-pDownload all files necessary to display the webpage
-ASpecify a comma-separated list of file name suffixes to accept

Using Wget on Windows

To use Wget on Windows, download wget.exe (for 32-bit or 64-bit systems) or wget64.exe (for 64-bit systems) from Wget for Windows. Place wget.exe in the directory C:\Windows\System32. If you downloaded wget64.exe, rename it to wget.exe.

In the directory where you want to download files, hold the Shift key and right-click. Then select “Open command window here.” This will allow you to use Wget in the Windows terminal, using the same commands as in Linux.

References

Leave a Comment