The wget command is a powerful utility used in Linux for non-interactive downloading of files from the web. It supports a wide range of protocols, including HTTP, HTTPS, and FTP, making it versatile enough for various tasks such as downloading single files, entire websites, or even files listed in a text file. wget is particularly useful for automating download tasks, performing recursive downloads, and resuming interrupted downloads. Its command-line nature allows it to be easily integrated into scripts and cron jobs, making it an indispensable tool for system administrators and power users.
This guide will explore 30 practical examples of using wget, demonstrating how it can be used to handle a variety of common and advanced download tasks. From basic file downloads to more complex scenarios like mirroring websites and handling authentication, these examples will help you master the wget command and make the most of its capabilities.
Understanding Wget Command Syntax
The basic syntax of wget is as follows:
wget [option] [URL]Within the above syntax:
- The command wget is the trigger for deploying the utility.
- [option] acts as a discretionary parameter, enabling you to define various configurations. Wget comes with a vast array of options that allow users to modify its operation. From enabling background downloads to limiting download speeds, a wide assortment of settings can be fine-tuned to suit your needs.
- [URL] signifies the web address of the file you are aiming to download. This is where you input the specific location on the web from where the utility fetches the file.
Wget Command Examples
Below, we will demonstrate various examples using the wget command.
1. Downloading a Single File
The most straightforward use of wget is downloading a single file from a specified URL.
wget https://example.com/file.zipThis command will download the file named file.zip from example.com and save it in the current working directory.
2. Downloading in the Background
For downloading large files, wget can be run in the background using the ‘-b’ option.
wget -b https://example.com/largefile.zipThis command initiates the download of largefile.zip in the background, allowing you to continue using the terminal for other tasks.
3. Limiting Download Speed
To prevent wget from consuming all available bandwidth, you can limit the download speed using the ‘–limit-rate’ option.
wget --limit-rate=200k https://example.com/file.zipThis command restricts the download speed to 200 KB/s, ensuring other online activities are not disrupted.
4. Downloading Multiple Files
Wget can download multiple files at once. This is achieved by creating a text file with a list of URLs and using the ‘-i’ option.
wget -i urls.txtIn this example, urls.txt is a text file containing the URLs of all the files to be downloaded. The command downloads all the files listed in urls.txt.
5. Downloading an Entire Website
Wget can be used to download an entire website for offline viewing. The ‘–mirror’ or ‘-m’ option is used to do this.
wget --mirror https://example.comThis command mirrors the entire example.com website, downloading all its pages and assets for offline use.
6. Downloading Files with Specific Extensions
To download files with specific extensions from a website, use the ‘-r’ option for recursive download and ‘-A’ to specify the file types.
wget -r -A.pdf https://example.comThis command downloads all PDF files from example.com and its subdirectories.
7. Downloading Files Under Different Names
Using the ‘-O’ option, you can save a file under a different name.
wget -O myFile.zip https://example.com/file.zipThis command downloads file.zip from example.com and saves it as myFile.zip in the current directory.
8. Saving Files in a Specified Directory
You can specify a directory to save the downloaded file using the ‘-P’ option.
wget -P /path/to/directory https://example.com/file.zipThis command downloads file.zip from example.com and saves it in the directory specified by /path/to/directory.
9. Setting Retry Attempts
Using the ‘–tries’ option, you can set the number of retry attempts in case the download fails.
wget --tries=3 https://example.com/file.zipThis command attempts to download file.zip from example.com up to 3 times before giving up.
10. Downloading via FTP
Wget can also download files via FTP. You need to specify the username and password.
wget --ftp-user=USERNAME --ftp-password=PASSWORD ftp://example.com/file.zipThis command downloads file.zip from the FTP server at example.com using the specified username and password.
11. Continuing Interrupted Downloads
If your download gets interrupted, you can continue using the ‘-c’ option.
wget -c https://example.com/file.zipThis command resumes the download of file.zip from example.com from where it left off.
12. Retrieving Whole Websites
You can download the entire content of a website for offline viewing using a combination of options.
wget --mirror --convert-links --page-requisites --no-parent -P /path/to/directory https://example.comThis command downloads the entire example.com website and its assets, converts the links for offline use, and saves everything in the directory specified by /path/to/directory.
13. Locating Broken Links
Wget can be used to find all broken URLs that display a 404 error on a specific website.
wget -o wget-log -r -l 5 --spider http://example.comThis command checks all links on example.com up to 5 levels deep and logs the results in wget-log.
14. Downloading Numbered Files
If you have files or images numbered in a specific list, you can easily download all of them.
wget http://example.com/images/{1..50}.jpgThis command downloads the images 1.jpg to 50.jpg from example.com/images.
15. Downloading Files with a Timeout
With the ‘–timeout‘ option, you can set a timeout for when you stop trying to connect or read from a site.
wget --timeout=30 https://example.com/file.zipThis command attempts to download file.zip from example.com but will stop if it cannot connect to or read from the site within 30 seconds.
16. Ignoring Case When Matching Files/Directories
The ‘-nc’ or ‘–no-clobber’ option prevents wget from overwriting existing files or directories.
wget -nc https://example.com/file.zipThis command downloads file.zip from example.com only if a file with the same name does not already exist in the current directory.
17. Downloading Only Certain File Types
Using the ‘-r’ and ‘-A’, you can instruct wget to download only specific file types from a site.
wget -r -A jpeg,jpg,bmp,gif,png https://example.comThis command downloads all JPEG, JPG, BMP, GIF, and PNG files from example.com and its subdirectories.
18. Downloading All Files Except Certain File Types
Using the ‘-R’ or ‘-reject’ option, you can instruct wget to download all files except certain file types from a site.
wget -r -R pdf https://example.comThis command downloads all files from example.com and its subdirectories, except for PDF files.
19. Downloading Files from Websites That Use Cookies
Some websites require users to accept cookies before they can access files. You can use the ‘–load-cookies’ option in this case.
wget --load-cookies /path/to/cookies.txt https://example.comThis command downloads files from example.com using the cookies stored in cookies.txt.
20. Downloading Files from Password Protected Sites
If a site requires username and password authentication, you can pass them using the ‘–user’ and ‘–password’ options.
wget --user=USERNAME --password=PASSWORD https://example.comThis command downloads files from example.com using the specified username and password.
21. Downloading Files from a List of URLs
You can download files from a list of URLs using the ‘-i’ option.
wget -i list.txtThis command downloads files from the URLs listed in list.txt.
22. Downloading Files to a Specific Directory
Using the ‘-P ‘ option, you can specify the directory where the downloaded files will be saved.
wget -P /path/to/directory -i list.txtThis command downloads files from the URLs listed in list.txt and saves them in the directory specified by /path/to/directory.
23. Downloading Files with a Specific File Extension
You can download files with a specific file extension using the ‘-A’ option.
wget -r -A .pdf https://example.comThis command downloads all PDF files from example.com.
24. Downloading Files from a Specific Directory
You can download files from a specific directory on a website using wget.
wget -r -np -nH --cut-dirs=3 -R index.html https://example.com/dir/subdir/This command downloads all files from the subdir directory on example.com.
25. Downloading Files with a Specific Pattern in Their URL
You can download files with a specific pattern in their URL using wget.
wget -r -np -nH --cut-dirs=3 -A '*.pdf' https://example.com/dir/subdir/This command downloads all PDF files from the subdir directory on example.com.
26. Downloading Files from a Website Using FTP
You can download files from a website using FTP with wget.
wget --ftp-user=USERNAME --ftp-password=PASSWORD ftp://example.com/This command downloads files from example.com using FTP with the specified username and password.
27. Downloading Files from a Website Using HTTPS
You can download files from a website using HTTPS with wget.
wget https://example.com/file.zipThis command downloads file.zip from example.com using HTTPS.
28. Downloading Files from a Website Using HTTP
You can download files from a website using HTTP with wget.
wget http://example.com/file.zipThis command downloads file.zip from example.com using HTTP.
29. Downloading Files from a Website Using a Proxy
You can download files from a website using a proxy with wget.
wget -e use_proxy=yes -e http_proxy=PROXY_URL:PORT https://example.com/file.zipThis command downloads file.zip from example.com using the specified proxy.
30. Downloading Files from a Website and Saving Them with a Different Name
You can download files from a website and save them with a different name using wget.
wget -O newname.zip https://example.com/file.zipThis command downloads file.zip from example.com and saves it as newname.zip.
Conclusion
By mastering the various features of the wget command through these 30 practical examples, you can greatly enhance your efficiency in managing downloads on Linux. Whether you need to download a single file, automate a series of downloads, or mirror an entire website, wget provides the flexibility and power to accomplish these tasks with ease. Regular use of wget can streamline your workflows, save time, and ensure that you are leveraging one of the most useful tools available in the Linux command-line toolkit.