It's possible that spoofing the user-agent header might help, but unlikely. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Collectives on Stack Overflow. Learn more. Asked 5 years, 8 months ago. Active 4 years, 1 month ago. Viewed 35k times. I'm having trouble using wget for my Debian 7. HTTP request sent, awaiting response Does anyone see a flaw I'm making in my command?
Improve this question. Logan Butler Logan Butler 1 1 gold badge 1 1 silver badge 8 8 bronze badges. Is the file size K as well, or was that just the transfer rate? No, the file size is 96kb. Add a comment. GNU Wget is a popular command-based, open-source software for downloading files and directories with compatibility amongst popular internet protocols.
You can read the Wget docs here for many more options. For this example assume the URL containing all the files and folders we want to download is here:. You might get all the pages locally but all the links in the pages still point to their original place. It is therefore not possible to click locally between the links on the pages.
You can get around this problem by using the -k switch which converts all the links on the pages to point to their locally downloaded equivalent as follows:. If you want to get a complete mirror of a website you can simply use the following switch which takes away the necessity for using the -r -k and -l switches. Therefore if you have your own website you can make a complete backup using this one simple command.
You can get wget to run as a background command leaving you able to get on with your work in the terminal window whilst the files download. You can of course combine switches. To run the wget command in the background whilst mirroring the site you would use the following command:. If you are running the wget command in the background you won't see any of the normal messages that it sends to the screen.
You can get all of those messages sent to a log file so that you can check on progress at any time using the tail command. To output information from the wget command to a log file use the following command:. The reverse, of course, is to require no logging at all and no output to the screen. To omit all output use the following command:. Open up a file using your favorite editor or even the cat command and simply start listing the sites or links to download from on each line of the file.
Apart from backing up your own website or maybe finding something to download to read on the train, it is unlikely that you will want to download an entire website.
You are more likely to download a single URL with images or perhaps download files such as zip files, ISO files or image files. With that in mind you don't want to have to type the following into the input file as it is time consuming:. If you know the base URL is always going to be the same you can just specify the following in the input file:. If you have set up a queue of files to download within an input file and you leave your computer running all night to download the files you will be fairly annoyed when you come down in the morning to find that it got stuck on the first file and has been retrying all night.
You might wish to use the above command in conjunction with the -T switch which allows you to specify a timeout in seconds as follows:. The above command will retry 10 times and will try to connect for 10 seconds for each link in the file.
Learn more. Asked 4 years, 6 months ago. Active 1 year, 10 months ago. Viewed 11k times. Thank you very much. Improve this question. Mostafa Tavakoli Mostafa Tavakoli 65 1 1 gold badge 1 1 silver badge 6 6 bronze badges. Add a comment.
Active Oldest Votes. Improve this answer. I actually stopped for a second to think if it could be Farsi, but found it statistically more probable to be Arabic
0コメント