Christian Long Tech notes and tidbits

Here is an example of using the command line cURL tool to automate logging in to a website and downloading some data. It does the same thing as my PycURL example.

But how do you discover what stimülüs to apply to the server to get the response you desire? Use a tool such as liveHTTPHeaders (Firefox) or ieHTTPHeaders (IE6 - I couldn't get it to install properly with IE7). Install the tool in your browser, and visit your target site as you would normally. The *HTTPHeaders tool will capture the actual HTTP traffic between your computer and the server, which you can then analyze to discover how to mimic a real client using cURL or libcURL.

Note that each call to curl in the following example should exist on its own long line, even though the lines might be displayed as wrapped in your browser.


@echo off
curl -s -S -o C:\Temp\temp.txt -L -b C:\Temp\cookieFile.txt -c C:\Temp\cookieFile.txt -A "Mozilla/4.0 (compatible; MSIE 6.0)" http://interesting.website.com/LogIn.asp

curl -s -S -o C:\Temp\temp.txt -L -e ";auto" -b C:\Temp\cookieFile.txt -c C:\Temp\cookieFile.txt -A "Mozilla/4.0 (compatible; MSIE 6.0)" -d "FormField=URL%20Encoded%20Value" http://interesting.website.com/LogIn.asp

curl -o downloaded_file.txt -e "http://interesting.website.com/referer.asp" -b C:\Temp\cookieFile.txt -c C:\Temp\cookieFile.txt -A "Mozilla/4.0 (compatible; MSIE 6.0)" --url "http://interesting.website.com/do_it.asp?do=0&something=0&interesting=0"

del C:\Temp\cookieFile.txt
del C:\Temp\temp.txt
echo Done!
pause