How to Use Curl User Agent to Get URL Source Code

There is a cool method you can use with a curl user agent to get the HTML & CSS source code from a URL. Using this curl user agent bash you can even the the source code for even the http header info as well. It’s important to note that some sites have different content or HTML that won’t always work with the curl usher agent with a specific Internet browser or HTML. You may also be interested in reading; Command Prompt Tricks and Hacks.

The good news is that in most cases, curl user agent on Chrome and curl user agent on Internet Explorer work most of the time. But in cases when the curl default user agent won’t work as plan, we can spoof the user agent of another browser version and operating system, and this allows web developers to quickly get access to those alternate variations of a sites source code. This is a great alternative for curl user agent on Android. The following is the command line by using curl user agent.

The basic syntax is as follows:

curl -A "UserAgentString" http://url.com

One of the most common situations of different source HTML and CSS are for websites with stripped down mobile versions, you could retrieve iPhone-specific source code with:

curl -A "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5" http://www.apple.com

Some sites do this with other browsers too. This would be Chrome 12 in Mac OS X 10.6.8:

curl -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.112 Safari/534.30" http://microsoft.com

Here’s another that spoofs the Mac App Store and Mac OS X 10.6.7 as a user agent and is useful for querying the App Store from a script (more about that on TUAW):

curl -silent -A "iMacAppStore/1.0.1 (Macintosh; U; Intel Mac OS X 10.6.7; en) AppleWebKit/533.20.25" http://ax.search.itunes.apple.com/

Yet another spoofs Windows XP with Firefox 3:

curl -A "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" http://yahoo.com

You can search for different types of php curl user agent strings on the Internet, it’s required to include quotes if you want to retrieve a sites source as that user agent. If you want to read more about user agents, Wikipedia has a good entry on the topic.

Note: This is intentionally done through the command line and aimed at those who like to work from the Terminal, but there are easy ways to do this through standard graphical applications and web browsers like Safari, Chrome, and Firefox. Safari is probably the simplest, because you can set different user agents directly from the Developer menu: