Significance of User-Agent Field: Detection of Automated/Programatic HTTP Requests....!

Suppose you request a webpage using a program, say a Python script. Have you ever wondered whether a web server can differentiate it from a manual request by a user via a web browser, say Firefox. Yes it is possible for the web server to differentiate using various methods. One such method is analyzing the value of the 'User-Agent' field in the HTTP request header from the client.

You can see the difference from the output of the following Python script:

The output of the above script is:

Python code to use Ultrasurf

At times we have the need to access a particular URL or fetch web pages using a program/code.

Suppose we want to perform this action using a proxy server and we would like to use a proxy server that Ultrasurf provides. This can be achieved in many programming languages.

The following code is written in Python Programming Language. In the following we are accessing the URL, http://remotalks.blogspot.ca with the use of the Python module urllib.

The function urlopen in the module urllib accepts two parameters, the URL and Proxy Server Address, respectively. Here 127.0.0.1 is the IP address and 9666 is the default port number of the proxy server created by Ultrasurf.


Check my previous post for more details on Ultrasurf and detailed steps on installing  Ultrasurf in Ubuntu!

Running Ultrasurf in Ubuntu

Proxy servers become handy when you want to unblock a website or when you want to automate website querying. Ultrasurf provides the functionality of a proxy server. It runs a local proxy server at 127.0.0.1:9666. Here 127.0.0.1 is the IP address (localhost) and 9666 is the port used by the local Ultrasurf proxy server. You can change your browser proxy settings to point to this proxy server and start browsing.

Ultrasurf is a very useful tool if you are looking for secure proxy connections and if you have the need to change the proxy servers frequently.

In this article I present the steps to run ultrasurf in Ubuntu.

I used the following in the steps described in this article:

1. ubuntu 12.04 lts

2. ultrasurf 13.01

3. wine 1.5

Steps:

1. Install wine 1.5 using terminal/command prompt - use folllowing commands in the given sequence

    $ sudo add-apt-repository ppa:ubuntu-wine/ppa
    $ sudo apt-get update
    $ sudo apt-get install wine1.5

Please check http://www.winehq.org/ for more information on wine!

2. Download ultrasurf from http://ultrasurf.us/ and extract to any folder you prefer.

3. Run ultrasurf by double-clicking on the executable ultrasurf file u1301.exe. Alternately, the application can be launched by the following command:
    $ wine u1301.exe

In one of my upcoming posts, I will explain how Ultrasurf can be used along with python programming language to auomate website queries.