Spoofing X-Forwarded-For... What Happens?

As explained in a previous post, X-Forwarded-For can be used by web servers to detect the use of a proxy server. By comparing the IP address of the client with the IP address in the X-Forwarded-For field, it is possible to say whether the end user is connecting to the server via an HTTP proxy server.

Suppose the end user IP address is x.x.x.x and proxy IP address is y.y.y.y. For the web server's point of view, proxy server is the client with IP address is y.y.y.y. Since the end user is connecting to the web server via the proxy server, the X-Forwarded-For field will have the IP address x.x.x.x when the request reaches the web server. At the web server, if a comparison is made the client IP address (y.y.y.y) is obviously different from the IP address in X-Forwarded-For field (x.x.x.x). Hence this indicates the use of a proxy server.

Now you may think by spoofing the X-Forwarded-For field by setting the value to y.y.y.y (which is the proxy IP) it is possible to hide the use of a proxy server. This is not always true, which can be seen from the following demo Python script.



Output of the above Python script is:

Significance of User-Agent Field: Detection of Automated/Programatic HTTP Requests....!

Suppose you request a webpage using a program, say a Python script. Have you ever wondered whether a web server can differentiate it from a manual request by a user via a web browser, say Firefox. Yes it is possible for the web server to differentiate using various methods. One such method is analyzing the value of the 'User-Agent' field in the HTTP request header from the client.

You can see the difference from the output of the following Python script:

The output of the above script is:

Testing an IP:Port Proxy Server

It is common to use proxy servers to query webpages using programs written in Python, Java, etc. In doing so, it is necessary to maintain a list of proxies which are guaranteed to work as and when we make a webpage query. This can be done by storing a list of proxies in a database and by running a daemon to periodically check whether a particular proxy does what it is supposed to do. There are two important things that you need to consider in testing a proxy server:

1. Is the proxy server available - this check refers to the proxy server connectivity.
2. Is the proxy server correct -  this check is performed to see whether a proxy server is providing the requested webpage or replies with a random webpage.

The following Python code for testing IP:Port proxy servers can be used as a guidance:


It is important to ensure that the webpage used for testing (http://www.testpage.com) is reachable and static. If the webpage is neither reachable nor static then the above test will indicate a proxy failure, which in fact is a false alarm!

• A list of IP:Port proxy servers can be obtained from Hide My Ass .

An alternative to changing remote proxy servers used by Ultrasurf

Ultrasurf is an awesome way to use remote proxies via a locally created proxy server. As mentioned in my previous post, you can either set the browser proxy settings to use 127.0.0.1 (localhost) as the IP address and 9666 as the port number. This is the default setting used by Ultrasurf, which acts as a local proxy server and connects to remote proxies.

There could be situation where we have such a requirement. Suppose you are accessing a website via Ultrasurf and the website blocks the specific proxy IP address. In this case you will have to change the remote proxy server used by Ultrasurf. One way of doing this is via the Ultrasurf user interface, which gives an option of choosing one of three servers. However, it is not easy to programmatically change the remote proxy server Ultrasurf uses.




Note that, if you close the Ultrasurf application and reopen it, most of the time it connects to a new remote proxy server. Hence, a solution for the above problem is to close and reopen the Ultrasurf application as and when you need to change the remote proxy server used by Ultrasurf. This can be achieved from the following simple python code.

Python code to use Ultrasurf

At times we have the need to access a particular URL or fetch web pages using a program/code.

Suppose we want to perform this action using a proxy server and we would like to use a proxy server that Ultrasurf provides. This can be achieved in many programming languages.

The following code is written in Python Programming Language. In the following we are accessing the URL, http://remotalks.blogspot.ca with the use of the Python module urllib.

The function urlopen in the module urllib accepts two parameters, the URL and Proxy Server Address, respectively. Here 127.0.0.1 is the IP address and 9666 is the default port number of the proxy server created by Ultrasurf.


Check my previous post for more details on Ultrasurf and detailed steps on installing  Ultrasurf in Ubuntu!

Running Ultrasurf in Ubuntu

Proxy servers become handy when you want to unblock a website or when you want to automate website querying. Ultrasurf provides the functionality of a proxy server. It runs a local proxy server at 127.0.0.1:9666. Here 127.0.0.1 is the IP address (localhost) and 9666 is the port used by the local Ultrasurf proxy server. You can change your browser proxy settings to point to this proxy server and start browsing.

Ultrasurf is a very useful tool if you are looking for secure proxy connections and if you have the need to change the proxy servers frequently.

In this article I present the steps to run ultrasurf in Ubuntu.

I used the following in the steps described in this article:

1. ubuntu 12.04 lts

2. ultrasurf 13.01

3. wine 1.5

Steps:

1. Install wine 1.5 using terminal/command prompt - use folllowing commands in the given sequence

    $ sudo add-apt-repository ppa:ubuntu-wine/ppa
    $ sudo apt-get update
    $ sudo apt-get install wine1.5

Please check http://www.winehq.org/ for more information on wine!

2. Download ultrasurf from http://ultrasurf.us/ and extract to any folder you prefer.

3. Run ultrasurf by double-clicking on the executable ultrasurf file u1301.exe. Alternately, the application can be launched by the following command:
    $ wine u1301.exe

In one of my upcoming posts, I will explain how Ultrasurf can be used along with python programming language to auomate website queries.