Spoofing X-Forwarded-For... What Happens?

As explained in a previous post, X-Forwarded-For can be used by web servers to detect the use of a proxy server. By comparing the IP address of the client with the IP address in the X-Forwarded-For field, it is possible to say whether the end user is connecting to the server via an HTTP proxy server.

Suppose the end user IP address is x.x.x.x and proxy IP address is y.y.y.y. For the web server's point of view, proxy server is the client with IP address is y.y.y.y. Since the end user is connecting to the web server via the proxy server, the X-Forwarded-For field will have the IP address x.x.x.x when the request reaches the web server. At the web server, if a comparison is made the client IP address (y.y.y.y) is obviously different from the IP address in X-Forwarded-For field (x.x.x.x). Hence this indicates the use of a proxy server.

Now you may think by spoofing the X-Forwarded-For field by setting the value to y.y.y.y (which is the proxy IP) it is possible to hide the use of a proxy server. This is not always true, which can be seen from the following demo Python script.

	import lxml.html as lh
	import urllib2
	url = "http://www.murl.mobi/headers.php"
	http_headers = {"User-Agent":"Mozilla/5.0 (Windows NT 6.1; rv:25.0)" +
	"Gecko/20100101 Firefox/25.0", "X-Forwarded-For":"65.182.107.98"}
	request_object = urllib2.Request(url, None, http_headers)
	xpath_s = "/html/body/table/tr[2]/td/table/tr/td[2]/div[2]/text()"
	#Installing proxy server details to be used by urllib2
	proxy = urllib2.ProxyHandler({'http': '65.182.107.98:3128'})
	opener = urllib2.build_opener(proxy)
	urllib2.install_opener(opener)
	doc = lh.parse(urllib2.urlopen(request_object))
	headers = doc.xpath(xpath_s)
	print 'Output of the request with the use of a proxy server and spoofed x-forwarded-for header'
	print '\n'.join(headers)

view raw x_forwarded_for_spoof.py hosted with ❤ by GitHub

Output of the above Python script is:

	Output of the request with the use of a proxy server and spoofed x-forwarded-for header
	Accept-Encoding: identity
	Host: www.murl.mobi
	Connection: close
	User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:25.0) Gecko/20100101 Firefox/25.0
	X-Forwarded-For: 65.182.107.98, xxx.xxx.xxx.xxx

view raw x_forwraded_for_spoof_out.txt hosted with ❤ by GitHub

X-Forwarded-For: Proxy Server Detection

Do you know it may be possible to detect your actual IP address (source IP) even though a proxy server is used? Yes...it is possible most of the time. Please read on and I will tell you how it may be possible.

The X-Forwarded-For (XFF) HTTP header field is used for identifying the originating IP address of a client connecting to a web server through an HTTP proxy (Wikipedia article on X-Forwarded-For HTTP header field). This field, if set/modified by the proxy server, enables a web-server to detect whether a client/browser is connecting directly or via a proxy server.

The following simple python script shows the difference in the HTTP request headers when a programmatic HTTP request is made via a proxy server.

	import lxml.html as lh
	import urllib2
	url = "http://www.murl.mobi/headers.php"
	http_headers = {"User-Agent":"Mozilla/5.0 (Windows NT 6.1; rv:25.0)" +
	"Gecko/20100101 Firefox/25.0"}
	request_object = urllib2.Request(url, None, http_headers)
	xpath_s = "/html/body/table/tr[2]/td/table/tr/td[2]/div[2]/text()"
	doc = lh.parse(urllib2.urlopen(request_object))
	headers = doc.xpath(xpath_s)
	print 'Output of the first request without the use of a proxy server'
	print '\n'.join(headers)
	#Installing proxy server details to be used by urllib2
	proxy = urllib2.ProxyHandler({'http': '65.182.107.98:3128'})
	opener = urllib2.build_opener(proxy)
	urllib2.install_opener(opener)
	doc = lh.parse(urllib2.urlopen(request_object))
	headers = doc.xpath(xpath_s)
	print 'Output of the first request with the use of a proxy server'
	print '\n'.join(headers)

view raw x_fwd_for_proxy.py hosted with ❤ by GitHub

Output of the above Python script is:

	Output of the first request without the use of a proxy server
	Accept-Encoding: identity
	Host: www.murl.mobi
	Connection: close
	User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:25.0) Gecko/20100101 Firefox/25.0
	Output of the second request with the use of a proxy server
	Accept-Encoding: identity
	Host: www.murl.mobi
	Connection: close
	User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:25.0) Gecko/20100101 Firefox/25.0
	X-Forwarded-For: xxx.xxx.xxx.xxx

view raw x_fwd_for_proxy_out.txt hosted with ❤ by GitHub

As we can see from the output, there is an extra field (X-Forward-For) present in the HTTP request header when the web request is made via an HTTP proxy server. Hence, using this field the web-server can easily identify that this request is made via a proxy server. A simple comparison of the value of 'X-Forward-For' with the source IP of the TCP connection (in this case it is the proxy IP of 65.182.107.98) will reveal the truth.

Note that, I have masked my source IP address (xxx.xxx.xxx.xxx). If you run the above script you will see your actual source IP address (without mask) as the value for the X-Forward-For field.

Check out this informative article on Typosquatting: Typosquatting

Significance of User-Agent Field: Detection of Automated/Programatic HTTP Requests....!

Suppose you request a webpage using a program, say a Python script. Have you ever wondered whether a web server can differentiate it from a manual request by a user via a web browser, say Firefox. Yes it is possible for the web server to differentiate using various methods. One such method is analyzing the value of the 'User-Agent' field in the HTTP request header from the client.

You can see the difference from the output of the following Python script:

	import lxml.html as lh
	import urllib2
	url = "http://www.murl.mobi/headers.php"
	xpath = "/html/body/table/tr[2]/td/table/tr/td[2]/div[2]/text()"
	doc = lh.parse(urllib2.urlopen(url))
	headers = doc.xpath(xpath)
	print 'Output of the first request without user-agent header supplied'
	print '\n'.join(headers)
	http_headers = {"User-Agent":"Mozilla/5.0 (Windows NT 6.1; rv:25.0)" +
	"Gecko/20100101 Firefox/25.0"}
	request_object = urllib2.Request(url, None, http_headers)
	doc = lh.parse(urllib2.urlopen(request_object))
	headers = doc.xpath(xpath)
	print 'Output of the second request with user-agent header supplied'
	print '\n'.join(headers)

view raw python_user_agent_demo.py hosted with ❤ by GitHub

The output of the above script is:

	Output of the first request without user-agent header supplied
	Accept-Encoding: identity
	Host: www.murl.mobi
	Connection: close
	User-Agent: Python-urllib/2.7
	Output of the second request with user-agent header supplied
	Accept-Encoding: identity
	Host: www.murl.mobi
	Connection: close
	User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:25.0) Gecko/20100101 Firefox/25.0

view raw python_user_agent_demo_out.txt hosted with ❤ by GitHub

Testing an IP:Port Proxy Server

It is common to use proxy servers to query webpages using programs written in Python, Java, etc. In doing so, it is necessary to maintain a list of proxies which are guaranteed to work as and when we make a webpage query. This can be done by storing a list of proxies in a database and by running a daemon to periodically check whether a particular proxy does what it is supposed to do. There are two important things that you need to consider in testing a proxy server:

1. Is the proxy server available - this check refers to the proxy server connectivity.
2. Is the proxy server correct -  this check is performed to see whether a proxy server is providing the requested webpage or replies with a random webpage.

The following Python code for testing IP:Port proxy servers can be used as a guidance:

	import urllib
	def is_proxy_working(ip, port):
	proxy_ip_port = 'http://%s:%s' % (ip, port)
	response = urllib.urlopen( "http://www.testpage.com",
	proxies={})
	test_page_len = len(response.read())
	#Start check for criteria 1.
	try:
	response = urllib.urlopen( "http://www.testpage.com",
	proxies={'http':proxy_ip_port})
	except:
	return False
	#End check for criteria 1.
	#Start check for criteria 2.
	test_page_len_with_proxy = len(response.read())
	if test_page_len == test_page_len_with_proxy:
	return True
	else:
	return False
	#End check for criteria 2.

view raw gistfile1.py hosted with ❤ by GitHub

It is important to ensure that the webpage used for testing (http://www.testpage.com) is reachable and static. If the webpage is neither reachable nor static then the above test will indicate a proxy failure, which in fact is a false alarm!

• A list of IP:Port proxy servers can be obtained from Hide My Ass .

'Stop Buy Order'

In a previous article I had explained the significance of 'Stop Sell Order'. In this article I will be explaining the details of a 'Stop Buy Order'.

As the name says, 'Stop Buy Order' is useful for a trader who wants to 'buy' a particular stock.

Similar to 'Stop Sell Order', 'Stop Buy Order' has 3 factors:
 

1. Stop price - the price at which you want to your stop order to become a limit order.
2. Stop limit - your 'bid price' for the limit order. (optional most of the time)
3. Quantity - the number of shares you would like to buy.

When the 'Stop Price' is triggered, the 'Stop Order' becomes a 'Market Order'. But in the presence of a 'Stop Limit' price, when the 'Stop Price' is triggered, the 'Stop Order' becomes a 'Limit Order'.

Suppose the price of a stock increases. By placing a 'Stop Buy Order' we can avoid paying a higher price. Suppose a stock is trading at $10.00 and you are deciding whether to buy at this price or wait for it to decrease further. However, there is a possibility that the stock price goes higher than $10.00. In this scenario you might end up paying a price which is much higher than $10.00. However, by placing 'Stop Buy Order', with a Stop price of $10.10, Stop limit of $10.10 and a quantity of 100, the maximum price you may pay for the stock is $10.10!

We do not need to keep checking the prices all the time to place a buy order. Stop order will take care of it.

Zillow (Z) !

Zacks named Z as the bull of the day on Fri, 11 Oct 2013 07:35:01 GMT (Zacks bull of the day). Z trades in NASDAQ stock exchange.

This article provides a summary of Z at the close of today's market, such as the summary of Z's activity during the day, and in the past 52 weeks, recommendations/opinions by different analysts and researchers, earning estimates, and recent news articles which talk about this particular symbol.

Zacks has a short term rank of 1 and a long term recommendation of Outperform for Z. Z closed at 81.48 on Mon, 14 Oct 2013, with a change of -0.87 (-1.06 %). The volume of shares traded was 909,550, which is a change of -458,144 compared to its average volume of 1,367,694. 52 week low and 52 week high of Z are 23.00 and 103.00 (Zacks).

According to Yahoo Finance, the mean price target of Z is 87.36 based on the opinion of 11 brokers. Moreover, the low and high price targets of Z are 50.00 and 115.00. Based on Analyst opinions, this week's mean recommendation for Z is 2.6, where 1.0 indicates strong buy and 5.0 indicates strong sell.

StockScouter, a stock rating sytem, gave a rating of 3 for Z, in a rating scale of 1 to 10, where 10 is considered the best.

Recent articles related to Z (Yahoo Finance RSS) as of the date of this blog post:

1. Bull Of The Day: Zillow
2. A Small Tech Stock With Huge Real-Estate Upside
3. Investors Take Note: The Worst Is Still Ahead

An alternative to changing remote proxy servers used by Ultrasurf

Ultrasurf is an awesome way to use remote proxies via a locally created proxy server. As mentioned in my previous post, you can either set the browser proxy settings to use 127.0.0.1 (localhost) as the IP address and 9666 as the port number. This is the default setting used by Ultrasurf, which acts as a local proxy server and connects to remote proxies.

There could be situation where we have such a requirement. Suppose you are accessing a website via Ultrasurf and the website blocks the specific proxy IP address. In this case you will have to change the remote proxy server used by Ultrasurf. One way of doing this is via the Ultrasurf user interface, which gives an option of choosing one of three servers. However, it is not easy to programmatically change the remote proxy server Ultrasurf uses.




Note that, if you close the Ultrasurf application and reopen it, most of the time it connects to a new remote proxy server. Hence, a solution for the above problem is to close and reopen the Ultrasurf application as and when you need to change the remote proxy server used by Ultrasurf. This can be achieved from the following simple python code.

	import subprocess
	#Python code to open the ultrasurf application
	ultrasurf_p = subprocess.Popen('ultrasurf.exe', shell=True,
	stdout=subprocess.PIPE,
	stderr=subprocess.STDOUT)
	while True:
	try:
	#Insert your code that raises an exception when
	#there is a need for a remote proxy server change
	use_current_remote_proxy_server()
	except:
	#Kill/Close the currently running Ultrasurf application
	ultrasurf_p.kill()
	#Reopen the Ultrasurf application
	ultrasurf_p = subprocess.Popen('ultrasurf.exe', shell=True,
	stdout=subprocess.PIPE,
	stderr=subprocess.STDOUT)

view raw ultrasurf_change_ip_proxy.py hosted with ❤ by GitHub

Herbalife (HLF)

Zacks named HLF as the bull of the day on Wed, 02 Oct 2013 07:22:01 GMT (Zacks bull of the day).

This article provides a summary of HLF at the close of today's market, such as the summary of HLF's activity during the day, and in the past 52 weeks, recommendations/opinions by different analysts and researchers, earning estimates, and recent news articles which talk about this particular symbol.

Zacks has a short term rank of 1 and a long term recommendation of Neutral for HLF. HLF closed at 73.09 on Wed, 02 Oct 2013, with a change of 0.25 (0.34 %). The volume of shares traded was 2,591,187, which is a change of -760,216 compared to its average volume of 3,351,403. 52 week low and 52 week high of HLF are 24.24 and 74.94 (Zacks).

According to Yahoo Finance, the mean price target of HLF is 76.50 based on the opinion of 6 brokers. Moreover, the low and high price targets of HLF are 69.00 and 92.00. Based on Analyst opinions, this week's mean recommendation for HLF is 1.6, where 1.0 indicates strong buy and 5.0 indicates strong sell.

StockScouter, a stock rating sytem, gave a rating of 2 for HLF, in a rating scale of 1 to 10, where 10 is considered the best.

Recent articles related to HLF (Yahoo Finance RSS) as of the date of this blog post:

1. Pershing���s Ackman Moves to Soften Risk on Herbalife Pyramid Bet
2. Bull Of The Day: Herbalife
3. The Man Bill Ackman Really Needs To Worry About In The Herbalife War

Recipe for a Homemade Coconut Milk Pudding!!!

Want to make a pudding out of Coconut Milk at home? You are at the right place.

I would like to share a recipe I used to make an awesome Coconut Milk pudding this weekend! It is easy and fast; does not take more than 30 minutes!

First off, make sure you have the following ingredients ready!

Ingredients:

1. 200 ml Coconut Milk
2. 100 ml Condensed Milk
3. 4 Eggs
4. 150g Brown Sugar
5. Cardamom
6. 50 ml Water

Now, follow these instructions in the given order and you will have an awesome Coconut Milk Pudding ready to eat!

Instructions:

1. Using a blender, blend 50 ml of water and 150g of Brown Sugar. 

2. Use a blender to beat 4 eggs and add that to the above mixture. Do not beat for too long to avoid the eggs getting cooked.
3.  Add 200ml coconut milk to the above mixture and mix it well.
4. Add 2-3 ground cardamoms.

4. Add 1 drop of vanilla flavor.
5. Using a conventional oven, bake it for 30-35 mins at 190 degrees Celcius

6. Let it cool and refrigerate for 4-5 hours.

7. Ready to eat!

MATLAB symbolic tool box

I find the symbolic tool box in MATLAB to be very useful. It is possible to do symbolic integration, differentiation, solving equations, simplifying symbolic formulae, etc.

In order to use symbolic functions in this tool box, we should first define symbolic variables. Symbolic variables a, b and c  can be defined as;

syms a, b, c;

Then we define a symbolic function as

f = a^2 + 2b +c;

The above function f, can be passed to symbolic tool box functions such as diff() or int(), to differentiate or integrate, respectively.

For example, function diff() takes two parameters, first the function to differentiate and second the variable with respect to which the function is differentiated. Thus,

diff(f, a) = 2a;
diff(f, b) = 2;
diff(f, c) = 1;

Similarly int() function gives the symbolic integration of a function with respect to thr specified symbolic variable.

simple() is another function that is very handy and I use quite often. simple(f), as its name says, simplifies function f. Another cool function in this tool box is pretty()! pretty(f) formats the way function f is displayed, such that it is more readable and pretty!

Python code to use Ultrasurf

At times we have the need to access a particular URL or fetch web pages using a program/code.

Suppose we want to perform this action using a proxy server and we would like to use a proxy server that Ultrasurf provides. This can be achieved in many programming languages.

The following code is written in Python Programming Language. In the following we are accessing the URL, http://remotalks.blogspot.ca with the use of the Python module urllib.

The function urlopen in the module urllib accepts two parameters, the URL and Proxy Server Address, respectively. Here 127.0.0.1 is the IP address and 9666 is the default port number of the proxy server created by Ultrasurf.

	import urllib
	#URL you would like to access via Ultrasurf
	url = "http://remotalks.blogspot.com"
	#Retrieve webpage at url via ultrasurf
	response = urllib.urlopen(url, proxies={'http':'http://127.0.0.1:9666' })

view raw ultrasurf_python_query.py hosted with ❤ by GitHub

Check my previous post for more details on Ultrasurf and detailed steps on installing  Ultrasurf in Ubuntu!

Running Ultrasurf in Ubuntu

Proxy servers become handy when you want to unblock a website or when you want to automate website querying. Ultrasurf provides the functionality of a proxy server. It runs a local proxy server at 127.0.0.1:9666. Here 127.0.0.1 is the IP address (localhost) and 9666 is the port used by the local Ultrasurf proxy server. You can change your browser proxy settings to point to this proxy server and start browsing.

Ultrasurf is a very useful tool if you are looking for secure proxy connections and if you have the need to change the proxy servers frequently.

In this article I present the steps to run ultrasurf in Ubuntu.

I used the following in the steps described in this article:

1. ubuntu 12.04 lts

2. ultrasurf 13.01

3. wine 1.5

Steps:

1. Install wine 1.5 using terminal/command prompt - use folllowing commands in the given sequence

    $ sudo add-apt-repository ppa:ubuntu-wine/ppa
    $ sudo apt-get update
    $ sudo apt-get install wine1.5

Please check http://www.winehq.org/ for more information on wine!

2. Download ultrasurf from http://ultrasurf.us/ and extract to any folder you prefer.

3. Run ultrasurf by double-clicking on the executable ultrasurf file u1301.exe. Alternately, the application can be launched by the following command:
    $ wine u1301.exe

In one of my upcoming posts, I will explain how Ultrasurf can be used along with python programming language to auomate website queries.

Recipe for a an awesome Condensed Milk Pudding!

I would like to share the recipe for the condensed milk pudding that I make every weekend! I make condensed milk pudding with simple ingredients and it takes me very little time, as low as 20 minutes!

Ingredients:

1. Condensed milk - 1 tin

2. 2% Normal Milk - 1.5 tins

3. Eggs - 3

4. Vanilla flavor

5. Food coloring

6. Cashew nuts

Steps:

1. Mix 1 tin condensed milk with 1.5 tin of milk. 

2. Use a blender to beat 3 eggs and add that to the above mixture. Do not beat for too long to avoid the eggs getting cooked.

3. Add few drops of vanilla flavor and few drops of food coloring (if u prefer colored pudding).

4. Boil it for 20 mins on medium. Or, steam for about 45 minutes (duration specified for electric steamer).

5. Add crushed cashew nuts.

6. Let it cool and refrigerate for 4-5 hours.

7. Ready to eat!

'Stop Sell Order' in stock trading

I would like share my experience with a particular stock I traded and the lesson I learnt from the process.

I had bought this penny stock (priced below a dollar) in Toronto Stock Exchange (TSE) and held it for couple of weeks expecting the price to go up. It did once for a short period of time but started dropping since then. However, I wasn't ready to sell it, instead I bought more of the same stock expecting the price to go higher.

After a few days, to my surprise and disappointment, the stock was delisted from TSE. It wasn't a total loss for me as the company did not go bankrupt; the stocks were transferred to another exchange. Even though I had the option of selling the stock at this other exchange, I was reluctant due to the very high trading fees we have to pay to trade at an exchange overseas.

From this I learnt it is important to minimize losses in stock trading in addition to focusing on making profits. Since then, I place a stop order at the very moment I buy a stock. Stop order helps to minimize losses.

Stop Sell limit order has 3 factors:
 

1. Stop price - the price at which you want to your stop order to become a limit order.
2. Stop limit - your 'ask price' for the limit order.
3. Quantity - the number of shares you would like to sell.

Note that, there is a difference between 'Stop Order' and 'Stop Limit Order'. The difference is 'Stop Limit'. In a 'Stop Order' there is no 'Stop Limit'. In this case, when the 'Stop Price' is triggered, the 'Stop Order' becomes a 'Market Order'. On the other hand, in the presence of a 'Stop Limit' price, when the 'Stop Price' is triggered, the 'Stop Order' becomes a 'Limit Order'.

By placing a stop order we can minimize losses. We do not need to keep checking the prices all the time to place a sell order. Stop order will take care of it.

I use the following to determine my stop price:

SP  =  (PP * Q + 2 * TF ) *  0.9 / Q

SP  - Stop Price
PP  - Purchase Price
Q    - Quantity
TF  - Trading Fee
0.9 - This value is chosen as my loss threshold

Let us consider an example. Suppose  we bough 100 shares at $10.00 each and the trading fee is $30 per trade. Thus, PP = 10, TF = 30 and Q = 100. Hence,

SP = (10 * 100 + 2 * 30) * 0.9 / 100 = $9.54