I am using CURL to check for the existence of a URL (HEAD request) but when I test it with www.google.com , it redirects me to www.google.co.uk - probably because my server is UK-based.
Is there a way you can stop this from happening? I don't want to remove the CURLOPT_FOLLOWLOCATION option as this is useful for 301 redirects etc.
Part of my code is below;
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_MAXREDIRS, 5);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_FORBID_REUSE, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 4);
curl_setopt($ch, CURLOPT_TIMEOUT, 4);
$output = curl_exec($ch);
// get data
$data = curl_getinfo($ch);
$data['url'] contains www.google.co.uk when I set $url as www.google.com
Source: Tips4all
You need to use curl with a cookie that simulate a similar behavior in a browser.
ReplyDeleteWhen you visit google.com from England it redirects you to google.co.uk, however there is a link on that page titled "go to google.com" that lets you go back to google.com and stay there. It uses a cookie to remember your site preferences.
For example, here are the cookies that I have after doing this (using firefox):
Try accessing www.google.com/ncr, it'll avoid the redirect to the .co.uk (or any other national) page.
ReplyDeleteA bit of a hack, but how about using an IP address? http://216.239.59.147/ http://66.102.7.104/
ReplyDeleteYou could use www.google.co.uk directly, no difference there. google.com/.net always redirect to your location but if you use a country TLD like .co.uk it will not redirect.
ReplyDeleteThere is no way (known to me) to prevent the redirect when using .com or .net.
One way to avoid Google from deciding what country you are in, is by setting a different IP address. Just get one of the many US proxy servers from the Web and do something like this:
ReplyDelete$ch=curl_init();
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch,CURLOPT_FOLLOWLOCTION,1);
curl_setopt($ch,CURLOPT_PROXY,"8.12.33.159");
curl_setopt($ch,CURLOPT_PROXYPORT,"80");
curl_setopt($ch,CURLOPT_USERAGENT,"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3");
curl_setopt($ch,CURLOPT_URL,$URI);
$results=curl_exec($ch);
curl_close($ch);
This way, Google will think you come form a US IP address and not redirect to a local Google.
You should turn off the follow location from curl (set it to false) and you won't be redirected anymore ...
ReplyDeletecurl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);