Trusted Proxies Blog
Dig much deeper and faster into Google, Yahoo & Bing
- Where you are
- Which version of Google you’re searching eg .com or a local domain eg .co.uk)
- Where your target audience is, eg your country, a different country, and whether one or both of those countries is specifically the USA or specifically not the USA
- Logged out of all Google accounts
- Use your browser in “Private” browsing mode, ie so that Google doesn’t “remember” your history or store cookies in order to“personalise” your results based on your previous queries, eg for these tests, I used FireFox in “Private Browsing” mode. (side note, I prefer Chrome, but FireFox lets me switch Proxy Servers with 2 clicks with the add-on we use)
- Exit the browser and restart it between each test. This is because by default, your browser may retain cookies between searches (FireFox does) and hence Google could be influenced to “personalise” your results where you don’t want it to
- All my tests have been performed in English only
First critical issue to address is Google’s current “modus operandi” is a relentless drive for “relevance” and personalisation to achieve that. They don’t have SEO’s, their customers, Keyword Ranking Reports, or Proxy Server companies in mind, so getting the info we want is getting less and less straightforward as a result.
Here’s a basic example:
As stated above, all these examples assume a new session, logged out of all Google accounts, and in Private Browsing mode.
If you are outside of the USA, (or use a non-US Proxy Server if you are in the USA), open a new session and go to google.com.
Google automatically redirects you to your local google, eg google.co.uk if you are in the UK.
That might sound strange at first as you may be used to going to google.com and not being redirected. That’s because when you are first redirected, Google offers you a link to “Go to google.com” which it then “remembers” via cookies the next time you go to google.com.
By automatically redirecting you to your local google domain, this is Google’s clear message of intent “You’re in the UK, we’re going to assume you want UK oriented results, rather than American, French or Japanese”.
This leads on to the second major point of Google “localization”: google.com in, say, the UK, is not the same as google.com in the USA.
Querying google.com from the UK (or any other country) gets different results than querying google.com from within the USA.
Interestingly, in my tests, results from google.com seem to be the same when you are sending the identical query from UK, France or Israel (these are the countries I have tested recently, I recollect testing others in the past and getting the same results).
In other words, it seems to me that Google is telling you there are two different versions of google.com and they would rename them like this if they could:
The flipside also seems to be true, based on customer reported experiences and our own tests:
Querying a local google domain gives you the same results no matter which country you query from, including your own:
For example and perhaps surprisingly, querying google.co.uk from France gives you the same results as querying google.co.uk from the UK.
What this tells me is that Google has a “rule” that says, “If someone queries a specific local Google domain, they must want results oriented to that country, so we’ll give them that data, no matter where they are querying from”.
In all both of these cases, I am relating what I have seen and what customers have told me. As is always the case with Google, “caveat emptor” – Google reserves the right to do whatever it wants and mess up your day at the drop of a hat. Ie it may not ALWAYS behave the way I have described.
By way of example, I have one customer in Germany for example who required a huge number of Proxy Servers for google.de. As it costs us more to host Proxy Server in Germany than the US, German Proxy Servers cost more than US ones.
For this client as there were so many Proxy Servers involved, they worked out that they could save a tidy sum by buying just US Proxy Servers, and have been very happy with them.
Therefore there’s an argument for saying Trusted Proxies doesn’t need to host Parallel Private Proxy Servers anywhere outside of the USA. Aside from being the lowest cost place to host them, they server the American market obviously, and by the sounds of it can be used for querying for any other country.
There are two issues with this:
a) That logic only works for local Google domains, eg you can query google.co.uk from anywhere and seemingly get the same results, but it doesn’t work for google.com – see above. Hence my opening statement that it depends where your customers are and which google they’re looking at. If you are non-US and your clients and their visitors for whom they’re optimising for are primarily using your local Google, you may well be able to use lower cost US only Proxy Servers
b) In our experience, and probably driven by the “caveat emptor” of Google unpredictably returning different results depending on your location, we have found that most clients still prefer the feeling of “safety” of using Proxy Servers in their own/target country(ies).
As an aside, this gives us an interesting quandary: If we offer local Parallel Proxy Servers, customers could suspect we’re trying to unnecessarily foist higher costs Proxy Servers on them. Or if we offer them US, they could suspect we’re offering them bad advice which could lead to inaccurate results.
So for the record, we’re happy to sell you whatever you want Maybe you should take both
Long as this post is, it’s still only half the story (isn’t good SEO about a deep understanding of subtle points?), and doesn’t explain how to handle results via the Big-G Stealth Extractor.
One of Google’s shiny new toys it likes to play with in its ever relentless drive towards “personalization” even when you don’t want it, is “Location Based Technology”.
As well as detecting your country, Google also wants to know where in the country you are.
And it does this using its own algorithms and methods that are not always the same as the rest of the industry. The vast majority of websites in the world tailor their content to local markets on the basis of the surfer’s IP address and the country it is in according to commercial “Geo-IP” databases. They tend to be 95% accurate for working out a visitor’s country, but only 50% to 80% for city.
I believe this is why Google uses their own “Location Technology” as it knows 50% to 80% is not good enough, and they think they can do better. Although we have proof they are prone to getting it 50% wrong themselves, but that’s for another post.
Here’s the thing some SEO’s may not have fully appreciated before. Every time you go to Google, it is trying to figure out your precise location, and wants to serve you results tailored, or at least influenced by your location. Every time.
Gone is the “luxury” of being able to rely on “universal” results for a country.
A bold supposition I want to make on this blog is that you can no longer run universal Keyword Ranking Reports anymore that will give you the same results in your office (with or without Proxy Servers), as your client sees, as their visitors see. Even logged out of Google and in Private Browsing mode, Google is still able to “personalise” your results – and is doing it more and more.
This is what I meat in my opening remarks about Google not having SEO’s or our reports in mind in its march towards ever greater “relevance” and “personalisation”.
Here’s an example that came up recently with a client:
The first search term was “chinese restaurant”. Not surprisingly, this is a keyword that has “local” written all over it, so Google shows a Local Pack in the SERPs but also may show local influences results in the generic SERPs too.
Given what we all know about Google’s recent march to “everything local” perhaps it doesn’t come as a surprise that you can no longer get “generic” KW rankings for “chinese restaurant”.
But this client also sent us “web hosting” to investigate. What could be “local” about that?
Nevertheless, if Google thinks you’re in Atlanta, it will include results for “Atlanta web hosting” and if it thinks you’re in Seattle, you’ll get results for “Seattle web hosting”.
So again, we see that there is no longer any such thing as a generic or universal KW rank for huge numbers of keywords now, especially in the USA. (I haven’t been able to verify it either way for local domains. It could be that Google have rolled this our in the US first, and will roll it out worldwide later).
Are there any work arounds?
I did find one that seems to work frequently, but not 100%.
Google has extra search tools in the column to the left of the SERPs. One of them is to be able to set the “Custom Location” (Google’s admission that their location technology sucks?).
You can set it to “United States” and it gives you what look like “universal” USA wide results. You can also set “United Kingdom” for google.co.uk for universal UK results. Indeed Google says that this is a legitimiate use (See near bottom of this page under “Can I turn off location-based customization?”
Yes and no!
Within the US I did find using Proxy Servers in different cities, with custom location of “United States” sometimes returned slightly different results. This may be the kind of variations that have always been seen in SERPs, and perhaps isn’t surprising given the size and population of the country.
So for American customers on google.com, it maybe that you and your clients can live with these occasional variations.
But the question for American customes is, can your software support it? Can your software set the Custom Location? IMO, and purely hypothesis, that based on some of the log files I have inspected for my own traffic, I don’t think the commercial KW Ranking software suppliers do yet. I would love to hear feedback on that.
Please note that most packages do have options in their UI’s to select countries with their Search Engines, eg “Google.com (USA)” but it doesn’t mean to say that under the bonnet [“hood”] that that’s what they’re doing.
If you have your own custom software, with a little extra coding, your developers should be able to emulate a human “clicking” on the Custom Location, and so long as they handle the cookies OK, you should be set.
Please note, that within google.com in the USA, setting the custom location is “remembered” by Google via a cookie, NOT by parameters in the url.
In my tests, using the famous gl=US parameter does not alter the results like this anymore. Hence my scepticism that commerical KW Ranking software is handling this correctly now.
[A small but relevant plug here for our US Metro Proxy Servers – if you are concerned about these occasional variations in results, even for generic custom location of “United States” you can test out results in several dozen different US Metro Areas with our US Metro Proxy Servers].
Outside of the US, using the UK as an example, the answer is a little more complicated.
For SERPs from google.co.uk, the same principle applies, you can set the custom location to “United Kingdom” and you’re done, it looks to me as though “local” results are removed.
But what if you need to check UK results on google.com?
When you/your customers first go to google.com you’re redirected to google.co.uk as described above, and you can click “Go to google.com”.
The SERPS appear to me to be “universal” American results, NOT UK oriented results. The same is true when checked from France.
This behaviour I believe is the same as what I described near the beginning, that you get the same results from a local google no matter which country you query from. In this case, you’re getting generic/universal “American” results from google.com, regardless of which non-US country you query from. In my opening example, this is the equivalent of “google.com.usa”.
Maybe you’re thinking you can just set the custom location to “United Kingdom” and then you’ll get universal UK results from google.com. But you can’t, you get an error message! Once you are in google “USA” – you can only set a custom location within the USA. The same is true for google.co.uk, you can only set a custom location for an address within the UK.
Several interesting conclusions are thrown up by this:
a) It probably doesn’t matter! If you’re in the UK and get US results from google.com, so are your clients and their visitors! So maybe my last few paragraphs are just a red herring!
c) Unfortunately, it’s not a complete panacea. Like I said with local google domains, you get the same results no matter where you query from, the results from this “google.com.usa” are slightly different to the generic American results seen by Americans with custom location of “United States”, it’s more like my “google.com.universal”. Hence having led you up the garden path with this, it isn’t really a reliable solution for American clients interested in American results for google.com. Rather it could be useful to know what google.com holds for your keywords around the world, withut needing to get Proxy Servers in every one of those countries.
Some other interesting asides:
If you are outside of the USA, and go to google.com and set the custom location to “United States” for a query, look at the search url.
A new parameter has been added, “tbs”.
There has been some chatter about the tbs parameter, mainly in connection with using it to get time filtered data.
But in our example query
our tbs parameter looks like this:
which is translated into:
I conclude from this that tbs is a “master” parameter telling google that one of the search tools on the left has been used, tbs=cloc probably means it is the “Custom Location” that has been set, and cl_loc is what it has been set to, in this case, “United States”.
At first sight this seems really exciting, as we have found a way to replace the gl=US parameter with Google’s “new way of doing things”.
However, for google.com it can only be used by non-American google.com, eg a “google.com.universal” reached via google.co.uk first. If you try to paste in a url with tbs set directly into a browser running within the US, you get an error message.
Therefore you could use it to get an idea of American results from outside the USA without the use of Proxy Servers, but it isn’t completely accurate and does often contain non-USA results.
This parameter can also be used to query “google.com.usa” for non-USA locations also, eg I did a test from France, with this query:
and got what looked like UK SERPs with a London Local Pack.
Taking into consideration my “misselling” disclaimer above, use Proxy Servers in the country(ies) for which you want to check results that your clients are in.
If you are concerned about the impact of “localisation” skewing your data (ie you want country-wide data, rather than local), then you’ll need to set the Custom Location.
If you’re working manually through a browser that’s easy. If you’re using commercial Keyword Ranking software, you need to check with your vendor whether they’re doing this or not.
If you’re using custom software, ask your developers to add some code to first visit google.com (or your local domain), perform the first search (and discard the result), then set the Custom Location, record that refreshed result, then carry on issuing queries from there, being careful to maintain the session and cookie data.
All of our Parallel Proxy Servers will geo-locate to some city or another. Google may or may not be able to work it out themselves, so may give you the correct city by default, the wrong city, or no city, just country wide if it can’t work it out.
The same is true for the Big-G Stealth Extractor. We built it to be “location neutral” but it seems that Google have updated their algorithms and so it does default to one US city at the moment (although it seems to have changed although our hardware didn’t).
Hence the Big-G Stealth Extractor should be used the same way as US Parallel Proxy Servers (editions for other countries may follow soon).
In practice that means using the Big-G Stealth Extractor without software that handles the cookies for setting Custom Location to “United States”:
- Can be used for US-wide results but may have changing localised data for some of the SERPs
- Can be used for any local google domain, eg google.co.uk and should get the same results (most of the time as far as we can tell) as, say, google.co.uk queried in the UK.
All of this info is of course subject to change at very short notice as and when Google update their algorithms!