How common is Cloaking? (Showing different content to search crawlers than to users)

Google often points out that sites should not show significantly different content for users and search crawlers. Penalties threatened. It might negatively impact ranking accuracy, and Google also needs to see differences in files to update their index (not mentioned by Google, but likely).

An easy way to see if cloaking happens is to compare variations of a page downloaded with different user agents (curl, googlebot, firefox, ....) and then compre the md5 hashs of the versions. If a site sends different data only to crawlers (and not based on other user agents, IE or FF for example) this indicates that the site might be cloaking.

How do some categories of sites fare in this?

1. Spamlist 2 (as explained previously) is basically a list of 2336 blogs on different large scale blog platforms like blogger, tumblr, wordpress, blogspot that have many attributes that might be indicators for spam.
2. The earlier spam list has similar results (spam based on industry and competitor related inbound links).
3. The Alexa* lists use the top 1000 urls, middle 1000 urls and last 1000 urls of the Alexa top 1 million list.

The results sorted by percentage of 'cloaking':

And the table of results, again sorted by 'cloaking' percentage (other combinations make the difference to 100%):

Discussion

It seems one group of spammers differentiates a lot by agent - they still don't have a lot of settings where only bots see different content. Quite interesting, too, that one spam group and the top alexa sites are more likely to cloak than other sites. (Again - this is only considering one factor on how the homepage is displayed with all the resulting vagueness.)

How to replicate

First - this is severely limited, as it only analyses the homepage, and only the core page, not elements that are loaded with the page (images, scripts, etc.).

First generate list of spam-like urls / domains. All lists are checked for 200 status with several options: the url as is, then as https, then with www, then www with https plus one test that tries to work around crawler detection. The resulting lists of unique 200 OK urls are used in the next steps.

Download the homepage (just the html part) with different user agents (googlebot,bingbot,FF,IE,Chrome, and whatever else pleases your heart). For each download build the md5 has of the file, store in a table.
With awk we can compare quickly if a hash is the same or different by user agent, which then just needs to be summarized.

Big social platform shares - added Stumble Upon likes, shares, lists

And another script to check for social shares...this time more details around stumble upon (no, not really active there, anymore, although it has led me to a few outstanding sites).  The older version of this script pulls stumble data, but this shows pageviews from stumblers, not amplification through stumble activities. So this was mixing traffic data with engagement data - fixed now.

The data for stumble is pulled in two steps - first get the badge ID for the url, and with that pull the data (likes, shares, lists) for it. They all go into just one variable below, tab delimited, and then per echo into the larger list. That's why there are not 3 fields immediately visible when adding the data, but just seemingly one.

echo -e "Url\tripples\tFB-comments\tFB-shares\tFB-likes\ttwitter\tlinkedin\tstumble_likes\tstumble_shares\tstumble_lists" > "${1}-all-social-shares".csv

while read -r shortline ;
 do

This line is to replace special characters with their html encoding...

line=$(echo $shortline | sed -e 's/\?/\%3F/g' -e 's/&/\%26/g' -e 's/#/\%23/g')

gpull="https://plus.google.com/ripple/details?url=${line}"
ripples=$(wget -qO- "${gpull}" | grep -o "[0-9]*\s*public\s*shares.<" | sed "s/[^0-9]//g"  | tr "\n" "\t" | sed 's/\thttp/\nhttp/g'| sed 's/\t//')
commentpull="https://api.facebook.com/method/fql.query?query=select%20comment_count%20from%20link_stat%20where%20url=%22${line}%22&format=json"
comment_count=`wget -qO- $commentpull | sed -e 's/^.*://g' -e 's/\}//g' -e 's/\(]\)//g'`
echo "comment count: " $comment_count
sharepull="https://api.facebook.com/method/fql.query?query=select%20share_count%20from%20link_stat%20where%20url=%22${line}%22&format=json"
share_count=`wget -qO- $sharepull | sed -e 's/^.*://g' -e 's/\}//g' -e 's/\(]\)//g'`
echo "share count: " $share_count
likepull="https://api.facebook.com/method/fql.query?query=select%20like_count%20from%20link_stat%20where%20url=%22${line}%22&format=json"
like_count=`wget -qO- $likepull | sed -e 's/^.*://g' -e 's/\}//g' -e 's/\(]\)//g'`
echo "like count: " $like_count
twitterpull="http://urls.api.twitter.com/1/urls/count.json?url=${line}&callback=twttr.receiveCount"
twitternumber=$(wget -qO- "${twitterpull}"  | grep -o 'count\":[0-9]*\,' | sed -e 's/count//g' -e 's/,//g' -e 's/://g' -e 's/"//g' )
echo "twitter: " $twitternumber
linkedpull="http://www.linkedin.com/countserv/count/share?format=json&url=${line}" #echo ${linkedpull}
linkednumber=$(wget -qO- "${linkedpull}" | grep -o 'count\":[0-9]*\,' | sed -e 's/count//g' -e 's/,//g' -e 's/://g' -e 's/"//g' )
echo "linkedin count: " $linkednumber
stumblepull=`wget -qO- "http://www.stumbleupon.com/services/1.01/badge.getinfo?url=${line}" | grep -o "publicid.*views" | sed -e "s/publicid//" -e "s/\":\"//" -e "s/\",\"views//"`echo $stumblepullstumblenumber=$(wget -qO- "http://www.stumbleupon.com/content/${stumblepull}" | grep "mark>" | sed -e "s/^.*\">//" -e "s/<.*$//" -e "2d" -e "4d" -e "6d" | tr "\n" "\t")

echo -e "${line}\t${ripples}\t${comment_count}\t${share_count}\t${like_count}\t${twitternumber}\t${linkednumber}\t${stumblenumber}" >> "${1}-all-social-shares".csv
done < $1 

cat -A "${1}-all-social-shares".csv

Data Visualization

Just finished reading “Envisioning Information, The visual Display of Quantitative Information”, from Edward Tufte (first edition I just realized, got to check the second edition!)

Tufte shows how information can be condensed, transformed, made visible.
From 3D environment to 2D picture, from n-dimensional arrays to 2 dimensional maps, envisioning the other dimensions becomes a crucial task for designers.

The book

It is a book full of wonderful maps and graphics showing different, unique ways to visualize data. The chapter titles are an indicator of the main categories:

1. escaping flatland
   
2. micro/macro readings
   
3. layering and separation
   
4. small multiples
   
5. color and information
6. narratives of space and time.

From complex train schedules from espionage to regular maps, from lists to dance pattern manuals, Tufte picked a wide variety to show what is possible in envisioning quantitative data.

Missing

I am missing some digital information, some more technical displays. How about oscillation, how about the interactive displays with mouse overs on the net or with flash? Might be difficult to show in a book, but this is a strong limitation. Would be great if there could be an "interactive 2 D envisioning" addendum on the net to this book.

Application in Web Analytics

Sure, I had quite some ideas how this could be used for web graphics and especially for the display of data in web analytics.

Just one example. Why not make a real map out of the sitemap?

• map=sitemap
  
• country=pages
  
• rivers=paths
  
• hight=pageviews
  
• population=conversion
  

There are quite some analogies and it might be helpful to users to depict it in a pattern each of us learned to read in school. On top, it might give us insight into patterns we cannot see looking at plain 2D maps or data tables.

The book:

Tufte, Edward R., Envisioning Information, subtitle The visual Display of Qantitative Information, 1990. More details and some pictures from inside from the author.

Google Analytics (GA) Improvement Idea

One of the ultimate questions web analytics is supposed to answer is ‘what do I have to do to earn as much as possible with my site’, or close to that. This is a possible, viable outcome of e-commerce and shop analytics already. I have not seen or read about similar clear results for non-shopping sites.

Cause and Effect

For information, entertainment and brand sites the most important step is to simply (Hah!) connect what happens on the web with what happens on the site in a cause-effect relation.

Combine Calendar with Analytics Reports

So I asked our team to key in major site updates and PR releases into a Google calendar (many freelancers). Then, when checking the reports, we could compare these calendars with what happened on site and were able to connect some movements with a likely cause. Sometimes this was very clear, sometimes more vague. Especially on sites offering high value services or products where a direct connect from campaign to sales is not possible, there are many other things to factor in and from offer to contract can take more than half a year.

Still the calendars were a great way to start thinking about cause and effect. Just knowing, remembering what happened in a 300 project per year environment helped to distinguish between seasonal changes and campaign effects.

Google Analytics (GA) Improvement Idea

Now we just need GA to integrate that calendar into the GA view. As a first step, just a very plain view would be helpful like in the screen.

Later on, it might be great to be able to add some more data like budget or estimated traffic into the calendar, and get some calculations back with analytics. Best would be if GA could find connections between calendar events and site effects by itself!