Big social platform shares - added Stumble Upon likes, shares, lists

And another script to check for social shares...this time more details around stumble upon (no, not really active there, anymore, although it has led me to a few outstanding sites).  The older version of this script pulls stumble data, but this shows pageviews from stumblers, not amplification through stumble activities. So this was mixing traffic data with engagement data - fixed now.

The data for stumble is pulled in two steps - first get the badge ID for the url, and with that pull the data (likes, shares, lists) for it. They all go into just one variable below, tab delimited, and then per echo into the larger list. That's why there are not 3 fields immediately visible when adding the data, but just seemingly one.

echo -e "Url\tripples\tFB-comments\tFB-shares\tFB-likes\ttwitter\tlinkedin\tstumble_likes\tstumble_shares\tstumble_lists" > "${1}-all-social-shares".csv

while read -r shortline ;
 do

This line is to replace special characters with their html encoding...

line=$(echo $shortline | sed -e 's/\?/\%3F/g' -e 's/&/\%26/g' -e 's/#/\%23/g')

gpull="https://plus.google.com/ripple/details?url=${line}"
ripples=$(wget -qO- "${gpull}" | grep -o "[0-9]*\s*public\s*shares.<" | sed "s/[^0-9]//g"  | tr "\n" "\t" | sed 's/\thttp/\nhttp/g'| sed 's/\t//')
commentpull="https://api.facebook.com/method/fql.query?query=select%20comment_count%20from%20link_stat%20where%20url=%22${line}%22&format=json"
comment_count=`wget -qO- $commentpull | sed -e 's/^.*://g' -e 's/\}//g' -e 's/\(]\)//g'`
echo "comment count: " $comment_count
sharepull="https://api.facebook.com/method/fql.query?query=select%20share_count%20from%20link_stat%20where%20url=%22${line}%22&format=json"
share_count=`wget -qO- $sharepull | sed -e 's/^.*://g' -e 's/\}//g' -e 's/\(]\)//g'`
echo "share count: " $share_count
likepull="https://api.facebook.com/method/fql.query?query=select%20like_count%20from%20link_stat%20where%20url=%22${line}%22&format=json"
like_count=`wget -qO- $likepull | sed -e 's/^.*://g' -e 's/\}//g' -e 's/\(]\)//g'`
echo "like count: " $like_count
twitterpull="http://urls.api.twitter.com/1/urls/count.json?url=${line}&callback=twttr.receiveCount"
twitternumber=$(wget -qO- "${twitterpull}"  | grep -o 'count\":[0-9]*\,' | sed -e 's/count//g' -e 's/,//g' -e 's/://g' -e 's/"//g' )
echo "twitter: " $twitternumber
linkedpull="http://www.linkedin.com/countserv/count/share?format=json&url=${line}" #echo ${linkedpull}
linkednumber=$(wget -qO- "${linkedpull}" | grep -o 'count\":[0-9]*\,' | sed -e 's/count//g' -e 's/,//g' -e 's/://g' -e 's/"//g' )
echo "linkedin count: " $linkednumber
stumblepull=`wget -qO- "http://www.stumbleupon.com/services/1.01/badge.getinfo?url=${line}" | grep -o "publicid.*views" | sed -e "s/publicid//" -e "s/\":\"//" -e "s/\",\"views//"`echo $stumblepullstumblenumber=$(wget -qO- "http://www.stumbleupon.com/content/${stumblepull}" | grep "mark>" | sed -e "s/^.*\">//" -e "s/<.*$//" -e "2d" -e "4d" -e "6d" | tr "\n" "\t")

echo -e "${line}\t${ripples}\t${comment_count}\t${share_count}\t${like_count}\t${twitternumber}\t${linkednumber}\t${stumblenumber}" >> "${1}-all-social-shares".csv
done < $1 

cat -A "${1}-all-social-shares".csv

Scraper for video pages to get all data for video sitemap

This scraper is based mainly on opengraph tags (which are used by Facebook, for example), so it should work well with many pages, not just Dell.com pages. More info on sitemaps at Google .

#! bash

#check if its called with a filename - a file containing urls for pages with videos

if [[ ! $1 ]] ; then
echo "need to call with filename"
exit 1
else

 #now make sure to have a unique filename, based on the file with the urls

filename=$(basename $1)
name=${filename%.*}-$RANDOM-

#header info for the file with the results. Need to pull page url, video title, thumbnail url, description.

echo -e 'url\tpage\tTitle\tThumb\tDescription' > ${name}video-sitemap-data.txt

#loop through the file and store in a variable

while read -r line; do
filecontent=$(wget -qO- "$line")

# echo results and clean up with sed, tr and grep, then append to the file that has the column headers already. It has 4 elements - and each is isolated in its own part. The parts are connected with &&, and everything in ( and ) - otherwise it only echos the last part into the file.

(echo "$line" | sed 's/\r$/\t/' | tr '\n' '\t'  && echo "$filecontent" | grep "og:video" | grep "swf" | sed -e "s/^.*content=\"//" -e "s/\".*$//" | sed 's/\r$/\t/' | tr '\n' '\t' && echo "$filecontent" | grep "og:title" | sed -e "s/^.*content=\"//" -e "s/\".*$//" | sed 's/\r$/\t/' | tr '\n' '\t' && echo "$filecontent" | grep "og:image" | sed -e "s/^.*content=\"//" -e "s/\".*$//" | sed 's/\r$/\t/' | tr '\n' '\t' && echo "$filecontent" | grep "og:description" | sed -e "s/^.*content=\"//" -e "s/\".*$//") >> ${name}video-sitemap-data.txt
done < "$1"
fi

I'd be delighted to know this helped someone else - why don't you drop me a note when you do?

This is one of the pages that I used for testing, just in case someone wants to test this:
http://video.dell.com/details.php?oid=RraTJsZDrsDKFAjNcj2WmTovA1ovugc-&c=us&l=en&s=bsd&p=learn .

Script to check social shares: Facebook comments, shares and likes, tweets, G+ ripples, linkedin and stumbles

Social shares on Facebook, Linkedin, Twitter, Google Plus are relevant - for users, and we assume as well for rankings in search engines. (likely not directly, but indirectly through user behavior). There's a post checking a few platforms - now with more than ever! %-)

This is the version I use most - it covers the biggest relevant platforms (here in the US), and pulls it in the right speed. With my current internet connection, it just pulls slowly enough to not trigger any blocks from the api-s.

I also realized it is easiest to use a random file name, rather than pulling some info in from the site scanned. The url list can come from a sitemap scan or from a sitescan - or any other url list.

This is how the results look like:

I usually sort by each column, to filter out 'outliers', and then use the results.

This is the script:

#!bash
# set all variables to zero at the beginnning! then value >, replace
# check where the tabs come from (likely one tab to many in one of the echos?
rm "${1}-all-social-shares".csv

echo -e "Url\tripples\tFB-comments\tFB-shares\tFB-likes\ttwitter\tstumble_upon\tlinkedin" > "${1}-all-social-shares".csv
while read -r shortline ;

 do
line=$(echo $shortline | sed -e 's/\?/\%3F/g' -e 's/&/\%26/g' -e 's/#/\%23/g')
echo $shortline
echo $line
 number=0
 re='[0-9]+'

gpull="https://plus.google.com/ripple/details?url=${line}"
ripples=$(wget -qO- "${gpull}" | grep -o "[0-9]*\s*public\s*shares.<" | sed "s/[^0-9]//g"  | tr "\n" "\t" | sed 's/\thttp/\nhttp/g'| sed 's/\t//')

commentpull="https://api.facebook.com/method/fql.query?query=select%20comment_count%20from%20link_stat%20where%20url=%22${line}%22&format=json"
comment_count=`wget -qO- $commentpull | sed -e 's/^.*://g' -e 's/\}//g' -e 's/\(]\)//g'`

sharepull="https://api.facebook.com/method/fql.query?query=select%20share_count%20from%20link_stat%20where%20url=%22${line}%22&format=json"
share_count=`wget -qO- $sharepull | sed -e 's/^.*://g' -e 's/\}//g' -e 's/\(]\)//g'`

likepull="https://api.facebook.com/method/fql.query?query=select%20like_count%20from%20link_stat%20where%20url=%22${line}%22&format=json"
like_count=`wget -qO- $likepull | sed -e 's/^.*://g' -e 's/\}//g' -e 's/\(]\)//g'`

twitterpull="http://urls.api.twitter.com/1/urls/count.json?url=${line}&callback=twttr.receiveCount"
twitternumber=$(wget -qO- "${twitterpull}"  | grep -o 'count\":[0-9]*\,' | sed -e 's/count//g' -e 's/,//g' -e 's/://g' -e 's/"//g' )

stumblepull="http://www.stumbleupon.com/services/1.01/badge.getinfo?url=${line}"
stumblenumber=$(wget -qO- "${stumblepull}" | grep -o 'views\":[0-9]*\,' | sed -e 's/views//g' -e 's/,//g' -e 's/://g' -e 's/"//g' )

linkedpull="http://www.linkedin.com/countserv/count/share?format=json&url=${line}" #echo ${linkedpull}
linkednumber=$(wget -qO- "${linkedpull}" | grep -o 'count\":[0-9]*\,' | sed -e 's/count//g' -e 's/,//g' -e 's/://g' -e 's/"//g' )

#echo -e "$line\t$value\t$comment_count\t$share_count\t$like_count" >> "$1-all-public-shares".txt
echo -e "${line}\t${ripples}\t${comment_count}\t${share_count}\t${like_count}\t${twitternumber}\t${stumblenumber}\t${linkednumber}" >> "${1}-all-social-shares".csv


done < $1

Script to get Facebook Comments, FB Shares and facebook likes for a list of urls

Another little helper script - this time tapping into the facebook api to get comments, shares and likes.

The access to this api does not require a login or account, but cuts off after ~ 400 requests. This is much more than I usually need, so no modification of this script to get over the limit. (One could think of a timer with 'sleep 600' after 400 loops or so).

#!bash

first add a header into the output file - dynamic pagename based on input file $1 (first parameter to call with the file), then start the loop for each url. Each loop includes 3 wget calls to the FB api for different values.

echo -e "FB-comments\tFB-shares\tFB-likes\tUrls" > "${1}"-store.csv

 while read -r line; do

#line="${1}"

generate the url

pull="https://api.facebook.com/method/fql.query?query=select%20comment_count%20from%20link_stat%20where%20url=%27${line}%27&format=json"

pull the data with wget, remove unnecessary parts with sed, and store in the variable comment count

comment_count=`wget -qO- $pull | sed -e 's/^.*://g' -e 's/\}//g' -e 's/\(]\)//g'`

 now the same with shares and likes

pull="https://api.facebook.com/method/fql.query?query=select%20share_count%20from%20link_stat%20where%20url=%27${line}%27&format=json"
share_count=`wget -qO- $pull | sed -e 's/^.*://g' -e 's/\}//g' -e 's/\(]\)//g'`
#echo $share_count
pull="https://api.facebook.com/method/fql.query?query=select%20like_count%20from%20link_stat%20where%20url=%27${line}%27&format=json"
like_count=`wget -qO- $pull | sed -e 's/^.*://g' -e 's/\}//g' -e 's/\(]\)//g'`
#echo $like_count

add all three variables into the file where the data is stored

echo -e "${comment_count}\t${share_count}\t${like_count}\t${line}" >> "${1}"-store.csv
done < ${1}

I choose to store the data in variables and then to concatenate to one line with echo, because it makes it really easy to just separate this with tabs - echoing each item separately into the file would have required to remove the line break (\n) each time. 

Does Facebook privacy and copyright disaster call for Microsoft counterstrike?

Privacy and copyright concerns skyrocket after the facebook privacy disaster made visible to everyone what is happening with terms and conditions, those huge long pages hardly anyone reads. The solution could be close – if only Microsoft would enhance Outlook (and IBM Lotus Notes) to a global communication platform where users can own and control all their data.

Need for a communication platform

Users want, perhaps need, to be everywhere: Twitter, Linkedin, facebook, live, Digg, reddit, plaxo, high5, stumble upon, you name it. People want to find and meet friends, colleagues, potential future employers, buddies and want to communicate easily. Contacts, calendar, mail, video, pictures and publishing are the main elements of a tool of choice.

All online platforms offer some of this ‘cross connecting’ already or are heading in that direction. GREAT, this really helps. They login to my email accounts and pull the data. Still ok. There are tools to publish across several platforms. Great. Calendar integration is on the way. So everything is fine, or not?

Privacy concerns and copyright disaster

Problem is, some, if not most platforms want to keep everything, analyse it, perhaps sell it or use it with other companies. Remember that facebook disaster? Use of my pictures, my mails forever? Use of the knowledge of my contacts and network forever? What happens, if Facebook fails or what if Google buys Twitter including all user data?

Proof is easy - just remember how often you can upload you contacts, and how rare it is that you can download them!

Does anyone want a possible employer to know everything about what they have done in private time? Just an easy example: Maybe someone is looking for a pharmacy marketer – and someone has marked something in stumble-upon saying this pill is bad, dangerous and there is a free alternative. Would they still hire that person? Or is it just a matter of time that this will find it’s way into the HR departments?

So what do we need? Requirements for a communication central

A tool needs to offer to:
• Maintain ownership and control over my address book and connections
• Maintain control of my own publications and posts
• To connect all mail accounts
• To connect all social networks, platforms, etc. and search for my contacts
• Connect people with other people, sort, group
• Post same content to different platforms
• Post different content to various platforms
• Get calendars and events for my calendar

Microsoft counterstrike? The basics

Take MS Outlook for a start. Using it for all mail accounts is easy. To integrate, group, separate contacts is good, too; not very handy, but manageable and completely in my ownership. Love that. A good calendar, some item tracking, some security features. Take the small business opportunity manager and there is a simple CRM tool to your use. Nice but solid basics.

Want more? Connect Outlook with ALL online platforms Bi-directional
1. First, please add connectors (Apis, tools, plugins) to connect with ALL online platforms, to check for existing contacts, to import and export contacts. (Don’t wait for others to do that, time is precious!)
2. Then add a way to integrate events from offline to online and vice versa.
3. Now add a tool to send messages to boards and platforms as well (like you had with the tool for publishing on news forums many years ago), and a way to keep track of that.
4. Then, give it a nice surface, to copy or move posts from one to another location per mouse. Drag and drop publishing.

Why not ‘new tweet’ or ‘new entry in facebook’ in the menu? Why not tweets as special kind of inbox message? Why not copy a picture from online platform to online platform via Outlook just per mouse click?
There is much more, but you get the idea.

Communication central with privacy control and content ownership

More then ever people are in need of a communication central, making the ever changing communication manageable. Online solutions may not be the best according to the terms and conditions with which they come often.
Microsoft – and a few others like IBM – have tools already, on which these platforms could easily be built. Please, don’t try to just copy what’s going on online, wake up and use your assets!