Thursday, February 27, 2014

Script to get Facebook Comments, FB Shares and facebook likes for a list of urls

Another little helper script - this time tapping into the facebook api to get comments, shares and likes.
The access to this api does not require a login or account, but cuts off after ~ 400 requests. This is much more than I usually need, so no modification of this script to get over the limit. (One could think of a timer with 'sleep 600' after 400 loops or so).

#!bash
first add a header into the output file - dynamic pagename based on input file $1 (first parameter to call with the file), then start the loop for each url. Each loop includes 3 wget calls to the FB api for different values.
echo -e "FB-comments\tFB-shares\tFB-likes\tUrls" > "${1}"-store.csv
 while read -r line; do
#line="${1}"
generate the url
pull="https://api.facebook.com/method/fql.query?query=select%20comment_count%20from%20link_stat%20where%20url=%27${line}%27&format=json"
pull the data with wget, remove unnecessary parts with sed, and store in the variable comment count
comment_count=`wget -qO- $pull | sed -e 's/^.*://g' -e 's/\}//g' -e 's/\(]\)//g'`
 now the same with shares and likes
pull="https://api.facebook.com/method/fql.query?query=select%20share_count%20from%20link_stat%20where%20url=%27${line}%27&format=json"
share_count=`wget -qO- $pull | sed -e 's/^.*://g' -e 's/\}//g' -e 's/\(]\)//g'`
#echo $share_count
pull="https://api.facebook.com/method/fql.query?query=select%20like_count%20from%20link_stat%20where%20url=%27${line}%27&format=json"
like_count=`wget -qO- $pull | sed -e 's/^.*://g' -e 's/\}//g' -e 's/\(]\)//g'`
#echo $like_count
add all three variables into the file where the data is stored
echo -e "${comment_count}\t${share_count}\t${like_count}\t${line}" >> "${1}"-store.csv
done < ${1}
I choose to store the data in variables and then to concatenate to one line with echo, because it makes it really easy to just separate this with tabs - echoing each item separately into the file would have required to remove the line break (\n) each time. 

Tuesday, February 11, 2014

Internet speed check - Austin / Round Rock - scripts and sad results

My internet provider has been trying to get me upgrade - I use a small and inexpensive connection. Why upgrade, if they cannot even provide the speed they promised?

Sure, they offer an upload / download speed test, but from the company itself, how reliable is that? The test had various results, and it is only a spot check, so I decided to make a small script, and run it every 10 minutes my laptop is up and running. That script writes the traceroute into a logfile, which I then cleaned up and sorted.

A pretty simple script to get the date, a brief description (the 'echo') and the output of tracepath into a file to accumulate.

echo -e "\n\n"
date >> speedcheck.log
echo -e "tracepath www.google.com\n" >> speedcheck.log && tracepath www.google.com >> speedcheck.log
echo -e "tracepath www.google.de\n" >> speedcheck.log && tracepath www.google.de >> speedcheck.log

To run this script over and over on schedule, I added a line in crontab to run it every 10 min. (check here for the crontab syntax). You can edit the file  with >> crontab -e:

*/10 * * * * bash ~/Documents/[folder]/speedcheck.sh

And the result is a pretty long traceroute log from all times the computer was running - every 10 min. (and no, no loss of internet performance through this that I could feel - ping and traceroutes are usually very small transfers, too.)


Traceroute shows time for each step to the target destination. Data filtered by steps over 100ms (which is abysmally slow):



And now sorted by size, to show the worst offenders. I filtered all steps / processes out already that completely broke, so this is a tame list...



top (=worst) 10 entries TWC:
 5:  agg22.dllatxl301r.texas.rr.com                      17680.452ms
 5:  agg22.dllatxl301r.texas.rr.com                      17680.452ms
 5:  agg22.dllatxl301r.texas.rr.com                      17680.452ms
 5:  agg22.dllatxl301r.texas.rr.com                      17641.492ms
 5:  agg22.dllatxl301r.texas.rr.com                      17641.492ms
 5:  agg22.dllatxl301r.texas.rr.com                      17641.492ms
 6:  ae-4-0.cr0.dfw10.tbone.rr.com                       17473.460ms
 6:  ae-4-0.cr0.dfw10.tbone.rr.com                       17473.460ms
 6:  ae-4-0.cr0.dfw10.tbone.rr.com                       17473.460ms
 7:  ae-3-0.cr0.lax30.tbone.rr.com                       17264.655ms asymm  9


top (=worst) 10 entries overall:
 3:  75.8.128.106                                        32080.074ms
 3:  75.8.128.106                                        32080.074ms
 5:  gar23.dlstx.ip.att.net                              31161.992ms asymm  7
 5:  gar23.dlstx.ip.att.net                              31161.992ms asymm  7
 5:  gar23.dlstx.ip.att.net                              31161.992ms asymm  7
 5:  gar23.dlstx.ip.att.net                              30287.261ms asymm  7
 5:  gar23.dlstx.ip.att.net                              30287.261ms asymm  7
 5:  gar23.dlstx.ip.att.net                              29391.217ms asymm  7
 5:  gar23.dlstx.ip.att.net                              29391.217ms asymm  7
 5:  gar23.dlstx.ip.att.net                              29391.217ms asymm  7

It looks pretty bad (without data to be able to compare, admittedly), but it seems not to be close to my uplink, but somewhere in the network, perhaps a data center. Which also means, I am likley not the only one being affected by stopping videos, bad voice connections.

So, while the uplink seems fine on the first steps, it very often falls apart on the next steps on the route to destination.
Bookmark and Share