Logfile Analysis for SEO: Get status codes

Ever wondered if bots use cookies? Or if there is a relation from 302's to search engine bot visits? Total sum of 4xx errors as a trend? Our current analytics setup does not show this data, log file analysis, so I wrote a few brief scripts, pulled the zipped logfiles from servers to a local folder, and then analyzed (scripts at the end of this post).

Next step I added the data to an xls and calculate the percentages, like 302s as a share of overall traffic or how many bot visits use a cookie (nearly all bots visits are cookied visits!).

Visualization of one time frame worth of  http status codes

And sure easy  to show trends of status codes, for example share of bots visits to overall visits and the number of 302's on  a site.

And here are the scripts. If you wonder why writing the results into a file, and not just count, I run more analysis on these resulting files.

1. Get all lines with a certain status code not 200 OK:

zcat *.zip | grep " 301 " > all-301.txt
zcat *.zip | grep " 302 " > all-302.txt
zcat *.zip | grep " 304 " > all-304.txt
zcat *.zip | grep " 403 " > all-403.txt
zcat *.zip | grep " 404 " > all-404.txt
zcat *.zip | grep " 410 " > all-410.txt
zcat *.zip | grep " 500 " > all-500.txt
zcat *.zip | grep "^2014" | wc -l > logfile-results.txt

2. Error codes, redirects encountered by bots. First, filter out all lines with bots, then get the status codes lines in separate files.

zcat *.zip | grep "bot" > bots-traffic.txt
grep " 301 " bots-traffic.txt > bots-301.txt
grep " 302 " bots-traffic.txt > bots-302.txt
grep " 304 " bots-traffic.txt > bots-304.txt
grep " 403 " bots-traffic.txt > bots-403.txt
grep " 404 " bots-traffic.txt > bots-404.txt
grep " 410 " bots-traffic.txt > bots-410.txt
grep " 500 " bots-traffic.txt > bots-500.txt

3.  Pull out all the visits with cookies, in this case only the cookies themselves. In these logfiles they are in field $15:

zcat *.zip | awk ' BEGIN { FS = " " } ; NR > 5 { print $15} ' > all-cookies.txt
awk ' BEGIN { FS = " " } ; NR > 5 { print $15} ' bots-traffic.txt > bots-cookies.txt
awk ' BEGIN { FS = " " } ; NR > 5 { print $15} ' all-500.txt > all-500-cookies.txt
awk ' BEGIN { FS = " " } ; NR > 5 { print $15} ' bots-500.txt > bots-500-cookies.txt

4. This pulls all urls with an 500 error code into files - to have Dev look at these:

awk 'BEGIN {FS = " "; OFS = ""  } ($8 == "-") {print "www.dell.com"$7 } ($8 != "-" ){ print "www.dell.com"$7,$8} ' all-500.txt > all-500-urls.txt
awk 'BEGIN {FS = " "; OFS = ""  } ($8 == "-") {print "www.dell.com"$7 } ($8 != "-" ){ print "www.dell.com"$7,$8} ' bots-500.txt > bots-500-urls.txt

It's filtered, so if field 8 is just a hyphen, it prints just the url stem, otherwise stem and pagename. 

awk: count items in n number of lines (including percentile calculations)

Using several scripts to check a number of sites on features, I needed a way to calculate the sum of occurrences of yes / no and such alike - but not just across the whole file, but for each x number of lines, each 100 lines, 1000 lines or so.

This is based on a much simpler version to calculate just the totals for a whole file, not n-numbers. The script is called with input file as first parameter and number of lines to summarize as parameter two: . runscript.sh filewithdata.txt 100. Filewithdata.txt is stored in $1, 100 in $2. Then the $2 is handed to awk with the -v counter=$2.
First step is to set variables, then count each field up with the 'if' a value occurs.
Third step happens when the number of lines is reached (NR-1) to account for the header. Values are printed, added to the totalvalue variables, reset to zero.
In the END section there are two elements - first to print the totals, then also to show if there are 'lines left' due to an the aggregation over x lines. If there are, the number of lines and counting values are printed.

awk -v counter="${2}" 'BEGIN {counturl = 0; counttitle = 0; countpub = 0; countschema = 0 ; printcounter = 0 }

{counturl++ } ( $2 == "yes" ) { counttitle++ } ( $3 == "yes" ) { countpub++ } ( $4 == "yes" ) {countschema++ }

(NR-1)%counter==0 { print "number of lines: "NR-1, "og:title: "counttitle, "rel publisher: "countpub, "schema: "countschema;
titletotal+=counttitle ; totalpub+=countpub ; totalschema+=countschema;
counturl=0; counttitle = 0; countpub = 0; countschema = 0 ; printcounter++ }

END { print "\nnumber of calculations: " printcounter; print "urls: " NR-1, "all og:title: " titletotal, "rel pub total: " totalpub, "all schema: " totalschema;
if(counturl) print "lines left: " counturl; if(counttitle) print "left og:title : " counttitle ; if(counttitle) print "left rel pub: " counttitle; if(countpub) print "left schema: " countschema ; print "\n" }' $1  

Apart from the usual man pages, this post on stackoverflow was very helpful especially for the 'left' calculation in the END section.

AWK - simple counter of yes / no in tables

Counting yes or no in a table (or replace with what you need to count)

For a project I write data into a tab delimited table, urls and properties of the pages under the urls. Using the script over and over, I wanted to make the counting at the end a bit more efficient, and added below lines to the script generating the table.

This is how the call and result look like:

And this is how it is set up:
In the BEGIN section the counters are set, set to zero. Then in the body it counts three parameters up, if the according field contains 'yes'. In the END section it prints the number of lines minus one for the header column, then the name and value of each counter. And last, as standalone script, $1 is the file scanned - which, if added to the tab generating script is just replaced with the filename into which the table is written.

awk 'BEGIN {counttitle = 0; countpub = 0; countschema = 0 } ( $2 == "yes" ) { counttitle++ } ( $3 == "yes" ) { countpub++ } ( $4 == "yes" ) {countschema++ } END {print "number of lines: "NR-1, "og:title: "counttitle, "rel publisher: "countpub, "schema: "countschema} ' $1

Should be easy to adjust for re-use!

Finally: free to use Sitemap Generator with automated 'priority' configuration

Easy to use - at least in an linux environment. Not sure if adjustments are necessary for various (bash) flavors, same with -nix emulators like cygwin. Sitemaps can have up to 50,000 links - so this should be sufficient for many sites. 

First, check if called with a file - ideally the list of urls from an earlier scan with one of the scanners of this site, or from somewhere else, perhaps a content management system. Then, grab the file name without ending and add a random number to avoid overwriting of existing files.

if [ -z $1 ]
then echo "needs a file to start with"
else
file=$(basename "${1}")
name=${file%.*}-$RANDOM.txt
fi

 Clean up the file from empty lines, then from lines that start with underscore or hyphen - errors I found a few times.

sed -i '/^$/d' "${1}"
sed -i '/^[_\|-]/d' "${1}"

Echo the header into the new file - xml definition for a sitemap.

echo ""  >  "${name}"

Then, check if the url has a http:// in front, if not, add it, otherwise the following calculations get tricky.
Next - count the number of "/ ", subtract the two dashes from the http://. The 'gsub' returns the number of replacements - and that's all I need here, so I replace a dash with a dash.

The next step calculates the priority based on the depth in the folder structure - the lower in hierarchy, the lower in priority. This is a setting that might need to be changed, depending on the structure of a site.

awk ' $1 !~ /^http/ { $1 = "http://"$1 }  { count=(gsub("/" , "/" , $1)-2) } count > 9 { print $1, "0.1" } ( count > 0 && count <=9 ) { print $1, (10-count)/10 } count <= 0 { print $1, "1" }' $1 | while read input1 input2

The little sed inset is then used to remove the additional decimals - this was the easiest way to do this.
Then just echo it into the overall structure of the sitemap entries, then end the while loop with done.

do priorityvalue=$(echo $input2 | sed 's/^\(...\).*/\1/')
echo -e "$\n$input1\n$priorityvalue\n" >> "${name}"

done 

Finally, add the sitemap footer per definition and show the result

echo "

" >> "${name}"cat $name

looks like this:










http://www.tage.de  1
http://www.directorinternet.com/  0.9
http://andreas-wpv.blogspot.com/2014/05/new-google-schema-implementation-for.html  0.7

Same script on Dropbox for easy use.

New Google Schema implementation for Phone Numbers - little helper for lots of phone numbers

Google is showing phone numbers now for company searches

Nope, not happy about the fact that Google is showing phone numbers for company searches - but not for phone searches. How much sense does that make?

I prefer online, usually, and call only as a last resort. Calls take much longer, sometimes it is hard to hear or understand what the other's say. And sometimes I find it incredibly aggravating to be on a call for hours to fix something that the company I am calling "broke" in the first place - or did not prevent. (Although I know there are many companies involved, in producing one product). And I also know, most companies (including the one I work for)  really try hard to give good online and phone support.

And now Google is showing support phone numbers for companies - no matter if it is an online only company.  And they show them as part of the knowledge graph - but NOT when someone searches for a phone number. How smart is that.

And they are currently  not showing for Chrome books or for google - how fair is that?

The schema documentation works just fine, watch for this:

• the visible number on the page needs to be the same as the one in the json / schema part
• for multiple contact points for the same organization, that part of the code can be iterated, comma separated
• phone numbers for several countries can be integrated on one page. 

Script for many phone numbers:

Ok, so I had to update ~ 80 country phone numbers, and got just one not very good list of phone numbers, including international calling code. First I started concatenating in my beloved MSFT Excel, but that's no fun, especially since quotes need to be escaped (doubled), so I ended up with a lot of double quotes - which I then had to remove in a later step. But this needed to be done quickly, once I had the final list, which made me do this little script.

This script loops through the list of phone numbers with awk, writes the two parameters into variables, and fills the info in the schema snippet. As a result, the script generates one schema.org snippet for each country and phone number.

Three files needed, the script (shown at the end) a list of phone numbers and the schema for this kind of phone numbers.

This is the schema info we were going to use, where I used two placeholders - changeNumber and changeArea:

The list of phone numbers looks like this, and is tab delimited:

And now the script:

awk '{print $1, $2}' tab-phone-numbers | while read input2 input1

do

awk -v var1="${input1}" '{ sub("changearea",var1)} {print $0}' phone-code-snippet |  awk -v var2="${input2}" '{sub("changenumber",var2);print$0}'>> phone-results

done 

What really helped me do this was the trick to use read to get the output into variables (bold). Found this first here (thanks duckeggs01) and confirmed on my trusted source for man pages online - ss64 . I am sure I'll use this more, very quick and easy, efficient.
First part is a loop with awk through the numbers assigning each field value to a variable. Then nested two awk commands - assigning the external variable to an internal. Then, replacing the first variable the first placeholder, piped into a second awk with the second variable replacing the second placeholder, ... done.

In this first run this script produced one complete snippet per country - good if you need the snippets on several pages. For usage on one page, the template for the schema needs to be changed to only include the contact point info, then run, and add header / footer afterwards to the whole section.