TKG Web Analytics Blog

Aaron Wall of www.seobook.com posted his findings on an AdWords split test that may baffle some.  

I'm not sure what led Aaron to do this, but he decided to run two identicle ads that supposedly had identical titles, copy, and display url.  He went on to clarify that the ads ran at the same time, used Google search delivery and without content targeting.  He posted a screenshot of the ads with the Google reporting results.  Visually, everything is identicle. 

The outcome, in my opinion, raises the "stink factor". 

Ad 1 was served 50.5% of the time, had 316 impressions, 12 clicks @ a 3.79% CTR.
Ad 2 was served 49.5% of the time, had 310 impressions, 4 clicks @ a 1.29% CTR.

There are four primary threats that affect test validity:

1. History Effects
2. Instrumentation Effects
3. Selection Effects
4. Sampling Distortion Effects

My Findings:

1. History - both ads ran at the same time.
2. Instrumentation - both ads ran at the same time, using the same reporting metrics.
3. Selection - on the surface, it looks like the ads match the profile of the test subjects.  I suspect that selection may be distorted due to broad, phrase, or exact matching for the keyword used.  The use or lack of negative keywords may also have an impact on selection effects.
4. Sampling - based on the information provided, sample size distortion does not appear to be a factor.  A sample size of 626 is sufficient enough to produce a 95% confidence level.

What are your suspicions?  Do you think that over time, extending the sample size will even things out, or compound the distortion?

Jacob Nielsen's recent Alertbox article nicely brings out the importance of using several different graphs when analyzing data.   

Visualizing Web Analytics Data: Drooping Tail and Log-Log Charts

"Summary: Using a linear diagram to plot data from website traffic logs can lead you to overlook important conclusions. Sometimes advanced visualizations are worth the effort."

After seeing another visual perspective on site traffic, Jacob formulates the following question:

"So, what would happen if our sample site could wag its traffic tail up to the straight line representing the traffic potential the theory predicts?" 

He theorizes that they could double traffic and increase page views by adding enough new pages to the site.  He eventually concludes that adding a large volume of new content would not be cost effective. 

I have no problem with his thinking, however, this is not the "only" possible solution, nor should it be the end of his digging.

There are several other cost effective ways of doubling site traffic and page views.  Here are a couple:

• Natural SEO
• Assuring site spiderability
    

More often than not, large sites have several technical problems when it comes to being open to the search engines.  One possible solution could be as simple as correcting a typo on the "robots.txt" file that is preventing proper site indexing.  

This was the exact problem of a Fortune 500 company that we had consulted two years ago. 

Imagine the traffic, page views and revenue that was lost during a time span of about a year.  The actual cost of finding and fixing the problem was pennies compared to the total loss of revenue.  

I'm sure that it was an embarrassing relief for them.

Is your site being properly indexed in the search engines?  Large or small, being fully indexed makes all the difference in the world.

Natural SEO is another often neglected tool which opens up a site to more traffic and page views.  It's also possible to have a site that is technically "wide open" to the engines, and yet "tightly closed" due to a lacking or improper SEO effort.  I've seen nicely optimized sites that are fully open to the search engines suffer in rankings and traffic due to weak off-page SEO efforts.

Are you scratching your head with a similar problem?  If you are, you're not alone.  It happens all the time.

Never fear…there are solutions for those who want an edge over the competition.

Yeah, I know…this wasn't the point of his article. 

PS. This post is not meant to take a cheap shot at Mr. Nielsen. 

I digress…