Subscribe: Comments on: A/B Testing at PBworks
http://blog.pbworks.com/2009/09/16/ab-testing-at-pbworks/feed/
Added By: Feedage Forager Feedage Grade A rated
Language: English
Tags:
chart  collier  kevin  making  new features  new  pbworks  results  site  test  testing pbworks  testing results  testing  time  tom collier  tom 
Rate this Feed
Rate this feedRate this feedRate this feedRate this feedRate this feed
Rate this feed 1 starRate this feed 2 starRate this feed 3 starRate this feed 4 starRate this feed 5 star

Comments (0)

Feed Details and Statistics Feed Statistics
Preview: Comments on: A/B Testing at PBworks

Comments on: A/B Testing at PBworks



The official blog of PBworks



Last Build Date: Thu, 14 Dec 2017 17:55:05 +0000

 



By: gglivvyql

Thu, 29 Sep 2011 16:23:10 +0000

AhOP8B kwiyzqfoqlng



By: Kalea

Wed, 28 Sep 2011 07:27:22 +0000

If not for your wrntiig this topic could be very convoluted and oblique.



By: A/B Testing Links « streamhacker.com

Thu, 28 Jan 2010 03:19:52 +0000

[...] A/B Testing at PBworks [...]



By: kevin

Mon, 21 Sep 2009 22:26:44 +0000

@Tom, sounds reasonable - you've a/b tested your presentation of a/b testing results . . . now if you could just present the glazed-over metrics in a more appealing chart format :)



By: Tom Collier

Mon, 21 Sep 2009 21:59:06 +0000

yohannes sitorus, while we boast a healthy readership of our blog, I can't guarantee that Mr. Weiss saw your comment. Try visiting your workspace (http://.pbworks.com/) and clicking the "Contact the workspace owner" link.



By: Tom Collier

Mon, 21 Sep 2009 21:56:13 +0000

bojanbabic, we like to keep our tests short (1 - 2 weeks) for some of those very reasons. Our mix of traffic can change quite dramatically from one week to the next. However, we run a control site concurrently with the test sites and randomize which site a new visitor will see. This allows us to account for any interesting events that happen during the test. For example, if we had a spike in traffic on day 3 of a test, both the control and the test site would see the same spikes.



By: Tom Collier

Mon, 21 Sep 2009 21:50:45 +0000

Kevin, if I were the only one involved in making decisions, aggregate + error bars would be sufficient. However, members from the whole team (engineers, marketing managers, etc) must ultimately buy into A/B testing so that the results can be incorporated into decision making. Many on the team aren't necessarily well versed in statistical jargon. Initially my talks about the null hypothesis and confidence intervals were met with glazed-over stares. (The language of statistics is far from intuitive.) But, when I presented a sunrise chart, everyone just seemed to get it. This chart tells a story about how the test played out over time. Wrapping a story around numbers makes the analysis more accessible and ultimately allowed us all to accept the results and uncertainty of the test.



By: yohannes sitorus

Sat, 19 Sep 2009 15:39:24 +0000

Mr.weiss i can't log in what is my user and pass



By: bojanbabic

Thu, 17 Sep 2009 07:49:11 +0000

I agree, if timing of A/B tests performed is perfect. Otherwise, A/B test can be misleading. Increase or decrease can be result of previous campaigns, new features or milliseconds gained due to code refactoring. What is if A/Bs are performed for very long time while new features are rolled-out (i.e Google BIG search box has bee tested for year before finally pushed )? Its very hard to distinguish results, very tough to make decision and balance between speed development and making right turns. Cheers



By: Kevin

Thu, 17 Sep 2009 06:06:04 +0000

Why are you showing the data as a time series? Wouldn't aggregate + error bars be more intuitive? Is this just chart porn?