Are you a Tweak(er)?

How to handle site improvements?

I was just on Seth Godin’s blog (excellent source for motivating, business chatter) and his latest post on tweaking some improvements (whether in software design or home design) was very interesting:

About tweaking he states, “I'm talking about turning an arrogant checkout into a useful one by turning off the button that automatically resets to opt in to the spam list every single time I return to the checkout. Or changing the size of the product photo from 144 pixels wide to 500, because making the product the star can triple clickthrough.”

Seth Godin

Two quick take-aways that I got from the small post…which got me thinking.

First Take-Away: This is iterative, user-centered design which mitigates development risk and quickly garners user acceptance or lack there of.

I agree with the nature of Seth’s post. Tweaking is good in that I firmly believe any long term strategy for development (particularly an E-Commerce site) should be iterative, spiral in nature and continuous. This means that you (and your team) should be continuously gathering user and market feedback, along with industry trends and then plan for small site improvements in strategic areas of your site. Examples can be how to display your product, how your inventory index updates to your WHS, how you categorize or layout portions of your site, how your search function works, etc.

Using a project management voice, “You should always have something in the beginning, middle and end of your pipeline”. Basically, be documenting your conceptual plans (prioritizing per resources and opportunity), be actively working on a system improvement and have something that just released or is about to (and follow it closely for the feedback loop).

This approach is obviously helpful in that you pick up low hanging fruit, get impact-ful results in weeks rather than months and you get true uncluttered feedback. But, this is largely a great approach because the last thing you want to do is what GAP, Banana Republic, Old Navy did last year (See USA TODAY)…..because who else besides GAP inc could afford it? What they did last year is basically take down their entire websites for over a month during the busy back-to-school season to launch “ground-breaking” AJAX heavy functionality and design. They could do this because they have the $$$, they have the steady customer base willing to work through the confusion and they have a plethora of stores. Yet, this illustrates that who can afford huge, change?

The bigger the change, the harder your team will have releasing it “bug-free”. Expect many, many delays.

The bigger the change, how do you isolate what is working and what isn’t?

The bigger the change, who will get it?

The longer you wait to introduce change and new functionality, the harder and more costly it will become to replace your “old” system.

Lastly, the bigger the change, are you being responsive enough and really listening to your customer base?

Second Take-Away: Don’t drink the TWEAKING KOOLADE!

I’ve worked in environments where our team tested EVERYTHING. And, no change was ever made without an exhaustive scientific approach (minus the hypothesis). The way we went about this is really the extreme and not the rule. But beware, it is a slippery slope. Here’s what happened:

Before releasing a new button, new page layout, new text, etc. we would A/B and multi-variant test it. For example we might want to change the shape, color, text and placement of a certain button (e.g. the add to cart button). We weren’t really sure it the button change was warranted, but we just wanted to see if there was a better way (Flag, have a reason to change, don’t just change).

Next, we would say the button would be 5 different colors (red, green, blue, yellow, purple), 5 different shapes (roundish, circle, square…you get the idea). Couple this with 5 different text on the buttons (“add to cart” “add” “buy”, etc) and 5 different placements and you get 5 to the 4^th power. Or, 625 different variations to test. Remember, going into testing, we didn’t have a “reason” to believe change would be a good thing.

We would run “rules” software package that would display each variation to a portion of our web traffic and, depending on traffic exposed to each variation, we would gauge if certain variations produced more sales/conversions.

Usually what would happen is that out of the 625 variations 2-3 would produce slightly better results than the rest of the variations. For example variation 34 and 123 would produce a 2.7% conversion result rather than our normal 2.2% result over a 6 week span. Extrapolated out over an entire year, this could be lots of $$$. Next, we would take these 203 variations and do A/B testing; exposing 50% of our traffic to version A and the other half to version B. This was done for validation purposes of the first testing period.

Finally, with a clear variation (no we didn’t always have a clear “winning” variation) we would roll this out to every one of our website visitors. Unfortunately we weren’t necessarily sure why this variation one and inevitably 3 months later someone would want to test orange, dark blue and gray for the buttons. And, of course, we would….

Now this is all fine and dandy. And, it’s hard to argue with increased conversion results…even when you don’t know why they are happening. Yet, this makes you such a TWEAKER and so myopic…that you loose site of the larger picture…which is WHY DO your customers value your site/product/service and why are they reacting they way they are. The above approach turns your entire team’s attention on so many small details that you loose the PROCESSES.