A / B Testing – Dan Siroker [Book Summary]

by Nick

We know that the internet has changed the way we do business. Almost every company today has its own website, but how effective is it used?

The book “A/B Testing” describes how, through simple experimentation and evaluation, to improve the work of the company’s website in order to attract more customers and increase profits.

Dan Siroker was the director of analytics for the 2008 Obama presidential campaign. He is now a co-owner of Optimizely, a company that helps businesses improve their online presence strategies; Pete Kumen, a former employee of Google, and currently a co-owner of Optimizely.

This book is available as:

eBook | Print

Learn How to Optimize Your Online Presence Strategy.

What do you do first when you create a new company? Decide on a name, possibly a logo, and create a website. Most potential buyers will find out about your company by going to the site; many of the site visitors will purchase something in your online store. Thus, the site is very important.

Therefore, you need to organize it in the best way, using the most suitable format. How to do this? Through A / B testing. The book describes how using this method can significantly increase the number of visitors to your site.

From the book you will learn:

  • how the A/B testing method helped Obama win the election;
  • why, when it comes to sites, the saying is true: “less is better”;
  • why even an unsuccessful experiment is also a good result.

If You Want to Optimize Your Site and Attract Visitors, Try A/B Testing

Thanks to the Internet, companies have an unprecedented opportunity to find new customers. All you need is a website, and then anyone can see your products and make a purchase. Despite the apparent ease, creating a website that will help people make purchases is not an easy task. Fortunately, there is A / B testing – a reliable way to find out what is needed for this.

The principle of A/B testing is simple.

By demonstrating various versions of your site to randomly selected test groups over a short period of time, you accumulate information about which options are most effective. This information provides organizations with tremendous opportunities in realizing the task of increasing website traffic.

For example, a site created for President Obama’s presidential campaign in 2008 allowed visitors to leave their email addresses in order to receive a newsletter or simply support a candidate. The main page of the site was a photograph of Obama surrounded by many of his supporters holding the flags of “Obama 2008”.

Obama’s team conducted A / B testing to select the most successful site presentation. It turned out that visitors were 40.6% more likely to leave their email addresses when there was a photo of the future president with his family on the site next to the “Learn more” button. For the Obama campaign, this meant an increase of 2.8 million subscribers and 57 million additional donations.

As you can see, A / B testing offers tremendous opportunities. And it is practically free, unlike other testing technologies, which are usually complex and expensive. In recent years, A / B testing has often been used to improve web pages. For this, web-based optimization platforms, such as Optimizely, are being developed that simplify the use of A / B tests.

In order to start A / B testing, you need to choose clear hypotheses and quantitative indicators of success

Are you ready to start optimizing your site? Fine! But first, you need to clearly define what exactly you will test. Without highlighting indicators whose values ​​you will check and clear hypotheses, the testing process will not produce the desired results. Therefore, start by determining quantitative indicators of success, the most important factors, with a view to their subsequent evaluation.

For example, you are the owner of an online store. Information about the simple number of views on your site will not be informative, because it does not reflect how visitors view its content. But the information about the number of people who shared the link with friends, comments and the number of repeated visits will give more useful information regarding what content on the site attracts visitors.

If you are engaged in e-commerce, the indicators of success can be the volume of purchases made, the number and cost of goods added to the basket, and the number of views of goods. Success rates may vary depending on your site. But, as we noted earlier, quantitative indicators of success are not the only thing that needs to be taken care of. You also need to determine a clear hypothesis.

To better explain this principle, consider an example. Clinton Bush Haiti Fund was created in 2010 during the earthquake in Haiti. To optimize the page with textual information about sponsors, the organization signed a contract with Optimizely.

It has been suggested that adding a picture over textual information about sponsors will increase donations, but the opposite has happened.

Therefore, a new hypothesis was put forward that the pictures take up a lot of space and lead to additional loss of time (to scroll the mouse wheel and get to the desired field) of potential sponsors. To test this hypothesis, the picture was placed behind the input field, and achieved great success! This optimization provided more than a million dollars in additional revenue in the form of donations. Imagine if the creators of the site just tried to do something at random, without testing they would never have found the right solution.

The Summary you might like: Zen and the Art of Motorcycle Maintenance

A/B Testing is a Great Way to Check the Effectiveness of Changes for Your Site.

A / B testing is great not only for setting up and finalizing the strategy of your presence on the Internet but it is also useful for structuring your site.

This is exactly what Disney did at one time. After conducting A / B testing on one of their television channels, ABC Family, they analyzed the search queries of the audience and found out that many of them were looking for specific shows. In this regard, the company did not dwell on the elimination of minor flaws but completely changed the page structure so that it was easier for users to find the show they were interested in.

Disney’s goal was to increase the number of views of the experimental page by 10-20%, which they easily achieved. As a result of A / B testing, the number of views increased by 600%.

Chrome apparel company also used A / B testing to make significant changes to its website’s design. There were three ad units on the company’s homepage. The developers of the site suggested that the left block is the most effective since visitors see it first. But it turned out that the most viewed is always the central unit. Based on the information received, the site design has been completely changed.

It is Important to Note That Conducting a/b Testing is an Important Element During Site Changes, and Not After the End of This Process.

Netflix used A/B testing in 2011 to modernize its user interface.

The initial interface contained only four names, each with asterisks for rating and a play button at the bottom. But another option was tried with many thumbnails that users could scroll through. This change was extremely effective: not only the time spent on the site but also the number of visitors increased.

Remove unnecessary elements from the design of your site to increase its traffic

Imagine that a lot of images, pop-ups, and requests with suggestions to register somewhere hit you at the entrance to the site. Without a doubt, you will try to leave such a site as quickly as possible and not visit it again.

Most of us intuitively understand this. Regarding the design of the site, you always need to adhere to the principle of “better, less is better.” A / B testing shows that the removal of insignificant fields has a huge positive effect on the level of attendance.

For example, Optimizely specialists, optimizing the site for the Clinton Bush Haiti Fund, noticed that the donation page had fields for entering the phone and name. Since such information is not very important for charitable organizations, and A/B test was conducted for a page variant without these fields. The result was impressive: this small change led to an increase in donations of 11% each time you view the page.

What can be done to try to reduce clutter on the site based on this information? You can use hidden functions, as well as split long forms that are placed on several pages. Using hidden functions (which appear only when they are accessed) for a promotional code and other purchase options, information about discounts increases the income from each visitor by 15.6 percent. Of course, there are times when you cannot hide or delete fields. In these cases, it is advisable to split the long form into several pages.

This approach has been shown to be effective in 2012 during the Obama campaign. Everything superfluous was removed from the donations page, and each remaining element was important. Obama’s team decided to take the form even shorter, for which it was divided into two separate pages – one related to the number of donations, the other had personal data. Together with other similar changes, this provided an additional $ 190 million in donations.

Using the Right Literary Language Attracts People and Improves Your Site’s Performance.

As the saying goes, it’s better to see once than hear a hundred times. Nevertheless, as A / B testing testifies, if you want to attract people, you must choose the right words.

And if the virtual visitor from what you read on the page will understand what you are doing, then your site will be more effective.

In the example of the Clinton Bush Haiti Fund, the A / B test showed that instead of the standard “Send” button, the text “Help Haiti” will be more effective on the donations page, which clearly reveals the contents of the operation. And the experts were not mistaken – it worked! The test page brought additional revenue from the sponsors for each view.

Another tip to make the most of your language, especially if you want to encourage visitors to take action, is to use verbs instead of nouns whenever possible.

This principle was also discovered with the help of A / B testing, the results of which have repeatedly shown that if you want someone to do something, you need to say this, and it is advisable to use the verb in a stimulating form.

For example, LiveChat, a software company that allows businesses to communicate with customers in real time, conducted an experiment to attract customers. In order to strengthen the influence of the company, designer Lucy Frank wanted to test whether the use of the verb would affect customer behavior.

Using the A / B test, she compared the effectiveness of the slogans “Free trial” and “Try it for free.” The second option was 14.6% more effective.

When It Comes to a / B Testing, a Bad Result is Also a Good Result.

Scientists often say that there is nothing more entertaining than dealing with something that does not work because you can learn a lot. This is precisely what happens when the hypothesis tested with the A/B test “doesn’t work” turns out to be false.

This principle holds true for any kind of failure, including minor ones. Because the accumulated information and experience will be useful in the future.

For example, a Chrome e-commerce company that has always used images to advertise bicycles has tried to increase its sales by replacing static images with video images. After a three-month

the trial period, A / B testing showed that using video images instead of images does not have any significant effect on sales.

You can say that this test did not lead to anything, although in fact, it was very revealing. Based on it, Chrome will not continue to use video images to promote products, as it now knows that this does not provide the desired effect.

Here is another way in which failure in an A / B test provides users with useful information.

This happened with the IGN gaming site. To encourage visitors to view ads on the site (which generated significant revenue), IGN tried moving the video link to the far left of the screen. This had disastrous consequences. Advertising views fell 92.3 percent.

But there is no silver lining! The company quickly realized what had happened: most of the regular visitors to the site could not find the link after it was moved. Therefore, although this experiment did not bring the company immediate gains, it taught her a valuable lesson.

Encourage Colleagues to Use Data-based Business Approaches

Just as political movements need activists and lobbies to bring about change, the A / B testing method needs people to “promote” it. Therefore, if you want to use this method in your organization, then you may need to talk about it.

One of the best ways to convince people of the effectiveness of the A / B testing method is to disclose the results that have been achieved with its help.

A simple, regular presentation of the test results will be enough to convince shareholders that the data-based testing method works with considerable profit.

For example, Scott Zakrajsek, an Adidas manager, and analyst, often uses safe “case studies,” that is, A / B tests that are likely to produce positive results, to help colleagues understand the testing method.

Such practice on a regular basis can lead to the fact that A / B testing will become part of the corporate culture of the company.

And here is the experience of Lizzy Allen (Lizzy Allen). When she first came to work as an analyst at IGN in 2010, she was surprised that no one had ever heard of A / B testing there. She devoted a month to talk about the concept in the company. To make sure that her colleagues were interested in this method, she held competitions. Employees were asked to predict usage performance.

testing in the company. Although the assumptions of most colleagues turned out to be incorrect, everyone got used to the idea of ​​user testing in the company. People realized that they can learn a lot from this approach, and as a result, A / B testing has become an important part of company policy.

The Final Words

If you want to optimize your site and attract visitors, try the A/B testing method.

In order to use this approach based on data analysis, you need to formulate a clear hypothesis and select estimated success rates.

This will allow you to attract users, employees, and increase the efficiency of the site.

Use the platform for testing.

If you do not have your own A/B testing technology, using a test platform, such as Optimizely, will allow you to conduct tests simply and easily, without having to hire a whole department of analysts.

Why You Should Read “A / B Testing”

  • To understand how to optimize your website and drive traffic with sales
  • To build a highly-successful blog
  • To apply scientifically based and proven strategies in business.

This book is available as:

eBook | Print