Making design data-informed

This project is about how I make data-informed design decision by using A/B testing.

Role

Interface design Experimental design

Year

August - October 2014

Context

PanelPlace is a platform that helps people track and manage paid survey opportunities from market research companies. In July 2014, the client was working on significant feature changes and wanted to redesign the homepage as part of this rebranding. The conversion rate at homepage was low as people were reluctant to register for a new service like PanelPlace. As PanelPlace was growing its user base, therefore, this was the most critical issue on the website. The client wanted to take this opportunity to communicate with users about the rebranding of its product and new features.

Reframed challenge

How can we redesign the homepage to convey trust and potential values of using PanelPlace to the new customers?

Role

I worked together with Esther Fan, our user researcher in this project. I facilitated the ideation workshop, designed the homepage and conducted the experiment to measure the design.


Problem research

Competitive analysis

As new customers like to search for different alternatives before deciding which one to buy/use, we wanted to understand our competitor and therefore set our value proposition right so people can know which is unique about PanelPlace

My colleagues, Esther quickly did an analysis of competitors in the local market and how the website help them achieve their objectives. The client also discussed with us on how different they want to be from other competitors.

User research

We want to understand what people think when they reach the homepage, and why they don’t register for new account. This phase is to understand their needs and goals at the homepage. We asked them the following questions:

  • How did you know about paid survey before joining PanelPlace?
  • How did you earn money in the survey panels recommended by PanelPlace?
  • What did you expect when signing up for PanelPlace?

We interviewed 6 users. The questions were structured to help us gain insights into their goals and needs in the platform. Some people joined the platform just because they had free time and want to spend on earning some money. Others are very systematic about how to manage their earning and want to be efficient when doing surveys. We identified a common theme among these users.

Common theme

People consider answering surveys as a way to earn side money, so they do it only when the process is convenient, flexible and hassle-free.


Ideation

Co-design workshop with stakeholders

The early involvement of stakeholders in wireframing is essential to the faster approval of design later since stockholder will have a better sense of ownership in the design process. This project is the first time we experiment collaboration workshop with stakeholders. We wanted clients and designers to align themselves with each other about what the design will look like.

We used Six Up method, which requires each person to draw 6 ideas. We then compiled and sorted them into catefories.

PanelPlace's Value Proposition

Our customers can focus on earning money as we help them choose and manage the optimal survey panels at their convenience.

Experimental design

As the approval of design was mostly done by stakeholders previously, which explained why some design works were not successful after launch. We wanted to use data to assist decision making in choosing the best design.

As the objective of the homepage is to increase the rate of account registration, the key metrics of the homepage is the conversion rate measured by how many percentage people click the signup buttons in the form located at a few places in homepage. We started with a multivariate test for different changes at the signup form. Among all elements in the homepage, signup redesign has the most impact on changing the conversion rate.

More communication of values and process

Setup

We wanted to know if simplifying the form, communicating more information of registration process would lead to a better conversion.

Screenshots of control and variation

In the original homepage, the value of being secure was not well received by our interviewed users. Moreover, We used heatmap to monitor the homepage and found out that people rarely use the social media signup.

The original homepage communicated its value about security, had unnecessary elements such as the Facebook button.

In the redesigned homepage, we conveyed the convenience of using PanelPlace by explaining the process of signing up. The form was simplified to contain only compulsory fields.

The redesigned homepage communicated its value about the ease of signing up and had a linear format of content leading to the form.

Result

Version Conversion Rate Conversion Improvement
Control 19.9% 164/824
Variation 23.68% 188/794 +19.0% (97% confidence level)

Result Graph

Clearer copy

Setup

After the experiment with main elements of the homepage, we decided to improve the signup button since it was the most prominent element in the signup form. In this experiment, we wanted to have more variations to validate against the control.

Screenshots of sign up button

There were one control and four variations. I ran the test for seven days. Within the first three days, two variations were lagging behind other variation, so I decided to stop traffic going through those variations. In the 5th day, another variation lagged far behind, and it was stopped due to its obvious non-improvement. In the 2nd variation, I saw a relative improvement of 14.8% at 96% confidence level within seven days.

Result

Version Conversion Rate Conversion Improvement
Control 23.43% 228/973
V1 26.90% 244/907 +14.80% (96% confidence level)
V2 22.10% 156/706
V3 20.61% 128/621
V4 16.39% 118/720

Result Graph

Final result

Two weeks after we launched the redesigned homepage, the improvement of signup rate was relatively smaller than the result in the experiments. The reason, we agreed, was that the conversion rate was decided by not just individual elements but as a whole with their complex influential relationship. Regarding the time and traffic constraints we had, the testings helped my team improve the design, and convince client to implement the new design.

All information in this case study is my own and does not necessarily reflect PanelPlace's view.