Testing methodology for CRO

Now that we already know what CRO is, how users behave, the techniques and tools we can apply, it is time to bring everything together and create a framework for working.

This is the final and most important point in the article, because it brings together everything we have been explaining so far. So pay attention.

Because this framework explains the process we can follow if we want to improve our project as a whole, instead of focusing on isolated aspects, by treating it as a methodology that forms part of the core of the business.

The one I suggest is this:

1.   Gather information about the site, the objectives being pursued and the types of users (personas, sociodemographics, etc): this will give us the necessary context and the first clues.
2. Study the metrics: normally the site will have a digital analytics tool. If it does not, it must be implemented as soon as possible and the Measurement Plan methodology covered in the first chapter should be followed.
3. Propose changes and launch the tests: with all the information analysed, we will propose changes that need to be tested with A/B tests (usually) or user tests (less frequently), depending on what is appropriate.
4. Analyse the results: see whether the proposed improvements actually improve things or not and return to step 1 again. It is an iterative methodology.

I will explain each step, combining the first two.

#1 and #2. Gathering information and analysing metrics

The objectives in these phases will be:

1. Understand the site we want to optimise.
2. Define the objectives.
3. Select the most suitable pages for testing based on the objectives set.
4. Analyse the problems on the page and how users use the site.
5. Formulate the hypotheses that will serve as the basis for the proposed changes.

A) Understand the site we want to optimise:

For this, we can create a project sheet similar to this one:

Project:  Yo pongo el hielo.

Site type: Ecommerce.

Main objectives:

- Product purchase.
- Newsletter subscription.

Relevant pages:

- Home.

- Product listing.

- Product page.

- Purchase process.

First impressions:

- The responsive carousel may be problematic (non-standard).

- Small banner copy.

- Checkout could be improved: different and repeated “1” steps

- No personalisation.

- In categories, products are shown in list mode rather than grid.

- Product page: many have the same generic text.

B) Define the objective

It will be based on:

• What the metrics say.
• Trend.
• Market studies.
• Competitor benchmarking.
• SWOT.
• Customer feedback.
• Any source of information that may seem relevant to us.

Ideally, we will end up with an objective that defines the KPI to increase, the time needed for it to materialise and the user segment, if necessary.

For example: Increase the monthly CR of new users by 1% within 6 months.

C) Select the most suitable pages for testing

Based on the objectives set:

• We will study the metrics available to us: Digital Analytics tools, heatmaps, surveys…
• We keep pulling on the thread of anything that catches our attention and look for trends (seasonality, YoY, channels and segments…).
• We can add techniques such as conversion funnels, recordings, heuristics, questionnaires or heatmaps.
• They should be pages with traffic that lead to conversions. If there is not enough traffic or conversions (let us consider 100 the minimum), the results will not be conclusive. This brings us to the possibility of investing in traffic acquisition.
• It is easier to start with: Pages with problems (high bounce and exit rates, low CR…)

• The closer we are to the conversion page, the sooner we will see results (Checkout)

We will end up with a selection of pages such as:

• Home page.
• Catalogue page.
• Product information pages.
• Steps in the purchase process.

D) Analyse the problems on the page and how users use the site

We will review the following:

Functional errors:

• Slow loading.
• 404.
• A part is missing.
• Errors that prevent progress (button that does not work).

Accessibility

• Sizes.
• Colours.
• Devices.
• Readers for blind people and people with disabilities.

Usability

• Too little information.
• Unclear value proposition.
• CTA located below the fold or not very visible.
• Copy and text that could be improved.
• Long forms.
• Requesting too much data.
• …

In short, all the points discussed in the heuristic analysis technique.

To collect user feedback we have these sources:

• Customer service.
• Forms.
• Feedback about the site (contact email, social media).
• User tests / interviews.
• Heatmaps.
• Recordings.

Once we have both the site's problems and user feedback, we will be in a position to establish the KPI to improve on the page.

For example, on a product page it will be the number of “Add to cart” actions, or on a product category listing page, the bounce or abandonment rate.

E) Formulate the hypotheses

These will serve as the basis for the proposed changes.

In this final point of this phase, we are going to use the scientific method: we identify a problem and suggest a possible solution (hypothesis). An experiment is designed to validate or reject our hypothesis

We already have the KPI selected in the previous step, so it is time to get creative and think about what changes will improve that KPI, formulating the hypothesis.

Example hypothesis:

“On the category page, by showing more products at a glance by switching from a “list” format to a “grid” format, users will be less likely to leave the site from that page.”

Important: hypotheses must always contain both the assumption we want to test and the result (otherwise it will be a “null hypothesis”).

#3. Test design and launch

Once we have formulated our hypothesis, it is time to test it.

For this, we will use a testing tool such as VWO, which has a free plan and is powerful.

You should know that there are several types of A/B tests:

• A/B (also called Split): a control page and one or more test pages (variants) where a single change is evaluated. For example, different colours for the “Pay now” button.
• Multivariate: a control page and several alternative pages, as many as there are possible combinations of different variables. Continuing with the previous case, in addition to button colours, we could test different copy such as “Pay now”, “Finish and pay”, “Go to the bank and pay”. Every colour and copy combination would be tested.
• Redirect: takes you to a completely different URL (valid for different checkout flows, in one step or several, for example).

Elements to test

The truth is that on a medium-sized website there are countless parts that can be tested. Here are a few:

CTA

• Copy.
• Shape.
• Colour.
• Size.
• Icon.
• Position.

Unique Value Proposition / Unique Campaign Proposition

• Copy.
• Main images.
• Distracting elements vs elements that help conversion.

Checkout

• Information: sufficient, appropriate, persuasive…
• Button copy => “buy” vs “add to basket”.
• Number and order of fields: fewer is not always better; it depends on lead qualification.
• Number of steps (URLs).
• Input and label format (does larger improve or worsen CR on mobile?).
• Add or remove images or videos (do they distract or reinforce?).
• Add upselling at the end.

User stage (knows what they want, is looking for information…)

• Information for each type of user: Usage guides / tutorials. “About us” and “terms” pages. Testimonials and social proof.

• Páginas “sobre nosotros” y “condiciones”.

• Testimonios y pruebas sociales.

• Add the number of references in each category.

Copy: do new users understand it, or do you need to be a professional looking for something specific? Headers. Different categorisations (by brand, categories, type of use…).

• Categorizaciones distintas (por marca, categorías, tipo de uso…).

• Category descriptions.

Personalisation: do new users see something different from returning and logged-in users? Recommendations.

• Home page design (different modules).

Usability

• Font size.
• Colours: Background of divs or modules. Icons.

• De iconos.

• Links.

Progress bars in the funnel.

Tooltips or information in each field of the funnel.

Look and Feel

• Amount of white space on the page.
• Fonts.
• Colour palette.
• Background colour / image.
• Rounded or curved corners on buttons and images.
• Quality and type of images (heavier, stock vs own).
• For external checkouts, test making them resemble the site's look and feel or embedding them as iframes.
• Number of products per page, in ecommerce.
• Number of text columns, in media sites.

“Findability” (ability to reach what you are looking for on the site)

• Show the most-read articles or best-selling products on the home page.
• Show products in searches with no results.
• Amount of information to show on the search results page (images, price, stock, short description?).
• Menu naming.
• Number of menu items and their order.
• Search bar: size, fixed, placement, copy…
• Search refinement methods: add “perhaps you mean XXXX”

Visual hierarchy

• Move important elements from one side of the landing page to the other.
• Sizes: absolute and relative.
• Number of elements.

Purchase

• Units in stock => sense of urgency.
• Order delivery date.
• Shipping price on the product page.
• Checkout with Paypal Express.
• Payment methods (segmented by country).
• Internal or external payment gateway (with or without iFrame).

Error prevention

• Error message: make it clear that it is the site's fault, not the user's.
• Highlight the field with the error.
• Content and copy on 404 pages.
• Remove optional fields from forms.
• Copy for the labels.

In short, as you can see, the list is long. And I have left quite a few things out…

#4. Results analysis

It is time to see whether our hypothesis was correct or not.

• If it was correct: we will need to think about the next iteration, on the same page or another one.
• If it did not produce a positive result: we will need to reformulate the hypothesis in order to improve the KPI.

In other words, the testing process never ends, whether the result is positive or negative.

It is important to bear in mind that, even if the test produced a negative result we have learned more about our site and its users, so it is never wasted effort.

Real example

To finish this section and this extremely long post, here is a real case from an ecommerce business I consulted for.

The complete process, summarised, was as follows. Exactly like this:

1. Heuristic analysis: during the review I saw that, although the website was generally good, (it generated more than 2 million euros in revenue, in my opinion the checkout process could be improved.
2. Metrics analysis: the checkout CR (users who buy / users who entered checkout) was 18%. I thought it could be improved.
3. Annotated prototype: a design suggesting some changes:
• Flow: registration (not login) took you out of checkout.
• Copy: some messages, such as the billing address message, were unclear, as were some error messages.
• Design: too many escape routes. We removed the top menu and bottom footer, and changed mandatory external links (“terms and conditions” and “instalment financing”) to pop-ups with the same content.
4. Experiment setup: in this case it was done with Google Optimize before Google killed it off. For simplicity, we left the flow change aside and only implemented the copy and design changes.
5. Test results: the checkout CR rose to 22%. And I am convinced that with a few more improvements, another point or two could probably be squeezed out.

I would be lying if I told you I analysed the revenue improvement, honestly, but given that increase in CR, it must have been noticeable.

So if you have an ecommerce business, you can replicate the process exactly.

Seriously, try it exactly like that and see what happens in your case 😉

And with that, we are done…

More about CRO…

Did you like the article?

Then you can continue here with the rest of the articles in the CRO series:

• What CRO is and the user matrix.

• Biases and persuasion in CRO and copywriting.

• The universal principles of CRO.

• CRO techniques to improve conversion.

• Tools for CRO.

• Testing methodology for CRO (the one you are reading).

• Bonus: What Microsoft Clarity is and how to integrate it with Google Analytics.

If you want to receive a new article by email every Thursday, subscribe here. So you do not miss a single one.

And if what you are looking for is help with your digital project, tell me here.