At the time of writing this article, I can assure you that 1 out of every 2 posts on LinkedIn is related to artificial intelligence. Undoubtedly, the emergence of this technology is making a significant impact on the global stage, and rightfully so. As I mentioned in my previous article “AI Marketer: Artificial Intelligence Applied to Growth Marketing,” and in greater depth in my e-book “AI for Marketers” artificial intelligence is a technology as transformative and disruptive as the advent of the Internet over 30 years ago.
If we could trace all the technological advances between the Internet and artificial intelligence, we would undoubtedly find everything related to data storage and analysis, which gave rise to a new commandment: being data-driven. I’m sure we can agree that prior to AI, 1 out of every 2 LinkedIn posts was about Data and how every personal or business decision should be data-backed.
In the professional realm of digital marketing, we can say that advancements in data have driven experimentation to the point of making it another commandment if we seek real growth. So much so that no one can call themselves a growth marketer or engage in growth without understanding how to experiment and analyze data. In other words, the most distinctive characteristic of a growth marketer is their ability to experiment, which is inherently linked to being data-driven.
What is experimentation?
Before delving into how artificial intelligence is impacting experimentation, let’s define this concept in simple terms: it involves applying the scientific method to establish causal relationships between the changes we make to our object of study. In growth marketing, this object of study is a digital asset such as an app, a website, an email, or advertisements.
The goal of identifying these causalities is to determine what works and what doesn’t by validating a hypothesis, allowing us to scale that learning or “discovery” without falling prey to the bias of assuming what might work. For instance, in general terms, you could experiment with the hypothesis that «placing a pop-up inviting users to subscribe to your newsletter will increase subscriptions by 20%.»
Additionally, experimentation has other inherent objectives that are crucial, especially in digital environments: minimizing risks and optimizing resources. These objectives are closely related, but they have distinct focuses. Minimizing risks involves testing whether something works with a small group of users before implementing it across the entire operation of a company. This approach ensures that potential negative impacts are confined to a limited scope, thereby safeguarding the broader system.
For example, consider a scenario where a company wants to test a new feature on its app. Instead of rolling it out to all users, they can implement it with a small, randomized segment. If the feature leads to unforeseen issues or doesn’t perform as expected, the damage is contained, and the overall user experience remains largely unaffected. This cautious approach prevents widespread disruptions and preserves the company’s reputation.
On the other hand, optimizing resources involves testing a change on a smaller scale and within a shorter timeframe than it would take to implement the change broadly. By doing this, companies can save both time and money. If the experiment proves successful, they can then proceed with a larger-scale implementation, confident in the knowledge that their resources are being allocated effectively.
The Scientific Method for Experimentation
It’s not the intention of this article to delve deeply into the step-by-step process of how to experiment (the course “AI for Experimentation” will soon be available where you can learn from scratch). However, we will say that experimenting is not about testing two versions of, for example, a webpage by making a change to one of them and testing it over an arbitrary period with an arbitrary number of users, declaring the one with better performance on a particular metric the winner.
In our toolbox for experimentation, we need to understand and establish certain concepts such as: sample size, conversion baseline, MDE (Minimum Detectable Effect), statistical significance, and statistical power.
These five variables are interrelated, and any change in one will affect the others. The goal is to understand how to ensure that the result of the experiment is reliable. If this is your first time encountering these concepts, don’t worry. It’s important to understand them, but there are calculators that help us adjust the variables for our experiments, such as the CXL calculator or the Optimizely calculator.
Let’s go through a quick practical example to make it super clear how to use experimentation variables to achieve reliable results. Following the example of implementing a pop-up on my website to increase subscriptions, we will use the CXL calculator to conduct an A/B test with the following hypothetical data for my website:
- Sample size: This metric depends on the traffic you can generate, let’s say, 500 per page.
- Conversion baseline: Let’s say 2% of users subscribe.
- MDE (Minimum Detectable Effect): 100%, this is the minimum change I need to see based on standard statistical significance and statistical power for reliability.
- Statistical significance: 95%, this is a standard reliability percentage.
- Statistical power: 80%, this is a standard reliability percentage.
(The image below is in Spanish because this article was originally written in Spanish)
If I mentioned that I could generate traffic of 500 users per page, why does it suggest that the required sample size per variant is «927»? Simply because below that level of users, the result would not be reliable. It even suggests that the experiment should last 20 days.
To familiarize yourself with how these variables are interrelated, I invite you to review the concepts, access the calculator, and experiment with it before implementing it in your daily routine.
I would like to conclude this brief introduction to experimentation by affirming something that may seem obvious: experimenting is much more than A/B tests. There are various methodologies, each with its own nature, such as Multivariate Testing or Funnel Testing. While the former involves experimenting with multiple pages with different variants (instead of just one as in A/B testing), the latter involves experimenting with a series of interrelated pages, such as an e-commerce checkout process.
AI for Experimentation
As expected, Artificial Intelligence can assist us in various tasks related to the experimentation process. As you may have noticed, when experimenting, we need to consider several factors:
- We need to have a clear understanding of our data.
- We need to understand where and what to experiment with by generating hypotheses.
- We need to determine which experimentation methodology is best for each of our hypotheses (A/B Test? Funnel Test?).
- We need to efficiently establish the variables for the experiment in question.
Artificial intelligence can help us with all four points mentioned, but in this article, we will look at a practical example to address points 2 and 3. We will see a sequence of simple prompts using as an example the startup where I currently lead the growth team: Glitzi (a Y Combinator Company), a home spa and beauty startup:
“You are an expert in experimentation and work for Glitzi, an app providing at-home spa and beauty services.
Identify the main consumption trends in wellness & beauty for 2023 and suggest hypotheses on how these trends might affect the demand for at-home massages in the near future.
Then translate these trends into concrete hypotheses and experiments aimed at generating more engagement with our customers and increasing the acquisition of new customers.
For each experiment, suggest:
- The most effective methodology such as A/B Testing, Multi-variable Test, or Funnel Test
- The most effective method for validating the hypothesis considering variables like: conversion baseline, MDE, sample size, statistical significance, and statistical power
- Consider asking me about any data you need to efficiently fulfill your task so that only the implementation of the experiments remains.
Use bullet points for your response to be concise and cite your information sources.”
Let’s review the response that GPT-4 achieved in 5 seconds by parts:
Trends: It provided me with 5 trends, with the most relevant for me being the «Increase in Personalization.» It argues that «consumers demand more personalized services and products that fit their specific needs.»
Hypotheses: It provided a hypothesis for each of the 5 trends, with the hypothesis for «Increase in Personalization» being a very vague option: «offering personalized massages according to customer needs will increase demand.»
Experiment and Methodologies: To validate the proposed hypothesis, it suggested:
- A “more detailed” hypothesis: «offering personalized massages will increase customer retention by 20%.»
- An experiment with an additional variable: giving users the option to choose the duration of their massages instead of the standard times of 60 or 90 minutes.
- A specific methodology: A/B Testing
- The variables to use in this experiment: base conversion rate, MDE, statistical significance of 95%, and statistical power of 80%.
It asked me for the base conversion rate and the expected conversion rate in this experiment.
I provided the requested data: my base conversion rate is 4%, and I hope to increase it to 6%. With this information, it concluded the process with the following response (The image below is in Spanish because this article was originally written in Spanish):
Something interesting to note: experimentation calculators like the CXL one we mentioned earlier can vary in the results provided and may even differ from GPT. This is because there are certain parameters that calculators do not reveal, such as statistical deviation.
I hope you found this brief introduction on how to apply artificial intelligence to experimentation useful. Please let me know if you have any comments, and let’s talk