Your users’ needs and preferences change as the world turns, and the applications of today have to keep up. Modern consumer and business apps are built on a bedrock of rapid experimentation as thesis A squares off against thesis B and the winner stays on, but not for long because another experiment is on the way. However, one part of the app that’s never part of these experiments is the actual authentication process. Until now, that is.
Earlier this year, Descope gave customers the ability to A/B test authentication and user journey flows with our drag & drop workflows and without touching their app’s codebase. We’ve been excited to hear the varied use cases and scenarios where our customers are deploying A/B tests with tangible business outcomes. Watch the video below for a simple A/B testing demo.
This blog will compile four popular use cases of Descope A/B testing where the ultimate goal is to improve user conversion during first-time and repeat visits.
If you’re not using Descope and don’t plan to, you can still read along since the A/B test examples are universal. Fair warning: if you’re planning to implement these A/B testing examples in-house, you won’t be getting a Holiday Card from your engineering team anytime soon.
1. Pilot passkeys
The rise of passkeys have brought with them both excitement and trepidation. Excitement at a potential passwordless future where users can log in to apps without feeling like they’re taking an exam, and trepidation about rolling out passkeys too quickly, or incorrectly, or with not enough user communication, and so on.
Here are some tips on how you can set up an A/B test to pilot passkeys for your users.
Thesis
Have a clear thesis for the experiment, such as “displaying passkeys as an auth option to 20% of our users will lead to an increased login success rate of xx%”. Here are some data points you may want to measure for this experiment and compare against existing benchmarks:
Login success rate
Login time
Number of auth-related support tickets
Condition
If you choose to run the experiment on random traffic, you can create a conditional step in your Descope Flow and use the abTestingKey variable as the filter. The screenshot below will send 80% of your login traffic to a non-passkeys login path and the rest to your new passkey login path.
Another condition you can choose – although this is less of a randomized A/B test – is to only show the passkey login path to users that have WebAuthn-enabled devices like smartphones or laptops that support biometrics. You can do this by setting the device.webAuthnSupport as the value to filter traffic.
Here’s what the CISO of Branch, a Descope customer, has to say about thoughtfully deploying passkeys:
“Descope’s flexible workflow approach has helped us add strong, phishing-resistant WebAuthn authentication when end-user hardware and software support it and fall back on other MFA options when it can't be supported. Visualizing the user journey as a workflow enables us to audit and modify the registration and authentication journey without making significant code changes.”
Flow and screens
Once you’ve decided what the A/B test trigger should be, setting up a new login screen that displays passkeys is very straightforward. In the example below, the default login screen shows only social login options, while the passkey login screen highlights passkeys and includes a backup button if the user wants to continue with social logins instead.
The screens and flows shown in all A/B tests here are just examples. Yours can be anything you want, whether you start with our 100+ templates or create something uniquely your own!
The screenshot below shows the start of a user journey flow with the A/B test steps included:
2. Geo-aware auth methods
Auth is universal but auth methods are not (apart from passwords, sadly). Users in Asia or Latin America may be comfortable with different auth methods than users from the US are. While users may still be able to authenticate using unfamiliar methods, providing a more personalized experience can play a key role in successful new market entry.
Thesis
Let’s say you have a US-based mobile app that accepts social login with Google and Apple. You now want to launch a dedicated app in the Indian market. You can go with the same auth methods, but what if you could supplement those options with WhatsApp, a messaging service that’s second nature to the entire country?
A working thesis for a geo-aware auth method A/B test can be:
Offering WhatsApp authentication to India-based users will increase signup numbers by xx%.
Offering WhatsApp authentication to India-based users will increase return visits / 30 day activity by xx%.
Condition
You can use the geo.country key in a conditional step to send users from a certain country to different onboarding paths. In the screenshot below, the conditional step divides users from India into their own path.
You can also use similar keys to filter based on the users’ city, source IP address, and whether they are initiating login from a mobile device.
Flow and screens
Using the conditional step above, you can show a different login screen to India-based users which have both existing social login providers (Google and Apple), but provides Descope nOTP WhatsApp-based authentication as an alternative.
The screenshot below shows the start of the user journey flow with the A/B test step included:
3. Progressive profiling
Inside you there are two wolves. One says you must do everything to improve user adoption and conversion. The other says you must collect as much information about the user as possible. You are confused (until now).
Here are some tips to run a progressive profiling A/B test to see if collecting the right information at the right time – rather than frontloading all questions to the first screen – can spur user growth.
Thesis
Let’s say you have an app that asks for the following information (all mandatory) during signup:
Name
Work Email
Company
While it’s not the longest signup form, mandating work emails for signing up can leave some users at the door who want to explore your app first before committing to sharing their work details.
Two theses for a progressive profiling A/B test can be:
Only asking for users’ email and not mandating work emails during sign up will improve conversion by xx%
Asking for users’ work email, name, and company during return visits will increase the number of enriched user accounts by xx%
Flow and screens
The conditional step for this A/B test can be randomized, similar to the passkey example shared in the first use case in this blog. With Descope Flows, you can easily create user screens to be shown at any point of the user journey.
For this experiment, let’s say you only ask the user’s email during initial signup. Moreover, you can choose to not mandate work emails, allowing curious souls to try your app out on its merits without any perceived baggage.
You can then ask for the user’s name, company, and work email* in a separate screen shown only to returning users after they authenticate a second time.
*Another neat play you can run with Descope is checking if the user’s existing email is from a free email provider. By using the isFreeEmailProvider key, you can show two screens with or without a “Work Email” input depending on whether you already have the user’s work email.
4. Adaptive MFA
My two-year-old son finds seesaws fun, but he would balk at the security-UX seesaw that most organizations get forced on when thinking about MFA. You may be hesitant to implement MFA at all because of potential user friction and churn. Or you may go too far the other way, implementing high-friction “always on” MFA that actually does hurt the user experience.
If you feel your current MFA implementation could do with improvement, you now have a way to tinker and find out!
Thesis
Let’s say your app currently has MFA enabled for all users every time. Instead, you may want to use a variety of risk factors to enforce MFA only if the login attempt is out of the ordinary. This way, returning users with low-risk login attempts are waved through without an MFA check, removing friction where it isn’t needed.
Some theses for an adaptive MFA A/B test can be:
Only enforcing MFA when the user logs in from a new device will result in xx% more successful logins.
Only enforcing MFA when a bot threat is detected will result in xx% more successful logins.
Condition and flow
Descope provides a variety of out-of-the-box risk signals which can be used to filter traffic into different user journey paths. For instance, the conditional step below shows three different scenarios where a login attempt would be considered risky:
riskInfo.trustedDevice: When a user logs in from a device that hasn’t been marked as a trusted deviceriskInfo.impossibleTravel: When a user logs in from two geographically far locations in a short space of time (impossible traveler)riskInfo.botDetected: When the login attempt is deemed to be from a bot
Conditional steps in Descope Flows can be stacked into numerous if-else-if statements. In this example, we are choosing to enforce OTP MFA when any of our chosen risk factors are true and log the user in directly if none of them are true.
Conclusion
I hope reading this blog has you thinking about all the experiments you can run on your user journeys. Descope Flows are like a box of Legos, so your imagination is your only restriction.
To keep up with more auth and developer content, subscribe to our blog or follow us on LinkedIn. Have an active CIAM project? Book a demo with our team.
