The Problem

Most redesign failures follow a familiar pattern:

1. Collect qualitative feedback (“Users said the page feels cluttered.”)
2. Make sweeping visual changes (“Let’s simplify everything!”)
3. Validate with a binary A/B test (“Did it win or lose?”)

In the following example, we explore how Evolv AI would approach a media company.

• Scenario: A brand came to Evolv AI with a redesigned pricing page inspired by qualitative feedback looking to run a standard A/B test to validate.
• Assumption: Sleeker layout = higher conversions.
• Validation Method: A/B Test
• Result: –15% drop in conversion drop.

What went wrong?

Like most teams, they:

• Built ideas with qualitative data
• Validated them with quantitative tests
• Never linked the two

And, the A/B test gave them a binary outcome with few insights on which changes hurt performance. Was it the layout? The CTA? The copy? Which were actually the right design choices? They had no visibility - starting them back at square one.

How Evolv AI Responded

Step 1: Deconstruct the Redesign

Break the pricing page experience into modular UX “blades”—like CTA, pricing section, and social proof placement.

Step 2: Unify All Relevant Data

Evolv AI synthesized:

• The original qualitative insights 
• The results from the A/B test 
• Business intelligence from Evolv AI, Adobe Experience Manager, GA4, and Heap

Step 3: Form Hypotheses per Blade
Using the synthesized insights to assess each blade, Evolv AI’s platform and UX strategists developed testable ideas related to: 

• CTA wording and color
• Pricing section placement
• Plan comparison format (grid vs. accordion)
• Copy tone and length
• Visual type: icons vs. text
• Placement of social proof and trust badges

From Static A/B to Dynamic MVT

The New MVT Strategy

Shifting the customer away from A/B testing, we opted for a multivariate experiment: 

• 12 key UX elements
• 23 variants per element

With Evolv AI’s Active Learning engine: (alt: Application of Active Learning)

• Variants were dynamically served in real-time
• Winning combinations of variants were surfaced automatically
• Losing variants were phased out
• Mid-test changes occurred without restarting

The outcome

• A sustained 5–6% relative lift vs -15% drop
• Clear attribution for every UX element
• Ability to reuse insights across other key touchpoints in the journey
• Deeper understanding of what resonates—enabling smarter, targeted testing
• Company-wide shift toward iterative, data-first optimization

The Redesign Optimization Playbook

Here’s how you can apply this approach to your next redesign:

1. Right-Size Your Experimentation Strategy

• Test volume should match your traffic capacity.
• Too many variants on low traffic = noise.
• Too few on high traffic = missed opportunity.

2. Expect Initialized Volatility

Early data will swing. That’s not failure. That’s signal. Evolv AI’s models dynamically reweight experiences to stabilize insights faster.

3. Pair qualitative and quantitative data together

Most teams use qualitative insights to ideate, and quantitative to validate. But without linking them, you never truly know why something worked.

With Evolv AI, your voice-of-customer data is embedded into hypothesis generation and prioritized for testing.