How to A/B Test Your Merchandising and Get Valuable Insights to Grow Revenue

Ellie Ho

July 2, 2025

•

3 min read

Start A/B testing your merchandising strategy

Install Boost AI Search & Discovery

Get started

As a store owner, you make dozens of decisions every day. However, one of the most powerful, and often overlooked, is merchandising. The way you arrange products on your collection and search pages is like designing the layout of a physical store. It guides your customers, tells a story, and can be the difference between a quick browse and a full cart.

For years, these merchandising decisions have been based on instinct. You might think boosting your “Bestsellers” is the best strategy, or maybe promoting “New Arrivals” will create more excitement. You have a hunch, but you don't have proof.

What if you could eliminate the guesswork? What if you could run a head-to-head competition between your best ideas and let your customers' behavior declare the winner?

This is now possible with A/B Testing for Merchandising, a new feature of Boost Commerce. This article will guide you through the exact steps to utilize this tool effectively, optimize your merchandising strategies, make informed data-driven decisions, and ultimately drive more sales.

Why A/B Testing Your Merchandising is a Game-Changer

Merchandising isn't just about sorting products; it's about shaping the customer experience. A/B testing this experience allows you to answer fundamental questions about your business.

“Merchants know their products best, but their customers know what makes them click 'buy.' We built A/B testing to bridge that gap, turning merchant intuition into proven, revenue-generating strategies.” - Jamie Do, Product Manager at Boost Commerce.

Check out: What is A/B testing in eCommerce.

By testing your merchandising rules, you can:

Discover Your Most Profitable Strategy: Is it better to boost high-margin products or high-converting ones? Now you can find out for sure.
Increase Conversion Rates: By showing customers more relevant products, you reduce friction and make it easier for them to find what they love, leading to more sales.
Boost Average Order Value (AOV): Test rules that encourage bundling or showcase complementary items to see which approach gets shoppers to add more to their cart.
Make Changes with Confidence: Stop worrying if a new idea will hurt sales. Test it on a portion of your traffic first. If it works, you roll it out to everyone. If not, you've avoided a costly mistake.

A Step-by-Step Guide to Your First Merchandising A/B Test

Ready to get started? Setting up a test in Boost is straightforward. Think of it in three simple phases: forming your idea, building the test, and learning from the results.

Step 1: Formulate Your Hypothesis (The Big Idea)

Every great test starts with a question. This is your hypothesis. It doesn’t need to be complicated. It’s just a clear statement about what you believe will happen.

Here are a few examples to get you thinking:

“I believe boosting New Arrivals will lead to a higher conversion rate than boosting Best-Sellers for my 'Summer Dresses' collection because my repeat customers are looking for what's new.”
“I believe boosting products with a higher conversion rate will generate more total revenue on my 'All Products' search results page, even if the conversion rate is slightly lower.”
“I believe hiding out-of-stock products will improve the overall shopping experience and increase conversions compared to leaving them visible.”

Once you have your hypothesis, you're ready to build the test.

Step 2: Set Up Your A/B Test in Boost (The Execution)

Inside the Boost app, you'll find a new A/B Testing tab. Creating a campaign is designed to be intuitive.

Name Your Campaign & Choose Your Target: Give your test a clear name (e.g., “New Arrivals vs. Best-Sellers Test”). Then, select the existing merchandising rule you want to test against. This rule is your Control (Version A).
Define the Variant (Version B): Now, create your challenger. You’ll use the same powerful merchandising tools to build a new set of rules for your Variant. This is where you bring your hypothesis to life.
Set Your Goal and Duration: Inform the system about what you're measuring. Is your goal a higher Conversion Rate or more Total Revenue? Then, set the date range for the test. The test can only run while the original merchandising rule is active.
Allocate Traffic: By default, traffic is split 50/50, meaning that half of your visitors will see the Control and the other half will see the Variant. You can adjust this, but we recommend keeping each version above 10% to get reliable results.
Review and Publish: Give everything a final look and launch your campaign! If the test is scheduled for the future, it will live in your “Drafts” tab until it's time to go live.

Check the comprehensive A/B Testing guide for Merchandising.

Step 3: Analyze the Results and Declare a Winner (The Learning)

You don't need to be a data scientist to understand the results. Our analytics dashboard is designed to give you clear, actionable insights. You'll see:

Key Metrics: A side-by-side comparison of your Control and Variant, showing views, conversion rates, and total revenue.
Trend Overview: A chart showing how each version is performing over time.
A Clear Winner: Once the test concludes, we'll automatically highlight the winning version based on the goal you set. In the rare case of a tie, we'll let you know that, too.

With this information, you can confidently make the winning version your new permanent strategy and start thinking about your next test.

Best Practices: Do's and Don'ts for Effective Merchandising Tests

As with any powerful tool, there are a few key considerations to keep in mind for optimal results. Here are some key best practices based on how the feature is designed.

Do's	Don'ts
Do plan your tests around active merchandising rules.	Don't attempt to test a rule with an expired date range.
Do let a test run its course to get reliable data.	Don't edit a live test's core rules. You can only change the name and extend the date.
Do check the impact before turning off a rule with a live test.	Don't be surprised if turning off or deleting a merchandising rule also ends its associated A/B test.
Do use the "Ended" tab to review past results and learnings.	Don't expect tests to be saved if you uninstall the app or downgrade from the Professional plan.

A crucial tip: If you need to edit a live merchandising rule that has an A/B test running, you are limited to changing its name or extending its date range. To make significant changes, you must first end the A/B test. Think of it as locking in the rules of the competition to ensure a fair result.

Real-World A/B Testing Ideas You Can Try Today

Need some inspiration? Here are some powerful merchandising tests you can run on your Shopify store:

The “New vs. Popular” Test:
- Control: Boost "Best-Sellers" to the top.
- Variant: Boost "Newest Arrivals" to the top.
- Goal: Conversion Rate. This indicates whether your customers are more motivated by popularity or novelty.
The “Closing Power vs. Cart Appeal” Test:
- Control: Boost products with the highest conversion rate.
- Variant: Boost products with the highest 'add to cart' ratio.
- Goal: Total Revenue. This helps you find the sweet spot between selling a lot and earning a lot.
The “Availability” Test:
- Control: Show all products, including those that are out of stock.
- Variant: Create a rule to bury all out-of-stock products at the bottom of the page.
- Goal: Conversion Rate. This answers the age-old question of how to handle sold-out items.

You may want to know:

4 Best Practices to Turn “Out of Stock” into Sales Opportunities

4 Simple Ways To Hide Or Burry Products Using Boost AI Search & Discovery

Conquer the Holiday Rush: How To Use Pre-orders to Supercharge Your Sales

Final Thoughts: Your Next Best Idea is One Test Away

Your online store is a living, breathing entity, and your customers' preferences are always evolving. A/B testing your merchandising strategies allows you to evolve with them. It transforms your hunches into hypotheses and your guesses into data-backed growth.

By adopting a mindset of continuous testing and learning, you're not just optimizing a webpage; you're building a smarter, more resilient, and more profitable business.

Trusted by fast-growing brands

Start your 14-day free trial

Get started

Frequently Asked Questions about A/B Testing for Merchandising

1. What happens if I want to edit a merchandising rule that has a live A/B test running?

For the duration of the test, the core rules of the merchandising campaign are locked to ensure a fair comparison. However, you can make two changes: you can edit the campaign's name for organizational purposes, and you can extend its date range. If you need to make more significant changes to the rule, you must first end the A/B test.

2. What happens to my A/B tests if I downgrade my plan from Professional?

A/B testing is a feature exclusive to the Professional plan. If you downgrade to a lower plan, any live or draft A/B test campaigns will be stopped and deleted. The associated merchandising rules will remain (as long as they fit within your new plan's limits), but the A/B testing data will be removed. Ended campaigns will be saved if you later upgrade back to Professional.

3. What happens if an A/B test ends and there is no clear winner?

Sometimes, a test results in a statistical tie, meaning neither the Control nor the Variant performed significantly better for your chosen goal. In this case, the analytics dashboard will show that there is no winner. This is still a valuable insight! It means your new idea didn't hurt performance, but it also didn't improve it, so you can stick with your original rule without worrying that you're missing out.

Trusted by fast-growing brands

Start your 14-day free trial

Get started

Ellie Ho

•

Content Marketing Specialist

July 2, 2025

•

3 min read

There's More

How to A/B Test Your Merchandising and Get Valuable Insights to Grow Revenue

Install Boost AI Search & Discovery

Start your 14-day free trial

Related Posts

How Much Cost To Open A Shopify Store For SMBs. 4 Tips To Save Money On Your First Years

How We Optimize Boost App To Improve Site Speed of Shopify Stores

9 Elements Of High Converting Shopify Fashion Stores