Stop Guessing. Start Testing! How to AB Test EVERYTHING on Shopify 📈

My Conversation With Shogun’s COO, Phill Moorman

TLDR Summary

Most Shopify stores are guessing. The top Shopify brands are testing. In this episode and breakdown, I dig into how A/B testing, personalization, and AI powered sections can give your Shopify store a real conversion advantage.

Key points in plain language:

• The average Shopify conversion rate is around 1.5 to 2 percent. Top performers hit 3.3 percent or more

• Brands that A/B test regularly can see conversion lifts from 20 percent to well over 100 percent

• You can now test layouts, content, and sections directly in the Shopify editor with no code

• Personalization by location, campaign, or login status is one of the easiest conversion wins

• AI can generate new Shopify sections in seconds so you can test faster than ever

If you run a Shopify store and you are still relying on your gut, this is your sign to stop guessing and start testing.

Topics Covered

• Shopify conversion rate optimization

• Shopify A/B testing strategies

• How to test Shopify templates, themes, pages, and sections

• Shopify personalization for location, campaigns, and influencers

• AI generated sections for Shopify Online Store 2.0

• How to think about metrics, revenue, and “vanity” numbers

• When design hurts conversions on Shopify

• How high traffic Shopify brands should structure continuous testing

• Recommended Shopify CRO tools and apps

Most Shopify merchants do not have a traffic problem. They have a conversion problem.

The average Shopify store converts around 1.5 to 2 percent of visitors. The top performing stores convert 3.3 percent or higher. That small gap between “fine” and “fantastic” is the difference between a struggling store and a serious Shopify business.

Here is the part that hurts a little. Brands that run consistent A/B tests often see conversion lifts from 20 percent to over 100 percent. Some studies show ranges as high as 27 percent to 550 percent when stores systematically test layouts, messaging, and offers.

Yet most merchants still trust their gut more than their data. It is like choosing a paint color blindfolded and hoping it matches the furniture.

So I invited Phill Moorman, the Chief Operating Officer of Shogun, onto the Shopify1Percent podcast.

Shogun is one of the original Shopify app companies. They launched their page builder back in 2015. Phill started in ecommerce working on sourcing and logistics, then moved into growth marketing at Sourcify, and eventually joined Shogun where he climbed from Head of Marketing to COO over six years. He lives in the world of experimentation, campaigns, and storefront performance.

We talked about A/B testing, personalization, and AI section building for Shopify. The conversation was packed with practical ideas you can apply quickly. Here are the biggest takeaways.

1. Stop Guessing. Start Testing.

If you are not A/B testing your Shopify store, you are making business decisions the same way people pick restaurants in a new city. You think the sign looks nice and hope for the best.

A/B testing is simply the practice of showing two versions of a page or section to different users and seeing which version wins on an important outcome. For Shopify, that is usually adds to cart, purchases, or revenue.

Phill sees merchants get 20 to 30 percent lifts in conversion regularly from simple A/B tests. Sometimes it is as basic as changing the layout, headline, or product image order.

The good news. A/B testing is no longer reserved for big teams with developers.

Shogun A/B Testing lets you:

• Pick any Shopify 2.0 page. homepage, product, collection, landing page

• Clone it into a variant directly inside the Shopify editor

• Change copy, images, sections, layouts, or even test a completely different page via URL testing

• Set your traffic split and goals

• Let the app measure which version drives more revenue

No code. No theme hacking. No dev tickets.

Actionable idea for your Shopify store:
This week, run one A/B test on your Shopify homepage or top product page. Test something simple but important like the hero headline or the primary call to action. Do not overthink it. Just get one test live.

2. Test for Revenue, Not Vanity Metrics

Clicks are cute. Revenue is better.

A lot of people get excited when they see more clicks, more scroll depth, or more “engagement” on a new variation. But if that variation does not lead to more purchases, you did not improve the business. You just created a more interesting page.

Phill talked about this directly. Shogun A/B Testing tracks:

• Revenue driven by each variant

• Conversion rate tied to the page

• Product level sales that originate from that test

If a page gets more clicks but fewer purchases, something is wrong with the funnel. Maybe the add to cart button is not visible. Maybe there are too many distractions. Maybe the copy is cute but not clear.

Actionable idea for your Shopify store:
When you set up an A/B test, make revenue or purchases your primary goal. Use click and scroll metrics as secondary indicators only.

3. Personalization Is a Conversion Multiplier

Personalization is one of those buzzwords that got overused early, then ignored, and now it is quietly becoming a real advantage again.

The idea is simple. Not every Shopify shopper should see the exact same experience.

In the episode, Phill walked through how Shogun Personalization works:

• You can segment visitors by location, city, region, country

• You can segment by referral source or UTM, like a specific campaign or influencer

• You can segment by whether a shopper is logged in or returning

• You can create variants of your page tailored to each segment

For example:

• Show different product selections to customers in different cities if inventory is local

• Show a different hero section for paid traffic versus email traffic

• Show returning customers a tighter, more product focused homepage and show first time visitors more brand story and social proof

There is data backing this up. Multiple studies have found that personalization can drive significant lifts:

• One well known consulting study showed businesses that excel at personalization generate about 40 percent more revenue from those activities than average competitors

• In Shogun’s own world, merchants have used personalization to reduce bad orders, avoid selling out of region products, and increase conversion on regional campaigns

Actionable idea for your Shopify store:
Create one personalized variant for a high value segment, such as visitors from your top city or customers coming from your top influencer. Swap the hero image and copy to match that audience more closely.

4. How to Combine Influencers and Personalization on Shopify

We also talked about influencer campaigns.

Here is a pattern that works.

Instead of sending all influencer traffic to your generic homepage, you:

• Create a variant of a landing page with Shogun

• Personalize it using a UTM parameter for that influencer

• Add their photo, name, a short quote, and a curated set of products they recommend

Now when someone clicks from an influencer’s Instagram, TikTok, or YouTube, they land on a Shopify page that looks like it was built just for them.

There are whole tools built around this concept. But you can recreate most of the effect on Shopify with Shogun’s personalization system and some smart templates.

Actionable idea for your Shopify store:
Pick one influencer who sends you traffic. Create a personalized page for that influencer’s UTM. Include their face, their quote, and their recommended products. Then compare the conversion rate to your generic landing page.

5. AI Generated Sections for Shopify

One of the most exciting parts of the conversation was Shogun’s AI Section Builder.

Historically, if you wanted a fancy new section on your Shopify store, you had two options:

• Buy a new theme or section

• Hire a developer and wait

Now, with AI, you can type a prompt like:

“Create a Shopify 2 column hero section with product image on the right, headline and subheadline on the left, plus a main CTA and a secondary CTA.”

The AI Section Builder generates a full Online Store 2.0 compatible section that you can then:

• Drop into any page in the Shopify editor

• Reuse across multiple pages

• A/B test against your existing sections

• Use with personalization variants

This matters because speed is everything in experimentation. If you can test two or three new sections this week instead of this quarter, you will win.

Actionable idea for your Shopify store:
Use AI to generate a new “conversion focused” section for your product page. For example, a benefits grid, a comparison block, or a bundle explanation. Then A/B test it against your current layout.

6. Do Not Let “Brand Aesthetics” Kill Your Sales

At one point, Phill told a story that made the marketer in me laugh and cry at the same time.

They had a customer whose test variant beat the control by around 30 percent in conversions. Real money. Real impact.

The brand looked at the winning page and said, “We do not like it, it feels off brand” and decided not to implement it.

Here is the tension.

Brand consistency matters. But so does cash flow.

There are valid reasons to make brand decisions that sacrifice some short term conversion. For example, when you are building a long term luxury positioning.

Just make sure you are honest with yourself about which you are doing.

Actionable idea for your Shopify store:
When you see a test winner that is “ugly but effective,” ask your team one direct question. Are we turning this down because of a real brand strategy or because we are emotionally attached to the old version?

7. High Traffic Shopify Stores Should Always Be Testing

If your Shopify store has low traffic, A/B testing still helps, but you have to be patient with sample sizes and careful about expectations.

If you have a lot of traffic, on the other hand, testing is not optional. It is a responsibility.

A store doing hundreds of thousands of visits per month can gain serious revenue from a one percent conversion increase. That is usually worth far more than most “big idea” projects.

Phill’s rule of thumb:

• Smaller stores. run a few high impact tests and focus on big changes like layout, messaging, and offer

• Bigger stores. treat experimentation as a permanent process and run tests regularly on high traffic, high impact parts of your Shopify funnel

Actionable idea for your Shopify store:
If you have decent traffic, build a simple six week testing roadmap that focuses on one part of the store per week. Homepage, top collection, top product, cart, free shipping threshold, etc.

Use Cases for Shopify Merchants

Clear use cases that come straight out of this conversation:

• Optimizing product pages for Shopify paid traffic from Meta and Google

• Personalizing Shopify landing pages for influencers and affiliates using UTM parameters

• Testing homepage layouts for new visitors versus returning customers

• Improving collection page conversions by removing hero banners or simplifying layout

• Using AI to generate new sections for seasonal campaigns and A/B testing them quickly

• Running price tests on Shopify to find profitable sweet spots for key products

• Localizing offers and products by city or region without creating separate Shopify stores

Recommended Tools for Shopify Optimization

Run A/B tests on Shopify pages and templates
Shogun A/B Testing: https://apps.shopify.com/shogun-ab-testing 

Personalize Shopify pages by location, campaign, or login status
Shogun Personalization: https://apps.shopify.com/shogun-personalization 

Build landing pages and sections visually
Shogun Page Builder: https://apps.shopify.com/shogun 

Generate AI based sections for Shopify Online Store 2.0
Shogun AI Section Builder: https://getshogun.com 

Increase average order value with upsells
Bold Upsell (if you want to include it): https://www.boldcommerce.com/shopify 

FAQ for Shopify Merchants

Q1. How can A/B testing improve my Shopify store performance?
A/B testing lets you compare two versions of a Shopify page or section to see which one creates more revenue. Testing headlines, images, layouts, and calls to action can easily unlock 20 to 50 percent conversion lifts, especially on high traffic pages.

Q2. What should I A/B test first on my Shopify store?
Start with your homepage hero, your top product page, or your main landing page for paid traffic. Test the headline, hero image, social proof, or primary call to action. These areas control first impressions and usually have the biggest impact.

Q3. Do I need a developer to run A/B tests on Shopify?
No. Tools like Shogun A/B Testing work directly inside the Shopify editor. You can duplicate pages, change content, set traffic splits, and track conversions without touching code.

Q4. What is an easy personalization win for my Shopify store?
Location based personalization is a great first step. Show different hero images, offers, or product blocks to specific cities, regions, or countries. You can also personalize by campaign UTM or returning vs new visitors.

Q5. How can AI help improve my Shopify conversion rate?
AI can generate new, on brand sections for your Shopify store in seconds. You can quickly produce a new hero, testimonials block, comparison table, or campaign section, then A/B test it. This speeds up experimentation and helps you find high converting designs faster.

Questions This Article Answers

• How do I increase conversions on my Shopify store?

• What should I A/B test first on my Shopify pages?

• How can I run A/B tests in Shopify without a developer?

• How do I personalize Shopify landing pages by location or campaign?

• How do influencer landing pages work on Shopify with personalization?

• What are the best tools for A/B testing on Shopify?

• How can AI help me build better converting Shopify sections?

• How should high traffic Shopify stores structure continuous testing?

Final Thoughts

A big thank you to Phill Moorman and the team at Shogun for continuing to push what is possible on Shopify.

The core message from this episode is simple.

Your Shopify store should not be a static brochure. It should be a living experiment. Every section, every page, every layout is a hypothesis you can test.

Stop guessing. Start testing. Then personalize it. Then let AI speed up the whole process.

That is how you move your Shopify store from “looks nice” to “prints money.”