Get Started

Email A/B Testing: How to Test Subject Lines, Content, and Send Time

Published

The best email is the one you tested

You wrote two subject lines and can’t decide which is better. You’re not sure if a button or a text link gets more clicks. You think Tuesday morning might outperform Friday afternoon, but you’re not certain. Every one of these questions has a simple answer: test it and find out.

A/B testing takes the guesswork out of email marketing. Instead of relying on intuition, best-practice articles, or what worked for someone else, you run a controlled experiment with your own audience and let the data decide. Over time, those small, evidence-based improvements compound into significantly better results.

What email A/B testing actually is

An A/B test splits your list into groups, sends each group a slightly different version of the email, and measures which version performs better. The key rule: change only one variable at a time. If you change the subject line and the button color and the send time all at once, and version B outperforms version A, you don’t know which change was responsible.

Common things to test in email:

  • Subject line — the highest-impact element to test. Try different lengths, tones, personalization, and framing. See Subject Line Best Practices.
  • Preheader text — the preview snippet that pairs with the subject line. See Email Preheader Text.
  • Send time — day of week and time of day. Your audience’s behavior may differ from the “best practices” you read online.
  • Call to action — button vs. text link, button color, button text, button placement.
  • Email length — short and punchy vs. detailed and comprehensive.
  • Design layout — single column vs. multi-column, image-heavy vs. text-heavy.
  • From name — company name vs. person’s name vs. combination.

Why it matters

  • Your audience is unique. What works for another company’s list may not work for yours. A/B testing reveals what your specific readers respond to.
  • Small improvements compound. A 5% improvement in open rate, applied across hundreds of sends per year, adds up to thousands of additional opens. A 10% improvement in click rate compounds into measurable revenue.
  • It replaces opinion with evidence. “I think the button should be green” vs. “the test showed the green button got 12% more clicks.” Evidence wins arguments and builds confidence.
  • It reveals surprises. You might discover that short subject lines outperform long ones, that Tuesday is actually your worst send day, or that your audience prefers text-heavy emails over image-heavy ones. These are insights you’d never get from guessing.
  • It prevents regression. Once you know what works, you can standardize it. Without testing, you’re always starting from scratch.

How to run effective A/B tests

  1. Test one variable at a time. This is the most important rule. If you want to test subject line and send time, run two separate tests — not one test that changes both.
  2. Have a hypothesis. Before you test, state what you expect and why. “I expect the shorter subject line to win because mobile truncation won’t cut the key word.” A hypothesis turns a random test into a learning opportunity.
  3. Use a large enough sample. If your list is small, a single test may not be statistically significant — the result could be random noise. Most ESPs calculate significance for you. If you can’t reach significance, run the test multiple times and look for consistency.
  4. Split your list randomly. The two groups must be comparable. Most ESPs handle this automatically, but if you’re manually splitting, make sure it’s random — not “first half vs. second half,” which may introduce bias.
  5. Test the subject line first. The subject line has the largest impact on open rate, and it’s the easiest variable to test (you don’t need to build two email versions). Start there for quick wins.
  6. Give the test enough time. Don’t measure results after 2 hours. Wait 24–48 hours (or longer for B2B audiences where opens happen over days) so the full picture comes in.
  7. Document the results. Keep a log of what you tested, what won, and by how much. Over time, you’ll build a playbook of what works for your audience — so you’re not re-testing the same questions.
  8. Send the winner to the rest of the list. Most ESPs support this workflow: test on a small portion (say 20%), identify the winner, and send it to the remaining 80% automatically.
  9. Iterate. One test answers one question. The next test builds on it. If short subject lines win, next test whether personalization in the short subject line helps further.

Common mistakes

  • Testing too many variables at once. You change the subject line, the design, and the CTA. Version B wins. Which change was responsible? You’ll never know.
  • No hypothesis. You test randomly without a reason. Even if you get a result, you don’t learn the why behind it, so the learning doesn’t transfer to future emails.
  • Small sample sizes. You test on 200 subscribers and declare a winner based on a 3-open difference. That’s noise, not signal.
  • Declaring a winner too early. You check results after 1 hour, see version B ahead, and send it to the rest of the list. Later opens would have told a different story.
  • Not documenting results. You run a test, get a result, and forget the details. Six months later, you’re wondering whether short or long subject lines work better — again.
  • Testing trivial things. You spend a week testing button color (a 1% difference) while ignoring your declining open rate (a 15% problem). Prioritize tests by expected impact.
  • Never standardizing winners. You discover that Tuesday at 10am works best, but you keep sending on random days because you forget. Turn winning insights into standard practice.

How to handle A/B testing with Temway

Temway is a builder and exporter — it produces the HTML for your emails. The A/B test itself (splitting the list, sending variants, measuring results) is managed by your ESP, not by the email builder.

The workflow for testing content variants: build version A in Temway, then duplicate it and make the single change you want to test (a different button label, a different layout, a shorter body). Export both versions or push both to your ESP, where you’ll set up the test: split the list, assign each variant, schedule the send, and measure the results.

For subject line and preheader tests, you don’t need two email versions at all — build one email in Temway, push it to your ESP, and test different subject lines in the ESP’s A/B testing tool. See Subject Line Best Practices and Email Preheader Text for what to test in those fields.

Where to go next