What A/B testing means for your Substack (and how to use it)

Title testing just launched on Substack. Here’s what it means for you!

This week Substack announced the introduction of title testing across the platform. This feature allows users to test as many as four title options before the post sends to the complete subscriber list.

It’s currently only available if you’ve 200 or more subscribers, which makes sense because smaller audiences are unlikely to make much use of this or gain much from the analytics. It’s also available solely for the website, not the app.

What is A/B testing?

When you hit ‘Continue’ in the top right of your page, you’re given the chance to check through a number of settings before publishing. Substack has now added a section called ‘Run a title test’.

The image below shows you exactly what this looks like.

Essentially, this gives you the opportunity to find the headline that makes your subscribers more likely to open your post.

You can test four different options on anything from 5-50% of your subscribers, and after the test duration Substack will automatically send the post to everyone else on your list, changing the title to whichever option performed the best.

How much does your title matter?

A great title can, to some degree, make or break the performance of your post. It’s the very first thing subscribers see, and often pops up on their phone before anywhere else. It’s often the only thing your subscribers consider before they decide whether to open your email or not (unless they already love your brand).

Even small tweaks in phrasing or tone can lead to big differences in open rates. For example, in the FWN’s twice-weekly opportunity newsletters, where I place ‘FWN’ in the title and whether I include the total number of opportunities or not has a big impact on open percentages, sometimes as much as 5%.

5% of my 13,000 readers is 650, and that’s a lot of opens and potentially paying customers that I’m leaving behind. With this new testing format, I can now fine tune exactly which title I want for each edition of the newsletter, leading to more email opens and potential paid subscribers that I might have missed otherwise.

Best practice for using the A/B title tool

My best piece of advice for using this tool is to make each option meaningfully different. Don’t change the odd word, but shift the entire structure, specificity or tone. And play around with your options.

One title could be direct, while another leads with a question. The goal is to find a winner between two different styles, rather than tweak your original title into something with one word changed. You’ll get better insights if your titles reflect different approaches.

For example, if I was writing a piece about why The Subway is Chappell Roan’s best piece of music yet (it is, just for what it’s worth), I could try a range of titles such as:

• Is this Chappell’s best song yet?

• Why ‘The Subway’ hits harder than any other Chappell song

• Who knew The Subway could break my heart

These are just to give you a very simple example. See how each title is styled differently? I could test these on 20% of my audience for an hour, and use the data to send out the one with the best open rate to everyone else. That, in turn, would get my writing seen by more people, which is ultimately why we’re all here.

It’s also worth considering your audience and subject when drafting each title. Just because you’re aiming for more opens, it doesn’t mean you have to throw clickbait at your readers. The tone of your writing is still important, and your title should still reflect the content inside the post.

I would also recommend keeping track of the titles which perform better over time. Substack doesn’t have analytics for this, but you can always keep a note yourself of the tone and vibe that gets more readers opening your work.

So how do you come up with titles?

I’m curious to know how you come up with your Substack titles. Do you have a process? Something you do consistently to create a title? Anything you base them on?

Come along and share your thoughts and ideas in the GYN Substack chat.