Last quarter, newsletter giant The Hustle ran a test that earned 43,876 additional clicks in their emails.
Not 44,000 total. 44,000 more.
If you’re like me, you’d do terrible things for that kind of engagement.
So I banged on their managing editor’s door and demanded to know his secrets.
Thankfully, the process is simple and repeatable — no terrible things required. So put down that crowbar, because I’m going to share a process that you can start today.
The How-To That They Do
Ben Berkley, the managing editor of The Hustle, perpetually sports an impish grin. Which is fitting, because behind that grin is a deceptively simple method — with some twists coming up.
Here’s how it works:
Every weekday morning, Ben cooks up two separate subject lines for each newsletter. At 5:30 am ET, each subject line is sent to only 30% of The Hustle’s audience.
After an hour, the variant with the most engagement is automatically sent to the remaining 40% of recipients.
“That’s it?” you’re thinking. “It’s just a stinkin’ A/B test?!”
Not so fast. I promised you twists:
- The variations are sent to four predefined subscriber segments.
- The winner is based on the click-through rate (CTR) — not the open rate.
- Ben’s secret sauce: The WTF factor.
Let’s dig in.
Twist 1: Segmenting Readers
The Hustle splits the original two test emails among four segments.
To help me understand this, Ben refers me to Kaylee Jenzen, The Hustle’s resident MarTech wizard.
(“If the Nobel Foundation ever adds a prize for newslettering, I’m nominating Kaylee so fast,” he says.)
“The Hustle uses HubSpot workflows to segment contacts with values ranging from 1 through 4,” Jenzen explains. “Email 1 is targeted at segments 1 and 2, while Email 2 is directed towards segments 3 and 4.”
In other words: One email, two variations, four sends. The reason?
“By distributing email sends over time, The Hustle avoids triggering spam filters and ensures more consistent delivery rates,” Kaylee says.
The takeaway: High send volumes are a signal that email service providers use to identify spammers. By reducing your recipient list and increasing send times, you make your emails safer in the eyes of email servers.
Twist 2: Targeting CTR, Not Open Rate
If you’ve tried A/B testing your subject lines before, I’ll bet you based the test on open rate, right?
That’s what nearly every tutorial will tell you, and it’s not a bad choice. But it’s not the only choice.
While open rate measures the number of readers who access the email, it can’t separate out those readers who delete it immediately. Click-through rate, on the other hand, measures the number of readers who actually engage with the email.
“By emphasizing CTR, The Hustle aims to not only capture the reader’s attention, but it also encourages them to take action and interact further with the content,” Jenzen says. “This approach reflects a strategic effort to drive meaningful engagement beyond just opens.”
See, The Hustle’s open rate is already above industry average, so tweaking the subject line only results in around a 1% difference in that metric. (#HumbleBrag?) So instead, its team makes the strategic choice to focus that time and effort on cultivating even deeper engagement.
The takeaway: Open rate might be the metric you want to focus on. But think about your ultimate goal. Optimize to achieve that, not what a YouTube tutorial told you to do.
Twist 3: The WTF Factor
“When people are looking through their inboxes, they’re often just looking to clear all the new stuff out,” Ben says. “And they get in a rhythm.”
That rhythm is the sound of your email being deleted.
“So the goal is to gently jostle someone out of their rhythm,” he says.
That’s where Ben’s secret sauce comes in.
“To do that, you need a little ‘WTF?’ factor sprinkled in.”
As the former executive editor of The Onion, Ben’s no stranger to the WTF factor. But that doesn‘t mean that he’s trying to recreate Onion headlines for The Hustle.
“I never go for shock value or try too hard to throw people off,” he cautions. “But you ultimately want them to see something unexpected and give them a quick pause. You just want them to glitch for a split second.”
Some of my favorite examples:
- 🎨 Believe in yourself, but not like this
- 🕹 Buried under cement in a landfill
- 🔓 Breaking out of toothpaste jail
The takeaway: Your subject line should be surprising enough to shake them out of autopilot and spare a few minutes for your email.
But what does that actually mean?
How to Write Subject Lines Like The Hustle
Ben shares with me what he calls his “Subject Line Manifesto.” These are actual guiding principles for The Hustle’s editorial voice.
1. Be unexpected.
You’re not likely to jostle someone out of their rhythm with a subject line that confirms something they already know.
“Having worked in comedy, I know the power of subverting expectations,” Ben says. “It’s at the core of getting laughs and creating something memorable.”
“Excluding all of the other actual unforgivable crimes, there’s just one unforgivable crime at The Hustle,” he adds. “Being boring.”
That doesn’t mean you have to try to be funny. In fact, that may be inappropriate for your brand. You can subvert expectations with new data, surprising ideas, or a thoughtful question.
What part of your email is adding something new to the conversation? That’s your subject line.
And speaking of which…
2. Strong subject lines come from strong stories.
“The subject line has to match the newsletter’s overall vibe, and also the specific content within each day,” Ben explains. “So really, the work of writing a strong subject line starts with the work of writing strong stories.”
So if your email doesn’t add anything new to the conversation, maybe pause the subject lines for now and revisit your content strategy.
3. Don’t go for cheap laughs.
If you do decide to use humor, don’t use the same pun that 10 of your competitors also used. Chances are, an example just popped into your head. Skip that one.
“Low-hanging fruit in comedy is low for a reason; it’s there because it’s what people expect,” Ben adds. “If you give it to them, maybe you get light laughs, or a pity laugh or two, but you’re probably not getting a belly laugh because they weren’t surprised. Valuable content comes from jolts.”
And while we’re on the subject…
4. Never stoop to clickbait.
“It’s so easy to have a subject line feel like clickbait, and that’s the last feeling we want our audience to have,” Ben warns. “In this hypercompetitive attention economy, if you sour that relationship, you’re never getting it back.”
5. Keep it brief.
The Hustle sees a noticeable lift in performance from subject lines that are only 11-15 characters long.
Their shortest subject lines enjoy an engagement rate almost 75% higher than emails with the longest.
“As I approach subject lines, I always remind myself: Those words are there to whet the palate, not overwhelm it,” Ben says. “You need to give them just enough context to pique their interest, but not so much that they feel like they already have the whole story.”
Now that you’re a subject line expert, let’s talk process.
How to Do the Test at Home
Sending multiple emails every day sounds like a ton of work, and it would be if you did it manually. Thankfully, most email marketing tools have some kind of automated A/B testing feature.
The Hustle happens to use Marketing Hub, so we’ll show you how to do it within HubSpot, but the process should be nearly the same no matter what tool you use.
1. Create your email as usual.
In HubSpot, as with most tools, this will automatically be version A of the A/B test. (So name it accordingly, and give it your first subject-line variation.)
2. Click on “Create A/B test” in the top left corner.
3. Enter a name for version B, then click “OK.”
Don’t overthink this. Naming conventions are one place where it’s safe to let your boring side show.
Even Ben forgives it in this screenshot below.
4. Edit the subject line for version B.
Remember that a best practice for A/B testing is to only change one element per test. Otherwise, you’ll risk skewing the results.
5. Under the version dropdown menu, click “Manage test.”
6. Set the A/B distribution to your desired mix.
The Hustle uses 30% of their recipients for each variation, with 40% reserved for the winner. But with over two million subscribers (#HumbleBrag confirmed), it can spare a few readers to get more accurate results.
If you have a smaller send volume, you may wish to stick with 20-25% for each test version. While your test will be based on a smaller sample size, the proven winner will be seen by more recipients.
7. Set the winning metric to “
Click
Through Rate.”
To replicate The Hustle’s method, you’ll want to choose CTR. That said, consider the goals of your campaign.
The tool will also allow you to choose “open rate” or “click rate.”
8. Set “Test duration” to 1 hour.
You’ll see a warning to set a duration of “at least 4 hours” in order to help reach conclusive results.
That’s good advice for most marketing emails, but it can be a problem for newsletters. HubSpot research shows that the best send times are between 9am to noon Eastern time.
You can see why a four-hour test might complicate that. But what you’ll lose in statistical significance, you’ll make up for in eyeballs.
9. Click “Save changes.”
10. Review your email settings and recipients as usual, and then send.
Get Your Hustle On
To assuage our legal department and prevent you from banging on my door: I can’t promise you 43,876 additional clicks.
But by following a data-driven approach, you’re almost guaranteed to see improvement. Pair that with Ben’s advice, and you’ll be jostling your readers out of their rhythm regularly.
At the very least, you’ll never be boring.
