Engagement is everything: Privacy and newsletter metrics that matter

Why tracking clicks is the only metric we need to know whether the Console developer tools newsletter is performing.
Engagement is everything: Privacy and newsletter metrics that matter

A few weeks ago, Apple announced Mail Privacy Protection for iOS 15, iPadOS 15, macOS Monterey, and watchOS 8. In the Mail app, Mail Privacy Protection stops senders from using invisible pixels to collect information about the user. The new feature helps users prevent senders from tracking when they open an email, and masks their IP address so it can’t be linked to other online activity or used to determine their location.

If you’re responsible for sending newsletters and optimising newsletter performance then this news was treated with concern. Indeed, for May 2021, 93.5% of all email opens on phones come in Apple Mail on iPhones or iPad (Source: Litmus). At Console, as a privacy-first company we were already not tracking opens, nor are we using any other types of tracking pixels to monitor engagement with the newsletter.

In the immediate aftermath, Console’s CEO, David Mytton, blogged Apple's Mail Privacy Protection is a necessary step that helps small publishers. Today, I’d like to go a little further to share how we monitor the Console newsletter’s performance in the absence of invasive tracking.

We use Mailchimp to manage our newsletter templates, subscriber list and email delivery. Notably, we modified Mailchimp’s default tracking features to only track clicks. To protect our audience’s privacy we do not track opens, or use any other types of tracking pixels to monitor engagement with the newsletter. We consider tracking clicks as the least invasive form of tracking we require to be able to commercially run Console.

Mailchimp Settings & Tracking for the Console Weekly Newsletter

Engagement matters most

We believe that longer term engagement statistics are ultimately more valuable metrics to obsess about than more granular statistics such as open rate. Apple’s latest privacy updates are a positive step forward for subscribers and shouldn’t concern marketers.

Split testing subject lines and preview text can work. Open rates will be improved by testing subject lines, but your open rates are ultimately just an input to your click through rate (CTR). Yes, increasing the open rate can improve the chance of increasing the CTR, but only if you chase short-term growth metrics on a week-to-week basis.

It’s easy to periodically increase your open rate, (‘🔥 7 New Developer Tools You Won’t Believe!’), but audiences are smart and if the quality of your content doesn’t match up to the hype then engagement will suffer. Long-term engagement drives compounding growth that will outstrip any short-term tactical wins.

Key metrics

We’re still an early-stage growing start-up and while rapidly growing subscriber numbers is exciting, subscriber numbers are really only a vanity metric without sustained engagement. That’s why long-term engagement is the key metric we monitor internally to determine the health of the newsletter.

We’re not suggesting we'll never change or test our subject line or preview text, as poor subject lines will affect overall performance, but monitoring engagement over prolonged periods of time is the metric that matters most.

Our weekly internal newsletter report compiles the stats you’d expect such as CTR, unsubscribes and bounces along with a breakdown of what people are clicking in the newsletter that tells us the performance of the tools and content featured. This analysis is useful as a snapshot, but for a more detailed understanding we also split out engagement over time and group by the signup channel.

Tracking signup channel is particularly helpful, as it allows us to understand which ones deliver the most engaged subscribers in the long run. For example, does someone who found us through Product Hunt connect with the newsletter more regularly than someone who found us on Reddit? This sort of information, helps guide our marketing efforts and can also help explain improving or declining engagement over specific periods of time.

The most important report we produce is our monthly engagement analysis which looks at engagement of users over consecutive four week windows to see whether engagement is increasing, declining or plateauing. All audiences typically plateau in engagement at some point, unless you radically change your product or mode of engagement. The question is: when does audience engagement level-off and at what level?

While we’d love subscribers to find a new tool or beta every week that they want to try out, we’re realistic. There’s only so many tools any developer will have an interest in over a certain time period.

As even our oldest subscribers are still less than six months old, we don’t have a view yet of how many tools an engaged subscriber should be clicking on every 'x' weeks to be considered an active subscriber, but we know that consistent weekly engagement is an unrealistic metric to chase. For now we consider an engaged subscriber to be someone who clicks on at least one tool every four weeks. This is the ‘north star’ metric we monitor most closely.

Quality content is the best driver of prolonged newsletter engagement

So far, we’ve been pleased to see that in the first few months, monthly engagement is growing rather than declining. This suggests that our open rate is consistently strong, with a subscriber typically finding a minimum of one tool of interest every four weeks. Having designed the newsletter to be purposely succinct and 'snackable', we know that it is possible to get value from the Console newsletter some weeks without ever needing to click a link.

There's lots of things we don't know yet. Notably, how many tools can we expect a subscriber to try out over a prolonged period of time e.g. every three months or annually? We're only starting to get a clearer picture of the types of tools and betas that really resonate with our subscribers.

Nevertheless, seeing consistent engagement gives us enough confidence that the tools we're selecting are of interest to our subscribers. It prevents us from worrying when engagement drops for a particular week or to avoid celebrating too early when we have a particularly strong week.

Greater privacy protects our subscribers and focusses our attention on the metric that really matter - consistently delivering the best tools to developers.

Discover the best tools for developers

Console Newsletter - A free weekly email digest of the best tools and beta releases for developers. Every Thursday.
See the latest email.