AI Writing Systems

Answering the question, "Does this sound like my client?"

Brand voice is the hardest thing to encapsulate and scale.

And since I like doing hard things, I decided I wanted to try to take the "magic" out of brand voice, and turn it into a number that answers one question:

"How closely does this piece of copy resemble the brand voice?" 

To do this, I took about 200 emails that I hand-wrote for one of my clients, Olivia, and ran them through an AI model called StyleDistance.

The result is exciting for anybody who's looking to capture and scale brand voice copy.

The reason I cared about quantifying brand voice into a number, not just a feeling, is what happens when you try to write at scale.

If you've ever:

• Onboarded a junior writer
• Cringed at a marketing team trying to write in a founder's voice
• Asked AI to draft something that sounds like a specific human

You know just how hard (and incredibly frustrating) this can be.

If I could solve this issue, it helps the writer show how close or far away they are from matching their brand's voice. Also, if I could get this number, I could use it as a feedback mechanism for AI.

"If the brand voice match is under X% retry."

And if we get a few more supporting details (discussed later), we could change that prompt to:

"If the brand voice match is under X%, use the supporting details to figure out why it doesn't match, and list the best edits to make in the next revision."

That's what got me excited enough to build this.

Quantifying brand voice is complicated, but not impossible!

How complicated, though? The easiest way I've found to explain this layer is to compare it to a spam filter. A spam filter asks a yes-or-no question. If it's spam, you put it in bucket A. If it's not spam, you put it in bucket B. EASY.

But brand voice has significantly more variables than a simple yes or no.

Instead of two buckets, StyleDistance creates a cloud of 768 dimensions. If you can't picture it, that's okay, neither can I.

The point is that it's much more complex than the spam folder, and that the experts have taken care of the math behind the 768 dimensions (thank goodness).

Under the hood, there's a machine learning model called StyleDistance that does one thing, it reads a piece of writing and pays attention to HOW it's written, not WHAT it's written about.

Put simply, StyleDistance measures style, not topic.

Most models that turn text into numbers do the opposite. They measure the topic and ignore the style.

But I wanted a model that notices when two pieces of text use the same diction, rhythm, tone, mood, energy, and sentence shapes, REGARDLESS of the topic.

The way it was trained is the part I find interesting. The researchers showed it the same content rewritten in different styles, over and over, until it learned to distinguish between the styles.

It read about the same idea in a formal voice, then in a casual voice, then in a clinical one.

This tells the model, "these pieces all say the SAME thing, so instead, you should group them apart by HOW they say it."

Do that enough times with enough examples and the model learns STYLE.

Once style is the organizing variable, the rest is math.

What the model gives you for any piece of text is a list of 768 numbers. More specifically, those 768 numbers are actually coordinates in a really high-dimensional style space.

Think of graph paper, but instead of two axes (X and Y), you have 768 of them. A piece of writing then becomes just a single dot somewhere in that space.

Here's the part that makes it all work: Two pieces of writing that sound alike land near each other on that graph. Two pieces that sound different land far apart. The closer together, the closer the brand voice match.

Now it's time to actually use the dang thing to score a draft.

There are three steps to seeing if this worked or not.

One: Take the new draft and run it through StyleDistance. This gives it the 768 numbers it needs and plots it on the same graph as the 200 emails I've written by hand.

Two: Measure how far the draft is from the 200 brand voice target emails. There's some nerdy math stuff here, but basically, you can calculate the direction of your drafted piece against the average of the example pieces.

If they point in the same direction in style space, the score number is near 1 (good brand voice match). If they point in unrelated directions, the number is near 0 (bad match).

Three: Pull the closest archived pieces to the draft and surface them as evidence.

These matching pieces pulled from the archive are the "supporting details" I mentioned earlier. They become meaningful context that we can use later to answer questions like:

• Can you show me what a good match looks like?
• Why does this draft match (or why not)?
• How can we edit this to make it match more?

Now it was time to ask the scariest question of all. Does it actually work??

I picked an email called Dec Newsletter 9, "The Power of Decision." It's the kind of piece that should match the brand voice with flying colors.

I yanked it out of the 200 examples that I used to train the StyleDistance model. I rebuilt the style signature from the other 199. Then scored the test email against the rebuilt signature.

It got a score of 0.988 (that's good)!

That puts the test email at the 91st percentile of the cohort, which means only about 9% of the other emails landed closer to their own style signature than this one did.

Put simply, the piece I pulled out of the pile sat back down almost exactly where it should have.

Then the receipt. The four closest neighbors from the rest of the cohort, by score, are:

• 0.990, April Newsletter 3, "Your house is your sugar daddy"
• 0.988, September Email 3, "How to negotiate interest rates"
• 0.988, January Email 5, "Chartering helicopters w/ the girls nbd"
• 0.988, Newsletter Week 5, "Become the main character in your life"

The closest matches are recognizably the same kind of writing as the test email. Same shape, same rhythm, same job. That's exactly what I want this tool to do!

That's just ONE test though...

A careful reader would (correctly) push back and ask whether Dec Newsletter 9 was a lucky chance.

So to really test the heck out of this thing, I ran the same test on all 200 emails.

For each email in the cohort, I rebuilt the style signature from the other 199 and scored the held-out one. 

The full distribution comes out at p5 = 0.937, median = 0.979, p95 = 0.990. Effectively the same shape as the in-cohort distribution.

This just means that the system reliably recognizes Olivia's brand voice across all 200 emails!

To test it a little bit more, I took some academic text from a guy named Steven Pinker, and tested it against my brand voice email cohort. If everything is working, this should NOT score well.

It landed with a score of 0.76!

The cohort minimum score is 0.84. Pinker sits well below the cohort's p5 of 0.937. CLEANLY not a match. The encoder IS NOT confused about what does and doesn't belong as brand voice for Olivia.

The calibration bands fall out naturally:

• ≥ 0.94: inside the cohort
• 0.85 to 0.94: drift zone
• 0.75 to 0.85: off-voice (Pinker text lives here at 0.76)
• < 0.75: different genre entirely

This in and of itself is cool, but it's not the end-all-be-all of scaling brand voice.

However, it gives us a concrete number and some extra context that ENABLES us to do much cooler stuff in the future.

Next, I want to work this into an agentic writing workflow as a source of feedback for agents.

AI, as it is now, is not very good at editing because it's honestly not that smart.

But by giving it this extra layer of feedback, we're giving it one more way to reason about whether a piece of copy matches brand voice or not. The score is a quantifiable number that helps AI not hallucinate, and the context it gives explains WHY it does or doesn't match and HOW to make it better. 

Forecasting how a draft is likely to perform before it ships.

The performance of every piece of copy (email, social posts, etc.) can feel like a gamble.

Maybe it's based on hunches you've had from past tests, maybe it's based on data, maybe it's based on something else...

The reality is that, until you publish a piece and get the numbers back, it's basically a guess.

But what if you could make a quantifiable prediction of how well a piece would perform BEFORE publishing it?

Something that says, "This will go viral and get around 1M views, but it won't move people further down the funnel."

If we knew that, it would be the marketing equivalent of time travel. Before we publish anything, we can go back and tweak it until it's optimized for the right thing.

How freakin' cool would that be?!

So that's what I tried to build here!

If you can't figure out what variables in your copy affect outcomes, this project is for you. 

Every piece of copy should optimize for ONE goal at a time. The way you find that goal is by asking, "What one action do we want the reader to take?"

If we want more leads coming in top of funnel, our goal should be to optimize for virality and follower counts.

If we want leads going deeper into the funnel, we need to optimize for opt-ins, booked calls, etc.

Therefore, we need this performance predictor to differentiate between goals.

So I started by asking two questions.

Question 1: "Will this piece go viral, but not convert?"

Question 2: "Will the audience actually take the action the copy asks them to take?"

To train the performance predictor on this, we split up pieces of copy based on their goal. Did it include a CTA to go deeper into the funnel? Or was it optimizing for reach?

Once you have both numbers, you can do something no off-the-shelf software does. You can determine the variables that contribute to reach and conversion learn and where they intersect.

To do this, we're using a machine learning model called LightGBM.

To start, the human creates variables that they believe might or might not affect the performance of copy. Then LightGBM takes those variables, creates a spreadsheet with a column for every variable, then creates one spreadsheet row for every piece of copy.

Then you let it loose to fill out every single column for every single piece of copy.

To give you an idea, here's a sample row for one IG post:

Row for one IG post
  caption_length              : 187
  hashtag_count               : 9
  has_manychat_keyword        : 1
  sentiment_arc_p1            : -0.12
  sentiment_arc_p10           : +0.34
  style_distance_to_centroid  : 0.41
  fw_to_ratio                 : 0.018
  dim_offer                   : "lead-magnet"
  ... about 100 more like this

I decided that I wanted to be able to predict 6 different copywriting objectives across IG, YouTube, Threads, and Email.

Here's the list:

After splitting all of the samples along these lines, and filling out the spreadsheet for each piece of copy, it's time to start training this thing!

What does "train" even mean?

Think of it like studying with flashcards (where each piece of copy is a single flashcard).

1. You take a stack of past pieces of copy whose outcomes are already known.
2. You show the algorithm one piece of copy (one row from the spreadsheet).
3. You ask it to guess the outcome.
4. You tell it the right answer (based on your real performance data).
5. You let it adjust, so its next guess on that kind of post will be a hair closer to correct.
6. Then you do it again. Then again. Thousands of times across all past posts.

Eventually the algorithm gets pretty good at guessing the outcome on posts it's already seen, and (if you set it up right) on posts it hasn't.

But it's not really flashcards.

Under the hood, this algorithm uses something called "decision trees".

A decision tree is a short flowchart of yes/no questions about variables, ending in a guess. Here's one the model might actually build for IG conversion:

One tree the model built (illustrative)

  Does the caption have a ManyChat keyword?
    No  → guess: 4 ManyChat triggers
    Yes → How many hashtags?
            ≤ 9   → guess: 12 triggers
            > 9   → How long is the caption?
                      ≤ 220 chars → guess: 8 ManyChat triggers
                      > 220 chars → guess: 6 ManyChat triggers

The power behind LightGBM and these decision trees is that they build on each others' learning.

For example, the first tree makes a guess to find it was wrong by a score of 10. Tree 2 focuses on figuring out what Tree 1 got wrong. Over time, this process gets closer and closer to finding the patterns that make predictions more accurate.

This is called "Gradient Boosting" (hence the "GB" in LightGBM).

How do we know LightGBM is actually working though?

We also trained a simpler version of every model as a sanity check, called ElasticNet. ElasticNet draws straight-line rules (every extra word does X to the score, that kind of thing).

LightGBM beat it on every one of the six datasets.

That's because LightGBM successfully catches the complex patterns where performance depends on multiple factors in any given piece.

And to prove that we're ABSOLUTELY SURE about the successful results, I ran the LightGBM process on each copywriting objective 15 times.

The ± number next to each score (something like ±0.025) is the spread across those 15 runs. A small number means a smaller spread and a higher confidence in performance. A bigger number means a wider spread and less confidence.

The top three are strong. This tells us that the model is able to predict higher performing copy around 64% to 71% of the time, which is statistically good enough to be useful in production.

Middle two (YouTube and Threads) aren't quite there yet, but I honestly haven't spent as much time on them.

The bottom one (IG keyword-path engagement at 0.08) is weak ON PURPOSE. We can use this as a way to predict what DOESN'T perform.

(Also, I left out the email click rate because there just wasn't enough data to get statistically significant predictions)

When an IG post has a ManyChat keyword in it, two prediction models run on it, so you get two predictions for one post.

This allows you to predict how the post will convert and how much reach it will get. Then you can plot the post on a grid.

Predicted virality on one axis. Predicted conversion on the other.

Let's take an actual IG post as an example:

Olivia hosted a women's investing retreat in Sedona last spring. The reel she posted afterward opened with this caption:

"I just hosted a retreat in Sedona and I am still sitting in the afterglow
☀ Comment LEAP and I'll DM you the details on the Leap Year Minimind."

The LEAP keyword was the ManyChat trigger for a Leap Year Minimind launch sequence. The objective of the post was to drive conversion to that launch.

When I ran the prediction model on this post, two predictions came back.

Predicted virality was in the top 5% for the cohort. Predicted conversion was in the bottom 1%.

How did the post actually perform?

Reach: 3,407. Likes: 137. ManyChat triggers: zero.

The model called it. High reach, no conversion. Lots of likes, no leads.

How often is the model right about which box a draft lands in?

On the most recent 20% of posts it never saw at training, the high-reach, low-conversion was predicted 49.1% against a 25% random-chance baseline.

That's statistically significant precision meaning the model reliably flags when something needs work. But of course, it's the human's call to push publish in the end.

Imagine writing your next piece of copy inside an editor that scores it as you type. The feedback would look like this:

• Your first sentence is longer than 90% of high-performers in this cohort.
• Your emotional sentiment dips negative around sentence 7. High-performers hold positive longer.
• Adding an emoji here historically lifted pieces like this by 0.4 points.
• Your hashtag count is one above the high-performer median for promotional reels.

Remember how LightGBM uses the spreadsheet row of 100+ variables to predict performance for a post?

A standard technique called SHAP cracks the prediction open and tells you which features pushed the score up, which pulled it down, and by how much.

For example, say you have a draft and you run the prediction model on it.

The IG conversion model scores it 0.65.

SHAP breaks that 0.65 score into parts.

• +0.18 came from the post's purpose tag (lead-magnet promo)
• +0.07 from a ManyChat trigger word in the caption
• -0.04 from the hashtag count being one too high

Basically, LightGBM gives you a prediction. SHAP gives you the WHY behind the prediction.

Run the same process across every post the model has seen, and you get the features it leans on most. Here's the aggregated view for IG conversion:

This opens the door to a powerful source of feedback.

You type a draft. The performance model scores it as you write. SHAP runs underneath and surfaces feedback line by line.

And even better, you can easily hook this up to an AI agent to help with planning, ideation, testing, writing, and beyond!

Turning tests into a compounding map of what actually causes performance.

Most marketing teams are running A/B tests, but are they running real, RIGOROUS testing systems?

• How do you choose what to A/B test?
• Is the test designed to measure the right thing?
• How statistically significant were the results?

Most people run tests on "hunches" and, when they're done, they end up with "hunches" about what worked or what didn't work.

Hunches don't survive or grow, and a year later, you still don't know what variables influence outcomes.

I want to build a system that fixes that.

An A/B test shouldn't get you a "winning variant". It should get you a more confident answer to a question like, "What variable(s) about that subject line CAUSED it work?"

Not a "hunch". Not a fluke. A real root cause.

Because the closer you get to discovering root causes, the more results compound.

Below is the system I'm planning to build, the math underneath it, why I'm confident it works, and what it unlocks.

It's a closed loop. Five pieces hand off to each other in order, and the last one feeds back into the first, so the whole system gets a little more confident every cycle.

1. The hypothesis store
   Holds every claim about your copy ("curiosity-gap subjects lift opens, +6%") with a confidence number on each one ("± 4 percentage points"). This is where the loop starts and ends.
2. The test designer
   Reads the hypothesis store and ranks every claim by impact, expected information gain, and how many future sends it affects. Then it picks the one test that would teach you the most.
3. The variant gate
   Builds the A/B pair for that test and checks it against a control and against Olivia's voice, so you know the send is actually testing what you think it's testing.
4. The test runs
   The actual send. The audience votes with their opens and clicks, and the result comes back as an outcome.
5. The Bayesian updater
   Takes that outcome, tightens the confidence interval on the claim you just tested, and pools the evidence across sibling hypotheses. Then it writes the sharpened numbers back into the store, closing the loop.

One more piece sits just outside the loop and feeds it: the hypothesis generator. This is a LightGBM model that reads factor importance across your entire copy archive and proposes brand-new variables worth testing, seeding fresh rows into the store for the loop to chew on.

The output of this isn't standalone "winning variants." Why? Because winning variants are isolated, and they can't be consistently repeated.

Instead, this testing system is a map of cause-and-effect that grows more confident every week. The result is consistently high-performing variants, with the guesswork removed.

The hypothesis store is a literal spreadsheet that can be imagined to look like this:

Each row is a testable claim worthy of testing.

The best estimate column calculates the impact of that claim.

The confidence column quantifies how unsure we are about the estimate (is it a coincidence or a root cause).

The evidence column tells us how much data we need to gather before we can make a statistically significant conclusion about that claim.

The job of the system is to:

1. Rank which tests would have the largest impact
2. Increase the confidence of our hypothesis one test at a time
3. Collect and update performance data in real time (the loop)
4. Explore new variables and combinations to discover new tests

The math underneath is called a Bayesian hierarchical model with partial pooling.

Put simply: it's the spreadsheet above organized so the rows can talk to each other.

With every test, it runs the math and updates every column automatically.

The "pooling" aspect is cool enough to explain.

Let's say you set out to test the "curiosity-gap on warm subscribers" hypothesis. The results affect that test most, but "pooling" means the results also subtly nudge other related tests like "curiosity gap on cold subscribers" too.

This means, even if you're sending to smaller segments (which is good practice), your results can still have a significant effect since every test is doing work for every related claim, forever.

It's like compounding interest, but for marketers!

While we're building this testing system, we might as well squeeze as much as we can out of it. Why not have it surface new hypotheses it thinks could work?

That's where LightGBM comes in. This AI model reads every email (or any other piece of copy) you've ever sent and learns which features (subject length, brand-voice consistency, emotional sentiment, emoji density, opening word, P.S., etc.) correlate with which outcomes.

And because it has access to all of those variables and data about how they predict performance, it can discover hypotheses worth testing through making new variable combinations.

It's able to make a test suggestion like, "subject length under six words seems predictive of opens." 

And when you combine the automatic hypothesis discovery and the "pooling" features, this system will collect data on this hypothesis in the background while you run other tests.

COOL!

"Okay, but that sounds like A LOT of tests, man... Where do I even start?"

No worries. I got you!

Prioritize the biggest potential impact hypothesis first. Narrow in from there.

We can calculate that with a formula that looks something like this:

If we're running an A/B test, every proposed A/B pair has to pass two checks before it passes:

1. Variants differ only on the variable under test.
   If you're testing curiosity vs. benefit-forward, the two subject lines should be identical in as many ways as possible: same length, same topic, etc. Otherwise you can't attribute the outcome to the variable you thought you were testing.
2. Both variants are high quality.
   Both A and B have to score high on the brand-voice scorer and clear the baseline performance predictor (both are separate tools I've already built). If a variant loses because it's straight up BAD, the variable being tested doesn't have the chance to make a difference.

"Bandits" are tools that dynamically shift traffic toward whichever variant is winning DURING a test. It's great at squeezing every last conversion out of this week's campaign. This sounds awesome at first...

However bandits can have a negative effect on testing programs.

Bandits starve losing variants of traffic which kills the precision needed to build a causal claim. Bandits learn which arm pays. They don't care about why.

A true detective tester wants to know which variant pays AND why. And the only way to make a real causal conclusion is to get enough evidence (losing variant data). 

Bandits enable this week's revenue, but they kill the compounding effect of next year's learning.

I haven't shipped this. So why should you (or I) trust the design? Three reasons.

The math is settled. Bayesian hierarchical models, gradient boosting, partial pooling, Expected Information Gain, these are decades-old, well-tested techniques.

• LinkedIn uses them for email-marketing personalization.
• Booking.com uses them for sequential testing.
• Optimizely uses them for their stats engine.

The novel part is intimately applying them to a company's marketing assets, sticking with it over time so it compounds, and using the data as context that can be leveraged by AI for future copy generation. 

The shift doesn't come from any singular component.

It's what happens when the full testing loop runs for a full year.

Every send becomes a data point. There's no "throwaway" campaign anymore, because even routine sends quietly sharpen the hypothesis store. Twelve months in, you have a growing map of cause and effect instead of a folder of non-scalable "winning variants".

At enterprise scale, this testing system creates institutional memory. All of the testing knowledge survives a marketer changing roles or a quarter ending. One team can read what another already proved and skip the test.

It also fixes the opportunity cost problem. Since we test the highest impact claims first, wasted sends drop, meaning we waste less revenue. And because this is structured data, it feeds straight into whatever AI writes your copy next.