Compare two creator batches by fixing who belongs in each batch, what counts as success and how long each creator has to achieve it. Keep creators with poor results in the denominator when they meet the original inclusion rule. Record missing evidence, exclusions and differences between batches beside the result. A fair influencer cohort analysis begins with that written agreement.
Decide what the comparison will change
Write one decision before you open the results. For example, "Should we use the revised brief for the next creator batch?" That requires a different comparison from "Which batch produced more attributed sales?"
NIST's guidance on experimental objectives starts with recording and prioritizing goals, then selecting the measures and design. Modash's measurement guide applies the same goal-first approach to influencer campaigns.
For a brief comparison, a useful primary outcome might be the share of contracted creators who deliver the agreed publication by its deadline. Write the acceptance requirements before launch. For a sales comparison, define the credited purchase event and attribution rules instead. Neither measure can stand in for the other halfway through the report.
If the decision remains unclear, first choose campaign metrics from the decision you need to make.
Freeze a comparison sheet
Use one row per creator and one definition sheet shared by both batches. The following is a hypothetical plan for comparing an old brief with a revised brief. The rules are recommendations for this example, not platform requirements.
| Field | Rule fixed before results |
|---|---|
| Decision | Whether to reuse the revised brief |
| Cohort A | Creators assigned the old brief on the locked roster |
| Cohort B | Creators assigned the revised brief on the locked roster |
| Inclusion | Signed agreement, one agreed publication, same product and market |
| Unit | One creator; repeat posts do not add denominator entries |
| Primary outcome | Creator publishes the agreed asset and meets the written acceptance checklist by the deadline |
| Denominator | Every eligible creator on the locked roster |
| Start | Scheduled brief-delivery timestamp, stored in UTC |
| Deadline | Start plus 336 hours for each creator |
| Evidence | Publication URL, observed publication time and checklist result |
| Reporting cutoff | After every creator's deadline and a fixed evidence-review period |
| Exclusions | Duplicate roster entries or creators ineligible under the original rule |
| Missing data | Separate unknown status with a reason and owner |
| Change control | Save the original definition; record every later correction |
Store each creator's cohort assignment before measuring outcomes. If someone receives the wrong brief, retain their assigned group in the planned comparison and flag the deviation. An additional comparison by brief received may help diagnosis, but label it separately.
Save both the initial roster and the final reporting roster. An exclusion log should contain the creator ID, original cohort, rule invoked, evidence, date and person making the decision. "Performed poorly" is not an eligibility rule.
Give each creator the same opportunity
A common extraction date does not give creators equal follow-up. Someone who started yesterday has had less time than someone who started two weeks ago.
Choose the start event to fit the question. In the brief example, starting the clock at actual publication would remove late delivery from the measure. The scheduled start keeps delays visible. In a post-performance comparison, publication time may be the appropriate start instead.
Do not call a calendar week a rolling seven-day window. Google Analytics cohort exploration groups weekly cohorts from Sunday through Saturday. Its inclusion criteria, return criteria and calculation types also produce different answers. GA4 cohorts concern site or app users and use device data without User-ID; they do not automatically represent your contracted creator roster.
For staggered launches, wait until both batches have completed the agreed follow-up. If an interim report is necessary, mark unfinished rows as pending and do not rank full-batch outcomes yet. Choose a reporting window before looking at results when you need to set the start event and cutoff for another goal.
See how one exclusion reverses the ranking
The following hypothetical data assume complete evidence for every creator. Each batch contains ten eligible creators. All twenty received their full observation window.
| Outcome at deadline | Batch A | Batch B |
|---|---|---|
| Met the publication checklist | 8 | 7 |
| Published but missed a requirement | 2 | 1 |
| Did not publish | 0 | 2 |
| Total eligible creators | 10 | 10 |
Under the agreed rule:
- Batch A completion rate = 8 / 10 = 80%.
- Batch B completion rate = 7 / 10 = 70%.
- Batch B minus Batch A = minus 10 percentage points.
Now suppose the analyst removes B's two non-publishers because they have no post to inspect. B becomes 7 / 8 = 87.5%, apparently above A's 80%.
That second calculation answers how often published work met the checklist. It no longer measures delivery across the contracted batch. You may report both questions, but preserve their distinct names and denominators. Do not replace the planned result with the more favorable one.
Missing evidence needs different treatment. An unreturned creator report does not establish that a publication failed. Show the number of unknown outcomes and withhold a complete-batch rate until you can resolve them. A clearly labelled range may help: with seven confirmed successes and two unknown outcomes among ten creators, the possible completion rate runs from 70% to 90%.
Record what else differed
A fixed denominator cannot make two different batches equivalent. Keep a short difference log alongside the sheet.
| Possible confounder | What to record | Reporting response |
|---|---|---|
| Previous partnerships | New or returning creator at assignment | Show those groups separately |
| Workload | Required asset count and format | Compare equivalent deliverables |
| Incentive | Fee terms and completion bonuses | State whether incentives differed |
| Timing | Holidays, launch dates and stock availability | Explain unequal operating conditions |
| Support | Brief delivery failures and staff assistance | Keep deviations visible |
NIST's randomized-block guidance explains how factors outside the main question can affect outcomes. For a future brief experiment, you could group creators by prior partnership status and randomly assign brief versions within those groups.
A historical split into new and returning creators is descriptive. It does not remove every difference in experience, audience or selection. The hypothetical table shows observed delivery rates; it does not prove that the revised brief caused a decline. If that causal question drives spending, plan an incremental-impact measurement separately.
Keep attribution settings with sales comparisons
When the goal is sales, add the event definition, report name, attribution model, lookback window and export timestamp to the sheet. Keep order-level counts separate from attributed event credit.
Google's attribution documentation says reporting-model changes affect historical and future data for reports using event-scoped traffic dimensions. User- and session-scoped traffic dimensions are unaffected. Lookback-window changes apply going forward. Consequently, two exports can differ because settings changed, even when you select the same reporting dates.
The observation window and attribution lookback window also answer different questions. One sets how long you observe a cohort. The other sets how far back an interaction can qualify for credit. Record both.
If a definition must change, preserve the original result and calculate a separately labelled revision for both batches where the data allow it. If you cannot reconstruct comparable inputs, state that the comparison is unavailable.
Before the next batch starts, copy the comparison sheet, name its owner and lock the roster, outcome and deadline. Ask a colleague to calculate one hypothetical creator's status from those rules. Any disagreement identifies a definition to fix before real results arrive.
