Label the praise and criticism separately, against the thing each describes. A comment can be positive about design and negative about delivery without either label canceling the other. When the wording leaves the intended meaning unresolved, keep an uncertain label. For sarcasm, read the surrounding exchange before deciding whether apparently positive words express criticism.
Choose the aspect before the sentiment
An aspect is the part of the experience being discussed, such as fit, packaging, delivery or support. The SemEval-2014 aspect-based sentiment task separates identifying these aspects from assigning their polarity. Its labels include positive, negative, neutral and conflict, with conflict covering both positive and negative sentiment toward an aspect.
That distinction gives a brand team a useful starting point. Use these working labels:
| Label | Coding rule |
|---|---|
| Positive | The author expresses approval of this aspect. |
| Negative | The author expresses dissatisfaction with this aspect. |
| Neutral | The author mentions this aspect without an expressed evaluation. |
| Mixed | The same aspect receives both positive and negative evaluations. |
| Uncertain | Available wording or context does not support a defensible interpretation. |
Here, mixed corresponds to SemEval's conflict category. Uncertain is our recommended operational addition. It records an unresolved interpretation and should stay separate from neutral.
Keep aspect names stable. If your codebook splits packaging into appearance and protection, label those separately. If packaging is your lowest level, retain both views under mixed. Do not change the level midway through a batch to make a difficult comment fit.
Sprout Social's sentiment examples describe grouping feedback by product and reviewing sentiment within those groups. Aspect coding takes that question inside each comment: which part received praise, and which part needs attention?
Twelve synthetic comments with aspect labels
Every comment below is invented for training. None represents a customer, a measured result or a real brand incident. Labels follow the available text; the notes identify where more context would matter.
| Synthetic comment | Aspect labels | Annotation note |
|---|---|---|
| 1. "The jacket fits beautifully. The zip keeps getting stuck." | Fit: positive; zip function: negative | Keep both judgments. A positive fit does not resolve the zip complaint. |
| 2. "Lovely, another delivery pushed back after I stayed home all morning." | Delivery: negative | The stated inconvenience supports criticism. Flag likely sarcasm; do not treat "lovely" as praise. |
| 3. "Nice work." | Target: unresolved; sentiment: uncertain | Without a clear target or exchange, do not decide between praise and irony. |
| 4. "The box looks gorgeous, but it did nothing to protect the mug." | Packaging appearance: positive; packaging protection: negative | Two packaging aspects receive different labels. |
| 5. "Support were kind, but I still can't log in." | Support manner: positive; account access: negative | Kindness and an unresolved access problem can coexist. |
| 6. "This price is ridiculous." | Price: uncertain | "Ridiculous" could reject a high price or celebrate a low offer. Request context. |
| 7. "I don't hate the new color; I preferred the old one." | Color preference: negative, relative to previous version | The comparison expresses a preference. Do not turn the word "hate" into strong hostility. |
| 8. "It arrived Tuesday in a cardboard sleeve." | Delivery timing: neutral; packaging type: neutral | No promised date or evaluation appears. Do not assume Tuesday means late. |
| 9. "The sizing is good on some pairs and awful on others." | Sizing: mixed | The same coded aspect receives explicit opposing judgments. Keep the variation note. |
| 10. "Everyone says the battery is terrible. Mine has been fine." | Battery: positive for the author | Record reported criticism separately if needed. It is not this author's stated experience. |
| 11. "Love the design. Shame the clasp broke on the first wear." | Design: positive; clasp durability: negative | Direct praise survives alongside the complaint. Do not flip the whole comment because it sounds disappointed. |
| 12. "Fantastic support, if you enjoy being sent back to the same broken link." | Support resolution: negative | The conditional phrase supplies the critical meaning. Flag likely sarcasm, with the broken-link detail as evidence. |
Examples 2 and 12 support negative labels even if reviewers disagree about whether to call the tone sarcastic. Example 3 supports no such conclusion. That difference matters more than forcing every comment into a sarcasm category.
Use a repeatable ambiguity check
Treat this as an annotation procedure, rather than a platform rule or a claim to know the author's private intent.
- Identify the speaker and target. Separate the author's view from quoted opinions, reported complaints and replies about someone else.
- Mark the evidence span. Save the clause that supports each label. Include negation and conditions, rather than extracting a sentiment word alone.
- Read available context. Check the parent post, preceding reply and relevant caption. Note missing media or an unavailable parent instead of inventing what it contained.
- Assign each aspect's label. Preserve clear labels even when another aspect remains uncertain.
- Record the unresolved question. Name the missing information, such as the advertised price, the referred-to product or the previous reply.
Keep a separate tone field with values such as "likely sarcasm", "no sarcasm cue" and "uncertain". This prevents a debate about tone from erasing a concrete complaint. It also prevents a sarcastic sentence from automatically making every mentioned aspect negative.
For local slang or language-specific ambiguity, ask a reviewer who understands that usage. Keep their reasoning in the record. Use the multilingual listening codebook guide to keep those decisions consistent across languages.
Check whether the conversation is complete
Missing context can come from the collection method. For example, YouTube's comment-thread documentation warns that a thread response may contain only some replies. Its documented route for retrieving replies is comments.list with the parent comment ID. Do not treat an embedded reply list as proof that you have the full exchange.
The YouTube list method also scopes results to a channel, video or specified IDs. Search terms filter those results. Time and relevance ordering are available, and further pages use a page token. Moderation-status requests require proper authorization; the default status is published.
Record the collection scope, ordering and missing context beside the annotation batch. Keep inaccessible material marked unavailable. A keyword-filtered or relevance-ranked selection should not become a claim about how all customers feel.
Resolve disagreements without deleting uncertainty
For disputed comments, have another reviewer code the aspects before seeing the first reviewer's labels. Compare the evidence spans and the codebook rule behind each decision. A disagreement about "mixed" may come from different aspect boundaries rather than different readings of sentiment.
If the reviewers cannot resolve the meaning from accessible evidence, retain uncertain and state why. Do not make majority preference stand in for missing context. For measuring the performance of an automated classifier across a larger set, use the separate guide to evaluating sentiment labels before acting on them.
Store each result as a comment ID, aspect, evidence span, label, context note and review status. Multiple aspect rows may share one comment ID. Keep that distinction when reporting counts so a single mixed comment does not silently become several customers.
Start the next review batch by giving two reviewers these twelve synthetic comments. Compare their aspect boundaries and their uncertain labels, then add the disputed rules to your codebook before labeling live comments.
