Reddit Advice Intent Classifier

Most NLP work on Reddit asks what a comment is about, or whether it is toxic. This project asks a different question: what is the commenter trying to do. Somebody replying to a stranger's problem can be telling them to take an action, warning them what happens if they do not, comforting them, telling their own story, or passing judgment on the situation, and the wording of all five looks roughly the same. We built a labelled dataset for that distinction and then had four of us each train a different architecture on it, so the comparison was between real approaches rather than between hyperparameters.

The five labels come from speech act theory: advice, warning, emotional support, anecdote and appraisal, one label per comment, chosen by dominant function rather than by keywords. That last part is the whole difficulty. "I'm sorry this happened. You should talk to your advisor tomorrow" is advice, even though it opens with sympathy, and "You are justified in feeling that way" is appraisal rather than emotional support, because it evaluates instead of comforting. Our guidelines settle mixed comments with an explicit priority order, advice over warning over emotional support over anecdote over appraisal, so two annotators reading the same message land in the same place for a reason they can point at.

The data is 2,000 comments pulled through Arctic Shift from four advice subreddits, 500 each from r/Advice, r/legaladvice, r/personalfinance and r/relationship_advice, so no single community could dominate the label distribution. Deleted and removed comments went first, then a pattern filter for bot replies and moderator messages and the sidebar redirects those subreddits are full of. What survived went through gpt-4o-mini as a pre-screen that answered keep, drop or unclear on each comment, with unclear kept deliberately, because the point of the screen was to throw out "lol" and spam, not to decide anything a human should be deciding.

Then four of us annotated 500 comments each, with 300 double-annotated so agreement could be measured instead of assumed. Cohen's kappa came out at 0.534, moderate agreement, on 65 percent raw agreement. Disagreements went to a third annotator through a small adjudication tool that reads both labels, takes the matching ones automatically, and only stops to ask when the two annotators actually differ, saving after every decision so a session could be resumed. The finished ground truth was split 1,400 train, 300 validation and 300 test, stratified, seeded, and published as a CodaBench competition with our own scoring program so submissions could be ranked on a test set nobody could see.

The label distribution is the thing that shaped every result that followed. Advice is about 42 percent of the data, appraisal and anecdote take most of the rest, and emotional support and warning are under 5 percent each, which is 14 and 19 instances in the test set. On a split that skewed a majority-class baseline already gets 42 percent accuracy, so accuracy stops being a useful number and macro F1 becomes the one worth reporting.

My model was a BiLSTM on top of frozen Flair embeddings. Flair is a pair of character-level language models, one reading forward and one reading backward, and concatenating them gives 2,048 dimensions per token that already carry context, which matters here because words like leave, fine and serious mean different things depending on whether the comment is directing, warning or reassuring. I kept the encoder frozen on purpose: 1,400 training examples is not enough to fine-tune a language model without destroying what it already knows, so it stayed a fixed feature extractor and only the classifier learned. Embeddings are computed once per split and cached, which turns the character LM from a cost paid every epoch into a cost paid once.

On top of that sits the part I actually designed. Sequences are packed before the LSTM so no computation is spent on padding, and the output is reduced to one vector by concat pooling: the true final hidden state of the last layer, a masked max over all timesteps, and a masked mean, glued into 3,072 dimensions and passed through a small feedforward head to the five classes. Each of the three views answers a different question, what the model ended up believing, the single strongest signal anywhere in the comment, and the overall drift of it. Loss is cross-entropy weighted by inverse class frequency, with gradient clipping, a learning rate that halves when validation macro F1 stalls, and early stopping on the same metric so the checkpoint that gets evaluated is the best one rather than the last one.

I ran three configurations as an ablation: two layers with no augmentation, two layers with EDA synonym replacement and swaps upsampling the minority classes, and three layers with EDA. They scored 0.395, 0.339 and 0.427 macro F1 on the test set. The middle result is the interesting one, because augmentation on its own made the model worse, and the same augmented data helped once the classifier was deep enough to use it. None of that was the first thing I tried either. I started with a ULMFiT setup, frozen and then gradually unfrozen, and it sat between 0.19 and 0.22 macro F1 across three runs, which is barely above sampling from the label distribution. Switching to Flair was the decision that made the model competitive.

Across the group the stacked ensemble finished at 0.506 macro F1, RoBERTa at 0.500, my BiLSTM at 0.427 and the TextCNN at 0.426, all well clear of the TF-IDF and logistic regression baseline at 0.342. On the CodaBench leaderboard my run placed fourth overall and above the BERT-base submission from another group. Every one of the four models failed in the same place: my best run got 3 of 19 warnings and 3 of 14 emotional support comments right, and the other three architectures have their own version of that hole. The honest conclusion of the report is not that a better architecture was waiting to be found. It is that 14 test instances of a class cannot support a real F1 score, and that a few hundred more labelled examples of the two rare intents would buy more than any model we could have swapped in.