Enhancing LLM-assisted move annotation: A focus on inter-model inconsistencies

Recent studies have shown the potential of Large Language Models (LLMs) to annotate text corpora for genre moves, but also cautioned against their errors and inconsistencies. To improve the accuracy of LLM-assisted annotation, this study presents a human-AI collaborative approach that involves human verification focusing on resolving inconsistencies revealed in separate rounds of LLM-generated annotations. Specifically, we examined the inconsistencies between GPT-4 and GPT-4o in annotating rhetorical moves in 800 abstracts from four applied linguistics journals. This approach uncovered similarities and differences in error patterns between the two models, indicating their different focuses in linguistic recognition, especially with instances marked by vagueness. This suggests the need to use different LLMs for annotation, as relying on a single model may lead to repeated errors of the same type. The findings show that a human-AI collaborative approach enhances annotation accuracy significantly. Moreover, analysing inconsistencies provides opportunities to calibrate the annotation scheme and refine linguistic theory.