2026-11-26 –, Sutherland Room
Large language models (LLMs), such as ChatGPT, are increasingly being considered as tools for coding in evidence synthesis. However, evaluations of LLM coding often rely mainly on overall agreement scores, which can conceal how and why model decisions differ from researcher judgement. This is especially problematic in conceptual reviews, where definitions, operationalisations and relationships between concepts may be implicit, spread across a text or open to more than one reasonable interpretation. This paper asks where the two coding approaches diverge, whether disagreements are systematically associated with particular coding functions and concepts, and whether they lead to different substantive interpretations.
The study is embedded in a large-scale scoping review of 6,205 peer-reviewed articles on refugee integration published between 2001 and 2025. For the comparison, 15 articles were purposively selected from 200 full texts coded with the support of NVivo to maximise variation in period, method, discipline, geography and conceptual engagement. ChatGPT applied the same codebook to the NVivo-identified passages, using each extract without the wider article context.
The paired dataset comprised 296 coding units covering eight concepts: integration, settlement, belonging, citizenship, adaptation, acculturation, social capital and social inclusion. Three coding functions were assessed: whether a passage defined a concept; identified how it was studied or measured; or explicitly related it to another concept. The analysis combined exact agreement, Cohen’s kappa, the direction of disagreement, concept-level cross-tabulations and comparison of the resulting qualitative syntheses.
Exact agreement was similar across functions, ranging from 53.4% to 54.4%, while agreement beyond chance was slight (κ = .064–.132). Yet these headline coefficients concealed strongly directional differences. NVivo-assisted researcher coding was broader for definitions (56.1% of passages compared with 19.6% for ChatGPT) and operationalisation (49.7% compared with 25.3%). ChatGPT applied a broader threshold to relationships between concepts (57.4% compared with 32.8%). Divergence also varied across concepts: ChatGPT identified no operationalisations for settlement or citizenship, whereas belonging and social capital showed especially strong substantive convergence.
Despite low agreement on individual passages, the two approaches produced similar higher-level interpretations. Both described integration as multidimensional, processual, relational and institutionally conditioned, and produced closely aligned accounts of belonging, social capital and the distinction between formal citizenship and lived membership. They often reached similar conceptual conclusions through different allocations of material across coding functions.
The paper argues that evaluating LLM coding requires more than a single agreement coefficient. It proposes a multilevel assessment combining code prevalence, direction of disagreement, concept-specific patterns and substantive convergence. Because the study used a researcher-derived passage set, a single model and prompt format, and no independently adjudicated standard, it evaluates consistency and coding thresholds rather than comparative accuracy. The findings show that LLM coding should not be treated as interchangeable with researcher judgement, but may offer a complementary perspective when its role, access to context and limitations are explicitly documented.
Gabriella D’Avino is a Marie Skłodowska-Curie Global Fellow at the University of Birmingham, hosted by the University of Technology Sydney. She leads UNPACK-NET, a comparative project examining how social and institutional networks shape refugee education pathways in Australia and the United Kingdom. Her wider research focuses on refugee resettlement, sponsorship and integration, using qualitative, comparative and mixed-methods network approaches.