BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//pretalx//conferences.acspri.org.au//2026//talk//UKCQSB
BEGIN:VTIMEZONE
TZID:AEST
BEGIN:STANDARD
DTSTART:20000326T040000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=3;UNTIL=20050326T170000Z
TZNAME:AEST
TZOFFSETFROM:+1100
TZOFFSETTO:+1000
END:STANDARD
BEGIN:STANDARD
DTSTART:20060402T040000
RRULE:FREQ=YEARLY;BYDAY=1SU;BYMONTH=4;UNTIL=20060401T170000Z
TZNAME:AEST
TZOFFSETFROM:+1100
TZOFFSETTO:+1000
END:STANDARD
BEGIN:STANDARD
DTSTART:20070325T040000
RRULE:FREQ=YEARLY;BYDAY=4SU;BYMONTH=3;UNTIL=20070324T170000Z
TZNAME:AEST
TZOFFSETFROM:+1100
TZOFFSETTO:+1000
END:STANDARD
BEGIN:STANDARD
DTSTART:20080406T040000
RRULE:FREQ=YEARLY;BYDAY=1SU;BYMONTH=4
TZNAME:AEST
TZOFFSETFROM:+1100
TZOFFSETTO:+1000
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:20000827T030000
RRULE:FREQ=YEARLY;BYDAY=4SU;BYMONTH=8;UNTIL=20000826T170000Z
TZNAME:AEDT
TZOFFSETFROM:+1000
TZOFFSETTO:+1100
END:DAYLIGHT
BEGIN:DAYLIGHT
DTSTART:20011028T030000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10;UNTIL=20071027T170000Z
TZNAME:AEDT
TZOFFSETFROM:+1000
TZOFFSETTO:+1100
END:DAYLIGHT
BEGIN:DAYLIGHT
DTSTART:20081005T030000
RRULE:FREQ=YEARLY;BYDAY=1SU;BYMONTH=10
TZNAME:AEDT
TZOFFSETFROM:+1000
TZOFFSETTO:+1100
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
UID:pretalx-2026-UKCQSB@conferences.acspri.org.au
DTSTART;TZID=AEST:20261126T111500
DTEND;TZID=AEST:20261126T113000
DESCRIPTION:Large language models (LLMs)\, such as ChatGPT\, are increasing
 ly being considered as tools for coding in evidence synthesis. However\, e
 valuations of LLM coding often rely mainly on overall agreement scores\, w
 hich can conceal how and why model decisions differ from researcher judgem
 ent. This is especially problematic in conceptual reviews\, where definiti
 ons\, operationalisations and relationships between concepts may be implic
 it\, spread across a text or open to more than one reasonable interpretati
 on. This paper asks where the two coding approaches diverge\, whether disa
 greements are systematically associated with particular coding functions a
 nd concepts\, and whether they lead to different substantive interpretatio
 ns.\nThe study is embedded in a large-scale scoping review of 6\,205 peer-
 reviewed articles on refugee integration published between 2001 and 2025. 
 For the comparison\, 15 articles were purposively selected from 200 full t
 exts coded with the support of NVivo to maximise variation in period\, met
 hod\, discipline\, geography and conceptual engagement. ChatGPT applied th
 e same codebook to the NVivo-identified passages\, using each extract with
 out the wider article context.\nThe paired dataset comprised 296 coding un
 its covering eight concepts: integration\, settlement\, belonging\, citize
 nship\, adaptation\, acculturation\, social capital and social inclusion. 
 Three coding functions were assessed: whether a passage defined a concept\
 ; identified how it was studied or measured\; or explicitly related it to 
 another concept. The analysis combined exact agreement\, Cohen’s kappa\,
  the direction of disagreement\, concept-level cross-tabulations and compa
 rison of the resulting qualitative syntheses.\nExact agreement was similar
  across functions\, ranging from 53.4% to 54.4%\, while agreement beyond c
 hance was slight (κ = .064–.132). Yet these headline coefficients conce
 aled strongly directional differences. NVivo-assisted researcher coding wa
 s broader for definitions (56.1% of passages compared with 19.6% for ChatG
 PT) and operationalisation (49.7% compared with 25.3%). ChatGPT applied a 
 broader threshold to relationships between concepts (57.4% compared with 3
 2.8%). Divergence also varied across concepts: ChatGPT identified no opera
 tionalisations for settlement or citizenship\, whereas belonging and socia
 l capital showed especially strong substantive convergence.\nDespite low a
 greement on individual passages\, the two approaches produced similar high
 er-level interpretations. Both described integration as multidimensional\,
  processual\, relational and institutionally conditioned\, and produced cl
 osely aligned accounts of belonging\, social capital and the distinction b
 etween formal citizenship and lived membership. They often reached similar
  conceptual conclusions through different allocations of material across c
 oding functions.\nThe paper argues that evaluating LLM coding requires mor
 e than a single agreement coefficient. It proposes a multilevel assessment
  combining code prevalence\, direction of disagreement\, concept-specific 
 patterns and substantive convergence. Because the study used a researcher-
 derived passage set\, a single model and prompt format\, and no independen
 tly adjudicated standard\, it evaluates consistency and coding thresholds 
 rather than comparative accuracy.  The findings show that LLM coding shoul
 d not be treated as interchangeable with researcher judgement\, but may of
 fer a complementary perspective when its role\, access to context and limi
 tations are explicitly documented.
DTSTAMP:20261013T080154Z
LOCATION:Sutherland Room
SUMMARY:Where NVivo-assisted researcher coding and LLM coding diverge: Evid
 ence from a large-scale scoping review - Gabriella D'Avino
URL:https://conferences.acspri.org.au/2026/talk/UKCQSB/
END:VEVENT
END:VCALENDAR