10th Biennial ACSPRI Social Science Methodology Conference in conjunction with the Australian Social Network Analysis Conference (ASNAC)

Large language models (LLMs), such as ChatGPT, are increasingly being considered as tools for coding in evidence synthesis. However, evaluations of LLM coding often rely mainly on overall agreement scores, which can conceal how and why model decisions differ from researcher judgement. This is especially problematic in conceptual reviews, where definitions, operationalisations and relationships between concepts may be implicit, spread across a text or open to more than one reasonable interpretation. This paper asks where the two coding approaches diverge, whether disagreements are systematically associated with particular coding functions and concepts, and whether they lead to different substantive interpretations.
The study is embedded in a large-scale scoping review of 6,205 peer-reviewed articles on refugee integration published between 2001 and 2025. For the comparison, 15 articles were purposively selected from 200 full texts coded with the support of NVivo to maximise variation in period, method, discipline, geography and conceptual engagement. ChatGPT applied the same codebook to the NVivo-identified passages, using each extract without the wider article context.
The paired dataset comprised 296 coding units covering eight concepts: integration, settlement, belonging, citizenship, adaptation, acculturation, social capital and social inclusion. Three coding functions were assessed: whether a passage defined a concept; identified how it was studied or measured; or explicitly related it to another concept. The analysis combined exact agreement, Cohen’s kappa, the direction of disagreement, concept-level cross-tabulations and comparison of the resulting qualitative syntheses.
Exact agreement was similar across functions, ranging from 53.4% to 54.4%, while agreement beyond chance was slight (κ = .064–.132). Yet these headline coefficients concealed strongly directional differences. NVivo-assisted researcher coding was broader for definitions (56.1% of passages compared with 19.6% for ChatGPT) and operationalisation (49.7% compared with 25.3%). ChatGPT applied a broader threshold to relationships between concepts (57.4% compared with 32.8%). Divergence also varied across concepts: ChatGPT identified no operationalisations for settlement or citizenship, whereas belonging and social capital showed especially strong substantive convergence.
Despite low agreement on individual passages, the two approaches produced similar higher-level interpretations. Both described integration as multidimensional, processual, relational and institutionally conditioned, and produced closely aligned accounts of belonging, social capital and the distinction between formal citizenship and lived membership. They often reached similar conceptual conclusions through different allocations of material across coding functions.
The paper argues that evaluating LLM coding requires more than a single agreement coefficient. It proposes a multilevel assessment combining code prevalence, direction of disagreement, concept-specific patterns and substantive convergence. Because the study used a researcher-derived passage set, a single model and prompt format, and no independently adjudicated standard, it evaluates consistency and coding thresholds rather than comparative accuracy. The findings show that LLM coding should not be treated as interchangeable with researcher judgement, but may offer a complementary perspective when its role, access to context and limitations are explicitly documented.

Egocentric network data are often elicited rather than directly observed. Name generators and alter-level questions require participants to define boundaries, retrieve alters, retain an expanding list and, where local structure is collected, assess a rapidly increasing number of possible alter–alter ties. These demands intensify when interviews are multilingual, interpreter-mediated, ethically sensitive, conducted in domestic settings and embedded in longer qualitative encounters. Participant-aided sociograms may reduce some of this burden, but they are often described mainly as engagement tools. Less attention has been paid to how they transform the production, observability and interpretability of network evidence.

This paper develops and evaluates a qualitative-led mixed-methods protocol centred on a low-technology, participant-aided target sociogram. It asks: how do participant-aided sociograms transform egocentric network elicitation under high-constraint field conditions, and what are the implications for data quality, boundary specification and interpretation? The five-stage protocol combines: (1) an ethically permissive contextual opening; (2) affective and hypothetical exchange name generators; (3) question-wise composition measures; (4) participant placement and movement of alters across four perceived-closeness rings; and (5) participant mapping of salient alter–alter ties and clusters, followed by qualitative probing.

The protocol was piloted and applied in 17 interview sessions with 23 Arabic-speaking refugees in England, including 10 sessions mediated by a trained Arabic–English telephone interpreter. The methodological evaluation integrates transcripts and field notes, photographed sociograms, documented additions and repositioning of alters, case-by-case audits of maps against node and edge files, and descriptive summaries of network size and map complexity.

The findings show that the sociogram redistributes cognitive and interactional work across participant, visual artefact, interviewer and interpreter through four linked mechanisms. First, the visible map supports memory offloading by keeping previously elicited alters available for inspection and iterative recall. Second, it enables comparative calibration: participants judge and revise the closeness of one relationship in relation to others. Third, spatial grouping makes perceived local structure feasible to elicit without a complete dyadic census. Fourth, interaction around the map makes parts of the reporting process visible. Hesitation, clarification, movement and decisions to omit a tie can be qualitatively probed, helping distinguish forgetting, boundary misunderstandings, taken-for-granted household relations, ethical withholding and uncertainty.

These gains do not produce a complete or objectively accurate network. The maps may privilege strong, salient and coherent ties; question-wise composition reduces repetition but sacrifices alter-level cross-classification; very large networks crowd the paper target; and a telephone interpreter cannot share the visual field. The protocol therefore supports claims about elicited network size, broad role composition, ordinal perceived closeness and salient perceived structure, but not total degree, valued ties or a complete adjacency matrix.

The paper contributes a replicable five-stage protocol, explicit conversion and reporting rules, and an implementation-based framework for evaluating participant-aided sociograms as measurement interventions as well as visual elicitation tools. It argues that, in high-constraint fieldwork, data quality depends not on maximising standardisation or detail in isolation, but on the documented fit among the research question, elicitation instrument and field encounter.