John Collins
John “Jack” Collins holds a PhD in Social Data Science from the University of Mannheim and a Master’s degree in Data Science from James Cook University. His research focuses on the application of machine learning to social research methods, with particular emphasis on debiasing survey samples and responsive/adaptive survey design. He also works in computational social science, especially on information extraction from text and image data. John is a member of the GENERATION longitudinal survey team at ANU. Prior to entering academia, he worked as an IT consultant, specialising in CRM systems and data-driven application development.
Session
Coding qualitative open-text data is a central concern across the social sciences. Survey researchers develop and apply codebooks to participants' open-text responses in order to derive structured data. More recent digital research methods scrape unstructured content and likewise iterate through text, images, and other media to describe a corpus in a structured manner. Numerous research papers have explored large language models (LLMs) as a means of automating this coding process. These studies suggest that reliable, complete agreement with 'gold-standard' codings (i.e., two independent experts code responses and reconcile their results) is unlikely to be attainable, even with frontier models. However, an LLM coder that achieves a level of inter-coder agreement with a human coder comparable to that achieved between two human coders may be attainable. Given that this level of performance appears possible with frontier models, a natural next step is to determine whether comparable performance can be achieved using smaller models.
This research presents a system that uses distilled and quantized language models small enough to run on locally deployed, GPU-accelerated commodity hardware. As a result, the approach can be effectively utilized on devices as modest as a laptop computer.
We present a locally running automated open-text coding agent built on a small open-weights model. This approach offers several advantages. First, it improves replicability. Frontier models do not guarantee that identical outputs will be returned indefinitely, whereas open-weights models can be configured to consistently produce the same results. Second, it enhances data privacy. Researchers often need to engineer systems that ensure sensitive data, including participants' open-text responses that may contain personally identifiable information, never leaves their control or is transmitted to third-party servers. A locally deployable system makes secure on-premises processing straightforward.
Third, open-weights models provide greater flexibility for customization. In the current implementation, we use a simple classifier to estimate the probabilities of binary "Yes/No" outputs per code definition and derive one-versus-one code probabilities. This enables classifications to be ranked by confidence and allows thresholding techniques to identify genuine "none of the above" responses. We can also implement guided reasoning workflows that improve the quality of coding justifications and evidence-based grounding.
Finally, deploying smaller models on hardware of the researcher's choosing allows per-token costs and energy consumption to be reduced. Researchers can also invest in greater GPU acceleration or parallelize inference across multiple machines to process large corpora quickly. This enables rapid iteration of codebook design and supports efficient large-scale qualitative analysis.