Bibliometric topic-modeling is essential for synthesizing scientific literature, yet its adoption by domain-expert teams is hindered by technical barriers. This pilot study benchmarks four topic modeling paradigms on 286 CO2 mineralization papers (2013–2025), with all methods implemented end-to-end on consumer hardware by researchers without prior natural language processing (NLP) training. Its principal methodological contribution is a four-level calibration protocol for zero-shot large language models (LLMs) used as innovation detectors. Naive zero-shot prompting flagged 71.0% of the corpus as innovative, over-attributing novelty by a factor of 22.7 relative to a calibrated statistical ensemble. A qualitative-strict prompt reduced this to 26.6%, while a structurally different unified prompt yielded 28.0%. Together, these results indicate a floor near 28% that prompt engineering alone does not overcome. A self-reflection pass, in which the model re-reads and revises its own output, instead reduced the flag rate to 3.5%, close to the ensemble’s 3.13%. The papers retained by self-reflection form a strict subset of those retained by the unified prompt, suggesting that the model applies a consistent discriminative criterion rather than simply adopting a stricter prompting frame. On this basis, we propose a transferable four-layer triangulation protocol combining statistical ensembles with complementary LLM regimes. The classical benchmark shows that no single pipeline dominates and we derive selection guidance accordingly.

Low-Resource Topic Modeling for Domain Experts: A Pilot Study in CO2 Mineralization Literature

Giuseppe Sabatino
;
Giuseppe De Rosa;Marcella Di Bella
2026-01-01

Abstract

Bibliometric topic-modeling is essential for synthesizing scientific literature, yet its adoption by domain-expert teams is hindered by technical barriers. This pilot study benchmarks four topic modeling paradigms on 286 CO2 mineralization papers (2013–2025), with all methods implemented end-to-end on consumer hardware by researchers without prior natural language processing (NLP) training. Its principal methodological contribution is a four-level calibration protocol for zero-shot large language models (LLMs) used as innovation detectors. Naive zero-shot prompting flagged 71.0% of the corpus as innovative, over-attributing novelty by a factor of 22.7 relative to a calibrated statistical ensemble. A qualitative-strict prompt reduced this to 26.6%, while a structurally different unified prompt yielded 28.0%. Together, these results indicate a floor near 28% that prompt engineering alone does not overcome. A self-reflection pass, in which the model re-reads and revises its own output, instead reduced the flag rate to 3.5%, close to the ensemble’s 3.13%. The papers retained by self-reflection form a strict subset of those retained by the unified prompt, suggesting that the model applies a consistent discriminative criterion rather than simply adopting a stricter prompting frame. On this basis, we propose a transferable four-layer triangulation protocol combining statistical ensembles with complementary LLM regimes. The classical benchmark shows that no single pipeline dominates and we derive selection guidance accordingly.
topic modeling, bibliometric analysis, CO2 mineralization, large language models (LLMs), zero-shot innovation detection, low-resource computing, SciBERT, research synthesis
File in questo prodotto:
File Dimensione Formato  
analytics-05-00045.pdf

accesso aperto

Tipologia: Versione Editoriale (PDF)
Licenza: Creative commons
Dimensione 3.65 MB
Formato Adobe PDF
3.65 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/20.500.14083/53744
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact