Calculating the strength of ties of a social network in a semantic search system using hidden Markov models
Bibliographic record
Abstract
The Web of information has grown to millions of independently evolved decentralized information repositories. Decentralization of the web has advantages such as no single point of failure and improved scalability. Decentralization introduces challenges such as ontological, communication and negotiation complexity. This has given rise to research to enhance the infrastructure of the Web by adding semantic to the search systems. In this research we view semantic search as an enabling technique for the general Knowledge Management (KM) solutions. We argue that, semantic integration, semantic search and agent technology are fundamental components of an efficient KM solution. This research aims to deliver a proof-of-concept for semantic search. A prototype agent-based semantic search system supported by ontological concept learning and contents annotation is developed. In this prototype, software agents, deploy ontologies to organize contents in their corresponding repositories; improve their own search capability by finding relevant peers and learn new concepts from each other; conduct search on behalf of and deliver customized results to the users; and encapsulate complexity of search and concept learning process from the users. A unique feature of this system is that the semantic search agents form a social network. We use Hidden Markov Model (HMM) to calculate the tie strengths between agents and their corresponding ontologies. The query will be forwarded to those agents with stronger ties and relevant documents are returned. We have shown that this will improve the search quality. In this paper, we illustrate the factors that affect the strength of the ties and how these factors can be used by HMM to calculate the overall tie strength.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.005 | 0.025 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.004 | 0.002 |
| Science and technology studies | 0.002 | 0.001 |
| Scholarly communication | 0.002 | 0.004 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.002 | 0.001 |
| Insufficient payload (model declined to judge) | 0.002 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".