Making the invisible visible: visualization of music and lyrics for deaf and hard of hearing audiences
Bibliographic record
Abstract
Music is a major art form present in all cultures. It is nearly ubiquitous in Western culture. As a shared social experience, music is a shared knowledge space. Some music is so well recognized that people may refer to it by name (e.g., Beethoven’s 5th symphony, the Star Wars theme). Signing Deaf communities have indigenous forms of lyrical music. While these may not be melodic in the Western expectation, they still conform to the idea of music as metrical lyrics accompanied by instrumentation. Music is a system made up of several elements and attributes containing discrete information chunks that can be perceived by a human listener. Music communicates complex emotional signals and, in some cultures, has specific conventions designed to manipulate emotions. However, deaf, deafened, and hard of hearing people are unable to fully access the richness of the aural musical experience. This dissertation explores ways to make information that is presented in an auditory modality accessible to those who cannot fully perceive it due to hearing loss or environmental barriers; making the invisible visible. A multi-disciplinary, multi-method project, this research examines the design of visualizations intended to present different kinds of otherwise invisible music information in a visible, entertaining form. Ninety-four people, 66 women and 28 men, who identified themselves as Hard of Hearing (46), Deaf (38), Cochlear Implantees (8), or Deafened (2), participated in one of four experiments. Eight of the 94 participated in a fifth experiment. Research questions included determining the fundamental properties of music that visualizations must present, determining what emotional responses users might have to visualizations of music information, and determining whether a music experience similar to that of hearing audiences can be experienced via visualizations of music information. Among several findings, this project learned that many deaf, deafened and hard of hearing people are active consumers of music; lyrics represent a key piece of music information hard of hearing people require of any visualization; different visualizations might be better suited to different musical genres or emotional moods; and the set of music visualizations tested convey some types of emotional information but do not obviously communicate the emotional component of the music.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.009 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.001 |
| Bibliometrics | 0.001 | 0.000 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.003 | 0.002 |
| Open science | 0.001 | 0.004 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.006 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".