Seeing music’s effect on the heart

Real-time and retrospective visualization, listening, annotation to better understand the mechanisms of expressive music effects on cardiorespiratory variables for music theranostics: music based digital therapeutics and precision diagnostics Music influences our physiology, altering our cardiac function, blood pressure, and breathing.1 It enters through our senses, without an incision or the insertion of a catheter, nor the need to ingest or inject drugs. Our hearts quicken (or slow) to its mounting tensions and releases; it makes us hold our breaths in awe, suspense, or anticipation of pleasurable sounds. Hence, music modulates the autonomic nervous system, producing measurable changes in heart rate variability, respiration, and blood pressure, making it a promising cardiovascular therapeutic and diagnostic approach. While music has been linked to positive cardiovascular outcomes, the mechanism by which music produces these effects remains vague.2,3 It’s not what you play, it’s how you play it. The effects of music on the heart can be subtle,4 specific, and context-dependent, making dissecting the music-cardiovascular interaction challenging. To overcome this, we focus on expressive musical structures. Performers imbue music with expressivity by shaping acoustic properties like loudness, tempo, and articulation in tandem with the music’s inherent structures like rhythm or harmony and tonal context for Western common-practice music (the composed music structures can be different for other kinds of music, but the expressive musical structures are more basic and broadly applicable as they draw on human communication paradigms). The performer’s communicative decisions shape the actual music that reaches listeners’ ears, producing meaningful constructs like phrases, transitions, and tension and release. The very same notes can be delivered with verve, prosaically or insipidly; the same music can have long phrases or short, be legato (smooth) or articulated, challenge or sooth, tickle or annoy. Thus, it is important to find measurable ways to represent performers’ expressive renderings of music if we are to explain music’s impact on human physiology. In a groundbreaking study by Bernardi et al.,5 participants had cardiovascular and respiratory variables monitored while listening to music with big expressive gestures like progressive crescendos (transitions from low to high sound levels) and 10 s phrases (corresponding to the period of Mayer waves). Crescendos and rhythmic phrases induced autonomic arousal, and 10 s phrases from two Verdi arias entrained the autonomic variables. This is only the tip of the iceberg. Can we find ways to see evidence of these changes? What are other musical structures that elicit autonomic reactions or entrain autonomic parameters? How can we find and use them for potential therapeutic effects? To understand and decode the mechanisms of music-heart effects, we need systematic measures of performed music and ways to link these transitory, felt events to physiological variables. As concrete steps towards achieving this goal, we have created HeartFM, a mobile and real-time visualization desktop application for capturing physiological information in sync with music, and CosmoNote, a web-based citizen science platform for music-based data visualization, annotation, and listening. CosmoNote (cosmonote.isd.kcl.ac.uk)6 leverages developments in music information research and the citizen science movement to enlist volunteer thinking to mark up and explain expressive structures. The platform enables the transfer of experiential music knowledge through its layered data visualizations (tempo, loudness, harmonic tension) and annotation tools (boundaries, regions, note groups, and comments). CosmoNote can concurrently display physiological time series; thus, integrating CosmoNote with PhysioNet has also produced PhysmoNote, a new visualization option for PhysioNet databases. In Chew et al.,7 pacemaker patients had their intracardiac electrograms downloaded from the left ventricular lead of their resynchronization ICD/pacemaker (CRT-D/P), re-programmed from CRT to dual chamber pacing at 80 b.p.m. or 10 above intrinsic, while listening to a live piano performance. One of the pieces, the Chopin Ballade No. 2, has many sudden changes in expression. With their heart rates held constant, the patients’ activation recovery intervals (ARI, proxies for their action potential durations, APD) were found to change significantly around some of the score boundaries, where many musical parameters also change.8 We use CosmoNote as a tool to test hypotheses underpinning the causes of individual responses. Upon completion of the web platform, we uploaded the pacemaker music study data to CosmoNote so as to concurrently listen to the music that the patients were listening to while cross examining the synchronous ARI, LF, HF and music feature information. We invite you to log in to CosmoNote—username: ehj-guest; password: music-heart—and explore the interface and the data in this (and the next) example. In CosmoNote, the music information layers viewable include the audio waveform, MIDI notes and pedal, loudess, tempo, and tension parameters; select Open panel to choose the physiological data to view with the music information. Listen and see the higher ARIs in the Andantino and the lull in the music, before they plummet and vanish in the stormy Presto con fuoco section; a higher ARI sneaks back in at Tempo I, with the return of the Andantino theme––the visuals are summarized in Figure 1. Pacemaker studies are labour intensive, requiring clinician time and device maker support, and are difficult to scale, which prompted our next project to provide live cardiorespiratory monitoring with music and where possible, concurrent blood pressure and other physiological measurements, at scale using mobile devices. HeartFM (heartfm.kcl.ac.uk) draws upon advances in wearable and mobile health technologies to enable tailored cardiovascular music therapies with physiological feedback. The mobile app plays music while monitoring and recording data from commercially available sensors like heart rate monitors and respiration belts. Its desktop visualization companion allows real-time tracking and facilitates science communication. Data captured by HeartFM are visualized both live and in CosmoNote. Music wields its magic through the choreography of expectation.9 A common expressive strategy is to withhold or delay an expected outcome, which causes stress to increase in the intervening pause or silence. One example of this strategy is the tipping point,10 which typically manifests as an extreme stretching of the musical pulse followed by a release back into the normal pulse. When exercised at the end of a concerto cadenza, the apex of the tipping point (the point of maximum pulse stretch) heightens the anticipation for the return of the original tonal context and theme. Chopin Ballade No.2 and pacemaker patient ARI in CosmoNote: (top) music data (audio signal, tempo, loudness), sections (Andantino, Presto con fuoco, Tempo I), annotation (lull before the storm); (middle) patient ARI data; (bottom) high/low ARI’s circled. Explore the data in cosmonote.isd.kcl.ac.uk (username: ehj-guest; password: music-heart). Figure 2 shows Richard Strauss’ Burleske (cadenza) and the player’s and listener’s heart rate variability in CosmoNote: (i) the player hammers in the A, A, A, A, …, A, prolonging the wait for the expected resolution; (ii) approaching the tipping point, the musical pulse slows, the music quietens, the player relaxes (RMSSD increases); and (iii) the delayed resolution increases the listener’s anticipation, causing their RMSSD to drop, indicating increased sympathetic tone. Finally, the player gives the listener the long-awaited resolution, D, and their RMSSD rise again. Explore the ECG, RR, LF/HF ratio, respiration, and music data in CosmoNote. See a live demonstration with the player and two listeners in Video 1 – the short video can also be viewed at bit.ly/HeartFM-demo-Barts900 (more at bit.ly/heartfm–demos) and the full length lecture, The Musical Heart, where the demo took place in Supplemental Material or at bit.ly/Barts900-MusicalHeart. Strauss Burleske (cadenza) and player + listener physiological signals in CosmoNote: (top) music data (audio signal, MIDI notes), boundary annotations (including tipping point), region annotation (tension build-up), bass note/chord annotations (A, …, A, D); (middle) player’s RMSSD; (bottom) listener’s RMSSD. Listen to the music and see more music and cardiorespiratory information at cosmonote.isd.kcl.ac.uk (username: ehj-guest; password: music-heart). HeartFM demonstration live: Richard Strauss’ Burleske (cadenza): real-time visualization of one player’s and two listeners’ electrocardiographic traces, RR intervals, respiration, RMSSD, and LF/HF ratios at the Barts900 presentation by Professors Elaine Chew and Pier Lambiase, The Musical Heart – Cardiac health and pathology through a musical lens – the YouTube short can also be viewed at bit.ly/HeartFM-demo-Barts900 (more at bit.ly/heartfm–demos) and the full lecture in Supplemental Material or at bit.ly/Barts900-MusicalHeart. Music is invisible, some say ineffable, but still has measurable effects on physiology. We have demonstrated how our tools can help make the expressive structures of music visible and readily quantifiable, making visible the unseen. By providing tools to chart meaningful moments and continuous parameters in music and to link them to time-varying cardiovascular autonomic measures, our aim is to enable the building of explanatory models to perform causal analysis to better understand the beneficial effects of music in cardiovascular therapeutics and diagnostics—music heart theranostics—ultimately leading to individually tailored approaches. With thanks to Corentin Guichaoua and Daniel Bedoya for the music data processing and development of the cosmodoit software; and, Vanessa Pope, Courtney Reed, Mateusz Soliński, and Natalia Cotic for help with the Barts900 HeartFM demonstration. The original pacemaker music study could not have happened without the support of Professor Peter Taggart, Peter Waddingham and Hakam Abbass (recruitment), Louise Sheil and Jan Mangual (Abbott), Holly Daw, Daniel Meese, Genine Sambile (physiologists), James Weaver (recording engineer), Courtney Reed, Shamindra De Zylva (annotation apps), Vanessa Pope, Sebastian Ruiz, Changhong Wang, Simin Yang (interviewers), and St. Bartholomew-the-Great Church. Supplementary data are available at European Heart Journal online. All authors declare no disclosure of interest for this contribution. This result is part of the COSMOS (cosmos.isd.kcl.ac.uk) and HEART.FM (heartfm.kcl.ac.uk) projects that have received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement nos. 788960 and 957532). Elaine Chew, an operations researcher and pianist, is Professor of Engineering in the School of Biomedical Engineering & Imaging Sciences and Department of Engineering at King's College London. Born in Buffalo, NY, and raised in Singapore, Elaine studied Music Performance (distinction) and Mathematical and Computational Sciences (honours) at Stanford and proposed the spiral array model in her PhD in Operations Research at MIT. A pioneering music information researcher, she is forging new paths between music and cardiovascular science. She is Principal Investigator of ERC projects COSMOS, using citizen + data science to probe music structures created in performance, and HEART.FM, making tools and techniques to study how these structures affect cardiac response. Her work has been recognized by the ERC, PECASE, NSF CAREER, and Harvard Radcliffe Institute for Advanced Study. Lawrence Fyfe, research software engineering in the ERC COSMOS project, developed the CosmoNote web-based visualization software and database infrastructure to harness volunteer thinking in the project's citizen science modules, and the sister application PhysmoNote for viewing, annotating, and time-map navigating of PhysioNet databases. Lawrence received his PhD in Computational Media Design from the University of Calgary and a master's degree in Music, Science and Technology from CCRMA at Stanford. Before joining the COSMOS project, he worked on a binaural telepresence system for the Digiscope project at INRIA. The Digiscope project connected various visualization labs around Paris via telepresence (audio and video conferencing) to facilitate collaboration. Charles Picasso, research software engineer in the ERC POC Heart.FM project, developed the heartfm mobile and real-time visualization desktop apps to deliver personalized music therapy to lower blood pressure based on physiological feedback. He spent nine years as a software engineer with IRCAM's Analysis/Synthesis Team and is currently a research engineer for the CNRS for the IRCAM STMS Laboratory Sound Systems & Signals team. As an electronic music producer and sound designer, he regularly engages in artistic projects that integrate real-time, visual, and immersive technologies. Pier Lambiase, Professor of Cardiology at University College London, is an expert in cardiac arrhythmias and technologies applied to treat these conditions. His research focuses on arrhythmia mechanisms and randomized trials of therapy—he has pioneered the use of subcutaneous cardiac defibrillators as a minimally invasive therapy. He is Chief Investigator of the BHF funded CRAAFT HF randomized trial of atrial fibrillation in heart failure—a partnership of the British Heart Rhythm Society and the British Heart Failure Society co-led with Professor Mark Petrie. Mechanistic studies are focused on the genetics of arrhythmias and mapping of ventricular tachycardia. He has co-written ESC Guidelines on SVT 2019, Ventricular Arrhythmias and Prevention of Sudden Death 2022, and published over 350 peer-reviewed papers in Cardiology (Feb. 2024 H index 66 -> 39 000 citations). He is an Editor of the Oxford Handbook of Inherited Cardiovascular Disease & Associate Editor European Heart Journal. He is CV Research Co-Director at the Barts Heart Centre.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.