Author response: How significant are the public dimensions of faculty work in review, promotion and tenure documents?

Article Figures and data Abstract Introduction Materials and methods Results: public and community Results: research and metrics Discussion Data availability References Decision letter Author response Article and author information Metrics Abstract Much of the work done by faculty at both public and private universities has significant public dimensions: it is often paid for by public funds; it is often aimed at serving the public good; and it is often subject to public evaluation. To understand how the public dimensions of faculty work are valued, we analyzed review, promotion, and tenure documents from a representative sample of 129 universities in the US and Canada. Terms and concepts related to public and community are mentioned in a large portion of documents, but mostly in ways that relate to service, which is an undervalued aspect of academic careers. Moreover, the documents make significant mention of traditional research outputs and citation-based metrics: however, such outputs and metrics reward faculty work targeted to academics, and often disregard the public dimensions. Institutions that seek to embody their public mission could therefore work towards changing how faculty work is assessed and incentivized. https://doi.org/10.7554/eLife.42254.001 Introduction Review, promotion and tenure (RPT) processes are a cornerstone of academic life at higher education institutions in the United States and Canada. They can influence where faculty focus their attention, the activities they choose to pursue, and choices such as the direction of their research program and the venues where they publish their work, especially during the pre-tenure period (Harley et al., 2010). Unsurprisingly, RPT has been the subject of much scrutiny (for examples see Gordon, 2008; Schimanski and Alperin, 2018). While previous studies (Gardner and Veliz, 2014; Youn and Price, 2009) have documented how expectations of faculty have expanded from having to excel in either teaching, research or service, to having to demonstrate excellence in all three, research continues to be the most highly valued aspect of faculty work (Acker and Webber, 2016; Green and Baskind, 2007; Macfarlane, 2007). Teaching is typically valued less than research, despite teaching duties often representing more than half of the workload (Diamond and Adam, 1998), and service activities come a distant third (Fischman et al., 2018; Foos et al., 2004). Where, then, in this context of ever-expanding responsibilities and emphasis on research, does a commitment to the public come into the RPT process? This depends, of course, on which concept of public one focuses on and what dimensions are emphasized. In 2010 one of us (GEF) and two colleagues offered four basic dimensions of publicness that are used in discussions about what it means for universities to fulfill their public missions (Fischman et al., 2010). Perhaps the most frequently used dimension is that which refers directly to the concept of public patronage in the sense that public universities in the United States and Canada belong to, and are administered by, federal, state or provincial agencies such as a state's appointed board of regents. A second dimension relates to the widespread notion that public universities should be as close as possible to free of cost, or the cost should not be a barrier to access through the use of financial assistance. A third dimension of publicness stems from the belief that universities should operate with the mission of addressing general social problems, promoting the common good, and emphasizing the social contributions of educational achievement beyond the individuals' benefit of access to higher education. Finally, the publicness of a university requires addressing the notion of accountability: to whom are higher education organizations accountable? Who represents the public interest in assessing the public effectiveness of an organization? Notably, the work of faculty members intersects with all these dimensions: a good deal of research and development activities are supported with public money (i.e., public patronage), even at private institutions (NSF, 2016); faculty labor in the form of teaching, research and service is supposed to serve the common good and address social problems (i.e., public good), for which universities in the US and Canada receive a tax-exempt status; and, perhaps now more than ever, faculty need to demonstrate the value of their work (i.e., public accountability), and are therefore subject to more intense public scrutiny. Faculty work is also related in multiple ways to keeping the costs of access (at least at public universities) as low as possible (i.e., public access). Among other things, faculty work intersects with this economic dimension through their salaries (which are directly linked to maintaining low fees and tuition), their work as administrators, and through the expansion of their fund seeking actions (including fundraising activities not related to research grants). As universities struggle to define their own publicness, how do faculty effectively manage their careers in ways that support the various dimensions of the public mission of universities? There appear to be organizational tensions between demands for demonstrating the public value of scholarship (i.e., public accountability) and the focus on "high prestige" or "high impact" publications by RPT committees. If publicness is interpreted as promoting public good, we might expect there to be calls for research outputs to take forms that are more ready for public consumption (not just more publicly available). Yet, determining the "prestige" of a publication venue is usually done at the discretion of evaluation committees (King et al., 2006; Seipel, 2003), through ranked lists or tiers supplied by academic institutions (Malsch and Tessier, 2015), or directly through impact factors and other citation metrics that measure use only within other scholarly works (Adler et al., 2009; Walker et al., 2010). These measures of prestige and impact reinforce the most commonly found publishing formats and venues (e.g., journal articles, books and conference presentations), which do not usually serve public needs in the way other forms do more directly (e.g., blog posts, podcasts, public outreach events). In the sense of public patronage, we might expect the emphasis on publications to move towards the use of open access (OA) models with the public gaining access to the work they are funding. OA has indeed grown, with around 50% of the most recent literature being freely available to the public (Archambault, 2018; Archambault et al., 2014; Piwowar et al., 2018). Yet, OA remains low on the priority lists of faculty (Dallmeier-Tiessen et al., 2011; Gaines, 2015; Odell et al., 2017), even when surveys indicate that many faculty believe open access to their published works is beneficial to their careers due to wider readership (Dallmeier-Tiessen et al., 2011; Gaines, 2015; Odell et al., 2017). It seems these faculty simultaneously hold the conflicting belief that traditional publishing is better for their careers overall because it is valued more in the RPT process (Migheli and Ramello, 2014; Peekhaus and Proferes, 2015; Peekhaus and Proferes, 2016; Rodriguez, 2014). The debate about OA and of where to publish has been complemented with a growing interest for scientific measures beyond citations (so-called altmetrics; Priem et al., 2010). Some hope that these new metrics might serve as indicators of societal impact (Bornmann, 2014; Bornmann, 2015; Robinson-Garcia et al., 2017b; Konkiel, 2016). However, despite predictions that there would be a movement towards using non-citation metrics to assess the influence of research findings for RPT (Darling et al., 2013; Piwowar, 2013), there are concerns, limitations and challenges in the use of these metrics that are hampering their uptake (Gordon et al., 2015; Haustein et al., 2016; Howard, 2013; Lopez-Cozar et al., 2012). Moreover, there is little evidence that mentions on social media are correlated with citations (see Konkiel et al., 2016 for an overview) or that they can serve as indicators of public uptake (Alperin et al., 2019a; Didegah et al., 2018; Robinson-Garcia et al., 2017a). It may be, however, that interest in developing and adopting these new metrics is not indicative of an interest in measuring the alignment between research and the public, but of growing calls for public accountability. A recent independent report commissioned by the Higher Education Funding Council for England (HEFCE) to assess the role of metrics in research assessment and management sees this as part of a "metric tide" that has been swelling in part because of "growing pressures for audit and evaluation of public spending on higher education and research" (p. 136, Wilsdon et al., 2015). Within the RPT process, there is little evidence for the inclusion of altmetrics within formal evaluation procedures (Gruzd et al., 2011; Howard, 2013), although this may be changing, as examples of altmetrics in faculty CVs have begun to emerge (cf., Webster, 2018). However, even in some documented cases where they were included (information science and medicine), department chairs did not value them towards promotion (Aharony et al., 2017; Cameron et al., 2016; Fischman et al., 2018). In contrast, there is evidence that institutions consider citation counts in their RPT process, which, by design, only measure uptake and use of the research by the academic community (Dagenais Brown, 2014; Harley et al., 2010; Reinstein et al., 2011). Another attempt to address the public dimensions of faculty work beyond publication and dissemination formats is manifested through concerted efforts to engage communities in the research process itself. Such efforts can be seen in the growing body of work about such practices (cf. a bibliography of over 600 articles on Community Engaged Scholarship; CES Partnership Resources, 2014), and in the various statements, toolkits and standards for documenting and evaluating community engaged scholarship in faculty RPT guidelines, including the Carnegie Foundation's Elective Community Engagement Classification, the Association of Public Land-Grant Universities' Task Force on the New Engagement, the Research University Engaged Scholarship Toolkit developed by The Research University Civic Engagement Network (TRUCEN), and a partnership of eight Canadian universities that developed Rewarding Community-Engaged Scholarship: Transforming University Policies and Practices. While many faculty have embraced such community engaged scholarship and related models, there is still little evidence these are valued across the academy. In particular, Harley et al. found that faculty who find ways to give back to the community and acknowledge the support of taxpayer funding, such as by participating in public education, generally receive recognition for these efforts regardless of institution type or field of study (Harley et al., 2010). However, these kinds of activities, while representing valid social contributions that can increase a university's accountability to the public, are often not recognized formally in the RPT process (Goldstein and Bearman, 2011). Although previous work provides a sense of how the dimensions of publicness outlined here (public patronage, public access, public good and public accountability) intersect with the RPT process (O'Meara, 2002), more empirical work is needed to understand how publicness is incentivized in faculty careers (O'Meara, 2014; O'Meara et al., 2015). To this end we set out to collect documents, including collective agreements, faculty handbooks, guidelines and forms, that describe RPT requirements for faculty at a representative set of higher education institutions in the US and Canada. We collected these documents to analyze the degree to which various terms and concepts, in particular those that relate to research outputs and assessment, are mentioned in the RPT process, and discuss how the presence of these terms may relate to different concepts of publicness in higher education. Materials and methods Selection of sample Request a detailed protocol We began by creating a stratified random sample based on the 2015 edition of the Carnegie Classification of Institutions of Higher Education for US-based institutions, with an eye to have representation of institutions identified as: 1) doctoral universities (i.e., research-focused), which we refer to as R-type institutions; 2) master's colleges and universities, which we refer to as M-type institutions; and 3) baccalaureate colleges, which we refer to as B-type. Each of these categories is made up of multiple subcategories. R-type institutions are subdivided into those with highest research activity, higher research activity and moderate research activity (R1, R2 and R3); the M-type institutions are subdivided into larger programs, medium programs and small programs (M1, M2 and M3); and the B-type institutions are subdivided into those that are arts and science focused and those from diverse fields (Bas and Bd). For Canadian-based institutions, we used the 2016 edition of the Maclean's Rankings, which similarly classifies institutions into: 1) doctoral (R-type); 2) comprehensive (M-type); and 3) undergraduate (B-type). We aimed to have enough institutions in each of the three broad categories to have statistical power of .8, assuming a small effect size (.25 of a standard deviation), when broken down by discipline. A summary of the number of institutions in each category, the number that we included in our random stratified sample, and the number for which we were able to obtain documents can be found in Table 1. Table 1 Sampling summary of universities from Canada and the United States. Overview of population of universities from the United States and Canada by type and sub-type, the number and percent randomly chosen for the stratified sample, and the number of institutions for which at least one relevant document was obtained. https://doi.org/10.7554/eLife.42254.002 Number in categoryNumber sampledPercent sampledNumber with documentsR-typeR11151715%15R21071615%15R31131715%14RCan1515100%12M-typeM1393174%11M2298124%10M314164%4MCan1515100%13B-typeBas259145%11Bd324175%5BCan1919100%17 We collected documents that applied to the institution as a whole, and also those that applied to specific departments, schools or faculties, which we collectively refer to as academic units. We made a concerted effort to collect documents from academic units from a wide range of disciplines. While there is no single accepted classification system for fields of study, we opted to use the structure of fields and their subfields provided by the National Academies Taxonomy to group disciplines into three main areas: Life Sciences (LS); Physical Sciences and Mathematics (PSM); and Social Sciences and Humanities (SSH; National Academy of Sciences, 2006). Collection of documents Request a detailed protocol We set out to collect documents from the institutions identified. In November 2016 we put out calls on social media and on several mailing lists related to issues of scholarly communications and librarianship, but when that method failed to yield many documents, we turned to a more proactive approach. Equipped with the randomly selected list of institutions, we searched the web for the documents. This method was especially fruitful for identifying documents about RPT that are set out by the institution, but not by individual academic units. For the latter, we searched for email addresses of faculty members of units at each of our target institutions by navigating from their university webpages to those of different faculties and their departments, making sure to look at departments from across the three fields. Given the variety of units, organization structures and naming conventions, our selection of which units to target was not perfectly systematic. It was impossible, for example, to target a specific unit by name across different institutions, since each university makes different decisions of whether to put a discipline within its own department, school or faculty (if it even has a unit to correspond with the discipline at all). Instead, we focused on the concept of an "academic unit" as any administrative unit within the university structure, and from those units listed on websites, our research assistant attempted to pick contacts across the three main field categories. In the end, we sent at least 915 emails to faculty from a dedicated project account between late 2016 and August 2017. In many cases, the persons contacted did not reply, the email address was no longer valid, or there was an auto-response. In many others, the faculty responded to let us know that they were not aware of any documents for their academic unit. In other instances, the person contacted responded with documents pertaining to their unit, and, in a few cases, with documents for several units at their institution. As a result of this process over an almost year-long period, we obtained 864 documents from 129 universities and 381 units, of which 98 (25.7%) are from LS; 69 (18.1%) are from PSM; 187 (49.1%) are from SSH; and 27 (7.1%) are from multidisciplinary units that could not be classified under a single category. A large proportion of the documents collected are undated, but some have dates that go back as far as 2000 and as recent as the year of collection. To the best of our knowledge, these are the documents that the sender believed to be the most recent or applicable. While these documents correspond to the different types of universities and fields, the units are not spread out across all the universities evenly. We have at least one unit-level document from 60 of the 129 universities. We were told that documents did not exist at the academic unit-level by at least one faculty member at the remaining 69. In the majority of cases, we have four or fewer unit-level documents from each institution, but there are 10 instances in which we have more than 10 unit-level documents per institution (with a maximum of 45). Identification of terms Request a detailed protocol We proceeded to load these documents into QSR International's NVivo 12 qualitative data analysis software as two separate sets: the documents corresponding to university-level policies and those corresponding to different academic units. First, we created an NVivo "case" for each institution and academic unit, and we included in these "cases" the content of their respective documents. We then searched the documents for terms of interest, sometimes grouping several terms under a single concept, using various strategies as described in the research methodology notes found in the public dataset (Alperin et al., 2018). The mentions of each term or concept were included in an NVivo "node." We subsequently performed a "matrix coding query" in NVivo to export a table with every university and academic unit as a row, and each of the nodes (terms and concepts of as a coding the between two lists of with each in the with whether at least one document from that university or academic unit at least one mention of the corresponding this we were able to a to the data with the sample and counts and of universities and academic units that mentioned each and those across university types and fields. We were also able to the of the university and academic unit-level to counts of whether a term was mentioned in at least one academic unit or one university-level document for each the that report this The used to these counts can be found in Alperin, at For each term and concept, we used a analysis of to whether the across categories were different from a For all the was that the overall proportion of documents the term or concept was the between the different categories. In Figures 1 and which institution the In Figures and which the Social The was that the proportion of documents the of interest was not across all different categories included in the significant are in Figures and with the and For all in Figures and and some of Figures and in the the data did not the of an of at least for all 1 of institutions public and community terms and concepts by type of institution. whether each term or concept terms and was identified within documents from universities master's colleges and universities and baccalaureate colleges The terms and and the concept of community appear less often in documents from B-type institutions than from and The of the were not for the term but the analysis the in presence of term and concept community are Public community of institutions public and community terms and concepts by institution whether each term or concept terms and was identified within documents of universities, from the most research to those that are less and as as the Canadian research universities The terms and the concept of community appear more at and R2 institutions than with universities in the However, sample for a to measure the of these of institutions public and community terms and concepts by discipline. whether each term or concept terms and was identified within documents of academic units from Social Sciences and Humanities (SSH; Physical Sciences and Mathematics Life Sciences and multidisciplinary units The terms and concepts appear more frequently in units than size for a were only for the concept of community where it that the in this is Public community The data that support the findings of this study are available in the with the (Alperin et al., 2018; Alperin, These data the list of institutions and academic units for which we have documents with an of whether each term and concept was found in the documents for the institution or academic unit. The data also the and The used for these can be found on The documents collected are available on from the corresponding author These documents are not publicly available due to Results: public and community We began our analysis with the terms and to understand the degree to which the public is and to a sense of the context their inclusion in RPT documents. We then focused on several terms and of terms (i.e., that intersect with the of publicness identified with the concept of and community the presence of which would be indicative of to work the public in ways that more research to the Given the to research in the RPT process and Alperin, we then turned our to terms and concepts related to research publications and their assessment, such as and all of which to the different of publicness outlined public and community In RPT documents for their inclusion of concepts related to the public and we found that of institutions mention the term in either the university or academic unit guidelines, while mention the term inclusion of the terms and is most common in institutions Within R-type institutions, we also found a towards inclusion of these terms at those institutions with the highest of research activity (i.e., documents at the included the terms and while of R2 institutions included both these terms and only of the Canadian R-type Within the academic units of R-type institutions, we found that of the disciplines the Life Sciences most frequently these with including and including To better understand the context in which the terms and were being we analyzed the most each these and other we a to be our term of interest it was within the or it of an The 2017). The 10 most used in of and The 10 most used around in of and of the term representation of the of the across all documents. The most is with other this that in the context of is most often with a service of the term representation of the of the across all documents. The most is with other this that the community most often to is that of other The of these terms that publicness in the RPT process in some Although both and appear the most and is mentioned and while is mentioned less than half as much and and the other in the are indicative of the terms and being more commonly with the service of which is the least highly of the RPT (Fischman et al., 2018; Foos et al., Harley et al., 2010). of the often included to as a dimension or set of activities within the service category, from For example, guidelines of the Faculty of of the University of state that duties of a faculty member teaching and related duties research, or duties in public of 2017). guidelines of

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.