Report on Responses to the Questionnaire
- Demographic Characteristics
- General Questions
- Questions on Peer Review
- Questions on Peer Review for Publishers and Editors of Journals
- Questions on Peer Review for Members of the HSSFC Aid to Scholarly Publications Programme, Academic Administrators, and Reviewers
- Questions for Publishing Scholars and Academic Authors
- Questions on Copyright
- Conclusions from the Questionnaire
In August and September of 2000, scholars, reviewers, and publishers in the humanities and social sciences were surveyed about their views on aspects of the credibility of electronic publishing. There were 336 responses to our questionnaire from across the country and across disciplines. The following is the report on the responses to our questionnaire. The aggregated data and statistical analysis is available as Appendix D of this report.
This report follows roughly the organization of the questionnaire which is available, in both official languages, as French: Appendix C1 and English: Appendix C2.
For this survey we had 336 valid respondents; most work in English (79%), are affiliated with a university (76%) and the larger percentage were male (64%). The spread of academic rank and age was reasonable. (68% were between the ages of 40 and 59, and 69% were either Associate or Full Professors.) The respondents were evenly divided between those disciplines that were classified under the Social Sciences (52%) and those under the Humanities (46%). The best represented disciplines were English (8.41%), Sociology (5.5%), Psychology (4.5%), Economics (4.21%) and History (4.21%).
In many of the demographic categories our distribution matched the percentages in the Archer report of 1999 (Electronic Publishing in the Humanities and Social Sciences: A Report to the Humanities and Social Sciences Federation of Canada on Survey Findings by Keith Archer, see http://ahdp.lib.ucalgary.ca/hssfc/ ) except in the areas like the distribution of disciplines and academic rank. In the Archer report 66.1% of the respondents were in the Social Sciences and 33.9% in the Humanities. In the Archer report 23.3.7% were Assistant Professors while only 13% were in our survey. The following table summarizes the demographic similarities and differences between the two reports.
Rockwell / Siemens
Language English 74.4% 79% French 25% 14% Both - 6% Gender Female 28.4% 33% Male 71.6% 64% Academic Rank Assistant Professor 23.3% 13% Associate Professor 28.3% 26% Full Professor 41.7% 43% Sessional Instructor/Instructor or Lecturer 3.6% 5% Other 3.0% 11% Disciplines Social Sciences 66.1% 52% Humanities 33.9% 46% Country of Study of Ph.D. Canada 63.3% 53% United States 20.5% 21% Other 13.2% 13%
It should be noted that in our survey for the question about Academic Rank the fourth option was worded differently than in the Archer survey, which may explain the difference in percentage. In our survey the option was worded "Instructor or Lecturer" in the Archer report it was worded "Sessional Instructor".
Some of the parts of the questionnaire were directed at subsets of the people contacted. The following is a summary of the percentage of respondents that answered questions in the different parts.
- Part 0: Demographic Characteristics 100%
- Part I: General Questions 100%
- Part II: Questions on Peer Review 100%
- Part III: Questions for Publishers and Editors 19%
- Part IV: Questions for Members of HSSFC ... 45%
- Part V: Questions for Authors 88%
- Part VI: Questions on Copyright 100%
In some questions we found statistically significant differences between demographic groups. This chart summarizes the location of those differences. Where there was a significant difference (p-value < .21) we have marked it by providing the p-value. For the full statistical analysis see the data aggregation appendix.
Country of Ph.D.
It is worth noting that there were no questions where Age was statistically significant, and only two places where Affiliation was (Questions 39 and 42). This goes against the accepted wisdom that older academics are less comfortable with electronic resources. Perhaps the most interesting demographic categories where there was a difference are Gender (18 Questions), Discipline (13 Questions), and Rank (7 Questions). Given the way we categorized the types of institutions using the Macleans categories (with an additional category for Emerging institutions) the differences around Institutional Affiliation (10 Questions) are less interesting.
In 18 questions there were significant differences between how Female and Male respondents responded. For example, for Question 3 (Have you ever made your scholarship available electronically in any way?), 56.76% of Female respondents answered "Yes" compared to 66.04% of Male respondents. It is difficult to interpret these differences, for which reason we believe it would be useful to conduct a follow up interview with selected respondents or a supplementary survey intended to explore such differences.
In 13 questions there were differences between how respondents from the Humanities and those from the Social Sciences responded. For example, for Question 12 (Peer review must take into account aspects of the electronic publication that are unique to the medium.), 67.32% of Humanities respondents agreed or strongly agreed compared to 59.77% of Social Science respondents. These differences between the Humanities and Social Sciences need to be studied further.
In 7 questions Academic Rank seemed to make a difference. Surprisingly it often seemed that senior faculty (Associate and Full Professors) were more likely to experiment with electronic publication. For example, for Question 3 (Have you ever made your scholarship available electronically in any way?) the percentages by rank broke down thus:
Yes to Q3
No to Q3
It is possible that tenured faculty feel they have more freedom to publish electronically in the light of credibility issues because they do not need to worry in the same way about their careers. These responses suggest that it is not true that senior and older colleagues are less likely to experiment with new technology compared to their junior, younger (and presumably untenured) colleagues. Again, further study of these differences needs to be undertaken.
Use of Electronic Resources
The majority of the respondents use on-line resources (69% use on-line scholarly journals, for example, and 82% use other on-line resources.) An extraordinary 96% use at least one of the types of on-line resources listed in Question 1. Our responses also show a marked increase in the use of electronic resources from the Archer Report where 50.9% used on-line scholarly journals and 67% reported using other on-line resources. It is clear that on-line resources are being used, and it appears that there has been an increase in use since the Archer Report which conducted its survey in the Fall of 1999. This chart summarizes the increase in use of different types of on-line resources. The increase in use of on-line newspapers is particularly noticeable. Our colleagues seem to be reading the news off the WWW more and more.
Rockwell / Siemens 2000
On-line scholarly journals 50.9% 69% On-line government resources 51.1% 68% On-line newspapers 39.7% 65% On-line archival materials 40.7% 62% Other on-line resources 67% 82%
It should be noted that these increases found between our survey and the Archer Report are consistent with the answers to Question 15 where 56% of the respondents either agreed or strongly agreed with the statement that "The number of effectively peer reviewed electronic publications available to academic authors and readers is increasing in most fields." (Only 1% disagreed while 31% didn't know and 11% neither agreed nor disagreed.) The Archer Report also found that a substantial number of respondents felt that their use of on-line resources had increased in the last 5 years. (For example 43.7% felt that their use of on-line scholarly journals had increased while only 0.6% found it had decreased and 6.8% felt it hadn't changed.) While it is possible that the increase between the two surveys is due to the other factors, there is a perception that there is an increase in availability and use of such resources.
As for what venues the respondents would be most likely to use for electronic resources (Question 10), 75% chose "on the Internet at a university-sponsored site", 58% chose "on the Internet from a reputable publisher", 50% chose "through library subscription", 35% chose "on a CD-ROM from a reputable publisher", and only 25% chose "on the Internet generally". This suggests that university sponsorship of sites may be an important way by which respondents distinguish worthwhile resources. University reputation may become important in the assessment of on-line resources. This is surprising given the complete freedom academics have over the mounting of research materials at university run sites.
Creation of Electronic Resources
While there is widespread use of on-line resources only 16% of the respondents had published on-line in a refereed outlet while 61% had made their scholarship available electronically and 70% had integrated electronic resources into their teaching. This suggests that respondents are more comfortable experimenting with the creation of electronic teaching materials and non-refereed materials than with submitting to refereed outlets. It could be that the way scholars develop their familiarity with the creation of on-line resources is through the development of teaching materials and informal research materials not intended for publication through a refereed venue. These numbers also suggest that familiarity with the skills needed to make information available on-line and to use such information is not an issue if 70% of the respondents have some experience with the creation of electronic resources and 82% report using other on-line resources. We no longer have a skill or access issue in the Social Sciences and Humanities.
The difference between those that have mounted research on-line (61%) and those that have attempted or succeeded in publishing in on-line refereed outlets (16%) is worth noting. Why are so few of those that create on-line research resources trying to publish them in refereed outlets considering that for Question 32 65% responded that they are rewarded at their institution "for publications that appear only in peer-reviewed academic forums"? Possible reasons for this difference are: a) scholars aren't aware of appropriate refereed outlets for their on-line work, b) scholars are producing research works that do not lend themselves to the existing refereed outlets (they are using the Internet to publish works in progress, for example), c) despite the rewards of peer reviewed publication, scholars choose to also publish in other venues, or d) it is simply easier to mount on-line research locally through a university WWW site than to submit it to another venue.
Compared to similar questions in the Archer Report there is a consistent increase in the percentage of respondents who have created electronic materials for teaching and research purposes. The following table compares the results from the two surveys, though it should be noted that the questions were not worded identically.
Archer 1999 Rockwell / Siemens 2000 Integrated electronic resources into courses 55.7% 70% Published in refereed electronic outlets 10.6% 16% Attempted to publish in refereed electronic outlet 2% 7% Made scholarship available in some way - 61%
Quality of Electronic Materials
When it comes to the quality of electronic materials (Questions 5 to 10) it is clear that respondents still have concerns about the quality and credibility of electronic publications. Only 26% felt that there was no difference between electronic and print materials while 39% disagreed or strongly disagreed. 86% felt that non-electronic outlets were more credible, though 61% felt that peer review ensures similar quality. There is however, a slight shift between the responses we got in our survey and those in the Archer Report that is summarized in this table.
Archer 1999 - Strongly Agree or Agree Rockwell / Siemens 2000 - Strongly Agree or Agree There is no difference in quality between material published electronically and material published in print (non-electronically.) 28.8% 26% Peer review of electronic publishing ensures that its quality is similar to that of non-electronic publishing. 50.6% 61% A problem with electronic publishing is ensuring its long-term accessibility through proper archiving. 74% 74.4% At the moment, publishing in non-electronic outlets is more credible than publishing in electronic outlets. 84.5 86%
It is worth noting that the respondents felt differently about quality and credibility. While a majority (61%) felt that "peer review of electronic publishing ensures that its quality is similar to that of non-electronic publishing", an even larger percentage (86%) still felt "publishing in non-electronic outlets is more credible than publishing in electronic outlets."
There seems to be the perception that whatever the quality of electronic publications they still do not have the credibility of non-electronic ones. The percentage of respondents who agreed with this actually went up slightly from the Archer Report, which is disturbing. One of the reasons acknowledged for the lack of credibility of electronic materials, other than lack of peer review, was the long-term accessibility through proper archiving. 80% agreed or strongly agreed that "A problem with electronic publishing is ensuring its long-term accessibility through proper archiving." Respondents were less worried with the dynamic character of electronic materials. 41% agreed or strongly agreed with the statement that, "The ease with which Web electronic documents can be altered, updated and corrected is a serious threat to scholarly publishing." 31% disagreed or disagreed strongly with this and 27% either didn't know or neither agreed nor disagreed, which suggests that the community is split over the virtues of dynamic documents. It is worth noting in this regard that for Question 12, 62% agreed or strongly agreed that, "Peer review must take into account aspects of the electronic publication that are unique to the medium." It is clear that the dynamic character of the Web will have to be taken into account. We will return below to this issue of credibility.
Peer Review and Electronic Publications
When it came to the questions on peer review, the majority (81%) felt strongly or agreed that peer review for electronic publications should operate "in a way identical to that in traditional publication". This coincides with the answer to Question 6, that peer review would guarantee quality of electronic publications similar to that of non-electronic venues. Further, for Question 13, 92% felt that peer review for the electronic medium should "operate in a manner that guarantees the same qualitative results". Only 1% of the respondents disagreed with this. The respondents overwhelmingly seem to feel that peer review should and could guarantee a level of quality to electronic publications similar to that of print publications.
What is interesting is that 62% felt that peer review for electronic publications should "take into account aspects of the electronic publication that are unique to the medium." It is difficult to know how to reconcile this with the 81% who felt that peer review for electronic publications should operate in an identical way to the processes used in traditional publication unless respondents felt that peer review processes could be identical while still taking into account unique aspects of electronic publication.
Existence and Increase in Available Peer Reviewed Publications
A significant number of respondents (56%) didn't know or didn't agree nor disagree with the statement that "There are effectively peer reviewed electronic publications available to academic authors and readers in most fields." This coincides with the answers to Question 17 where 49% didn't know or neither agreed nor disagreed with the statement that "There are recognizable and prestigious academic publishers who currently offer their imprimatur to electronic publications." It seems that while most authors are presumably aware of traditional peer reviewed venues, a majority do not know about or are not sure about the availability electronic peer reviewed publications. This lack of knowledge or lack of agreement or disagreement coincides with the lack of knowledge we found in our questions about relevant policies at institutions. Knowledge about the existence of peer reviewed electronic publications is one of the issues affecting their use and credibility.
As noted above, while 56% respondents felt that the number of peer reviewed electronic publications is increasing, fewer seemed think that there was a comparable increase in the number of recognizable or prestigious academic publishers willing to offer their imprimatur to electronic publications. Only 39% agreed or strongly agreed with this, while 58% neither agreed nor disagreed or didn't know.
Reputation and Electronic Publications
As for reputation, Question 16, 82% felt that the notion of imprint or press/publisher/journal reputation is as important for electronically published materials as for those in print. For Question 17, 45% agreed or strongly agreed that there were "recognizable and prestigious academic publishers who currently offer their imprimatur to electronic publications", while 49% didn't know or neither agreed nor disagreed. Imprint or reputation is important, but a significant number of people don't know if reputable publishers are publishing electronic documents in their field. This suggests that there is both an opportunity to translate prestige for print publications into an electronic publication reputation and the danger that amateur experiments in electronic publication could affect the reputation of publishers from whom scholars expect quality.
Organizational Background of Respondents of Part III
Some 65 respondents, 19% of the total group, identified themselves as publishers and/or editors of journals that publish academic materials exclusively (Q19: 48%), as an central part of their operation (31%), or as a minor part of a larger operation with other areas of focus (21%). 75% of the respondents worked for organizations that had been publishing academic materials for more than 11 years and 92% of the respondents reported that their organization follows standard academic peer review processes. Of those that follow standard peer review processes most (88%) of them had specific guidelines, and 43% had guidelines approved by an authoritative body. (Q21d: Some of the authoritative bodies mentioned were, Association of Canadian College and University Teachers of English and other Canadian learned societies, the Modern Language Association, the Council of Editors of Learned Journals and the Canadian Association of Learned Journals, various university press editorial boards, the boards of their sponsoring or host institution, AAUP or ACUP, CRSH or FCAR, SSHRC, and guidelines circulated by the HSSFC Aid to Scholarly Publication program. For a full list see the aggregated data appendix.) Overwhelmingly, these respondents stated that their organization had a recognized imprimatur in the academic area it serves (Q21c: Yes 88%, No 12%).
Questions on Peer Review Specifically Intended for Academic Publishers and Journal Editors
Just over a third of the respondents to this part represented organizations that publish material in both print and electronic media (Q21d: 38%). Of those who do publish both print and electronic media, 87% carry out a level of peer review on materials to be published electronically that is equal to that of materials to be published in print (83%) or greater than that of materials to be published in print (4%).
This subset tended to derive their peer review process for these materials from processes of peer review associated with standard academic print publication (Q21f: 91%) or from new processes of peer review that are evolving in the electronic medium (9%), and just under half followed practices set out by an authoritative body (Q21g: Yes 43%, No 57%) such as those noted above (in Q21d), with the exception of one respondent whose institution has a Digitization Review Board. This group felt that establishing a peer review process for electronic materials was a straightforward or unremarkable process (Q21h: Strongly Agree 13%, Agree 35%, Neither Agree Nor Disagree 26%), some two thirds being confident that the peer review process they adopted for electronic materials offered the same qualitative assurances as their process for print materials (Q21i: Strongly Agree 22%, Agree 43%, Neither Agree Nor Disagree 22%); none felt that the peer review process they adopted for electronic materials did not offer the same qualitative assurances as their process for print materials, and none felt that concerns about carrying out peer review processes on electronic materials were a barrier to their organization publishing electronically (Q21j).
Questions Specifically Intended for Academic Administrators and Reviewers
Some 294 respondents, or 88% of the group, identified themselves as publishing scholars or academic authors. While under half this group published only in peer-reviewed academic forums (Q31: Yes 39%, No 61%), two-thirds registered that they were rewarded at their institution only for publications that appeared in peer-reviewed academic forums (Q32: Yes 65%, No 33%) with just under a third stating that they had, themselves, been rewarded at their institution for publishing in academic forums that are not peer reviewed (Q33: Yes 29%, No 45%, Don't Know 24%). Roughly half either stated that their institution did not have a policy concerning how electronic documents are to be evaluated in tenure, salary, and promotion procedures (Q34a: 47%) or did not know (48%), with 4% stating that their institution did have such a policy; answers to a similar question, 34b, registered approximately the same.
Of this group, 40% had published in an electronic forum (Q35): registering the types of electronic publications they had published in, 35% responded that they had published in an electronic-only journal, 6% in an electronic-only monograph, 48% in a print publication (journal, monograph, newspaper, etc.) also distributed in electronic form, and 40% in other electronic venues. Of the most prominent of these venues, 60% published peer-reviewed pieces (Q36) -- two thirds noting that peer review was a significant factor for them to publish in that venue (Q36a: 69%) and just over half noting that imprint was a significant factor for them to publish in that venue (Q36b: 54%). The majority found that their impression of the level of peer review carried out on their material was equal to what you would expect for a print publication (Q36c: 79%), with a small number finding that it was less or greater than they would expect for a print publication (Less 11%, Greater 3%). Notable, however, is that of those who had published in a peer reviewed electronic format, only 54% was able to claim that the work they published in this way was considered as if it were equal to print publication by those who reviewed scholarship in their academic unit (36d: 54%).
Of those who had not published in an electronic forum (Q35: 60%), one third were discouraged from publishing electronically because they felt there was a lack of peer reviewed electronic venues for them (37b: 31%), and approximately the same number felt discouraged from such publication because there was a lack of electronic publication venues with an established imprimatur (Q37c: 34%); even so, the majority of those who had not yet published in electronic form did not find these factors discouraging, and just over half were also aware of hybrid forms of print electronic publication in their area, such as peer reviewed print publications that archive and redistribute in electronic form (Q37a: 56%).
Some 150 respondents, 45% of the group, identified themselves as academic administrators or reviewers (including those who functioned as members of the HSSFC Aid to Scholarly Publication Programme, the SSHRC Grants Programmes, and others).
Overwhelmingly, this group felt that they considered peer review to provide a standard level of qualitative assurance (Q22: Yes 90%, No 10%), the majority basing their understanding of the peer review process on standard scholarly practice (Q23: 93%), with several suggesting that it derived also from guidelines of the professional organizations akin to those mentioned above (see Q21b).
A strong majority stated that they were inclined to evaluate academic materials on their own merits, regardless of the medium in which they are published (Q29: Strongly Agree 35%, Agree 45%), with the same number noting that they were inclined to evaluate materials that bear equal levels of peer review and equivalent imprimatur on their own merits, regardless of the medium in which they are published (Q30: Strongly Agree 29%, Agree 51%).
Two-thirds felt that, for electronic materials, their being subject to processes of peer review was the most significant factor ensuring their quality (Q25: Strongly Agree 22%, Agree 45%), but less felt that, for materials published in the electronic medium, publisher's imprint also was a significant factor in guaranteeing their quality (Q26: Strongly Agree 4%, Agree 51%, Neither Agree Nor Disagree 17%, Disagree 7%, Don't Know 13%); note that, in evaluating any published materials, they considered the publisher's imprimatur to provide a degree of qualitative assurance that is less than that provided by peer review (Q24: 54%) or equal to that provided by peer review (39%).
Of the electronically-published materials they evaluated, a significant number didn't know if those materials were peer reviewed. (Q27: Strongly Agree 5%, Agree 22%, Neither Agree Nor Disagree 27%, Disagree 3%, Disagree Strongly 1%, Don't Know 28%) A similar indifference was registered when asked the same about the electronic materials they evaluated bearing a significant imprint, though almost half did not mark an opinion at all (Q28: Agree 17%, Neither Agree Nor Disagree 3%, Disagree 7%, Don't Know 27%; the rest did not answer).
Institutional Reward for Publication
For Part V we had 294 respondents or 88% of the total group. A minority of the respondents (39%) publish only in peer reviewed forums while 61% responded that they do not. Of those responding to this Part of the questionnaire 65% are rewarded for publications that only appear in peer reviewed forums while 29% reported having been rewarded for non peer reviewed forums.
Policies Concerning Electronic Publication
What was striking was how few respondents knew if their institutions had policies regarding how electronic documents are to be evaluated in tenure, salary, and promotion procedures. Only 4% responded that their institution had such a policy and only 5% responded that their department had such a policy. Close to half the respondents (48%) reported not knowing if their institution has such a policy and 33% reported not knowing if their department has a policy. Either such policies are there, but are not well advertised, or administrators feel that electronic publications should be treated like traditional publications, or institutions have not dealt with the issue. The lack of institutional policies may explain concerns people have about publishing in electronic venues.
Questions for Respondents Who Have Published Electronically
Of those who responded to this section (Publishing Scholars and Academic Authors) 35% have published in an electronic-only journal, 6% and electronic-only monograph, 48% have published in print publications also distributed electronically and 40% in another electronic venue.
We had 117 respondents or 40% of the subset respond to Question 36 and 70 respondents answered "Yes" - that their most prominent electronic publication was peer reviewed. These 70 respondents went on to answer Questions 36a to 36d. Of these 69% considered peer review a significant factor for choosing to publish in this venue while only 54% considered the publisher's or journal's imprint significant. 79% of these considered the level of peer review equal to what they would expect in a print publication while only 11% considered it less that what they would expect. By contrast only 54% of these thought that the work published in electronic format would be considered the equal of a print publication by those that review scholarship in their area.
Questions for Respondents Who Have Not Published Electronically
Of those who had not published in an electronic forum 56% were aware of print publications that archive and redistribute in electronic form and 43% were not. As for what discouraged the respondents from publishing in an electronic forum, 31% felt the lack of peer review discouraged them from publishing electronically (61% did not respond that this was a factor) while 34% felt the lack of venues with an established imprint or name was a discouraging factor (and 57% did not feel that was a factor).
Existence of Copyright and IP Policies
While 68% of respondents reported that their faculty or institution has a policy on the ownership of copyright (24% didn't know), only 20% are at an institution that has a policy on the ownership of copyright in the electronic medium (61% didn't know). As for intellectual property policies, 60% reported that their institution has a policy concerning its assignment (32% didn't know.) The large percentage of respondents who do not know if their institution has a policy regarding copyright in the electronic medium is significant and coincides with the answers about policies regarding the evaluation of electronic documents (see Policies Concerning Electronic Publication above).
Ownership of Copyright
As for who should hold copyright on scholarly work, 79% felt the author should hold it when the medium was print, while 82% felt the author should hold it when the medium was electronic. The rest felt either that the institution the author works for should hold copyright (2%) or that it should reside in the public domain (5%). 69% felt copyright is valuable or very valuable in the field of scholarly work while 27% felt it was of little value.
Security of Copyright
A significant number (51%) of the respondents felt copyright in the print medium was insecure or very insecure (46% felt it was very secure or secure). This increased when asked about copyright in the electronic medium - 81% felt copyright in the electronic medium to be insecure or very insecure while only 12% felt it was secure or very secure. Given the value placed on copyright in scholarly fields by the respondents, the perception of it being insecure in the electronic medium is significant. The respondents generally both value copyright and feel that it is difficult to protect in the electronic medium.
The questionnaire which was used for this survey was designed jointly by the research team, first collated and administered under the leadership of Joanne Buckley (McMaster U), then developed further and its delivery overseen by Ray Siemens (Malaspina U-C), Michael Best (U Victoria), and a team based at both the University of Victoria and Malaspina University-College.
We used as a starting point the questions in the report by Keith Archer (noted above) on their survey findings. In particular we tried to gather the same demographic information and use the same list of academic disciplines.
New questions were solicited from members of the team and, once the questionnaire was finished, it was translated into French and put on the WWW in July of 2000.
The on-line questionnaire is available, in both official languages, as French Appendix C1 and English Appendix C2.
We received a small number of complaints about the French translation and our translators choice of computer vocabulary.
The questionnaire and related information about how the results would be stored were submitted to the Research Ethics Boards of McMaster University, the University of Victoria, and Malaspina University-College for approval. A research assistant was hired to conduct the survey and the majority of the responses were collected in August and early September 2000. (We stopped gathering responses on September 8.)
Four methods were used to generate potential respondents.
- 1. In order to have a representative disciplinary, geographic, and linguistic sample we developed a list of universities and colleges to contact. We started with the categories of institutions that Macleans uses (as an arbitrary starting-point) and added a category for Emerging Institutions. This gave us four categories of institutions; Comprehensive, Medical Doctoral, Primarily Undergraduate, and Emerging. We then selected a sample of institutions evenly distributed across these categories and also evenly distributed geographically. Institutions participating in this project were removed. For each institution chosen we then randomly chose three or four departments from the list of disciplines (see the questionnaire) to make sure there was an even distribution of disciplines. We then contacted an administrator at each of these departments and asked for the names of three or four members who were available. These people were then contacted by phone and asked if they would complete the questionnaire. The people contacted were given the choice of completing the questionnaire by phone, on the WWW or on paper.
- 2. The HSSFC specifically asked us to poll their own reviewers. They distributed a message inviting reviewers to complete the questionnaire.
- 3. SSHRCC offered that we contact the reviewers of their grant programs. SSHRC provided us with a list of people to invite to complete the questionnaire.
- 4. Finally, we sent an e-mail message to each of the presidents of the learned societies that are part of the HSSFC inviting them to forward, if appropriate, a copy of the invitation to their members.
We had 336 valid respondents. The data collected has been aggregated and made available by Lynne Siemens as part of this report. This is available as Appendix D.
The following are some tentative conclusions that we can draw from the responses to our questionnaire.
- 1. Most respondents have experience using electronic resources and making scholarship available electronically, though only 16% had published in peer reviewed electronic venues.
- 2. A significant number of respondents feel that there is a difference in quality between electronic and traditional publications and most feel that non-electronic outlets are more credible. Some of the possible reasons for this perception are:
- 2.1 The dynamic character of the WWW where documents can be altered easily.
- 2.2 The long-term accessibility and archiving of electronic publications.
- 2.3 The lack of peer reviewed electronic publications or the lack of visibility of those that are there.
- 2.4 The lack of publications that bear a reputable imprint or the lack of visibility of those that are there.
- 2.5 The lack of institutional policies (see next point.)
- 3. Institutions should develop policies regarding the evaluation and copyright of electronic publications if they wish to encourage such publication. Where institutions have such policies they need to better educate people about these policies.
- 4. The existence of suitable peer reviewed or reputable outlets for electronic publications needs to be better advertised in the scholarly community. Many academics either do not feel there are suitable venues or they are not aware of the ones that are there.
- 5. Despite concerns about quality and credibility, most reviewers are likely to judge an electronic publication on its own merits especially if it is in a peer reviewed electronic publication or from a reputable publisher.
- 6. Peer review and the reputation of the publisher are important to the credibility of electronic publications. In particular, peer review processes that are the equivalent to those followed for print publications, while taking into account the electronic medium, can guarantee an equivalent level of quality for electronic publications.
- 7. Authors are more likely to submit documents to electronic publications if they are peer reviewed and/or published by a reputable organization.
- 8. Given concerns about the long-term accessibility of electronic publications it is important that standards and archiving services be developed to reassure authors that their publications will continue to be available over the long term. It is possible that well advertised and well understood standards for archiving would alleviate concerns about the credibility of electronic publishing which might then lead to such publications being taken seriously for tenure and promotion.
- 9. Authors concerned about the evaluation of electronic publications should consider taking the following steps:
- 9.1 Publishing in peer reviewed electronic publications and documenting the review processes followed for those that evaluate them.
- 9.2 Publishing in electronic publications from reputable publishers or institutions.
- 9.3 Publishing in forums that have a well documented archival process.
- 10. Further work needs to be done analyzing this extensive data set and supplementing the questionnaire with a new one aimed at identifying why different groups might have answered certain questions differently. In particular the following follow-up studies could be run:
- 10.1 Where we found a significant correlation between answers for questions and the gender of the respondent we could devise a follow-up survey.
- 10.2 Where we found a significant correlation between answers for questions and the discipline of the respondent we could run a follow-up survey.