The Credibility of Electronic Publishing

 

Peer Review and Imprint 

Draft: June 11, 2001
Jean-Claude Guédon, with Raymond Siemens


Introduction

The history of scholarly publishing generally conflates the creation of the Journal des Sçavans in Paris and that of the Philosophical Transactions of the Royal Society of London as two parallel events separated by a few weeks referring. This treatment is somewhat inaccurate because the essential part of the Parisian journal was to report news of ideas, theses, theories, discoveries, observations, etc... in the best tradition of mail gossip typical of the exchanges within the Republic of Letters. The Philosophical Transactions, on the other hand, corresponded to a selection of all the presentations made at the Royal Society and, as such, it can be emblematically used as the beginning of the peer review process. By contrast, the Journal des Sçavans behaves more like Scientific American nowadays.

As scholarly publishing developed, various habits of selection occurred until a kind of « best practice » stabilized around the idea of peer review. Whenever a journal became more than the particular vehicle for an individual or a tightly-defined school, it resorted to having a peer-review process in place. As much a way to protect the editor-in-chief and his (or her) decisions as an attempt to guarantee some objective and serene judgment in the midst of intense competition, the peer-review process nevertheless rested on procedures that put the editor-in-chief in absolute control, albeit in an acceptable way. Receiving a submission, the editor can immediately reject especially if it can be argued that the manuscript does not fit with the journal coverage, or its style, etc. If the manuscript is accepted for review, it is then dispatched by editors to various reviewers. The choice of these reviewers is his or her own and one can easily imagine that a suitable choice ensures a good deal of the outcome. This arbitrariness is facilitated by the fact that, more often than not, the correspondence between editors and reviewers is done privately and cannot be consulted in a public manner. In short, the peer-review process, while often used absolutely honestly and to good effects, can nevertheless harbour enormous zones of occult arbitrariness. For this reason, some journals have attempted to make their review process more objective, more rigorous, more neutral, for example by having the manuscripts anonymized before sending them to the reviewers. Alas, anyone capable of evaluating research in a given specialty generally knows that specialty sufficiently to identify the probable author of the manuscript under review. These attempts point more to the deficiencies of the peer review process than to their potential cures.

This said, peer review, despite its weaknesses, cannot be easily dismissed. A little like democracy, it is the least objectionable form of evaluation we have. And, of course, all research communities rely on it as a way to keep the process of evaluation as good as possible, despite everybody's knowledge that mistakes are constantly made. And the process is kept because all research communities also know that other concurrent forms of evaluation are at work at the same time or later in the career of a paper. The advent of the Science Citation Index (SCI) in 1963 has allowed measuring something called the impact factor of an article by looking at how this papers is cited over a certain period of time. This in turn has allowed establishing impact factors for collections of articles generally known as journals and these are even published by ISI, the company producing SCI (see http://www.isinet.com/isi/hot/essays/7.html).  However, at this juncture, a strange aberration became custom : individual scientists being evaluated for promotion began using the journal impact factor where they had published as an approximation of their own impact factor. In other words, the limousine in which a given scientist managed to slip into became the standard by which that scientist was personally evaluated. No matter if it was cold fusion or water memory, the appearance in Nature or Science ensured a very positive evaluation for the individual scientist. In other words, « L'habit fait le moine »!

The direct consequence of this practice has been to build up the importance of the imprint over the peer review process and has contributed to warping the whole evaluation results in a not insignificant way. Add to this that, since the invention of Bradford's Law in librarianship, back in 1933, the notion of core journals has become ever more prominent. SCI, resting as it does on a limited set of a few thousand titles, in effect has materialized this hitherto fuzzy notion of core journal by publishing the list. Impact factors and belonging to the core list are of course correlated and one can see from there what economic and power consequences can ensue from this setup. Let us not forget that a Social Science Citation Index also exists and although its present importance is far more limited than its natural science counterpart, it is nevertheless pointing to some future which will be difficult to avoid.

The consequences of all this are twofold. The first one does not concern us directly as it relates to commercial publisher strategies to buy or engineer core journals. Social science and humanities journals (SSHJ) have largely escaped this fate so far because the notion of core journals is more difficult to establish in fields where several paradigms may coexist at the same time. As a result, the market for SSHJ has proved far more elastic than its natural science counterpart, and this has prevented the creation of de facto cognitive monopolies that have translated into enormous subscription or licensing costs in the science, technology and medical science (STM) publications. However, the writing is on the wall and it is only a question time before this area follows the example already set up in STM. A quick glance at the evolution of certain fields such as economics and quantitative social sciences reveals similar trends, albeit with a delay.

In conclusion of this part, peer review is so important not so much because it allows for the selection of the best materials, but rather because it is widely used to manage personal careers. And the practices generally in place have contributed to creating a very strong link between individual evaluation and imprint through the conflation of individual value with journal impact factors. No wonder that electronic publishing is viewed with caution as it obviously is bound to modify a number of rules that will affect power structures within scientific communities and economic revenues for a number of large, rich and powerful commercial publishers. It is important to remember this in reviewing the literature on the topic as much of it is embedded in this latent struggle for preeminence or revenue streams.

 

Electronic Scholarly Publishing and Peer Review: What Is In the Offing?

Scholarly journals have progressively come to play three roles, one of which, in the print form, has also and almost gone out of business : the communication rôle at the cutting edge of knowledge. Beside this rôle, one can also note an archival rôle and the already analyzed career-management through the imparting of some « symbolic capital ». In the print world, the latter two roles are largely conflated to the point that it takes some effort to reconstruct the complete institutional and social value of journals. Moving into the digital environment unpacks this system and leads to analyzing each element in its own right.

Communication

Scholars have long given up on journals as a way to keep at the forefront of research. Printing delays, most of the time, make these results almost obsolete by the time they actually hit the libraries' shelves or - although more rarely because of the cost - the researcher's desk. People know their « invisible college » and they keep up with what is going on in a particular strategy through the interchange of pre-prints. Laboratories put themselves on the mailing lists of other laboratories and they automatically receive all that has been vetted by prior internal review before being released in a pre-print form to the outside world. (Most scientific articles, unlike SSH articles, are the result of large teams and, as a result, anything submitted to a journal has already been intensely reviewed by internal seminars. Similar trends are developing in SSH, but more slowly and less intensely, particularly in the humanities which remain a largely individualistic enterprise.)  Conferences, seminars and direct communications (including phone, fax and e-mail) complete the picture.

The memory of science is made up of the large library collections that have been lovingly maintained by a number of institutions throughout the world. One must note, however, that due to lack of space, libraries tend to get rid of older, less used documents. One must also note that despite bibliographies and other search aides, the retrieval tools of information in the print world are crude and painfully slow.

The prestige, authority and visibility of individual papers come largely through the value attributed to the periodical vehicle which harbours them, as well as the name of the lead author. Being printed in a core journal certainly lends a presumption of value to an article and this is something print papers do very well, too well perhaps in view of some the earlier remarks above.

In designing an electronic scholarly publishing system, it is important to separate these functions and see how each layer or level can be accommodated. Ideally, a design that could satisfy all three functions, while keeping them perfectly autonomous with regard to the other levels, would be ideal, of course.

Recent history shows that a consensus is being reached on the communication needs. From the commercial publishers such as Springer, Academic Press or Elsevier to the Open Archive Initiative, it is by now completely clear that the present basic unit of scientific communication is the article and that the task is to create an archive. ("Present" basic unit, but not necessarily future. It will be interesting if the step-like, incremental nature of scientific knowledge will tend to decrease as digitized means of communication facilitate and in fact invite mind synergies rather than validated authorship.)  The most recent and telling element sin this evolution are e-biomed and, perhaps even more spectacular, the Elsevier chemistry preprint server, Chemweb (see http://www.chemweb.com/preprint?URL=/CPS). From Paul Ginsparg to the big commercial publishers, the notion that the communication of science has little to do with the printing of knowledge, or even its prior validation, seems to be coming strongly to the fore.

This trend which is now in full swing in the natural sciences, is beginning to spread also to some SSH areas. Economics is a good example of this. RePec (see http://repec.org/) is an archive maintained by over 100 volunteers spread over the planet in order to enhance communication in economic studies. We may expect it to spread over to other fields of studies in the next few years.

Archiving

Paper has been moderately adequate for the preservation of knowledge. Durable until about 1860, paper making processes after yielded highly acidic papers that have the bad habit of self-destructing within a century. Recently, attention has been brought to bear on this problem and many publications since at least the '80's have been printed on reasonably acid-free paper.

The digital world is much more vulnerable to archiving questions, to the point that many question the ability of preserving anything in digital formats beyond a dozen years. This said, a convergence is appearing about the necessity to multiply sites so as to increase the durability of digital formats. At Stanford University, in the context of HighWire Press, the LOCKSS (Lots of Copies Keep the Stuff Safe) project, led by Vicky Reich, studies this very strategy while, in parallel, IBM scientists are trying to promote new forms of legislation that would grant special copyright privileges to bona fide archives so that the number of digital mirrors could increase considerably and not remain exclusively in private hands as is the case at this moment with the commercial sites such as Science Direct.

The credibility of journals depends also on their being maintained as a durable archive and this question, therefore, is more than an arcane concern. Interestingly enough, one can see that documents that can circulate freely are more immune to time than documents restricted by copyright rules that ensure a very limited number of copies at best, all in the hands of a single institution, which is the situation bred by the site-licensing system.

If the durability of digital material requires innovative approaches to be solved, the retrieval of information is much enhanced in the digital world. Search engines improve all the time and many can be applied to electronic corpora. On this score, digital material win hands down. The impact of this capacity on their credibility, however, is indirect at best although it cannot work but positively. How positively is hard to say, although enhanced retrieve-ability leads to greater visibility, which feeds directly into prestige and authority.

Authority Lending, Prestige Building and Visibility

Although core journals do help bring about all three characteristics above, each one deserves being looked at in a specific way.

Visibility is a function of prestige and technology. A very prestigious journal will always be very visible, but an average-prestige journal can see its visibility enhanced by technology as well as a change in language. In the bilingual journal on geology and physical geography published in the Université de Montréal, the same author sees his articles cited twice as much if he or she publishes in English rather than French. A journal that is better distributed, promoted, located within good baskets of similar periodicals, well referenced by many bibliographies, etc., will obviously be more visible and there is no question that a powerful publisher capable of printing lavish catalogs, offering easy subscription services to dozens of journals through one procedure, etc., will make a journal more visible than would a small university press or a small learned society. Big publishers have often played on this capability to convince interesting journals supported by weak infrastructures to move into their private collection.

Small, weak journals will probably see their visibility increase quite a bit through electronic publishing, and this is probably true for a fair proportion of HSS journals in Canada. And this increased visibility will also probably result in enhanced prestige and authority. However, one must keep in mind that such elusive variables as prestige or authority are built only gradually and that it takes time. As a result, a paper journal going toward a digitized version can expect temporary set backs in visibility, particularly if its reader community is known to be somewhat more resisting to technological innovations than is the norm. This also means that particular attention must be paid to the transition period through which we are presently going.

Prestige is very much tied to tradition and/or the prestige of the evaluators themselves. It is clear that if an electronic journal in literature were supervised by several Nobel prizes that would allow themselves to be actively engaged in its day to day activities, there is no question that such a journal would not suffer from any lack of prestige, on the contrary. The problem that new electronic journals face is that they have not had real time yet to establish a reputation and, as the technology, until recently, filtered the reading public in way that had nothing to do with their excellence, but rather with their proficiency in the new technologies, such journals were suffering -- and may still be -- suffering from an artificially truncated audience. This situation is evolving positively and ever more rapidly now, but the question nevertheless remains to some extent.

Authority is different from prestige and, on that score, digital material do not have to envy their paper counterparts. In effect, any electronic journal that would have authoritative editorial boards, that would develop particularly rigorous (and also innovative) peer review processes and that would manage to throw effective light on their best published pieces would certainly gain unquestionable authority without many problems. And if a journal enjoys visibility plus authority, prestige is not far behind.

Imprint, in a sense, is short hand for prestige. I have an article published in the American Historical Review and that means a lot more than the Canadian Historical Review for some mysterious reason, one of which being the relative size of the communities and the degree of competition to be accepted (measured in rejection rates). Presently, the Canadian and USA efforts work in parallel and with communities that overlap only slightly. However, in a distributed, digitized environment, being present at the same time as the other journal is not a threat, but an advantage on the contrary, as the strategy of web-rings amply demonstrates. In other words, articles from these two journals (and many other similar journals) could be joined in the same, large, archive. But what of Canadian individuality, identity, etc., questions that loom large in the HSS field (and factually repeated, so to speak, in the French-language journals of Québec and the rest of French Canada)? The answer, actually, is simple :nothing prevents one or several Canadian editorial board to select specific articles that they find particularly interesting and do or invent all they want to bring attention to these publications. In other words, electronic publishing distinguishes between the phase where documents are placed at the disposal of the public (publishing proper) and the phase where « distinctions » are being attributed. It used to be that being printed was « the » distinction; electronic publishing changes this and leads us to think of the distinction phase completely separately from the publishing phase.

However, doing so changes the means by which distinction is imparted and imparting distinction is a sure sign of power. In other words, those who now hold that privilege are afraid of losing it (« gate keepers ») and they will every possible argument to protect it without, if possible, ever mentioning it.

Text:  J.-C. Guédon (September 18, 2000)

A Brief Overview of Questionnaire Input Related to Peer Review and Imprint

The questionnaire compiled and administered as part of this report asked a significant number of questions related to peer review and imprint, and asked those questions of several different target groups.  The answers of those groups are surveyed, briefly, below, and are used to suggest certain trends of thinking within those groups.

To see the questionnaire itself, in both official languages, please see Appendix C1 and Appendix C2.  For a report on the questionnaire, please see Section 5 of this report, and for a full analysis of the data collected by the questionnaire, please refer to Appendix D.

 

Questions Asked to All Respondents

While there was very strong agreement with the statement, "At the moment, publishing in non-electronic outlets is more credible than publishing in electronic outlets" (Q8: Strongly Agree 45%, Agree 41%; compare Archer's results of 28% and 56.5%, respectively), only slightly more than half our respondents found that peer review of electronic publication was a guarantee of its quality (Q6: Strongly Agree 21%, Agree 40%; compare Archer's results of 10.2%, 49.4%).  An overwhelming majority felt that "Peer review processes for academic publication in the electronic medium should operate in a way identical to that in traditional academic publication" (Q11: Strongly Agree 46%, Agree 35%), but also many were receptive to the notion that "Peer review must take into account aspects of the electronic publication that are unique to the medium" (Q12: Strongly Agree 21%, Agree 41%) -- so long as those peer review processes operated in a manner that would guarantee the same qualitative results as they do in traditional academic publication (Q13: Strongly Agree 60%, Agree 32%).  Even so, responses further suggested a concern with appropriate peer-reviewed venues for publication.  While a slim majority agreed that that "The number of effectively peer reviewed electronic publications available to academic authors and readers is increasing in most fields" (Q15: Strongly Agree 10%, Agree 46%), a good number simply didn't know (31%) or neither agreed or disagreed with that statement (11%); this concern is echoed, further, by responses to the statement "There are effectively peer reviewed electronic publications available to academic authors and readers in most fields" (Q14: Neither Agree Nor Disagree 14%, Disagree 17%, Don’t Know 42%).

As with peer review, a good majority felt that a publisher's imprimatur, or imprint, was important; an overwhelming majority agreed with the statement "The notion of imprint–or press/publisher/journal reputation carried via its name–is as important for materials published electronically as it is for those published in print" (Q16 Strongly Agree 33%, Agree 49%).  Still, awareness of venues able to bestow a recognized or prestigious imprint to work in electronic format was high -- but suggestive also of a growing awareness of such venues.  To the statement "There are recognizable and prestigious academic publishers who currently offer their imprimatur to electronic publications" we received a split of responses suggestive both of some agreement and, also, indifference or lack of knowledge (Q17 Strongly Agree 11%, Agree 34%, Neither Agree Nor Disagree 13%, Don’t Know 36%); responses to the statement "The number of recognizable and prestigious academic publishers willing to offer their imprimatur to electronic publications is increasing" were similarly suggestive (Q18 Strongly Agree 10%, Agree 29%, Neither Agree Nor Disagree 14%, Don’t Know 44%).

 

Questions Specifically Intended for Academic Publishers and Journal Editors

Some 65 respondents, 19% of the total group, identified themselves as publishers and/or editors of journals that publish academic materials exclusively (Q19: 48%), as an central part of their operation (31%), or as a minor part of a larger operation with other areas of focus (21%). 

As a group, the majority worked with an organization that had been publishing academic materials for some time (Q20: less than 2 years, 9%; 3 to 10 years, 12%; 11 to 20 years, 26%; more than 20 years, 49%).  Overwhelmingly, these respondents stated that their organization had a recognized imprimatur in the academic area it serves (Q21c: Yes 88%, No 12%) and noted that they followed standard academic peer review processes when evaluating materials under considering for publication (Q21: Yes 92%,  No 6%) and, of those that did, their organizations had specific guidelines for peer review (Q21a: Yes 88%, No 12%) -- just under half of them having those guidelines based on peer review methods approved by specific authoritative bodies (Q21b: Yes 43%, No 53%), bodies that had approved or suggested practices as Association of Canadian College and University Teachers of English and other Canadian learned societies, the Modern Language Association, the Council of Editors of Learned Journals and the Canadian Association of Learned Journals, various university press editorial boards, the boards of their sponsoring or host institution, AAUP or ACUP, CRSH or FCAR, SSHRC, and guidelines circulated by the HSSFC Aid to Scholarly Publication program.

Of that group, just over a third represented organizations that publish material in both print and electronic media (Q21d: 38%) -- a very strong majority of that group carrying out a level of peer review on materials to be published electronically that is equal to that of materials to be published in print (Q21e: 83%) or greater than that of materials to be published in print (4%).  They tended to derive their peer review process for these materials from processes of peer review associated with standard academic print publication (Q21f: 91%) or from new processes of peer review that are evolving in the electronic medium (9%), and just under half followed practices set out by an authoritative body (Q21g: Yes 43%, No 57%) such as those noted above (in Q21d) -- with the exception of one respondent whose institution has a Digitization Review Board.  This group felt that establishing a peer review process for electronic materials was a straightforward or unremarkable process (Q21h: Strongly Agree 13%, Agree 35%, Neither Agree Nor Disagree 26%), some two thirds being confident that the peer review process they adopted for electronic materials offered the same qualitative assurances as their process for print materials (Q21i: Strongly Agree 22%, Agree 43%, Neither Agree Nor Disagree 22%); none felt that the peer review process they adopted for electronic materials did not offer the same qualitative assurances as their process for print materials, and none felt that concerns about carrying out peer review processes on electronic materials were a barrier to their organization publishing electronically (Q21j).

 

Questions Specifically Intended for Academic Administrators and Reviewers

Some 150 respondents, 45% of the group, identified themselves as academic administrators or reviewers (including those who functioned as members of the HSSFC Aid to Scholarly Publication Programme, the SSHRC Grants Programmes, and others).

Overwhelmingly, this group felt that they considered peer review to provide a standard level of qualitative assurance (Q22: Yes 90%, No 10%), the majority basing their understanding of the peer review process on on standard scholarly practice (Q23: 93%), with several suggesting that it derived also from guidelines of the professional organizations akin to those mentioned above (see Q21b). 

A strong majority stated that they were inclined to evaluate academic materials on their own merits, regardless of the medium in which they are published (Q29: Strongly Agree 35%, Agree 45%), with the same number noting that they were inclined to evaluate materials that bear equal levels of peer review and equivalent imprimatur on their own merits, regardless of the medium in which they are published (Q30: Strongly Agree 29%, Agree 51%).

Two-thirds felt that, for electronic materials, their being subject to processes of peer review was the most significant factor ensuring their quality (Q25: Strongly Agree 22%, Agree 45%), but less felt that, for materials published in the electronic medium, publisher’s imprint also was a significant factor in guaranteeing their quality (Q26: Strongly Agree 4%, Agree 51%, Neither Agree Nor Disagree 17%, Disagree 7%, Don’t Know 13%); note that, in evaluating any published materials, they considered the publisher’s imprimatur to provide a degree of qualitative assurance that is lesser than that provided by peer review (Q24: 54%) or equal to that provided by peer review (39%).

Of the electronically-published materials they evaluated, many were unsure if, or thought it unremarkable that, those materials were peer reviewed (Q27: Strongly Agree 5%, Agree 22%, Neither Agree Nor Disagree 27%, Disagree 3%, Disagree Strongly 1%, Don’t Know 28%); less strong explicit indifference was registered when asked the same about the materials they evaluated bearing a significant imprint, though almost half did not mark an opinion at all (Q28: Agree 17%, Neither Agree Nor Disagree 3%, Disagree 7%, Don’t Know 27%; the rest did not answer).

 

Questions Specifically Intended for Publishing Scholars and Academic Authors

Some 294 respondents, or 88% of the group, identified themselves as publishing scholars or academic authors.  While just under half this group published only in peer-reviewed academic forums (Q31: Yes 39%, No 61%), two-thirds registered that they were rewarded at their institution only for publications that appeared in peer-reviewed academic forums (Q32: Yes 65%, No 33%) with just under a third stating that they had, themselves, been rewarded at their institution for publishing in academic forums that are not peer reviewed (Q33: Yes 29%, No 45%, Don't Know 24%).  Roughly half either stated that their institution did not have a policy concerning how electronic documents are to be evaluated in tenure, salary, and promotion procedures (Q34a: 47%) or did not know (48%), with 4% stating that their institution did have such a policy; answers to a similar question, 34b, registered approximately the same.

Of this group, 40% had published in an electronic forum (Q35): registering the types of electronic publications they had published in, 35% responded that they had published in an electronic-only journal, 6% in an electronic-only monograph, 48% in a print publication (journal, monograph, newspaper, etc.) also distributed in electronic form, and 40% in other electronic venues.  Of the most prominent of these venues, 60% published peer-reviewed pieces (Q36) -- two thirds noting that peer review was a significant factor for them to publish in that venue (Q36a: 69%) and just over half noting that imprint was a significant factor for them to publish in that venue (Q36b: 54%).  The majority found that their impression of the level of peer review carried out on their material was equal to what you would expect for a print publication (Q36c: 79%), with small number finding that it was less or greater than they would expect for a print publication (Less 11%, Greater 3%).  Notable, however, is that of those who had published in a peer-reviewed electronic format, only 54% was able to claim that the work they published in this way was considered as if it were equal to print publication by those who reviewed scholarship in their academic unit (36d: 54%).

Of those who had not published in an electronic forum (Q35: 60%), one third were discouraged from publishing electronically because they felt there was a lack of peer-reviewed electronic venues for them (37b: 31%), and approximately the same number felt discouraged from such publication because there was a lack of electronic publication venues with an established imprimatur (Q37c: 34%); even so, the majority of those who had not yet published in electronic form did not find these factors discouraging, and just over half were also aware of hybrid forms of print electronic publication in their area, such as peer-reviewed print publications that archive and redistribute in electronic form (Q37a: 56%).

Summary: R. Siemens (June 11, 2001).

Recommendations

Recommendations on the topic of peer review and imprint remain largely implicit, in large part because of the complex web of associated and evolving issues that inform our understanding of academic quality as it has existed in print publication for some time, and has been given via peer review and publishers' imprimatur.  That said, a suggestive overview of current thought is provided, as is an annotated bibliography and a summary of the data gathered on these topics via the questionnaire that was circulated among a representative sample of publishers, scholars, representatives of the Aid to Scholarly Publication Programme, reviewers for SSHRC and other bodies, university administrators and academic authors.  This overview, these works, and the questionnaire results will, together, act as the best guide to issues relating to peer review and imprint in electronic scholarly publication.

Explicitly expressed, our findings suggest this: where similar qualitative guarantees exist -- guarantees of peer review, publisher imprimatur, and beyond -- electronic academic publication must given the same value as print publication.  Indeed, it is the mechanism of peer review (as well as that which lies behind the process of assigning imprimatur) that makes the claim of qualitative assurance, and not the medium.

Academic administrators must recognise this, and academic authors concerned about the evaluation of their own electronic publications should bear this in mind when they select publication venues appropriate to their work and its audience.

This is a recommendation made for the immediate present; as the dynamics of scholarly publication change with the increased acceptance of the electronic medium in the dissemination of academic work, issues related to the qualitative assurance of published materials in all media are sure to change as well.

Select and Annotated Bibliography

The body of literature touching on the subjects of peer review and imprint -- as they relate both to materials in print and electronic form -- is overwhelmingly large.  The below bibliography does not (and cannot), therefore, attempt to be exhaustive.  Rather, it attempts to highlight a number of works central to, and of most use to, those interested in the area(s) treated by this section; with few exceptions, each of these works provide extensive lists of further references.  Brief annotations follow most selections.

  1. Arms, William Y.  "The Institutional Implications of Electronic Information."  Presented at Technology, Scholarship, and the Humanities: The Implications of Electronic Information (September 30 - October 2, 1992; U of California, Irvine).  Rptd. Coalition for Netoworked Information <URL: http://www.cni.org/docs/tsh/Arms.html>.

    • Suggests implications of electronic publication and other types of electronically-facilitated exchanges on infrastructure and policy, and beyond; treats peer review.

  2. Armstrong, C.J. and Ray Lonsdale.  The Publishing of Electronic Scholarly Monographs and Textbooks. Aberystwyth: Centre for Information Quality Management, Information Automation Limited, and Department of Information and Library Studies, University of Wales Aberystwyth, 1998. <URL: http://www.ukoln.ac.uk/dlis/models/studies/elec-pub/elec-pub.htm>.

    • Pragmatic overview of possibilities; some typology; some recommendations; bibliography and appendix.

  3. Austin, Philip et al. “From Here to Eternity: Creating, Protecting and Sharing Digital Scholarly Resources at UBC”.  Working Paper: ACIT Policy subcommittee. May 1996: 27pp.  6/7/00. <URL: http://www.geog.ubc.ca/~acitpo/working.html>.

    • Detailed analysis of the challenges facing electronic publishing institutionally, at UBC.

  4. Baker, Shirley K. et al. “Principles for Emerging Systems of Scholarly Publishing.” May 10, 2000: 7pp. Association of American Universities, the Association of Research Libraries, and the Merrill Advanced Studies Center of the University of Kansas. 6/14/2000. <URL: http://www.arl.org/scomm/tempe.html>.

    • Detailed outline of principles developed and agreed upon at Tempe meeting.

  5. Beattie, David, and David McCallum. “Electronic Publishing Initiatives at Industry Canada.” Beyond Print: Scholarly Publishing and Communication in the Electronic Environment. Online Posting.  September 26-27. <URL: http://citd.scar.utoronto.ca/EPub/talks/IC-EPPP.html>.

    • Describes Industry Canada’s initiatives, programs and interest in scholarly electronic publishing; addresses the issue of  credibility, suggesting that related organizations adopt McCallum’s "Proposed Resolution" (see McCallum).

  6. Butler, H. Julene.  " Research into the Reward System of Scholarship: Where Does Scholarly Electronic Publishing Get You?"  167-177 in Ann Okerson, ed. Scholarly Publishing on the Electronic Networks.  Proceedings of the Fourth Symposium Association of Research Libraries, November 5­7, 1994.  Washington, DC: Office of Scientific & Academic Publishing, Association of Research Libraries, 1995.

    • Evaluation of e-publication in professional academic environment.

  7. Butler, H. Julene.  "Where Does Scholarly Electronic Publishing Get You?" Journal of Scholarly Publishing 26 (July 1995): 174-186.

    • As above.

  8. Chan, Leslie, and William Barek.  Beyond Print: Scholarly Publishing and Communication in the Electronic Environment.  Online Proceedings. Sept. 30, 1998. E-Publishing Symposium: Sept. 26-27, 1997.   6/15/00. <URL: http://citd.scar.utoronto.ca/epub/1997.html>.

    • Home page of conference with pertinent topics addressed; links to many presented papers on key issues, as well as web resources on e-publishing.

  9. Chan, Leslie K.W. “Electronic Publishing and Scholarly Communications on the Internet.” June 5, 2000: 10 pp. University of Toronto. 6/13/00. <URL: http://citd.scar.utoronto.ca/CAPA/E-publishing.html>.

    • As above; resource page carrying extensive links to areas covering issues facing electronic publishing.

  10. Cohen, Andrew G. “A System for Electronic Peer Review.” The Impact of Electronic Publishing on the Academic Community. (Session 3). 1997: 1-3. The Proceedings of the International Workshop Organized by Academia Europea and the Wenner-Gren Foundation. Portland Press, 1997. 6/15/00. <URL: http://tiepac.portlandpress.co.uk/books/online/tiepac/session3/ch2.htm>.

    • Offers a system for implementing electronic peer review; standards are the same as the traditionally-accepted practices.

  11. Day, Abby, and John Peters. Quality Indicators in Academic Publishing.  Library Review. 43.7 (1994): 3-72.
    • Full issue, dedicated to a study which identifies "those factors which indicate that a particular [academic] journal is of high quality” (4); results intended to reveal tools available for assessing the quality of academic journals; covers referee reports and 'quality' in academic and practitioner journals; includes representative data, methodological discussion, &c., with reading list.
  12. Ensor, Pat, and Thomas Wilson.  "Testing the Promise."  Journal of Electronic Publishing 3.1 (1997).  <URL: http://www.press.umich.edu/jep/03-01/pacs.html>.

    • Discusses pragmatic and other issues related to the peer review process and e-journal production.

  13. Evans, Paul. “What is the Peer Review Process?” Internet Free Press. (1998):6 p. 6/11/00.  <URL: http://www.free-press.com/resources/content/peerrev.html>.

    • Detailed description of the “conventional” peer review process and concerns related to it; offers an alternative process involving preprints and electronically-facilitated peer review processes.

  14. Fjällbrant, Nancy.  "Scholarly Communication -- Historical Development and New Possibilities."  Presented at the IATUL Conference, University Library of Trondheim, 1997.  <URL: http://educate.lib.chalmers.se/IATUL/proceedcontents/fullpaper/nfpaper.html>.

    • Historical overview and contextual development of scholarly and e-publication; rationale for practices; extensive documentation.

  15. Fraser, Murray, et al. “The Changing World of Scholarly Communication: Challenges and Choices for Canada”. November 1996. University Affairs. (June – July 1996): 6. 5/26/00. <URL: http://www.aucc.ca/en/briefs/aucccarl.htm>.

    • Final report of the AUCC-CARL/ABRC Task Force on Academic  Libraries and Scholarly Communication; stated purpose to heighten awareness about the crisis informing scholarly communication, and to discuss the dynamic and democratized potential for publication and interaction made possible by the electronic medium; recommendations and rationale, discussion, &c., on issues dealing with the challenges inherent in establishing and maintaining electronic publishing.

  16. Fraser, Murray, et al. "Towards a New Paradigm for Scholarly Communication.”  September 1995.  Association of Universities and Colleges of Canada and Canadian Association of Research Libraries. 6/6/00. <URL: http://www.aucc.ca/en/briefs/scholarly.html>. 

    • Earlier than that noted above, outlines the crisis in scholarly communication, discussing challenges facing Canadian libraries as well as  issues central to electronic publishing; copyright initiatives, governmental initiatives; addresses areas requiring further consideration and also short-, medium-, and long-term recommendations; initially provided reading list.

  17. Ginsparg, P. “Winners and Losers in the Global Research Village.” (Session: Scientist’s View of Electronic Publishing and Issues Raised, Wed. 21 Feb. 1996). Electronic Publishing in Science: Conference held at UNESCO HQ, Paris, France. 19-23 Feb. 1996.   6/15/00. <URL:   http://xxx.lanl.gov/blurb/pg96unesco.html>.

    • Distinguishes between trade and scholarly publishing, to explain the importance of the electronic medium in scholarly communication; explores problems with the traditional peer review, and pitfalls of using the electronic medium without proper understanding; draws on experience with automated archives to provide details of electronic preprint processes; expresses the need to transform the traditional peer review process.

  18. Green, Diana.  What Is Quality in Higher Education?  Buckingham: Open University Press, 1994.

    • General, and detailed, discussion of qualitative issues in all facets of our endeavour.

  19. Greenberg, Douglas.  "You Can't Always Get What You Want: Technology, Scholarship, and Democracy." New Technologies and New Directions. Westport, CT: Meckler, 1993: 11-25.

  20. Harnad, Stevan. "Free at Last: The Future of Peer-Reviewed Journals." D-Lib Magazine 5.12 (December 1999) <URL: http://www.dlib.org/dlib/december99/12harnad.html>.

    • Provides overview of newly-implemented peer review process; explores positive aspects of electronic publishing; distinguishes between types of journals; questions efficiency of print while addressing arguments against electronic publishing (namely, monetary); praises revolutionary potential of electronic publishing and its interactivity, cost savings, and ability to reach wide audiences.

  21. Harnad, Stevan. "Implementing Peer Review on the Net: Scientific Quality Control in Scholarly Electronic Journals." In Scholarly Publishing: The Electronic Frontier, ed. Robin P. Peek and Gregory B. Newby, 103-118. Cambridge, MA: The MIT Press, 1996.  <URL: http://cogsci.soton.ac.uk/~harnnad/Papers/Harnad/harnad96.peer.review.html>.

    • Collaborative model of publication, electronic facilitation; role of peer review; costs and efficiency. 

  22. Harnad, Stevan. “Learned Inquiry and the Net: The Role of Peer Review, Peer Commentary and Copyright.” Antiquity 71. (1997): 1042-8. <URL: http://citd.scar.utoronto.ca/EPub/talks/Harnad_Snider.html>.

    • Explains ideal intentions of the scholar / scientitist in creating and disseminating knowledge; notes the continuing necessity of peer review, while encouraging "open peer commentary" as an addition to it; addresses copyright.

  23. Harnad, Stevan. "The Invisible Hand of Peer Review." Nature, 5 November 1998, Web Matters Section. <URL: http://helix.nature.com/webmatters/invisible/invisible.html>; [expanded]  Exploit Interactive 5 (April 2000) <URL: http://www.exploit-lib.org/issue5/peer-review/>.

    • "[R]efereed journal literature needs to be freed from both paper and its costs, but not from peer review, whose 'invisible hand' is what maintains its quality."

  24. Judson, Horace Freeland.  "Structural Transformations of the Sciences and the End of Peer Review."  Journal of the American Medical Association 272 ( 1994) <URL: http://www.ama-assn.org/public/peer/7_13_94/pv3112x.htm>.

    • Intentionally provocative; "although peer review and refereeing seem rational, indispensable, and immutable, the histories demonstrate that they are social constructs of recent date"; suggests "an evolution toward a form of publication that will be a continuing open dialogue and collaboration among contributing scientists, editors, expert commentators, and readers."

  25. Kahin, Brian.  "Scholarly Communication in the Network Environment Issues of Principle, Policy, and Practice."  [Paper prepared for the Information Infrastructure Project at Harvard and the Coalition for Networked Information, February 18, 1992],  <URL: http://www.eff.org/pub/Intellectual_property/kahin_scholarly_communication>.

    • Good contextual overview of issues.

  26. Kassirer, Jerome P.  "Peer Review: Crude and Understudied, But Indispensable."  Journal of the American Medical Association 272 ( 1994) <URL: http://www.ama-assn.org/public/peer/7_13_94/pv3089x.htm>.

    • Defines and discusses peer review, and suggests possible improvements.

  27. Mathews, G. Jason and Barry E. Jacobs. “Electronic Management of the Peer Review Process.” Fifth International World Wide Web Conference, May 6-10, 1996. Paris, France. 21 pp. 5/25/00. <URL: http://www5conf.inria.fr/fich-html/papers/P55/Overview.html>.

    • Addresses  problems associated with implementing peer review on the internet; step-by-step description of a successful process, termed the Electronic Management System (EMS); list of electronic and non-electronic resources.

  28. McCallum, David L. “Proposed Principles For Electronic Scholarly Publishing Projects.” (25/02/97).   Online Posting. 6/15/00. <URL: http://www.schoolnet.ca/vp-pv/rcis/e/princips.htm>.

    • Principles developed to encourage discussion toward establishing desirable traits for successful electronic academic publishing in Canada.

  29. McCallum, David L. “Resolution for Consideration by Canadian Academic Associations In Support of Electronic Publications.” (September 7, 1997). Ottawa, Ontario. 6/15/00. <URL: http://www.schoolnet.ca/vp-pv/rcis/e/resolute.htm>.

    • Proposed resolution for adoption by Canadian Learned Societies: "This association recognizes the legitimacy of scholarly material published in electronic form when such information conforms to  broadly accepted standards of peer review”; includes further links to relevant material.

  30. McCarty, Willard.  "Because It's Time: A Commentary on the Program Session."  Presented at Internet-Accessible Scholarly Resources for the Humanities and Social Sciences (1996 ACLS Annual Meeting).  American Council of Learned Societies Newsletter 4.4 (1997) <URL: http://www.acls.org/n44mccar.htm>.

    • Suggests a useful typology of activities for scholarly internet (e-published) resources; includes evaluative steps and publication; role of institution.

  31. “MLA Guidelines for Evaluating Work with Digital Media in the Modern Languages.” MLA Committee on Computers and Emerging Technologies in Teaching and Research. Proceedings of the May 19-20, 2000 MLA Executive Council meeting. Online Posting. 6/22/00. <URL: http://www.mla.org/reports/ccet/ccet_guidelines.htm>.

    • Outline for institutional committees  developing guidelines which support and encourage scholarly electronic publishing.

  32. “MLA Guidelines for Institutional Support of and Access to Information Technology for Faculty Member and Students in Modern Languages.”  MLA Committee on and Emerging Technologies in Teaching and Research. Proceedings of the MLA Council meeting. May 19-20, 2000:2p.   Online Posting. 6/22/00. <URL: http://www.mla.org/reports/ccet/ccet_support.htm>.

    • Outline of features necessary to encourage and support scholarly electronic publishing, among other ventures.

  33. Nadasdy, Zoltan. "Electronic Journal of Cognitive and Brain Sciences: A Truly All-Electronic Journal: Let Democracy Replace Peer Review." The Journal of Electronic Publishing 3.1 (1997) <URL: http://www.press.umich.edu/jep/03-01/EJCBS.html>.

    • Suggests process alternative to peer review, electronically-facilitated.

  34. Peters, John. “The Hundred Years War Started Today: An Exploration of Electronic Peer  Review.”   The Electronic Peer Review Conference. April – August 1995: 9 p. Internet Research: Electronic Networking Applications and Policy 5.4 (1995): 3-9.  <URL: http://www.press.umich.edu/jep/works/PeterHundr.html>. 6/9/00.

    • Notes that the future of e-publishing is unknown; provides criticisms of electronic publishing and detailed look at role of peer review in eliminating most of these problems; suggests areas for further exploration; very well documented.

  35. Pope, Barbara Kline. “National Academy Press: A Case Study.” The Journal of  Electronic Publishing. Adapted from the 1999 National Electronic Publishing Seminar sponsored by the University of Virginia and the Library of Congress. 5/16/00. <URL: http://www.press.umich.edu/jep/04-04/pope.html>.

    • Brief exemplary overview of National Academy Press and why they chose e-publication; NAP offers free full text online while simultaneously marketing print versions.

  36. Roberts, Peter.  "Scholarly Publishing, Peer Review, and the Internet."  First Monday 4.4.  6/28/00.  <http://www.firstmonday.dk/issues/issue4_4/proberts/index.html>.

    • Documents significance of internet, reception by international academic community, and appropriateness of scholarly e-publishing; typology of acceptable forms of e-writing; arguments in favour of moving from print-publishing to e-environments assessed; reinforces need for rigorous systems of peer review in all scholarly work.

  37. Schauder, Don. "Electronic Publishing of Professional Articles: Attitudes of Academics and Implications for the Scholarly Communication Industry." Journal of the American Society for Information Science 45 (1994): 73-100.

  38. Shum, Simon Buckingham ed. “JIME Reviewer’s Guide.” 1998. Journal for Interactive Media in Education. 6/15/00. <http://www-jime.open.ac.uk/docs/reviewer-guide.html>.

    • Overview of the conversational/debating model of peer  review implemented by JIME.

  39. Smith, Alastair G. "Testing the Surf: Criteria for Evaluating Internet Information Resources." The Public-Access Computer Systems Review 8.3 (1997): <URL: http://info.lib.uh.edu/pr/v8/n3/smit8n3.html>.

    • Suggested process for evaluating internet resources.

  40. Stodolsky, David S. "Consensus Journals: Invitational Journals Based upon Peer Review." The Information Society 11.4 (1995): 247-260.

  41. Sumner, Tamara, and Simon Buckingham Shum. “Open Peer Review & Argumentation: Loosening the Paper Chains on Journals.” Adriadne.   (September 11, 1996):1-9. Knowledge Media Institute of Open University. 6/11/00. <URL: http://www.ariadne.ac.uk/issue5/jime/>.

    • Fuller context for Shum, above; explores attempts to develop a successful, interactive peer review process; examines shortcomings of traditional peer review process, document lessons learned and principles derived from previous computer-supported collaborative argumentation; documents JIME peer review process; extensive list of additional references.

  42. “UVIC Report on Academic Computing Recognition.” May 20, 2000: 1-3.  University of Victoria. 6/9/00.  <http://web.uvic.ca/shakespeare/Foyer/CompRecog.html>.

    • Statement recognizing the validity of electronic publishing; guidelines for candidates seeking recognition, committee responsibilities in evaluating computer-related activities, and further considerations; links to additional related resources.

  43. Willis, Jerry. “Some Decisions To Be Made in the Near Future.”  Bridging the Gap Between Traditional and Electronic Scholarly Publishing. 1995 EDUCOM Conference. 1-10. 6/22/00. <URL: http://www.coe.uh.edu/~brobin/Educom95/EducomJW/decision.html>.

    • Detailed overview of issues related to the transition from traditional and electronic scholarly publishing; examines role of library; further resources documented.

  44. Wood, D. J. "Peer Review and the Web: The Implications of Electronic Peer Review for Biomedical Authors, Referees and Learned Society Publishers." The Journal of Documentation 54 (March 1998): 173-197.

  45. Zinn-Justin, J. “Peer Review and Electronic Publishing.”  The Impact of Electronic  Publishing on the Academic Community. (Session 3).   1997:1-5. Proceedings of the International Workshop Organized by Academia Europea and the Wenner-Glen Foundation. Portland Press, 1997. 6/15/00. <URL: http://tiepac.portlandpress.co.uk/books/online/tiepac/session3/ch3.htm>.

    • Discusses problems of peer review in print publication and the crisis of the referee system in physics; suggests open refereeing via a specific type of database.

Bibliography: R. Siemens, with L. Johnston (June 11, 2001 ).