Scholarpedia:Policy revision proposal

This proposal aims to redefine the notions of Curatorship and authorship, specifies the concept of article re-publication and introduces the article scholar index.

The article Curator would see his tasks and privileges extended to include, in addition to the "old role" of responsibility for the article's contents, the "new role" of vetting or overseeing the writing of the next published version of the article. This supervision task would include the power to decide (together with the Category Editor) who among the article's contributors deserve(s) authorship. No matter what their level of contribution, the work of Curators would be acknowledged at a minimum as the "last authors" of the article re-publications that they oversaw.

The possibility of republication of an article with the same title (after significant and substantial modifications), changes the notion of author of a Scholarpedia article to that of author of the N-th republication of a Scholarpedia article.

This system of multiple frozen republications, together with permanent existence of the "development version" of the article (the last Curator-approved version at any given time) reconciles the standard system of citations and authorial responsibility with the current practice and value of having Scholarpedia articles that are timely and up to date.

The "article Scholar Index" (aSI) would aim to primarily represent a particular contributor's work on a particular article, rather than the contributor's work with Scholarpedia as a whole; it would inform the decision regarding whose name appeared in the formal citation of the re-published article. This is intended to encourage the active development of articles after their initial publication. Counter-balancing this, the article's Curator (and only its Curator) would determine what revisions are accepted or not. The community of Article Developers -- all individuals participating in the article's development -- would evaluate the contributions of others, increasing (or potentially decreasing) the contributor's aSI, with each evaluator's rating weighted by that evaluator's aSI.

Background and motivation

Beginning in February 2010, Editor-in-Chief Eugene Izhikevich asked for comments on revisions to the "Scholar Index" (SI) system.

Text of the initial request for feedback:

When Scholarpedia was originally designed we instituted a "Scholars Index" (SI) in an effort to prevent vandalism and maintain the quality and authority of the articles written. The idea has been that users grow their SI via their contributions to Scholarpedia, which gives the users greater say in elections as well as added editing privileges (http://www.scholarpedia.org/article/Scholarpedia:Scholar_Index). The long-term vision for the SI is that it will permit Scholarpedia to eventually become "run itself" with negligible need for executive input while remaining authoritative.

While certain aspects of the SI system have been employed, other aspects have gone unused. At present, Scholarpedia runs as a normal print encyclopedia would, with editors joining through invitation or self-nomination. These editors (you) engage in the normal process of choosing article titles, authors, and reviewers, and then ensuring that the peer-review process goes smoothly. In these activities the SI has thus not played a significant role -- apart from preventing vandalism, it has only played a relatively minor role through its effect on elections for the authorship of a contested article.

This email thus solicits feedback from the Scholarpedia editorship on what future role we believe the Scholar Index should have. It is probably fairly clear that we would like authors, reviewers, editors, copyeditors, and other contributors to the effort to receive some aggregated credit for their work. The question is over what should be the effect of having accumulated said credit. Should this accumulated credit give the holder added permissions, or should it simply be a symbol of the work they have done, a quantitative marker of status and say within Scholarpedia? Is it possible to invent a credit system that would allow Scholarpedia to "run itself" successfully?

We are extremely privileged to have an editorship with so many brilliant scientists and scholars -- I have little doubt that we will be able to develop a similarly brilliant plan for Scholarpedia's future. Your comments, thoughts, and ideas are therefore invited on this matter.

After some discussion, a proposal has been developed that aims to make better use of the Scholar Index metric, while enhancing other aspects of Scholarpedia's operation.

Components

Article

Article development is the primary work of Scholarpedia. A Scholarpedia Article is a topic with associated elucidation and explanation. Currently, articles are written by leading experts and their co-authors, and after review and publication the article is Curated by those who initially authored it. The primary change to articles in this proposal is that, after sufficient revision -- no doubt to in part reflect the constant progress of science -- a new version of the article is eligible to be reviewed and published. What this means is that, while the title may be identical or nearly identical, the article's new content can now receive recognition within a scholarly publication (namely, Scholarpedia) and be cited by other scholarly publications.

First published version of an article: the original article, i.e. the first version of the article to have passed peer review. It has an associated set of original authors.
N-th published version of an article: the N-th successful review and publication of an article.
Approved version of an article: a version that received Curator approval. All approved versions are stored in the article's history and are uniquely identified by a revision number. The last available approved version is called the development version and is visible in the main article page.
Unapproved version of an article: an article with edits that have not received Curator approval. Unapproved versions are stored in the article's history but never shown in the main article page.

Development version of an article

The last available version of an article which has received Curator approval is known as the development version. Only Curators (and developers that have received his/her explicit authorization) may approve a proposed revision of the development version. This version will be shown in the main article page, together with a list of links to all previous republications (on which the latest version is likely to be based).

(Imaginative) Example: The top of the development version of the article "Classical Mechanics", having revision number 12033, and being curated by John W Smith would show something like

Revision 12033 of Classical Mechanics, curated by Smith, JW and based on:

   --  Brown W, Rossi A (2010). Classical Mechanics. Scholarpedia 5(4): 1.

   -- ...

   -- Lagrange J-L (1766). Classical Mechanics. Scholarpedia 4(4): 785.

   -- Newton I  (1687). Classical Mechanics. Scholarpedia 3(11): 384.

   -- Galilei G (1632). Classical Mechanics. Scholarpedia 1(1): 20.

Strictly speaking the development version of an article has no journal number and is under the responsibility of the Curator, but, if necessary, it can be cited as the last published version of the article followed by the revision number and the name of the curator. E.g. in the previous example one would cite the development article as

Brown W, Rossi A (2010). Classical Mechanics. Scholarpedia 5(4): 1 (revision 12033 curated by Smith, JW).

Citation of the published version of an article

In Scholarpedia, the formal bibliographic reference of a peer-reviewed and published version of an article (PvA), as would appear in a scholarly bibliography, is known as that PvA's citation. Each PvA has its own set of authors and Curators permanently affixed to it. Within the citation would be the names of the most significant contributors (perhaps no more than five, and ideally three or four), including the PvA's Curator. As is often the case in large collaborative efforts, the "Principal Investigator" -- in our case, the Curator -- would likely appear as the PvA's "last author". Similarly, the first author listed might be the individual who contributed the bulk of the actual writing of the PvA, with additional PvA authors being those developers who made other critical contributions.

article Scholar Index

Article Scholar Index (aSI) is meant as an estimate of a user's contributions to a particular article . Article Developers with higher aSI can be said to have made more, or more substantive, contributions to an article's development. Increases in aSI may lead to titles (e.g. "Assistant/Associate Article Editor") meant to signify the amount of contribution that a user as made to a particular article.

A user's article Scholar Index may be adjusted each time he or she makes a contribution to an article. The net adjustment is determined by a rating performed by his or her peers; each article developer can evaluate revision of any other article developer and assign a numerical grade to it (e.g., currently, the grades are "-10=wrong, -1=irrelevant, 0=OK, 1=useful, 5=valuable contribution, 10=major contribution). Such evaluations do not result in approval of the revision, which can only be done by the curator. Ideally, the article developers would evaluate each other, and once a revision receives overall positive grade, the curator will receive an email and is asked for the approval of the revision.

Contribution judgments are weighted such that evaluations by Article Developers with a higher aSI are more strongly weighted than the evaluations by Article Developers with a lower aSI.

Ultimately, aSI would likely play a role in determining whether an Article Developer receives author-level recognition at the time of publication of a version of the article.

Article "back pages"

Along with discussion, as is there currently, an article's back pages serve as the locus for the community of Article Developers. A listing of the various titles and degree of contribution of each Article Developer would be displayed in recognition of their work. This might also be the site where different edits are rated by members of the community.

Roles

Editor of a Category of Scholarpedia

As is presently the case, the Category Editor would propose a list of articles for his/her category and seek out individuals to write these articles. One slight change to the Category Editor's role might be in how they relate to article Curators. Currently, their role is to invite someone to author an article, and after doing so this individual becomes the author's Curator. Under this proposal, the Category Editor would first seek out an article Curator who may or may not be the individual who does the majority of the writing for the article. Moreover Category Editors would decide if an article is worth re-publication and supervise the curators' choice of authors for the republication.

Scholarpedia Advisory Editors

Not all editors in Scholarpedia have the responsibility of a category. An editor without a category will have the title of Scholarpedia advisory editor. This title is merely honorific and should only be attributed to e.g. world-acknowledged experts that work in an advisory role in the background (in this case a section "Advisory editorial board" could be added in an "Encyclopedia of .." page to acknowledge even more their contribution).

Assistant/Associate Scholarpedia Editor

These roles already exist in Scholarpedia. Unlike honorary titles given to Article Developers, these roles are not, and will not, be meant to reflect past contributions but rather a set of responsibilities that a particular Scholarpedia Editor currently has (it is similar to Category Editor status in this way). With the contributions of an Asst./Assoc. Scholarpedia Editor his/her Scholar Index would increase, giving him/her greater weight in decisions that bear on Scholarpedia as a whole (such as the choice of new articles, Category Editors, and Curators).

Curator(s)

The Curator will be defined as the individual ultimately responsible for the contents of their particular article. The Curator is to be the only individual with article revision approval rights, and there will ideally be only one Curator per article in order to minimize disagreement and focus responsibility. In order to mitigate the workload associated with revision approval, the Curator will be free to delegate revision approval rights to a party whom he or she trusts. While the Curator might well be the article's primary author, his or her role in its initial and continued development could range from sole-authorship to a more hands-off, "Principal Investigator/last author" role. When a Curator believed an article had received significant and substantial revision from its last published version, he or she would propose to the Category Editor the possibility of republication, and nominate a list of individuals to be listed as the authors of the new version of the article.

Author

An author is defined as a contributor that co-signs at some time a published version of an article. "Author" would not be an actual role within Scholarpedia per se. Rather, an article's authors are individuals judged to have contributed sufficiently to have his or her name included in that published article's citation. As the final arbiter on the article's content, the Curator would always qualify as an author. During initial article development, the Curator might pick one or two individuals with whom to co-author the article, and these individuals' names would appear as the article's authors at time of publication. However these individuals would not be immediately made Curators, and while they would no doubt have been major contributors to the article, initial authorship status would not guarantee "author" status in publications of the article.

Article Developer

Any individual who made a successful contribution to an article's development would immediately earn the title of "Article Developer" for that article. Clearly, the article's Curator will be an Article Developer, but so too will be the article's reviewers (if they reveal their identities), the writers of the initial article, the Category Editor, and perhaps also the user who suggested the article title or Curator. The set of Article Developers would constitute a community of article contributors. While only the title of "Curator" is associated with additional rights, Article Developers could potentially gain titles that reflected the degree of their contribution (their aSI) to an article's development. The contribution of one non-anonymous accepted revision would, perhaps, lead immediately to the title of "Assistant Article Developer", perhaps followed by "Associate Article Developer" and then simply "Article Developer". Beyond this could be additional titles, such as "Assistant Curator", "Associate Curator", "Assistant Vice-Curator", "Associate Vice-Curator", and "Vice-Curator" (if needed).

Reviewers

The role of reviewers would change little under the present system. Reviewers of subsequent publications of the article would, in addition to judging the article's quality, would also evaluate whether the revised article was sufficiently different from the original that it merited re-publication. After initial publication of the article, if reviewers chose to reveal their identities they could immediately become members of the community of article contributors, perhaps with the title of full "Article Developer".

Processes

Initial authorship

As mentioned above, it would be entirely up to the invited Curator as to how he or she would like to begin the writing of the article. He or she would be welcome to write the entire article him or herself, or engage very little in actual writing and instead supervise the writing process. The initial authorship procedure for each article will differ depending on the preferences of the Curator and Category Editor, as well as potential interest in helping with authorship from members of the wider academic community.

One way the development of an article might proceed could be:

Category Editor invites Curator
Curator determines team of initial writers
Article is written. Curator follows-through on authorship promises to article writers
Peer-review and publication

But it might also proceed as

Category Editor invites Curator
Curator begins article, sketches its organization, and invites participation from others
As article develops, to help manage the work, the Curator gives approval privileges to an existing Article Developer
Curator continues vetting and advising on the article's development, indicating when he/she believes it ready for publication
Curator, Category Editor, and article community use aSI to inform a discussion of who should achieve authorship-level recognition
Peer-review and publication

(A potential problem here is that there is nothing to approve before the article is published for the first time, so items 3 and 4 may be irrelevant). (why not have the first development version of the article still qualify as an "article" like any other? why should there be nothing to approve?)

Peer review

This process would proceed as before. The Category Editor would invite anonymous reviewers to vet the publication for quality and accuracy. After the process was completed, the reviewers would be invited to join the article community and immediately awarded with a high aSI for the article.

Initial publication

Would proceed more or less unchanged.

Article revisions

Revisions, submitted by Article Developers or other contributors, are either approved or denied by the article Curator. All users have equal privileges in submitting article revisions (except the Curator and perhaps his/her designate). As is currently the case, the revision is visible in "article revisions" tab, however it does not become the by-default approved version unless the revision itself is approved. All revisions are subject to evaluation by the community of developers, ideally before the Curator decides whether to accept the suggested revision. Indeed, the Curator _may_ choose to only receive notification of revisions that have achieved a certain evaluation threshold from the article community.

Revision evaluation

Every revision (perhaps even those by the Curator?) would be subject to evaluation by the set of Article Developers. The reviser's aSI would be updated to reflect the aggregate opinion of the article community, irrespective of whether the revision was accepted by the Curator, however the Curator's evaluation of the revision would carry significant weight (perhaps half?).

The sequence of events might go as follows:

Revision submitted by reviser
Before the Curator decides whether to accept the revision, the entire article community is eligible to evaluate the revision
Each evaluator's rating is weighted by his or her aSI, and an aggregate revision score is tallied. This is then added to (or subtracted from) the revisor's aSI. Mechanism design is used to mitigate gaming of the evaluation process
The Curator (or his/her designate) approves the revision.
(optional) Evaluation proceeds for a certain number of days (7? 14?) after the initial submission

Article republication

On a case-by-base basis, after substantial updating and revision, the now-novel content of each article would have the potential to be published. The decisions over whether the revisions to the article are worth publishing would be reached ideally by consensus among the Category Editor, the article Curator, and the article's community of developers. Publication merit would lastly vetted by the article's (potentially anonymous) reviewers.

Open Problems

Automatic Selection of Authors

It would be nice to have a fair and automatic way to select authors of a new re-publication of an article. This way, people would know in advance whether or not they want to become article developers and what else they should do to earn the title of the article author. For example, the procedure could be:

The authors are the article developers with top 25% of aSI plus the article's curator (with some restrictions on the number of authors).
The article developers elect authors
The authors are assigned by the curator and the article category editor

That said, the ideal situation is one where there is simple consensus among the "major" article developers, Curator, and category editor, as to who should gain the title "author". If there is not consensus, there could be some fall-back mechanism, but this might be a problem we could (and perhaps should) put off till later.

My hope is that, if it's blindingly obvious who contributed how much (through visualizations of who did what to an article, the aSI, etc.) then the determination should be "easy".

A perhaps larger problem is whether aSI should "reset" at time of article publication, or whether it should persist. If it does persist, what role does it play? Ultimately, how do we weigh contributions that span article "publishings". ... how do open source projects manage this?

Scholarpedia:Policy revision proposal

Contents