Skip to content

Peer Review and Biblical Studies Scholarship

October 21, 2013

In a number of comment-exchanges and in a few postings on this site I’ve emphasized that genuinely scholarly claims are submitted for review by other scholars competent to judge matters.  This is how real scholarship works, not by rushing into print with some sensational claim (for which the author is scarcely qualified), by-passing scholarly review critique.  I thought it might be helpful to say something more about how scholarly work is rightly done, with special reference to Biblical Studies and allied areas that I know personally.

“Peer review” is the essence of scholarly work and discourse.  It means, simply, that claims, findings, interpretations, etc., are submitted for the criticism and assessment of other competent scholars before one makes any further claim about them.  And the more innovative, dramatic, or significantly different one’s purported findings or interpretation, the more such critical review is necessary, certainly before one can make public claims about the validity of one’s proposed view.

This peer review takes place in a variety of ways.  It may well commence with sending one’s essay or other written work to another scholar for comment and critique.  It may also involve presentation of one’s work in a conference or seminar where other scholars are present to comment.  Indeed, often these seminars may involve the work being provided to other scholars in advance, so that they have the chance to check data, etc., and so come better prepared to engage the work.

In publishing one’s work, peer review is done this way.  Peer-reviewed journals are those that send out submitted articles to other scholars in the field for review as to whether to publish them, revise them for publication, or reject them outright.  This process now typically is “double-blind”:  I.e., the name of the author is removed from the article, and the names of the reviewers are also confidential.

I understand that there is now a certain concern in at least some fields in the Sciences about falsification of data, and inadequate peer-reviewing, which has allowed articles to be published that have been shown later to rest on phantom data or rigged results.  I suspect that part of the problem in peer-reviewing articles based on experimental data is that one would really need to try to replicate the experiment to test the claims, and that would be difficult to do in many cases.  (Reviewers are themselves engaged in their own set of experiments on other questions, and couldn’t easily gear up for some other one.)  There are other factors, perhaps:  e.g., peer-reviewing typically isn’t remunerated, but is done as part of one’s participation in the field.  So, there may be a temptation to give a “light touch” and not spend the time required really to test things, e.g., re-running the statistics, etc.

But in Humanities fields, typically, the data are there and available:  A body of relevant texts, or images, or inscriptions, etc.  So what is involved usually is assessing how adequately and cogently the data are addressed, and also how adequately the author has engaged prior scholarship (e.g., giving adequate reasons why previous views are likely faulty).

There are, however, various questions that can be posed to the data, and various methods and standpoints from which this can be done.  So, a good part of assessing a work involves how explicitly the approach is stated, assumptions identified, method correctly followed, and arguments laid out clearly.

In some cases, it simply isn’t possible to settle the matter under dispute in some empirical manner.  So, assessing the work involves how cogently the author has presented what is at least a plausible view, at least as plausible as alternatives, and preferably more plausible/likely than other views. Judging this requires a knowledge of relevant issues, evidence and alternative viewpoints, as well as first-hand ability in the data.

After work has survived this and been published, the peer-review doesn’t stop.  This is especially the case with scholarly books.  Publishers of academic books will typically also get outside reviews prior to publication.  But then after publication of books the really crucial book-review process kicks in.  As well, other scholars (including PhD students) engage work and subject it to testing as they pursue their own research analysis of the data.  “Post-publication review” of books has a major role in determining whether they manage to achieve positive recognition as valuable studies.

However one may cherish one’s view of a given thing, as a scholar one can’t really claim or know that it’s really valid until this peer-review work has been done.  If your work survives this process, it just might be valid.

This scholarly critique should form part of doing a PhD, with supervisor(s) and examiners assessing the validity of the thesis as it progresses and then is submitted for examination.  This is why a PhD in the field is a good earmark of someone with competence and also someone who understands the scholarly process.

This peer reviewing all takes time, and a lot of behind-the-scenes work, scholars quietly going about the business of trying to produce good work, and trying to review and assess the work of fellow scholars.  It’s not glamorous, and it doesn’t typically hit the press or TV.  It doesn’t typically generate big royalties or public speaking fees.  But, for those who care to follow discussions of a given subject that are more reliable and worth the time, this is the kind of work, and these are the kinds of people who count.

From → Uncategorized

26 Comments
  1. Dear Larry,

    You only answered my first question. Would you kindly like to answer #2 and 3#, please?

    And please do try hard not to be so sarcastic in your old age? Please, I am almost as old as you so it is not called for.

    Thank you with cordiality,
    Neil

    • No sarcasm, Neil. You’re perhaps being a bit tetchy? Honestly, no desire to offend. But what do you mean by posting about how peer-review doesn’t work (supposedly)? And what were your questions 2 & 3??

  2. P.S.

    Dear Larry,

    I am not the two-horned, pitch-forked devil you seem to take me for. I actually work in academia and support academics like you. My professional dedication is directed towards supporting your citation rates and international scholarly profile. You are misguided, terribly, by your visceral biases if you think I am out to undermine the entire scholarly peer-review process per se.

    Please be nice and try to read what I actually write and not impute your conspiratorial notions into someone who actually dares to challenge some of your pre-suppositions!

    Neil

    • Neil. I see no mention of a conspiracy in my comments. Please try avoid imputing to me claims and attitudes that I haven’t articulated. I don’t articulate presuppositions. In stating the crucial role of peer review, I’m stating a fact, and one affirmed in virtually all scholarly circles.

  3. Don permalink

    Here is the second peer-reviewed book (after Brodie’s Beyond the Quest for the Historical Jesus 2012) that has been published that provides evidence that Jesus did not exist: http://www.amazon.co.uk/there-was-Jesus-God-Philosophical/dp/1492234419/ref=sr_1_1?ie=UTF8&qid=1383128159&sr=8-1&keywords=there+was+jesus

    The third one by Richard Carrier has just been accepted by Sheffield Phoenix Press.

    Have mythicists now become part of the scholarly world? Certainly it is the most popular position on the internet.

    • Don: “Most popular position on the internet”? That’s supposed to mean something? Would you then agree that religions must be true because that’s the overwhelming view of people around the world?
      Moreover, is there a scientifically based study backing your claim? Or is this puffery on your part?
      In any case, Lataster’s book (which has only appeared a few weeks ago, and so has not been subjected to scholarly review), seems less a scholarly study of the historical question and more a broadside polemic against religion as such. That’s theological polemics, not sober historical analysis.
      And a few curiosities: Lataster is described as holding a MA from “the Department of Studies of Religion” in the University of Syndey, but there is no such department. As for Brodie and Carrier, we’ll see what reviewers make of their books, IF they submit them to scholarly review. It remains the case, so far as historians are concerned, that a historical figure, Jesus of Nazareth, is commonly accepted. What Christians came to think of him is another matter (on which I’ve written a good deal myself), but denying the historical figure is . . . an eccentric view.

  4. Ljhooge permalink

    Do some scholars skip the peer review process and simply write books? E.g., I’m thinking here of N.T. Wright and his volumes on Jesus, and his recent one on Paul. Or are his books based on previously peer reviewed articles? When is it ok to skip peer review? E.g., when the topic is so big it would be too large to ‘fit’ within the pages of a scholarly journal?

    • When it comes to books, typically one’s manuscript (esp. if it is one’s first book-length production) will be considered by an editor at the publisher, and may well also be sent out for comments by scholars in the field. One one has become an “established” scholarly book-author, the process might involve simply the publisher’s editor judging a manuscript. But in any case, serious scholarly writers always know and want the post-publication review of their work by other scholars. This post-publication review is a distinctive feature of Humanities scholarship, in which books (and how they’re received among scholars in the field) form the crucial medium of major scholarship.

      • Ljhooge permalink

        Thank you for your answer. That makes sense.

  5. I want to say I agree with you, however I have a question. First, I understand where you are coming from, for instance, if I am having a discussion with someone, and I wish to make an authoritative point, I would most likely never quote a non-scholar, and would even prefer not to quote an unknown scholar if I can help it. I would want to quote a well known scholar. And this scholar is well known because he probably has numerous publications. And all these publications have made it through the peer-review process, indicating that the scholar must be reasonably competent to have passed the critical review of a variety of experts.

    Now my question. Could you clarify what kind of publications you are referring to in your post? I can see how if someone wanted to write a paper or book about Jesus having a wife, this should be subject to peer-review. But what if someone wanted to write about the history of the English Bible, would that need to be peer-reviewed?

    • Howard: Scholarly journals in any subject are typically peer-reviewed, regardless of whether an article is about the history of the English Bible or making some extraordinary claim (e.g., Jesus’ wife). As for books, publishers of academic books also often/typically use peer review, especially for first-time books/authors. Again, regardless of the subject. But if the thrust of the book is a new or radical revisionist claim, all the more will wise academic publishers seek peer review. Some “trade” publishers, however, may simply judge that the book will sell, perhaps more so because it promotes some quasi-sensational claim, and so will publish it. In these cases, the validity of the book is simply not an issue: All that counts in these cases is the likely sales figures.

  6. Stephen permalink

    Thanks for outlining the basics of scholarly processes for folks. If I may add a few additional thoughts (with which I suspect you will agree?) and questions:

    (1) An additional way of theorizing the importance of peer-review is to note what psychologists have termed “the confirmation bias;” i.e., how we (including highly educated scholars) intuitively prefer and seek out evidence and arguments that favor our positions. Correspondingly, we intuitively discount and ignore evidence and arguments against our positions. Scholarly practices such as peer-review are necessarily precisely to help fields, as a whole, overcome the effects of basic psychological dynamics such as the confirmation bias. For example, I and many colleagues who largely agree with me may have rigorously analyzed some set of (to stick with our field) texts, claims, or other ancient data and argued with it for X, Y, and Z positions of ours about that ancient data. But it’s necessary for scholars [A] who are equally well informed about the relevant data and scholarship on it and [B] who disagree with me to likewise be able to assess my arguments…as they are likely to think of counterarguments, other ways of looking at the data, etc. etc. etc. that I just do not see or that I find various ways of dismissing without adequate consideration. And, as such, fields of knowledge as a whole can advance.

    (2) I belabor the above point because the same basic issues also illustrate the problem of producing so called scholarship in and for settings where dissenters are ignored or marginalized. This, for example, is a fruitful way of thinking about the problems with much American Evangelical biblical scholarship (or, at least, the so-called scholarship in the parts of the US Evangelical world that I was associated with in my past; I bring up this example simply because I am familiar with it). To the extent “review” happens in such settings, it is largely a in-house affair, and line-towing Evangelicals’ work is not submitted to any substantial critical scrutiny. This is particularly the case among US “Inerrantist” Evangelicals.

    (3) The results of true peer-review scholarly settings are often misunderstood and an occasion of much misrepresentation by amateurs and non-specialists who want to make the kinds of sensationalist claims of which you speak. E.g., functioning critical peer-review settings mean that it would be difficult for outsiders (whether scholars in other fields or non scholars) to look at the field in question and discern widespread consensus on most issues. Thus it is easy for purveyors of the sensational to represent the situation as though, “Look, even the scholars don’t agree with each other. They are changing their opinions all the time!,” with their implicit logic entailing that, therefore, folks should instead listen to the purveyors of the sensational and/or advocates of some “traditional” view that lacks much scholarly support. Again, this is a misunderstanding of the dynamics of a peer-review setting. Truly critical scholarly fields are not about achieving “assured results” on all matters, but about many positions remaining up for (re)consideration in the light of new evidence and/or in the light of new analyses of previously available evidence.

    (4) Switching gears, are there instances where you see peer-review in our scholarly settings more commonly breaking down? For example, to what extent do you think it’s potentially common for some recognized “Big Name” senior scholars (especially those who have attained some kind of relative “celebrity” status) to no longer really have their feet held to the critical scholarly-review fire? The issue here would not be one of X senior scholar having become incompetent or lazy, but simply of (at least practically speaking) having had the various critical-review quality checks removed from his or her path? IMO, this would result in a significant quality reduction in the work of excellent scholars…especially if such scholars have such a situation become the norm for them.

    Sorry for the long comment.

    • Stephen, It is a long comment, but I’ve published it because to introduce some valid observations. Yes, “in house” peer review (i.e., reviews only by those already inclined to agree) are of limited usefulness, and it’s important to get others who don’t necessarily agree to review one’s work (so, e.g., when I wrote my chapter on “Q” in my book, Lord Jesus Christ, I sent it to two scholars I knew took a different approach and view, asking them to point out faults in my discussion.) But this means that “peer review” can be of varying quality, not that the process is fundamentally flawed, or that there is any better system for trying to assess scholarly work.
      As for your final question, I don’t think that there is much fear of this. As now a “senior” (in age) scholar myself, I certainly know that it is no insurance against quite vigorous critical engagement, often by younger scholars. And “double blind” assessment of articles for journals helps avoid the problem too.
      There are no doubt contrary instances, but I don’t think it’s an “industry wide” problem. In fact, younger scholars often seem to relish the prospect of proving some “established” scholar wrong!

  7. Right on, Larry! Thanks very much for your helpful delineation of the process by which knowledge and understanding are refined over time. I am glad to be a part of the people who both believe in and serve the quest for truth.

  8. Dr. Hurtado,
    Thank you for this timely post as I have recently found myself mulling over some of these same ideas as an outsider to the scholastic world. My thoughts were provoked in no small measure by your post on Joseph Atwill, and the comments following it.

    I agree whole-heartedly with the value of peer-review in weeding out nuts such as Atwill. My question is how does one break into the circle of scholarship without the advantages of position, finance or mere luck, required to get the ticket required to sit at the table? What happens when peer-review serves to weed out the good, as well as the bad? I am reminded that few of the subjects of biblical scholarship would have passed muster in their day. One doubts Gamaliel would would have approved Paul’s letter to the Romans for publication.

    While I might aspire to be a “Hurtado,” it is clear I am perceived as having more in common with an Atwill. As a 57 year old science teacher with little hope of being taken seriously in my field, where Christian confession is a disadvantage I find it depressing to think that I have little chance of a hearing among scholars who love my Lord.

    I am vain enough to think that I have seen some things worthy of scholarly review. I understand that my lack of credentials do not merit it. I understand that I cannot embark on a PhD program. In its absence I have taken advantage of today’s technology and published independently. I do not seek fame or money.

    I may be misunderstanding, but it seems that this is the accusation I receive when I seek to give away copies to those who are qualified to critique it. My desire is only to glorify my Lord. What is your counsel to me?

    Laymen say it is too complex. Pastors say no scholars agree, so you must be wrong even though I can’t show where or why. Scholars say you have no training so I won’t waste my time.

    These are not sour grapes. I realize for me, the train of scholarship has left the station. I realize no one thinks our current system is perfect, even though it serves a valuable purpose. I simply want to encourage you all to beware of the danger of throwing the baby out with the bath water. To consider the possibility that in filtering out the nuts, you may also be tossing some nuggets of gold.

    Again—thank you.

    • Chris: Your lament is touching, but it’s primarily concerned with trying to get a chance to have a say. My posting was primarily for the benefit of untrained *readers*, who might want some criteria by which to assess purportedly scholarly claims. “Credentials” isn’t the main issue, as if some piece of paper was what counts. A proper scholarly “credential” should indicate that one has received the appropriate training in knowledge, methods, reasoning, etc. But the critical test is the work produced. And the way we test it is by submitting it to the formal review by other scholars qualified to assess it.
      That’s if you want to contribute to scholarly knowledge and understanding, if you want to help shape a given subject or field. If you think that you can spot some widely accepted view that needs correcting, then work up your analysis and submit it to an appropriate journal. You should know that it will be judged (“blind”) on the basis of how well it conducts a scholarly analysis itself. So, if you don’t know how to do this, or aren’t able to do it well, don’t be surprised if the piece is rejected. That’s not persecution, so don’t feel sorry for yourself. It’s just the demanding standards for scholarly publication.
      If, however, you want simply to disseminate scholarship more widely, to “lay” readers, then (1) ensure that your work reflects good scholarship on the subject, and (2) communicate it effectively.

  9. Larry you wrote:

    “However, one may cherish one’s view of a given thing, as a scholar one can’t really claim or know that it’s really valid until this peer-review work has been done. If your work survives this process, it just might be valid.”

    How much science has been peer-reviewed, generally accepted as valid, then much later been found to be false? Peter Higgs had to wait about 50 years. The 84 year old Higgs said he feels uncomfortable being likened to other Nobel winners because his work on the particle which carries his name only took a very short time.

    • Geoff: Focus, please. Yes, some scholarly work, including experimental Science work, has later been shown to fallacious in method or assumptions or interpretation, and some recently shown to be fraudulent. But this proves my point: Where this has been shown it is through a better experimental method, or critical review by competent scholars in the subject. I.e., peer-review!

  10. PS — did you write this post after reading Vridar’s post and “inspired” by it?

  11. @ Larry
    I would totally agree that all methods of improving knowledge are flawed. I further agree that peer-review is a valuable method among many. But like all methods, it comes with its problems and we need to be honest about them. Meta-studies (in medicine, for example) are incredibly helpful but there pitfall is publication bias. Admitting this, has helped improved how to weigh meta-studies.

    I agree that conscientious execution is part of the solution, the other part is to realize that some bias are inherent in the method itself and further checks need to be incorporated to check them.

    You are ungenerous in your evaluation of Vridar’s intent — I’d be suspicious that it is as nefariously nihilistic as you paint it. Likewise, your idealistic, term-loaded description of the review process in the opening (as I illustrated) gave me pause to wonder about your intents.

    Peer-review needs checks it does not have, mere “diligence” is not enough. The deep flaws need to be clearly seen. More intelligent balances for those flaws need to be designed so to further prevent knowledge from being confused with propaganda. And I am also talking from out Medical field, but the parallels to Biblical Studies are obvious to me.

    • Sabio: First, my only intent in my posting was to explain how peer-review works and why it is important. I am not ungenerous in my characterization of Vridar, at least based on his previous comments and postings, in which he simply impugns the peer-review process with no alternative process by which scholars are to conduct the necessary challenge and critique of work.
      I am not idealistic and nothing in my posting justifies the term. I explained how peer-review works, in my experience (some 40 yrs of it now). The “deep flaws” that I know of are not in the process but in the conscientiousness of those asked to perform it. In the experimental Sciences, as I’ve said, the additional problems are the difficulties of repeating experiments to test results, the sometimes incredible complexity of statistical methods employed requiring specialists in that, and the pressures simply to produce publications and so justify grant applications. These do not map directly onto peer-review process in the Humanities, in which the major problems are ensuring that reviewers really have the particular competence to perform the task, and who are willing to devote sufficient time/effort to do it right.

  12. Interesting: I just read a fantastic review of the process of peer-review done by Vridar a few days ago. Is this in response to him?

    Larry, you said,

    genuinely scholarly claims are submitted for review by other scholars competent to judge matters.

    But isn’t it the case that these modifications would improve your claim:

    [in the academy, both] genuinely scholarly and ungenuinely scholarly claims are submitted for review by [a few] other scholars competent [and incompetent] to judge matters.

    Otherwise the way you phrase it look all pre-loaded to bias some answer.
    Having been part of peer review processes in the past, I can tell that when you say, ” so that they have the chance to check data, etc., and so come better prepared to engage the work.” This is done far less than we’d like to imagine.

    • Sabio: The piece on “Vridar” displays again the incomplete grasp of things by its author, and suggests, again, the defensive purpose: which is to pose flaws in (some) peer-reviewed work, in justification for dispensing with it and erasing any difference between real scholarly work and simply blogging one’s views without any critical testing.
      Any human activity is going have its share of flaws, mistakes, half-hearted effort, etc., but this is no logical basis for throwing out the process. Instead, it’s a call for more conscientious execution of it.
      The only alternative is what “Vridar” seems to want, which appears to be to scuttle the whole scholarly process in favour of an undifferentiated world of claim and counter-claim and no scholarly criteria for judging them.

      • Er, Larry, did you actually read my post? Where did I even suggest that the whole process be scuttled? Would you like to repeat here what I actually did say and what views I did quote? No?

      • Yeah, I read your post, and have read your several comments here, Neil. And you simply posit criticism of peer-review, questioning its effectiveness (based on selective use of opinions of others). Or have you retracted those comments, and expressed the view that peer-review (properly handled) is still the best/only way by which to filter rubbish from relatively sound scholarly claims? Forgive me if I missed it.

Comments are closed.

%d bloggers like this: