Review or not review, that is the question

As you may or may not know, I am a PhD student. This involves carrying out research and, when you have some interesting findings, writing papers. These papers are submitted to conferences with a view to having them published and presenting the work to your peers.

Most if not all of these conferences have a reviewing process in which relevant experienced researchers in the field form a programme committee. They then anonymously review the submitted papers and either recommend a given paper to be accepted or rejected. Normally, a review consists of a mark from 1-5 on a number of criteria such as "Technical Correctness", "Novelty", "Significance" etc. followed by specific comments from the reviewer justifying their decision.

I recently submitted two papers to a conference and a workshop and both were rejected. Far from being bitter, I understand not everyone's papers can be accepted. What I am extremely unhappy about, however, is the quality of the reviews I received.

In general, the comments in these reviews summarise the paper and highlight specific issues the reviewer had. My gripe is with the lack of consistency in this process across different reviewers and conferences. In my opinion, if a reviewer is going to reject a paper from the conference, the review should contain everything the paper needs to change/include -- in the reviewer's expert opinion -- in order to reach the publication standard. The idea behind this is that the researcher who has had their work rejected can take these comments on board, improve their paper and resubmit it to another conference down the line. The papers I had rejected lacked anything remotely approaching this.

One review was only about five lines long and did nothing but repeat my conclusions. The other reviewer obviously had some sort of personal gripe with my approach and borderline insulted me in his comments. Firstly, I only got two reviews and this was a large international conference. If I am going to spend months doing work and decide to submit it to your conference, the least you can do is oblige me with 3 reviews as is standard for conferences of this size. But more importantly, or frustratingly, there was absolutely nothing constructive I could take from the reviews to incorporate into my paper. So I'm left with a paper that was rejected by supposed experts and I'm none the wiser and to what to do with it so that it's not rejected the next time I submit it somewhere.

I understand the reviewing is a time consuming process, especially for larger conferences, but there really needs to be some sort of quality control. I'm not suggesting a review of the reviews or anything preposterous like that (although I am aware of some instances of programme chairs returning reviews to reviewers asking them for more substantial and useful reviews) but I think there are certainly some measures than can be taken.

Rather than present the reviewer with an unrestricted box in which to place comments, they could be more directed. If they are going to accept a paper, they can be asked to leave specific comments as to how the final paper can be improved. More importantly still, if they are going to reject a paper they should be asked to explain exactly what the paper needs to be brought up to standard. It's the least the person who submitted the paper deserves. Also, the 1-5 system of marking specific criteria can be improved. If as a reviewer you give a mark of 3 for "Clarity" you should have to explain why it wasn't a 5 -- what was unclear and how can you make it clearer. Why wouldn't you anyway? Laziness? Lack of motivation? If you don't really want to review, don't agree to be on the programme committee. It's a responsibility, not just another item to add to the "Professional Activites" section on your website/CV.

Another interesting approach is currently being adopted by one of the major conferences in my field. That is, the work is reviewed and returned to the submitter. They then have a few days to take on board the comments and resubmit the paper to the same reviewers who then decide whether the relevant changes have brought the paper to acceptable standard. This level of interaction between the reviews and the submitter is very beneficial. It will only lead to a higher quality conference proceedings and, for those who had work rejected, a justified rejection with the consolation of a clearer picture as to how to proceed. This all make take a little more time and effort on everyone's part but that shouldn't be an issue. The issue should be the betterment of the quality of research carried out on the field.

Now you may see these as the views of a young naive researcher who has yet to become disillusioned with the whole process, but I challenge you tell me something about my suggestions that wouldn't benefit anybody.