How to succeed with peer review for scientific conferences: A checklist

What is a peer review and why is it important for a conference?

The peer review is the cornerstone of scientific quality assurance. It is the process by which peers (fellow experts) examine and evaluate submitted conference papers. The assessment is carried out by experts who are familiar with the topic through their own research or professional practice.

Peer review generally begins as soon as the deadline for the call for papers has passed.

For conferences and conventions, this usually happens in two stages:

1. External evaluation by independent reviewers, who provide a well-founded recommendation.
2. The final decision by the program committee, which decides whether a submission is accepted, rejected, or sent back for revision.

A well-structured peer review process is far more than just a formality. It ensures transparency, quality, and fairness. It also helps position the conference as a platform for excellent research.
In this article, we show you how to plan your peer review workflow, find the right reviewers, and manage the entire process digitally.

Planning your peer review: How should it work?

Before reviewers evaluate the first submission, you need to answer this question: How complex should your peer review be?

The process varies depending on the type and size of the conference. Some organizers prefer a simple procedure where the program committee or conference management reviews all submissions themselves.

Advantage: It is usually clear quickly which submissions will make it into the conference program. Only a few people are involved in the decision-making. With many reviewers, delays are more common, and large-scale coordination is not required.

Other organizers prefer a multi-stage process. In this case, each submission is evaluated by several reviewers.

Feedback loops are also crucial. You can either be strict and only distinguish between "accepted" and "rejected." Or, you can add a third category: "sent back for revision" for papers that are strong in content but need a little polishing.

Checklist: Finding the best workflow for your peer review

• Are you using a simple review process?
  Only a few people are involved here. For example, the conference chair reviews all submissions and decides which ones will be part of the program.
  This saves time, as information does not need to circulate between multiple groups and you don't have to answer follow-up questions.
• Or is your review more complex?
  A paper goes through several stages after submission. It is evaluated by at least two reviewers. Later, a central authority (a committee) decides how to proceed with the paper.
  If so, roughly outline the different stages:
  • Which and how many people (or groups) are involved in the peer review?
  • How many and what kind of stages does a paper go through?
  • At what points do you need to notify which people?

External evaluation of submissions

Once the process is in place, the next step is the external review of the submitted contributions.

This is where the reviewers take the lead. They evaluate the papers from a technical perspective, provide recommendations, and thus form the basis for the program committee's final decision.

The evaluation usually takes place in a secure online area, which reviewers access via a personal login. As the organizer, you define the questions that reviewers must answer for each paper. These typically concern the relevance of the content, methodological quality, or the originality of the paper's topic and content.

Whether you conduct this external review depends, as described in Part 1, on the ambition and scope of your conference. For very small conferences with a manageable number of submissions, organizers sometimes decide to make the review an internal task.
For larger congresses, however, a multi-stage review by external experts is the standard. It ensures transparency, fairness, and scientific quality.

Speaking of fairness: A 2024 study by Pleskac, Kyung, Chapman & Urminsky found that single-blind reviews at conferences tend to favor well-known authors and disadvantage scientists with Asian names. This at least suggests that reviewers judge differently when they know whose paper they are looking at.

Checklist: Organizing an external review

Preliminary considerations

First, it's about the basics of the review process: Who evaluates what, and how exactly does it work?

• Who are the reviewers?
  Compile a list of all reviewers as soon as you know who is involved. For the first step, names, email addresses, and organizations are sufficient.
• Are these individuals already in the system?
  If a list already exists, it should be in the software you are using to conduct the review before distributing the contributions. If the individuals are not yet in the system, check how you can proceed faster than typing in every name individually.
  Some abstract management systems offer a CSV template. This is a spreadsheet that you fill with the reviewer data and then simply upload.
• Which reviewer is an expert in which topic?
  It is best to add this to the list immediately and assign topics to the individuals.
• How many experts do you need per topic area?
  Multiple experts ensure different perspectives and reduce the risk of one-sided decisions. Also, consider which individuals might be able to cover more than one topic.
• What questions do reviewers answer for an abstract?
  Sometimes it is enough for them to vote "accept" or "reject." In other cases, they answer additional questions about the paper.
• How many contributions is it reasonable to assign to one reviewer?
  This depends, among other things, on the number and scope of the questions. As a rule of thumb, plan for 4–8 submissions per reviewer. If the timeframe is very tight, aim for the lower end; for less in-depth reviews or smaller meetings, the upper end (up to ~10) is also acceptable.
• What if not all reviewers complete their task?
  Plan time in case you need to redistribute contributions. Alternative: Are the judgments of the other reviewers sufficient?
• Are reviewers allowed to evaluate their own contributions?
  The review should be as neutral as possible. When it comes to their own work, authors are often "blinded by familiarity." Others provide a more neutral judgment.
• Should a reviewer decide on the presentation format?
  Sometimes authors indicate a preferred presentation format. Decide whether reviewers should see this or make their own decision.
• Should reviewers see how many versions someone has submitted?
  This is very useful, especially for multi-stage reviews. This allows reviewers to understand what authors have changed since the last version.
• Should reviewers be allowed to download their contributions?
  This can be an option for those who travel a lot and might want to enter their evaluation all at once. Nowadays, this is rather uncommon. If you want to offer this, you should communicate that the results must be entered into the system by the deadline.
• Should reviewers be able to see who wrote the paper?
  Do you prefer a 100% anonymous evaluation process? Or one where author data is known to everyone? Or one that is limited only to the authors' names?
• Should reviewers see who else is evaluating the contribution?
  Transparency has two sides: On one hand, it can lead to more consistent evaluations; on the other hand, reviewers sometimes influence each other.
• Should reviewers be able to see all conference contributions?
  An overall overview makes it easier for reviewers to recognize thematic connections and better classify contributions. However, always design access so that no confidential information is disclosed.
• Should reviewers be able to send messages to authors?
  The program committee decides whether the messages are sent along with the evaluation result using placeholders. It is definitely helpful if an author is supposed to revise the contribution and there is a second round. Then they receive feedback.
• Should reviewers be able to send messages to the program committee?
  This can be useful. Often, however, the answers a reviewer provides during the evaluation are sufficient.

• ‍

Evaluation process

Let's fast-forward to the evaluation. There can be four options for each paper. You decide which of these become part of your peer review process:

• Accept
  The paper is accepted and will appear later in the conference program.
• Reject
  The reviewers (and the committee) consider the paper unsuitable. It will not be part of the conference program.
• Waitlist
  If more contributions have been accepted than you have presentation slots, a waitlist can be useful. Sometimes presenters cancel at short notice. Contributions on the waitlist then automatically move up.
  When you inform authors that their paper has been accepted, include a link where they can immediately accept or decline.
  Inform authors on the waitlist about when they can expect a decision.
• Revise
  If a feedback loop with revisions is part of your peer review, work backwards from a fixed deadline. For example, take the date on which you publish the conference program. This is the best way to keep track of all intermediate steps.
  Then, consider who will check the revised contributions. The same reviewers? That is the fastest way, as they know the first version and know what to look for in the next round. Once round two is complete, the final decision is made by the program committee.

Influencing the evaluation

This is about the criteria according to which reviewers evaluate the papers.

• Which questions should reviewers answer as part of the peer review?
  Does the contribution fit the conference theme? Is the contribution based on current scientific findings?
• Mandatory information for the evaluation
  Very important: What assessment do you absolutely need from the reviewers? Usually, this is the recommendation to accept or reject.
  However, this can also include other questions, such as whether the chosen presentation format is appropriate.
• What should the final evaluation be based on?
  There are two options here: Either the individual criteria (answers) from the reviewers are incorporated or you assign points. Each answer is worth points, which together result in an average grade for the paper.
• Do you want to group the criteria?
  If there are many questions, group them by theme. Conferences often distinguish between "content-related" and "formal" criteria.

The final evaluation of the contributions

Almost there! The final say lies with the program committee. Sometimes this is just one person, sometimes a group. For very large conferences, there may even be multiple committees, which makes the review process a bit more complex.

In any case, the committee already has a rough idea of what an interesting conference program looks like from a scientific perspective and how many slots are available for each topic area.

Checklist: Planning the final evaluation

• Who makes the final decision on submissions?
  Usually, this is the task of the program committee. Sometimes a single person makes the final call. Before that, the team typically meets to discuss the evaluations.
• What information do program managers need to provide?
  Clearly, there is a final result at the end. If a submission is to be included in the program, the scientific committee checks which topic area it belongs to. Part of the decision also involves the presentation format: Should the submission be presented orally, as a poster, or turned into a workshop?

Communicating with authors during the peer review process

If you are using software to support your peer review, you can send the results directly to the authors via email.

Checklist: Messages to authors

• Who should be notified of the review result?
  Determine whether only the submitting person receives the email or the entire list of authors.
• What else do you want to communicate?
  Create various email templates in advance for different scenarios—accepted, rejected, waitlisted, revisions required. Add information that is relevant to the authors in each case. Think about deadlines for revisions or for confirming participation.
• Do you also need different email texts for the presentation formats?
  Here, too, it makes sense to differentiate: What information is important for speakers? What do poster authors need to know? What concerns only workshop leaders?
• Do you need an active confirmation from the authors?
  This is very important for submissions on the waiting list. Who could step in if necessary, and for whom is it not an option?

How is AI changing the conference peer review process?

Don't worry: Machines won't be taking over the entire peer review process anytime soon. However, there is an interesting initiative by the Association for the Advancement of Artificial Intelligence (AAAI), which is launching an AI pilot program for its 2026 conference.

This could be groundbreaking. Large language models (LLMs) are intended to augment the review process without compromising human expertise or replacing human reviewers.

Specifically, this involves two parts of the workflow:

• In the first phase of the review, an AI-generated evaluation for each paper comes into play. This is intended to provide an additional perspective alongside the reviewers' assessments.
• The program committee also receives assistance: An AI summarizes the reviewers' findings. This allows the committee to see more quickly where the reviews agree and where further discussion is needed.

In the long run, these results could also influence other conferences. It is worth keeping an eye on technical innovations, especially when they make life easier for organizers.

FAQ

How does conference peer review differ from journal reviews?

For conferences, the review process is usually more time-constrained and geared toward presentation formats (talks, posters, etc.), whereas reviews for scientific journals focus more on depth of content and publication readiness. Conference reviews often also evaluate the contribution to the conference theme and the relevance to the audience.

Can a reviewer evaluate their own submission?

No. For reasons of independence and fairness, reviewers are strictly prohibited from evaluating their own submissions. Peer review software should technically prevent this combination from the outset to avoid conflicts of interest and ensure the integrity of the process.

How do I select suitable reviewers?

Select reviewers based on their area of expertise, experience, and availability. A structured reviewer pool with clearly defined subject areas makes assignment easier. If you have a fixed list, regularly cross-reference it with the individuals' current research focus.

How do I handle late or missing reviews?

Build in buffer times. Communicate clear deadlines with automated reminders.
If reviews are not submitted, you can reassign submissions if necessary or—if multiple reviewers are assigned to a paper—use the reviews already received.

How much time should authors be given for revisions?

The deadline depends on the extent of the required changes. In practice, a period of one to two weeks is standard for minor adjustments. Most importantly: plan your deadline by working backward from the date you intend to publish the program.

More tips for a successful peer review