nigelsmart.github.io/PCs.html at master · NigelSmart/nigelsmart.github.io · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
<html>

<head>
<title>Personal Views on Programme Committees</title>
</head>

<h1>Personal Views on Programme Committees</h1>

Having served on many programme committees, been chair of
two conferences and listening to other peoples experiences
I have come up with the following notes.

<p>
Please ignore these, if you want, they are just my opinions.

<h2>Selecting a Committee</h2>
Keep the balance as wide geographically as possible.
Different parts of the world often emphasize different aspects
of a subject, hence keeping a geographical balance helps. <p>

For example in Cryptography it is more common for stream
and block cipher design to be done in Europe, whilst work
on modes-of-operation etc is more likely to be done in the
US.
Cryptanalysis is less likely to come from the North America,
where as theoretical work is more likely to come from
North America.  <p>

Age should also be balanced. In my view younger PC members
have more time and do a more thorough job than older
PC members.
However, they are also more aggressive about rejecting papers
which are not quite up to scratch. They also don't take the
long-view as much as older members, either accepting papers
which are technically brilliant but which will have no
impact, or rejecting papers which will have huge impact but
which lack the technical depth of others. <p>

There is a surprising number of PC members who take the mickey
out of the entire process,
either handing in non-existent or ridiculously short reviews.
Some hand in scores, with no justifications. One would get
the sack if one did this for students, but apparently it is
acceptable for reviewing papers!
These are unlikely to be younger reviewers. So I suggest
before inviting someone on a committee who you
have not seen operate on a committee, that you ask a PC chair
who has asked them before to comment on whether they contributed.

<h2>Conflicts of Interest</h2>
Clearly PC members should not have access to the reviews
or comments related to their own papers.
Many programme committee's also insist that they also do
not have access to the papers from their own institution,
which is also perfectly sensible. <p>

A novel, and to my mind very good innovation, made by
Antoine Joux at EuroCrypt 2009, was the following.
No PC member should have access to the reviews and comments
of any paper by an author with which the PC member has
published multiple papers.
This is quite easy to create such a <i>conflict</i> list for
a PC member, for instance using to the table of co-authors on DBLP. <p>

Why is this a good idea?
There is a tendency to support papers from your friends.
In some committees this is done quite blatantly by certain
cliques, bringing the whole procedure into disrepute.
Since one is more likely in the Internet age to have a
virtual lab with people in other countries, as opposed to
a real lab with people in your own building, excluding
common co-authors from reviewing each others papers makes
a lot of sense.

<h2>Blind or Double-Blind Reviewing?</h2>

By blind reviewing I mean the paper is anonymous,
by double-blind I mean that the PC does not know who is
reviewing different papers.
Usually in Crypto we adopt blind reviewing, only a few
conferences have been double-blind and this was a disaster.
Double-blind did not work as one could not judge whether
a fellow PC's members comments were due to expertise,
incompetance, or known-bias.<p>

There are many arguments against blind reviewing which essentially
go along the following lines
<ul>
  If someone is famous, they have built up a reputation by producing
  many years of work, so this should be taken into account when
  reviewing their work.
</ul>
However, I feel this is a perfect argument <i>for</i> blind reviewing.
If knowledge of an author inclines one to accept or reject
with no basis on the actual quality of the work, then clearly
marking the paper anonymous is a must.
A PC is judging the current papers, and not the papers from twenty
years ago after all. <p>

Blind reviewing is now policy for all IACR conferences, but other
conferences adopt either a non-blind policy or a mixed policy
(where you can choose to be anonymous).


<h2>Having a PC Meeting</h2>
I had a PC meeting when I was chair of EuroCrypt 2008. It was
really nice having a day discussing papers and meeting people
with different interests and opinions. It was great fun.
<i>But</i> I would not do it again! <p>

Why?
Firstly, it is hard as a chair to make sure that a balanced
programme is produced when you are simultaneously chairing a meeting.
Secondly, some people are more eloquent than others at supporting
or not-supporting papers.
Thirdly, it is never the case that everyone can attend, so some
papers may be treated unfairly by not having their champion present.
<p>

Again Antoine Joux's EuroCrypt 2009 is a good case study.
There was no PC meeting, and the result was a relatively well balanced
programme. A fact which was commented upon by almost all attendees
which I spoke to in Cologne.

<hr>
Nigel Smart <br>
August 2009.
</body>

</html>