We decided to submit an update of Jeroen’s work on caching for model counters, based on his master thesis work. It was accepted, so last week Jeroen presented our work.
While I was naturally happy to see our work presented, I also very much enjoyed the rest of the program. It has been a while since I’ve seen so many interesting talks in a row. I particularly liked the contributions from Tuukka Korhonen and Lucas Bang.
Actually, that is not true. I am an outstanding reviewer! Or so say the Association for the Advancement of Artificial Intelligence and the AAAI-21 Program Committee.
If I’m perfectly honest, I was quite insecure about reviewing for AAAI. It was my very first time as a PC member, and I was not quite sure that I was knowledgeable enough to be able to be sufficiently critical of the contents of a paper and to recognise flaws, especially those related to novelty and obscure theorems that have pages-long proofs.
My colleagues encouraged me, by reminding me that I had read and graded roughly 150 student papers/reports/theses already. For many of those, the feedback I provided led to new and improved versions, and in some cases even to scientific publications. All in all, my colleagues reckoned I was ready.
I then simply did the best I could. I did not receive any feedback from the AAAI-21 Program Committee, so I can only guess why they picked me among the thirteen outstanding PC members out of all those thousands. Just in case you are reading this looking for tips on how to be a good reviewer, here are my recommendations for reviewing anything, be it a scientific paper submitted for publication, or a student report for an (under)graduate course.
All good feedback is actionable. For example: instead of saying “Section 4 is unclear”, say “I would find it easier to understand Section 4 if concept X was formally defined.” Note that this also reflects the fact that your feedback is your opinion, not fact. It also indicates what made Section 4 unclear to you, and what would improve Section 4’s clarity for you.
If authors have the opportunity to write a rebuttal, it is helpful if you number your points of critique and your questions. They can then easily use the numbers to refer to and respond to these critiques and questions, saving valuable rebuttal space. Conversely, you can easily find their responses and check if they have addressed all your points.
It is also helpful if you indicate, at least to your fellow reviewers, but maybe also to the authors, what you took into account in your final assessment of the work. Especially if a paper ultimately is rejected, it is good to make explicit that this was, e.g., because the quality of the experiments was lacking and not because there were too many typos.
I firmly believe that if I help you thrive, it makes me thrive, too. I want my students to do well, because a) their success reflects well on me and on my university, and b) it’s simply much more fun if they succeed than if they fail.
Similarly, I want the papers at the conferences where I (try to) publish my work to be of high quality. I genuinely tried to come up with feedback that would make the next version of the work I was reviewing better, regardless of whether it would be accepted or not.
Naturally, it does not hurt to also compliment the authors on anything they did particularly well. We all struggle to get our work published, so encouragement and appreciation are always welcome. Furthermore, you will also benefit if people learn what makes a paper interesting or pleasant to read for you, so you can also compliment authors for thoroughly selfish reasons!
Reviewing papers is a great opportunity to learn, which can be a great motivator to put some serious effort into it. You get a glimpse into the latest advances in your (or a related) field, which I trust you find just as exciting as I do.
On top of that, you can also learn a lot about other people’s writing process. While reviewing, ask yourself if the language and the structure in the paper works for you. Are there helpful analogies? Are the acronyms clearly explained? Is there a particular framing that you find compelling, or quite the opposite? Take notes and learn!
Especially if it is your first time reviewing, you may not be familiar enough with all the material to assess its quality. Try to learn as you go, but also don’t overwork yourself. If you have reached the end of your knowledge, just be open about that to your fellow reviewers and area chair. Maybe they can fill you in, or at least take into account that your knowledge is lacking.
You may also not fully understand the process or the conventions, especially because not all of them are always clearly communicated. It is absolutely okay to indicate to the other reviewers and the area chair that you are unsure if a certain paper is out of scope or if certain conditions (like anonymity) are not met, as long as you indicate how and where you have looked for the answer and what the result was of that search. You may not be the only one who is confused, but you also do not want to cause a paper to be rejected just because something was not communicated clearly.
Out of all the people they could have asked to review this paper, they chose you. You are part of the community. You have knowledge. You have ideas. You have a vision. You are qualified to give it your best shot. You have a lot to offer, so show it to them.
You are not the only reviewer and it is okay to disagree with the others. If, during the discussion, you find that somebody makes a good argument, let yourself be pursuaded and admit that you have changed your mind. Being able to change your mind when presented with compelling new information is a strength, not a weakness. If the new information is not compelling to you, it is also perfectly fine to not change your mind. You are entitled to your opinion.
Be a benevolent critic
This is really the summary of the points above. It is a phrase that prof. Holger Hoos (my PI) likes to use, and I find it very helpful. We also use it in our group, when we give each other feedback on our work. With any feedback I give, I ask myself if I am indeed being critical enough, and not slacking off or shying away from uncomfortable truths. Additionally, I ask if the feedback I give is constructive and respectful, and indeed likely to help the other person succeed, instead of demotivating them or dragging them down.
The award came as a complete surprise to me, if not to others. As one of my colleagues put it: “You cannot do anything without winning an award for it, can you?” Clearly that is untrue, but I am not going to deny that I wasn’t a bit flattered to hear that. Ultimately, it is nice to be appreciated by your community.
Last week I attended the Benelux Conference on Artificial Intelligence and Machine Learning (BNAIC) for the first time. This year, it was held in Brussels, in my beloved Belgium.
This year’s edition was framed around AI Synergies: business meets academics. This was very well reflected in the excellent collection of demonstrations, where we learned about self-driving wheelchairs and puzzle-solving technology.
My presentation was scheduled in the very first session on the very first day. I was afraid that nobody would turn up, but luckily roughly forty people did. True, a lot of them were my amazing colleagues from the AIA group at UC Louvain, and a few of my (former) colleagues from LIACS in Leiden, but there were also a lot of very unfamiliar faces there.
Then, in my presentation, I used the example of a viral marketing campaign for chocolate, by means of ‘seeding’ certain people in a social network by giving them a free sample of the chocolate, and relying on them to promote the product to other people through word-of-mouth.
As a reward for the people who showed up at this early hour, I handed out Belgian chocolates to my audience. I hope they appreciated the joke 🙂
Since I’m spending this summer at UofT, I had to travel for just under 30 hours to get from Toronto to Macau, where I shared an airbnb with my colleague Marie and our master student Daniël. I arrived around midnight, in a 30-degree humid heat, with a 12-hour jet lag and still a bit flustered by the protests I encountered at Hong Kong airport. While the airbnb wasn’t particularly fancy (we are on a budget, after all) I was grateful for the AC, the cool shower, and the opportunity to check the wheels of my suitcase for crushed cockroach.
After a short night on my part, we walked to the west entrance of The Venetian: the hotel/casino that would host the conference. The Venetian was… an experience. During one of the breaks, I sneaked upstairs to check out the replica of Venice they built on the third floor. It was… an experience. All the stores and all the people made me dizzy, and the building seemed to have been designed by M.C. Escher: no matter where I went, I always ended up at the shops or the casino.
I quickly learned that it was better to stay outside. The outside temperature never went below 30 °C, and with the high humidity, hot sun and occasional thunderstorm, most of us were soaked within a minute of leaving the venue. Therefore, we tended to stay inside, checking out the food courts of the adjacent hotels during our lunch breaks.
All the kitsch and replicas, as well as the mimes in the Parisian and the singing gondoliers in the Venetian created such a surreal feeling of dreaming in me. A very weird place to go for doing your job as a computer scientist, indeed.
After Daniël presented his paper (I am so proud!) and Marie presented hers (she has so much authority!) in the morning, Holger gave an invited talk about Cooperative competition: A new way of solving challenging problems in AI and beyond, giving us all food for thought for the reception afterwards. Of course, our mascotte, little Ada, was there to support us. Later, she came with us for a celebratory drink, showing her support for CLAIRE, too.
My IJCAI highlights
One of my main highlights was getting to see professor Adnan Darwiche talk about his work. Throughout my short career as a PhD student, I kept returning to his work. In particular I have been building on his ideas of knowledge compilation: clever ways of representing solutions to problems that allow us to reason about those problems in a very fast manner. Darwiche inspires me as someone who, even at this stage of his career, still publishes papers all by himself. He is a great example to me, and it was incredible to get the opportunity to attend his talk on Reasoning about the behaviour of AI systems.
I was also very excited to attend talks by so many of the amazing fellow PhD students and postdocs I’ve met over the years: dr. Sebastijan Dumančić presented two papers in the Understanding Intelligence and Human-level AI in the New Machine Learning Era track and received an honorable mention for the EurAI Distinguished Dissertation Award 2018, and Alberto Camacho presented so many papers on Linear Temporal Logic (LTL) in both workshops and main conference, that I frankly lost count. Aside from these presentations, I also attended those of dr. Emir Demirović, Mohit Kumar, prof. dr. Lars Kotthoff and many, many more. All in all, there were so many presentations by these research friends, that they even overlapped, and I had to choose! (And no, I won’t tell you whose talks I did not get to attend 😉 )
This year was the second time that I attended the Women’s Lunch. I don’t know if this was intentional or not, but I got the impression that the organisers made a real effort to include the younger generation of female AI researchers in the community. Almost all of the senior women present at the lunch approached me to introduce themselves or to just have a chat.
Finally, of course, there was my own presentation. Unfortunately, I was awarded a time slot at the end of a session at the end of the day at the end of the conference, so the audience wasn’t as alert and attentive as I would’ve hoped. Despite this, they asked some interesting questions and laughed at my jokes, so that was good.
When I think back at how much effort went into this paper, it is hard to believe. I might actually write a blog post about it one day, for I reckon it could be interesting for someone outside academia to get a glimpse of what goes into publishing a bit of research at a conference. For now, the most important thing to say is that I feel very, very fortunate to have worked on this project with two of the most dedicated, smartest, kindest researchers I know: dr. Behrouz Babaki and prof. dr. Siegfried Nijssen.
Time for some R&R
Occasionally, we have a little time for being a tourist. I went to the northern part of Macau on two different evenings, sampling the local food. On another evening we went to visit the fake Eiffel Tower and looked out over all the lights in the city.
On our last day we went to Macau tower for lunch. It has a rotating restaurant and we very much enjoyed the view, both of the city and of the people bungee-jumping off the tower!
After lunch we went to see the Guia Lighthouse and Lou Lim Loc garden. Unfortunately all in the pouring rain, but we still enjoyed the views and the turtles. We bought some bubble tea and took a taxi back to the hotel. From there, I took a bus to the ferry to Hong Kong, to continue my travels there.