Coding framework

In the first pilot, two coders working from the codebook alone each coded the same ten documents, in five languages, on the core dimensions. The codebook's author had read five of the documents while drafting it and had not read the other five. Both coders were instances of the same language model, which probably makes their agreement higher than two people would obtain. We plan to compare their codes with codes made by hand by a member of the team.

In a second pilot, two coders coded ten further documents on the revised second draft. Neither the codebook's author nor the first pilot had used these documents, and they include more council motions, orders and general racism plans than the first pilot's documents. A difference in agreement between the two pilots may therefore come from the revised rules or from the different documents. Agreement rose on definitions mentioned, indicators of success and the specificity of measures, and fell on how the problem and the solution are described and on the placement of antisemitism among other hatreds, mostly in documents that say little about antisemitism as a problem.

The table gives, for each document-level dimension, the mean overlap between the two coders' sets of codes (1 means identical sets), the number of documents in which the sets were identical, and Cohen's kappa on whether each category was present.