diff options
| author | adamjanovsky | 2023-02-23 13:00:03 +0100 |
|---|---|---|
| committer | adamjanovsky | 2023-02-23 13:00:03 +0100 |
| commit | 2cc17ac4134fc223e8bb0a2aed38e235a9ad3a7b (patch) | |
| tree | e08264f478741b639da241ba38d89b4040e9e7eb /data/reference_annotations/readme.md | |
| parent | ca82101078c2ecbc971483722537afb833637e30 (diff) | |
| download | sec-certs-2cc17ac4134fc223e8bb0a2aed38e235a9ad3a7b.tar.gz sec-certs-2cc17ac4134fc223e8bb0a2aed38e235a9ad3a7b.tar.zst sec-certs-2cc17ac4134fc223e8bb0a2aed38e235a9ad3a7b.zip | |
more notes to ref. annotation methodology
Diffstat (limited to 'data/reference_annotations/readme.md')
| -rw-r--r-- | data/reference_annotations/readme.md | 6 |
1 files changed, 6 insertions, 0 deletions
diff --git a/data/reference_annotations/readme.md b/data/reference_annotations/readme.md index a6541d7e..fbb3fd4a 100644 --- a/data/reference_annotations/readme.md +++ b/data/reference_annotations/readme.md @@ -28,6 +28,12 @@ After manually inspecting random certificates, we have identified the following 4. Label all references in [manual_annotations](manual_annotations/) as follows: - The reference meaning is recovered based on certification report and security target pdf. - The annotator should visit certification report first. If data in certification report is ambigous or does not allow precise annotation, the annotator should further visit the security target pdf. + - The annotator should set `None` label on instances that he/she was unable to decide. + - The annotator should label the `source` as: + - `report` if he/she decided purely based on the data from certification report + - `target` if he/she decided purely based on the data from security target + - `report+target` if both artifacts were needed for resolution + - Note that this label is only internal, not intended to be used for research - Artifacts of the referenced certificate are not examined. - The labeling is done by a pair of co-authors, the inter-annotator agreement is measured with Cohen's Kappa |