diff options
| author | adamjanovsky | 2023-02-23 13:00:03 +0100 |
|---|---|---|
| committer | adamjanovsky | 2023-02-23 13:00:03 +0100 |
| commit | 2cc17ac4134fc223e8bb0a2aed38e235a9ad3a7b (patch) | |
| tree | e08264f478741b639da241ba38d89b4040e9e7eb | |
| parent | ca82101078c2ecbc971483722537afb833637e30 (diff) | |
| download | sec-certs-2cc17ac4134fc223e8bb0a2aed38e235a9ad3a7b.tar.gz sec-certs-2cc17ac4134fc223e8bb0a2aed38e235a9ad3a7b.tar.zst sec-certs-2cc17ac4134fc223e8bb0a2aed38e235a9ad3a7b.zip | |
more notes to ref. annotation methodology
4 files changed, 9 insertions, 3 deletions
diff --git a/data/reference_annotations/manual_annotations/test.csv b/data/reference_annotations/manual_annotations/test.csv index 6208f2c0..fd43357f 100644 --- a/data/reference_annotations/manual_annotations/test.csv +++ b/data/reference_annotations/manual_annotations/test.csv @@ -1,4 +1,4 @@ -dgst,referenced_cert_id,location,label,comment +dgst,referenced_cert_id,source,label,comment 00a2b91e5b58cef1,ANSSI-CC-2014/23,None,None,None 00ad121fa63e5dfb,BSI-DSZ-CC-0973-V2-2016,None,None,None 00ad121fa63e5dfb,BSI-DSZ-CC-0996-2018,None,None,None diff --git a/data/reference_annotations/manual_annotations/train.csv b/data/reference_annotations/manual_annotations/train.csv index d7c82b2b..b29c609e 100644 --- a/data/reference_annotations/manual_annotations/train.csv +++ b/data/reference_annotations/manual_annotations/train.csv @@ -1,4 +1,4 @@ -dgst,referenced_cert_id,location,label,comment +dgst,referenced_cert_id,source,label,comment e676143c80802a59,BSI-DSZ-CC-0891-V4-2019,None,None,None 0bf7a19b22163465,ANSSI-CC-2016/44,None,None,None ab3af998dff7a2ef,ANSSI-CC-2017/47,None,None,None diff --git a/data/reference_annotations/manual_annotations/valid.csv b/data/reference_annotations/manual_annotations/valid.csv index 07f30a91..09368347 100644 --- a/data/reference_annotations/manual_annotations/valid.csv +++ b/data/reference_annotations/manual_annotations/valid.csv @@ -1,4 +1,4 @@ -dgst,referenced_cert_id,location,label,comment +dgst,referenced_cert_id,source,label,comment a21d17fd2d8b8edc,BSI-DSZ-CC-0813-2012,None,None,None 657f8d0cc39bbcdc,BSI-DSZ-CC-0946-2014,None,None,None 0efe2627f1ce8ac2,BSI-DSZ-CC-0351-2006,None,None,None diff --git a/data/reference_annotations/readme.md b/data/reference_annotations/readme.md index a6541d7e..fbb3fd4a 100644 --- a/data/reference_annotations/readme.md +++ b/data/reference_annotations/readme.md @@ -28,6 +28,12 @@ After manually inspecting random certificates, we have identified the following 4. Label all references in [manual_annotations](manual_annotations/) as follows: - The reference meaning is recovered based on certification report and security target pdf. - The annotator should visit certification report first. If data in certification report is ambigous or does not allow precise annotation, the annotator should further visit the security target pdf. + - The annotator should set `None` label on instances that he/she was unable to decide. + - The annotator should label the `source` as: + - `report` if he/she decided purely based on the data from certification report + - `target` if he/she decided purely based on the data from security target + - `report+target` if both artifacts were needed for resolution + - Note that this label is only internal, not intended to be used for research - Artifacts of the referenced certificate are not examined. - The labeling is done by a pair of co-authors, the inter-annotator agreement is measured with Cohen's Kappa |