Add cert_id_evalution notebook, improve cert-id heuristics.

4 files changed, 473 insertions, 281 deletions

diff --git a/data/cert_id_eval/cert_id_eval.ipynb b/data/cert_id_eval/cert_id_eval.ipynb
new file mode 100644
index 00000000..1c1b369c
--- /dev/null
+++ b/data/cert_id_eval/cert_id_eval.ipynb
@@ -0,0 +1,459 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "collapsed": true,
+    "pycharm": {
+     "name": "#%% md\n"
+    }
+   },
+   "source": [
+    "# CC certificate id evaluation\n",
+    "This notebook can be used to evaluate our heuristics for certificate id assignment\n",
+    "and canonicalization.\n",
+    "\n",
+    "It looks at several aspects & issues:\n",
+    "\n",
+    "1. Certificates with no id assigned.\n",
+    "2. Duplicate certificate id assignments (when two certificates get the same ID assigned).\n",
+    "3. Certificates that have the same certification report document (an issue of the input data that we get\n",
+    "   that explains some of the duplicate certificate id assignments)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "from sec_certs.dataset import CCDataset\n",
+    "import csv"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "# dset = CCDataset.from_web_latest()\n",
+    "dset = CCDataset.from_json(\"../../cc_dset/CommonCriteria_dataset.json\")\n",
+    "# dset._compute_normalized_cert_ids()\n",
+    "# dset.to_json()"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "## Certificates with no id\n",
+    "\n",
+    "Here we report the number of certificates in our dataset that we have no certificate ID for."
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%% md\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "missing_id_dgsts = set()\n",
+    "for cert in dset:\n",
+    "    if not cert.heuristics.cert_id:\n",
+    "        print(cert.dgst, cert.heuristics.cert_id, cert.scheme)\n",
+    "        missing_id_dgsts.add(cert.dgst)\n",
+    "print(f\"Total: {len(missing_id_dgsts)}\")"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "### Check manually evaluated missing\n"
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "i = 0\n",
+    "with open(\"missing_ids.csv\", \"r\") as f:\n",
+    "    reader = csv.DictReader(f)\n",
+    "    for line in reader:\n",
+    "        try:\n",
+    "            cert = dset[line[\"id\"]]\n",
+    "        except:\n",
+    "            continue\n",
+    "        if line[\"cert_id\"] and line[\"cert_id\"] != cert.heuristics.cert_id:\n",
+    "            i += 1\n",
+    "            print(line[\"id\"], line[\"cert_id\"], cert.heuristics.cert_id, line[\"reason\"])\n",
+    "print(f\"Total: {i}\")"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "\n",
+    "The following cell checks which manually analyzed missing certificate IDs were since fixed."
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%% md\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "manual_missing_ids = set()\n",
+    "i = 0\n",
+    "with open(\"missing_ids.csv\", \"r\") as f:\n",
+    "    reader = csv.DictReader(f)\n",
+    "    for line in reader:\n",
+    "        manual_missing_ids.add(line[\"id\"])\n",
+    "        if line[\"id\"] not in missing_id_dgsts:\n",
+    "            i += 1\n",
+    "            print(\",\".join(line.values()))\n",
+    "print(f\"Total: {i}\")"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "The following cell lists missing certificate IDs that *went missing* since manual analysis."
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "new_missing_ids = missing_id_dgsts.difference(manual_missing_ids)\n",
+    "for idd in new_missing_ids:\n",
+    "    cert = dset[idd]\n",
+    "    print(idd, cert.heuristics.cert_id)\n",
+    "print(f\"Total: {len(new_missing_ids)}\")"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "## Duplicate certificate id assignment\n",
+    "\n",
+    "Here we report the number of certificates in our dataset that have a duplicate certiticate\n",
+    "ID assigned."
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "id_mapping = {}\n",
+    "for cert in dset:\n",
+    "    if cert.heuristics.cert_id is not None:\n",
+    "        c_list = id_mapping.setdefault(cert.heuristics.cert_id, [])\n",
+    "        c_list.append(cert.dgst)\n",
+    "\n",
+    "duplicate_id_dgsts = set()\n",
+    "for idd, entries in id_mapping.items():\n",
+    "    if len(entries) > 1 and idd:\n",
+    "        print(idd, entries)\n",
+    "        duplicate_id_dgsts.update(entries)\n",
+    "print(f\"Total: {len(duplicate_id_dgsts)}\")"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "## Duplicate report documents\n",
+    "\n",
+    "Some certificates have erroneously uploaded certificate reports, here we check their\n",
+    "hashes and report such duplicates in the input data."
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%% md\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "duplicate_docs = {}\n",
+    "\n",
+    "for cert in dset:\n",
+    "    if cert.state.report_pdf_hash is not None:\n",
+    "        r_list = duplicate_docs.setdefault(cert.state.report_pdf_hash, [])\n",
+    "        r_list.append(cert.dgst)\n",
+    "\n",
+    "duplicate_doc_dgsts = set()\n",
+    "for hash, entries in duplicate_docs.items():\n",
+    "    if len(entries) > 1:\n",
+    "        print(hash, entries)\n",
+    "        for entry in entries:\n",
+    "            duplicate_doc_dgsts.add(entry)\n",
+    "print(f\"Total: {len(duplicate_doc_dgsts)}\")"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "The following prints the amount of certificate id duplicates due to input data (two or more\n",
+    "certificates share a certification report document)."
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%% md\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "duplicate_ids_due_doc = duplicate_doc_dgsts.intersection(duplicate_id_dgsts)\n",
+    "print(len(duplicate_ids_due_doc))"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "The following prints the amount of certificate id duplicates that are not due to input data."
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%% md\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "duplicate_ids_issue = duplicate_id_dgsts.difference(duplicate_doc_dgsts)\n",
+    "print(len(duplicate_ids_issue))"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "### Check manually evaluated duplicates\n",
+    "\n",
+    "The following cell checks that id collisions that were manually analyzed in the past have been fixed.\n",
+    "A `True` means that we have now assigned the correct ID."
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%% md\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "i = 0\n",
+    "with open(\"duplicate_ids.csv\", \"r\") as f:\n",
+    "    reader = csv.DictReader(f)\n",
+    "    for line in reader:\n",
+    "        try:\n",
+    "            cert = dset[line[\"id\"]]\n",
+    "        except:\n",
+    "            continue\n",
+    "        if line[\"true_id\"] != cert.heuristics.cert_id:\n",
+    "            print(line[\"id\"],line[\"result\"], line[\"true_id\"] == cert.heuristics.cert_id , line[\"true_id\"], cert.heuristics.cert_id, line[\"fixable\"])\n",
+    "            i += 1\n",
+    "print(f\"Total: {i}\")"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "The following cell lists those duplicates that were fixed by changes since manual analysis."
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%% md\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "manual_duplicate_ids = set()\n",
+    "i = 0\n",
+    "with open(\"duplicate_ids.csv\", \"r\") as f:\n",
+    "    reader = csv.DictReader(f)\n",
+    "    for line in reader:\n",
+    "        manual_duplicate_ids.add(line[\"id\"])\n",
+    "        if line[\"id\"] not in duplicate_id_dgsts:\n",
+    "            i += 1\n",
+    "            print(\",\".join(line.values()))\n",
+    "print(f\"Total: {i}\")"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "The following cell lists duplicates that were *created* since manual analysis."
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%% md\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "new_duplicate_ids = duplicate_id_dgsts.difference(manual_duplicate_ids)\n",
+    "for idd in new_duplicate_ids:\n",
+    "    cert = dset[idd]\n",
+    "    print(idd, cert.heuristics.cert_id)\n",
+    "print(f\"Total: {len(new_duplicate_ids)}\")"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
\ No newline at end of file
diff --git a/data/cert_id_eval/duplicate_documents.ipynb b/data/cert_id_eval/duplicate_documents.ipynb
deleted file mode 100644
index c189b197..00000000
--- a/data/cert_id_eval/duplicate_documents.ipynb
+++ /dev/null
@@ -1,267 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "collapsed": true,
-    "pycharm": {
-     "name": "#%% md\n"
-    }
-   },
-   "source": [
-    "# CC document duplicates"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "outputs": [],
-   "source": [
-    "from sec_certs.dataset import CCDataset\n",
-    "import csv"
-   ],
-   "metadata": {
-    "collapsed": false,
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Downloading CC Dataset: 100%|██████████| 133M/133M [00:18<00:00, 7.40MB/s] \n"
-     ]
-    }
-   ],
-   "source": [
-    "dset = CCDataset.from_web_latest()"
-   ],
-   "metadata": {
-    "collapsed": false,
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "4cc94ac9efb882f9fb446352dff34cf97b4505f437bd7ed12b5103a28fcdb04a [('088504703b9eedc1', 'FR', 'ANSSI-CC-2019/20'), ('c36783a79de002c5', 'FR', 'ANSSI-CC-2019/20')]\n",
-      "07164a214dfd06454e9353499d8985182139580740dc316560f4ae9a200d038c [('ba5cbf3c600d99ac', 'FR', 'ANSSI-CC-2014/46'), ('da7fa774f888542b', 'FR', 'ANSSI-CC-2014/46')]\n",
-      "665a19820a817fbd0b588e6681a70db35033cf575b55b50ebbff3d9eacd04834 [('4c33266562bb9b24', 'SE', 'CSEC2017009'), ('99ebfcfea90e5e75', 'SE', 'CSEC2017009')]\n",
-      "522e0b7a67ea60bed93e9506729aca9e4786a4433b70779b540ae11bc125946e [('887139711564fdb3', 'DE', 'BSI-DSZ-CC-1139-V3-2021'), ('793075287bf70a81', 'DE', 'BSI-DSZ-CC-1139-V3-2021')]\n",
-      "93617722f9726ead858d55aabdd0d65f150ac7e81c608df9c496d1f34904c9f0 [('4e246ff7fd346630', 'DE', 'BSI-DSZ-CC-1053-2018'), ('a6629b6ce5514a43', 'DE', 'BSI-DSZ-CC-1053-2018')]\n",
-      "d6ca43922026b6117e90b5dd12aa95b286ea4ac6f406927602f9645d70bf9581 [('4df9f5ce6006aa63', 'DE', 'BSI-DSZ-CC-1030-2018'), ('83b56f938e262858', 'DE', 'BSI-DSZ-CC-1030-2018')]\n",
-      "f406d07297102db5c6c2f2045821bca9f03afa97c2ba3145ce1eaae091175094 [('f254bdb3e177193d', 'ES', '2016-47-INF-2126'), ('b1f92dfbc5d51547', 'ES', '2016-47-INF-2126')]\n",
-      "dcb1304f44e60a78854edabc65a041362ab6241ba59164d973f08887934124d8 [('20f851314c4f57d9', 'US', 'CCEVS-VR-VID11024-2019'), ('b4288d36bc129d2f', 'US', 'CCEVS-VR-VID11024-2019'), ('dfa12d7893c9efdf', 'US', 'CCEVS-VR-VID11024-2019')]\n",
-      "87eca384777d14af604f86b556ca96daf173a396505fea4878162650e7f8f89e [('27baf39406deb767', 'FR', 'ANSSI-CC-2018/41'), ('10b17081dd7cad8f', 'FR', 'ANSSI-CC-2018/41')]\n",
-      "0187adf4b8885d9a13f1d2d889363991bb7b74aa84d3db5b53ee3a224c3156e4 [('11a73a5b4bff867e', 'US', 'CCEVS-VR-06-0008'), ('b33406a237a97bf0', 'US', 'CCEVS-VR-06-0008')]\n",
-      "6d440f398975c2035b540fe52b597925ed2646c404d024346944ecb5f842d561 [('8bf284d3b482ad65', 'US', 'CCEVS-VR-05-0124'), ('3ad601a9cb30dfef', 'US', 'CCEVS-VR-05-0124')]\n",
-      "00cbcb05882299c3d51548f012718c5fad96beaf6a62221ef148f3bf6d31ca2f [('3c24bbe7724dbfdd', 'CA', '383-4-198-CR'), ('0a3793d207c47790', 'CA', '383-4-198-CR')]\n",
-      "a9eca7bb6bfe081b54f9f4c7f42a2a09f5165fdc1ee97b346ce8761639502817 [('e2f3a1cdd592ab05', 'US', 'CCEVS-VR-10425-2011'), ('52f07fb00a81780e', 'US', 'CCEVS-VR-10425-2011')]\n",
-      "550f0378f93f161739070d162b30872cd4e71b7bb5006f4e9672a6e092a57c41 [('1c7087cf1e7ad810', 'US', 'CCEVS-VR-05-0137'), ('feeecec3ac6f0503', 'US', 'CCEVS-VR-05-0137')]\n",
-      "a59eeb860d9e4d777126ade1269cc3ed26111306c279531e04c8c13383f1789b [('6f0a50d3bd7db7c7', 'US', 'CCEVS-VR-10610-2015'), ('e93207ee36a5fabf', 'US', 'CCEVS-VR-10610-2015')]\n",
-      "959baa0c8c0862e7546c946665174c7dd05276191772e4ecd86af0cc6e9a1d43 [('f43765c81befb944', 'FR', 'ANSSI-CC-2014/36'), ('8d6bca0d7f228191', 'FR', 'ANSSI-CC-2014/36')]\n",
-      "9511d9e692e3cb8efb5d77a46bd257cb73f7ea9b4a7bd8d641ed0fc58d98cb4a [('bf7b3c78598f74cd', 'US', 'CCEVS-VR-VID10003-2008'), ('8f538d2eb9a1c6c7', 'US', 'CCEVS-VR-VID10003-2008')]\n",
-      "30010ed4e1b294c92f22a16304bfdedfa1c88716601449e96d49cd6a22fcfb84 [('031667f4e242da61', 'DE', 'BSI-DSZ-CC-0588-2009'), ('03b4ff588fb88dae', 'DE', 'BSI-DSZ-CC-0588-2009')]\n",
-      "59f3ef546ddd2ef0721b82bf6f5c565f8d50709b1c5780e499a4ace57a887fe6 [('355a5b9240e32cc9', 'US', 'CCEVS-VR-05-0091'), ('c79dafc7b0da855e', 'US', 'CCEVS-VR-05-0091')]\n",
-      "bc3b61739ac1082c81c477653ecb76ea81fc6c233650ad87bec544e6ead28c9a [('a67fe4216fd88917', 'US', 'CCEVS-VR-05-0102'), ('ec25be3458d728cd', 'US', 'CCEVS-VR-05-0102')]\n",
-      "51e598cb3a466f45f5fe2c519b8d8ddbc3767e1dbb8b65942de2ee8fa3bb8ca6 [('ee35dcafbe1b9532', 'FR', 'ANSSI-CC-2013/09'), ('a5adb726852f5cc5', 'FR', 'ANSSI-CC-2013/09')]\n",
-      "710cab8e86bd4f0e056c286191dcb0e8d225123f29ba5c8568800dccaf9d2d22 [('18e68a485e648ddc', 'FR', 'ANSSI-CC-2013/16'), ('9d15e7bf1444f8b8', 'FR', 'ANSSI-CC-2013/16')]\n",
-      "fc1fc60b6e1c812449536868fda1fe78b8f986110edcfeaeeb7cfd2e4255a166 [('69c75283a4ccf934', 'FR', 'ANSSI-CC-2010/36'), ('5e4c757231135ba1', 'FR', 'ANSSI-CC-2010/36')]\n",
-      "5baf1a55e4c08e022766f929ae84707cc9a91b12308549ab1630893d573e634c [('4e9bb649bce14c86', 'FR', 'ANSSI-CC-2010/39'), ('30ae700d801bd41c', 'FR', 'ANSSI-CC-2010/39')]\n",
-      "b6fb530f0674580e992b734b075f98e39daa8b2d92ce83d48fb7eb8513ea7aa5 [('2f53e1d94ca1a010', 'FR', 'ANSSI-CC-2010/34'), ('27ccec0740bb7915', 'FR', 'ANSSI-CC-2010/34')]\n",
-      "e06bcf3cd06cb1a44e069023d118d12284ed5e72fad3849fbc39eb1c285da8c2 [('41bd2924e9afced5', 'FR', 'ANSSI-CC-2011/07'), ('796445972ef2ff5b', 'FR', 'ANSSI-CC-2011/07')]\n",
-      "a2a59212a999759c5a75f24408d520d850d50b5591932d3d1673a75992a5ffdc [('0372b733c28f305a', 'FR', 'ANSSI-CC-2010/24'), ('b9789028f97490ce', 'FR', 'ANSSI-CC-2010/24')]\n",
-      "84c57cdd98ddae7edf245e4246866e98956ab657ccec4169ea36ecddfad7ea85 [('5d13cfce75a45cd1', 'FR', 'ANSSI-CC-2010/23'), ('74727e1e9f40b1db', 'FR', 'ANSSI-CC-2010/23')]\n",
-      "1d75ccb32d4998770a997486e9c3b7113a31d07c32f67b96da58446127bc1fcd [('8826a8ae0369901a', 'FR', 'ANSSI-CC-2010/03'), ('67a79bfc84b4071b', 'FR', 'ANSSI-CC-2010/03')]\n",
-      "89ac72c45dd136d36fc2cb90386d0feca9d48a2cdaabc95dd998e3ef27285e5f [('baead73ecc6e4a33', 'FR', 'ANSSI-CC-2009/46'), ('d76f210e41d5589c', 'FR', 'ANSSI-CC-2009/46')]\n",
-      "63b270e4758326c0f42369f0ffdd4882e46a110389d8901fc6aa02fc0a395af7 [('80ae5f71b5fa278a', 'FR', 'ANSSI-CC-2009/47'), ('0cbcee9f1cde47a8', 'FR', 'ANSSI-CC-2009/47')]\n",
-      "0d0cb46d860cce3b9783d0459d29c07e3bdf162f93de971187457b1110efa121 [('1b5a04adf5047f67', 'FR', 'ANSSI-CC-2009/49'), ('1188aebd8681e4fe', 'FR', 'ANSSI-CC-2009/49')]\n",
-      "695a144a7fe6f6327bd25b4500871eb89e7af20cb9dd18627ca70ec518cba88c [('fad651102525c5a4', 'FR', 'ANSSI-CC-2009/36'), ('928e364f71d64595', 'FR', 'ANSSI-CC-2009/36')]\n",
-      "9d832fa6cedfa8779c12b333355be2352e66c7909edef8714bc399b1cad3ab75 [('a52f807db0eb7fdb', 'FR', 'ANSSI-2009/24'), ('e6a8cbbc8c557bed', 'FR', 'ANSSI-2009/24')]\n",
-      "0e25deccc55eb6817213e132ba56d7ba524bf0e8d3a97ab06e16884504770e42 [('0b0648a9b9a26747', 'ES', '2004-3-INF-71'), ('1bb3da30bc8f4750', 'ES', '2004-3-INF-71')]\n",
-      "839967798c7b3851313702fe92f0fa2f58288c89a8cad7c90893138028850615 [('c3f8e8ddb1dabb02', 'FR', 'BSI-DSZ-CC-0223-2003'), ('4408217c4ce524fd', 'FR', 'BSI-DSZ-CC-0223-2003')]\n",
-      "528f44ec9a6fb8d0cea099f50e1430be4d104f4e88d06adc5044b7ba1dde2625 [('41708ba21f2bb4a7', 'DE', 'BSI-DSZ-CC-0196-2003'), ('063d4ed52b084441', 'DE', 'BSI-DSZ-CC-0196-2003')]\n",
-      "4997e90f46b56c022c3b6d2cf0f1190ae930eab0ba5ac38706afc978227a3545 [('a79f15874e255007', 'UK', None), ('20c98a49ea86a519', 'UK', None)]\n",
-      "a629fab094efe1b0b56f1abe80f62c190cc94a0c3db1f07694fa81f6122de634 [('e510cd16d4f9f4a3', 'UK', 'CERTIFICATION REPORT No. P141'), ('b802ce16d43a545e', 'UK', 'CERTIFICATION REPORT No. P141')]\n",
-      "0d27e41592c28820b996b2f3e5086db67d97a602c9a87b81c66eef3235f77b8e [('aa49720eb88f291c', 'US', 'CCEVS-VR-VID10559-2014'), ('449ea70a6342d680', 'US', 'CCEVS-VR-VID10559-2014')]\n",
-      "4468f2b1f1887ec6d2bd958e7468d038d63d019d476abfa157dfff3448fa270e [('31b7cfe0a8a73263', 'FR', 'ANSSI-CC-2010/16'), ('1bb49ac0d6f7944d', 'FR', 'ANSSI-CC-2010/16')]\n",
-      "2fd0d0500ae3fd6f256a20649ec3dab60e3706598b7d8a9238733d9a65a36d68 [('d42974d5cce56320', 'US', 'CCEVS-VR-04-0074'), ('1bcac3b368582ae1', 'US', 'CCEVS-VR-04-0074')]\n",
-      "7b99b70d97a108f3ad77dbe670ce3e690ba9e7e00c7fca52a0b97ce30460cd12 [('370d590d42b7d9f3', 'US', 'CCEVS-VR-VID11039-2020'), ('11162192b8001047', 'US', 'CCEVS-VR-VID11039-2020')]\n",
-      "9ec8209bac6e8f36122e2f94de65d61a3f026bcaf748add5196a925199aed39e [('8ba5d4d02ea73d3d', 'CA', '383-4-100-CR'), ('c4e1391e8ba34694', 'CA', '383-4-100-CR')]\n",
-      "9e16b403f88963dd417e3bcd95f2bc133daebe222762d13fa0f4292d928638d4 [('a5fd51390f814a46', 'NO', 'SERTIT-014'), ('8278a9ba639a08d0', 'NO', 'SERTIT-014'), ('3ba740ca6ee8ed75', 'NO', 'SERTIT-014')]\n",
-      "177de21de10e98c1e16ff7edb0957b06d4173c3697a822e0e3aba1a60e0df5b6 [('0f6c8bde7c6574e3', 'US', 'CCEVS-VR-VID10327-2009'), ('c80e1ca4d8641a3d', 'US', 'CCEVS-VR-VID10327-2009')]\n",
-      "e0a94363a3a69502dd53a983f764943d01ccc32eaf94321e7bc351ad673dee1d [('686005d0b5ff5c5c', 'US', 'CCEVS-VR-06-0024'), ('b901bbd0ce1f0a3b', 'US', 'CCEVS-VR-06-0024')]\n",
-      "7bd5b517b9bad7d57aa509ba4c24bf61aa31e1a2c7c07954dd3552a92388817f [('f3bb1641b4bb0aa0', 'FR', 'Rapport de certification 2005/01'), ('749fae44017016c5', 'DE', 'Rapport de certification 2005/01')]\n",
-      "61f65021cd05268cd26b56e5e4d3aff4f43cb2ed876e03a4c1ac9e0780688059 [('814a2ac0aeae08b1', 'US', 'CCEVS-VR-03-0044'), ('02a31d0ed42e4ff1', 'US', 'CCEVS-VR-03-0044')]\n",
-      "49\n"
-     ]
-    }
-   ],
-   "source": [
-    "hashes = {}\n",
-    "\n",
-    "for cert in dset:\n",
-    "    if cert.state.report_pdf_hash is not None:\n",
-    "        r_entry = (cert.dgst, cert.scheme, cert.heuristics.cert_id)\n",
-    "        r_list = hashes.setdefault(cert.state.report_pdf_hash, [])\n",
-    "        r_list.append(r_entry)\n",
-    "\n",
-    "dups = set()\n",
-    "i = 0\n",
-    "for hash, entries in hashes.items():\n",
-    "    if len(entries) > 1:\n",
-    "        i += 1\n",
-    "        print(hash, entries)\n",
-    "        for entry in entries:\n",
-    "            dups.add(entry[0])\n",
-    "print(i)"
-   ],
-   "metadata": {
-    "collapsed": false,
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 18,
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "CCEVS-VR-10425-2011,e2f3a1cdd592ab05,,,,\n",
-      "CCEVS-VR-10425-2011,52f07fb00a81780e,,,,\n",
-      "CCEVS-VR-05-0124,8bf284d3b482ad65,,,,\n",
-      "CCEVS-VR-05-0124,3ad601a9cb30dfef,,,,\n",
-      "CCEVS-VR-05-0102,a67fe4216fd88917,,,,\n",
-      "CCEVS-VR-05-0102,ec25be3458d728cd,,,,\n",
-      "383-4-198-CR,3c24bbe7724dbfdd,,,,\n",
-      "383-4-198-CR,0a3793d207c47790,,,,\n",
-      "CCEVS-VR-VID10327-2009,c80e1ca4d8641a3d,,,,\n",
-      "CCEVS-VR-VID10327-2009,0f6c8bde7c6574e3,,,,\n",
-      "ANSSI-CC-2010/36,5e4c757231135ba1,,,,\n",
-      "ANSSI-CC-2010/36,69c75283a4ccf934,,,,\n",
-      "Rapport de certification 2005/01,f3bb1641b4bb0aa0,,,,\n",
-      "Rapport de certification 2005/01,749fae44017016c5,,,,\n",
-      "2004-3-INF-71,1bb3da30bc8f4750,,,,\n",
-      "2004-3-INF-71,0b0648a9b9a26747,,,,\n",
-      "CCEVS-VR-VID10559-2014,aa49720eb88f291c,,,,\n",
-      "CCEVS-VR-VID10559-2014,449ea70a6342d680,,,,\n",
-      "ANSSI-CC-2010/24,0372b733c28f305a,,,,\n",
-      "ANSSI-CC-2010/24,b9789028f97490ce,,,,\n",
-      "ANSSI-CC-2010/34,2f53e1d94ca1a010,,,,\n",
-      "ANSSI-CC-2010/34,27ccec0740bb7915,,,,\n",
-      "CSEC2017009,4c33266562bb9b24,,,,\n",
-      "CSEC2017009,99ebfcfea90e5e75,,,,\n",
-      "BSI-DSZ-CC-1030-2018,4df9f5ce6006aa63,,,,\n",
-      "BSI-DSZ-CC-1030-2018,83b56f938e262858,,,,\n",
-      "CCEVS-VR-06-0008,11a73a5b4bff867e,,,,\n",
-      "CCEVS-VR-06-0008,b33406a237a97bf0,,,,\n",
-      "ANSSI-CC-2014/36,8d6bca0d7f228191,,,,\n",
-      "ANSSI-CC-2014/36,f43765c81befb944,,,,\n",
-      "CCEVS-VR-VID10003-2008,bf7b3c78598f74cd,,,,\n",
-      "CCEVS-VR-VID10003-2008,8f538d2eb9a1c6c7,,,,\n",
-      "ANSSI-CC-2009/46,baead73ecc6e4a33,,,,\n",
-      "ANSSI-CC-2009/46,d76f210e41d5589c,,,,\n",
-      "BSI-DSZ-CC-1053-2018,4e246ff7fd346630,,,,\n",
-      "BSI-DSZ-CC-1053-2018,a6629b6ce5514a43,,,,\n",
-      "ANSSI-CC-2018/41,10b17081dd7cad8f,,,,\n",
-      "ANSSI-CC-2018/41,27baf39406deb767,,,,\n",
-      "ANSSI-CC-2009/49,1b5a04adf5047f67,,,,\n",
-      "ANSSI-CC-2009/49,1188aebd8681e4fe,,,,\n",
-      "CCEVS-VR-05-0137,feeecec3ac6f0503,,,,\n",
-      "CCEVS-VR-05-0137,1c7087cf1e7ad810,,,,\n",
-      "ANSSI-CC-2013/09,a5adb726852f5cc5,,,,\n",
-      "ANSSI-CC-2013/09,ee35dcafbe1b9532,,,,\n",
-      "ANSSI-CC-2010/03,67a79bfc84b4071b,,,,\n",
-      "ANSSI-CC-2010/03,8826a8ae0369901a,,,,\n",
-      "BSI-DSZ-CC-0196-2003,063d4ed52b084441,,,,\n",
-      "BSI-DSZ-CC-0196-2003,41708ba21f2bb4a7,,,,\n",
-      "383-4-100-CR,8ba5d4d02ea73d3d,,,,\n",
-      "383-4-100-CR,c4e1391e8ba34694,,,,\n",
-      "ANSSI-CC-2019/20,c36783a79de002c5,,,,\n",
-      "ANSSI-CC-2019/20,088504703b9eedc1,,,,\n",
-      "CCEVS-VR-10610-2015,6f0a50d3bd7db7c7,,,,\n",
-      "CCEVS-VR-10610-2015,e93207ee36a5fabf,,,,\n",
-      "CCEVS-VR-04-0074,d42974d5cce56320,,,,\n",
-      "CCEVS-VR-04-0074,1bcac3b368582ae1,,,,\n",
-      "CCEVS-VR-06-0024,686005d0b5ff5c5c,,,,\n",
-      "CCEVS-VR-06-0024,b901bbd0ce1f0a3b,,,,\n",
-      "ANSSI-CC-2011/07,41bd2924e9afced5,,,,\n",
-      "ANSSI-CC-2011/07,796445972ef2ff5b,,,,\n",
-      "ANSSI-2009/24,a52f807db0eb7fdb,,,,\n",
-      "ANSSI-2009/24,e6a8cbbc8c557bed,,,,\n",
-      "2016-47-INF-2126,f254bdb3e177193d,,,,\n",
-      "2016-47-INF-2126,b1f92dfbc5d51547,,,,\n",
-      "ANSSI-CC-2010/39,4e9bb649bce14c86,,,,\n",
-      "ANSSI-CC-2010/39,30ae700d801bd41c,,,,\n",
-      "ANSSI-CC-2014/46,da7fa774f888542b,,,,\n",
-      "ANSSI-CC-2014/46,ba5cbf3c600d99ac,,,,\n"
-     ]
-    }
-   ],
-   "source": [
-    "with open(\"duplicate_ids.csv\", \"r\") as f:\n",
-    "    reader = csv.DictReader(f)\n",
-    "    for line in reader:\n",
-    "        if line[\"id\"] in dups:\n",
-    "            if not line[\"result\"]:\n",
-    "                print(\",\".join(line.values()))"
-   ],
-   "metadata": {
-    "collapsed": false,
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "outputs": [],
-   "source": [],
-   "metadata": {
-    "collapsed": false,
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   }
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 2
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython2",
-   "version": "2.7.6"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}
\ No newline at end of file
diff --git a/data/cert_id_eval/duplicate_ids.csv b/data/cert_id_eval/duplicate_ids.csv
index 7fab5ae3..400583bf 100644
--- a/data/cert_id_eval/duplicate_ids.csv
+++ b/data/cert_id_eval/duplicate_ids.csv
@@ -8,10 +8,10 @@ ANSSI-CC-2012/70,aabd0ff0d82fff72,tp,ANSSI-CC-2012/70,,
 ANSSI-CC-2012/70,bcac7e3f9f614ec6,fp,KECS-ISIS-0675-2015,number of mentions of component certificate higher than actual,scheme mismatch
 ANSSI-CC-2012/70,f2327302df1aa660,fp,KECS-ISIS-0435-2013,number of mentions of component certificate higher than actual,scheme mismatch
 BSI-DSZ-CC-0645-2010,a60b819509d62b84,tp,BSI-DSZ-CC-0645-2010,,
-BSI-DSZ-CC-0645-2010,720aab4f782f4c60,fp,2011-14-INF-1095v1,number of mentions of component certificate higher than actual,scheme mismatch
-BSI-DSZ-CC-0645-2010,9e7c513fdd35df17,fp,2011-13-INF-1092v1,number of mentions of component certificate higher than actual,scheme mismatch
-BSI-DSZ-CC-0645-2010,2fb4fdf2a570c366,fp,2011-15-INF-1098v1,number of mentions of component certificate higher than actual,scheme mismatch
-BSI-DSZ-CC-0645-2010,a5e8267322d0d7a6,fp,2011-12-INF-1089v1,number of mentions of component certificate higher than actual,scheme mismatch
+BSI-DSZ-CC-0645-2010,720aab4f782f4c60,fp,2011-14-INF-1095,number of mentions of component certificate higher than actual,scheme mismatch
+BSI-DSZ-CC-0645-2010,9e7c513fdd35df17,fp,2011-13-INF-1092,number of mentions of component certificate higher than actual,scheme mismatch
+BSI-DSZ-CC-0645-2010,2fb4fdf2a570c366,fp,2011-15-INF-1098,number of mentions of component certificate higher than actual,scheme mismatch
+BSI-DSZ-CC-0645-2010,a5e8267322d0d7a6,fp,2011-12-INF-1089,number of mentions of component certificate higher than actual,scheme mismatch
 BSI-DSZ-CC-1110-V3-2020,5346d5a4418466ab,fp,KECS-ISIS-1031-2020,number of mentions of component certificate higher than actual,scheme mismatch
 BSI-DSZ-CC-1110-V3-2020,a7840a951b1de6c2,fp,ANSSI-CC-2020/87,number of mentions of component certificate higher than actual,scheme mismatch
 BSI-DSZ-CC-1110-V3-2020,09220515a7edf3f9,tp,BSI-DSZ-CC-1110-V3-2020,,
@@ -35,9 +35,9 @@ BSI-DSZ-CC-0223-2003,c3f8e8ddb1dabb02,fp,DCSSI-CC-2003/12,number of mentions of
 ANSSI-CC-2019/01,f4b4b026e87b5086,fp,KECS-ISIS-0937-2019,number of mentions of component certificate higher than actual,scheme mismatch
 ANSSI-CC-2019/01,fbc78e2407910a4b,fp,KECS-ISIS-0936-2019,number of mentions of component certificate higher than actual,scheme mismatch
 ANSSI-CC-2019/01,84c8bf1f809380bc,tp,ANSSI-CC-2019/01,,
-ANSSI-CC-2018/12,a0c38b4389ad7cf7,fp,2017-63-INF-3232-v1,number of mentions of component certificate higher than actual,scheme mismatch
+ANSSI-CC-2018/12,a0c38b4389ad7cf7,fp,2017-63-INF-3232,number of mentions of component certificate higher than actual,scheme mismatch
 ANSSI-CC-2018/12,10107b83e7393c50,tp,ANSSI-CC-2018/12,,
-ANSSI-CC-2018/12,7dfcefd1ffeaa941,fp,2017-62-INF-3233-v1,number of mentions of component certificate higher than actual,scheme mismatch
+ANSSI-CC-2018/12,7dfcefd1ffeaa941,fp,2017-62-INF-3233,number of mentions of component certificate higher than actual,scheme mismatch
 ANSSI-CC-2021/49,6fd73557577cf9ed,tp,ANSSI-CC-2021/49,,
 ANSSI-CC-2021/49,037577fc2019fcfa,fp,ANSSI-CC-2022/28,number of mentions of component certificate higher than actual,frontpage match + file metadata
 ANSSI-CC-2021/49,d55eb8c97d71e843,fp,ANSSI-CC-2022/24,number of mentions of component certificate higher than actual,frontpage match + file metadata
@@ -59,8 +59,8 @@ ANSSI-CC-2021/52,536aac44b608f951,tp,ANSSI-CC-2021/52,,
 ANSSI-CC-2021/51,a47713a44e4c656b,tp,ANSSI-CC-2021/51,,
 ANSSI-CC-2021/51,940a18f62c79ffae,fp,ANSSI-CC-2022/30,number of mentions of component certificate higher than actual,frontpage match + file metadata
 ANSSI-CC-2021/51,5787ff8207fd751d,fp,ANSSI-CC-2022/26,number of mentions of component certificate higher than actual,frontpage match + file metadata
-BSI-DSZ-CC-0945-2017,457fcec1ad31841f,fp,ANSSI-CC-2022/37v22,number of mentions of component certificate higher than actual,scheme mismatch
-BSI-DSZ-CC-0945-2017,0b36924a005c2ae2,fp,ANSSI-CC-2022/36v22,number of mentions of component certificate higher than actual,scheme mismatch
+BSI-DSZ-CC-0945-2017,457fcec1ad31841f,fp,ANSSI-CC-2022/37v2,number of mentions of component certificate higher than actual,scheme mismatch
+BSI-DSZ-CC-0945-2017,0b36924a005c2ae2,fp,ANSSI-CC-2022/36v2,number of mentions of component certificate higher than actual,scheme mismatch
 BSI-DSZ-CC-0945-2017,5103648897ba1b3a,tp,BSI-DSZ-CC-0945-2017,,
 CCEVS-VR-05-0091,c79dafc7b0da855e,tp,CCEVS-VR-05-0091,clone,
 CCEVS-VR-05-0091,355a5b9240e32cc9,tp,CCEVS-VR-05-0091,clone,
@@ -128,8 +128,8 @@ BSI-DSZ-CC-0950-V2-2018,a46522d0a592c1fe,tp,BSI-DSZ-CC-0950-V2-2018,clone + remo
 BSI-DSZ-CC-0950-V2-2018,6f90b0bf6bbe884b,tp,BSI-DSZ-CC-0950-V2-2018,clone,
 Rapport de certification 2005/01,f3bb1641b4bb0aa0,tp,Rapport de certification 2005/01,clone,
 Rapport de certification 2005/01,749fae44017016c5,tp,Rapport de certification 2005/01,clone + mismatched scheme,scheme mismatch
-2004-3-INF-71,1bb3da30bc8f4750,tp,2004-3-INF-71v2,clone,
-2004-3-INF-71,0b0648a9b9a26747,tp,2004-3-INF-71v2,clone,
+2004-3-INF-71,1bb3da30bc8f4750,tp,2004-3-INF-71,clone,
+2004-3-INF-71,0b0648a9b9a26747,tp,2004-3-INF-71,clone,
 CCEVS-VR-VID10559-2014,aa49720eb88f291c,tp,CCEVS-VR-VID10559-2014,,
 CCEVS-VR-VID10559-2014,449ea70a6342d680,fp,,wrong certification report,
 ANSSI-CC-2021/30,c62b878b1ac43df9,tp,ANSSI-CC-2021/30,,
@@ -206,7 +206,7 @@ ANSSI-2009/24,a52f807db0eb7fdb,tp,ANSSI-2009/24,clone,
 ANSSI-2009/24,e6a8cbbc8c557bed,tp,ANSSI-2009/24,clone,
 ANSSI-CC-2013/47,34370b67b5e675c3,fp,KECS-ISIS-0489-2014,number of mentions of component certificate higher than actual,scheme mismatch
 ANSSI-CC-2013/47,6aa2f56da553d95d,tp,ANSSI-CC-2013/47,,
-ANSSI-CC-2010/02,7d4585a4b5b6e873,fp,KECS-ISIS-0419-201,number of mentions of component certificate higher than actual,scheme mismatch
+ANSSI-CC-2010/02,7d4585a4b5b6e873,fp,KECS-ISIS-0419-2012,number of mentions of component certificate higher than actual,scheme mismatch
 ANSSI-CC-2010/02,dbbf02a1cd0ad33b,tp,ANSSI-CC-2010/02,,
 2016-47-INF-2126,f254bdb3e177193d,tp,2016-47-INF-2126,clone,
 2016-47-INF-2126,b1f92dfbc5d51547,tp,2016-47-INF-2126,clone,
@@ -220,5 +220,5 @@ CSEC2016007,13e306eb69c8bf95,tp,CSEC2016007,clone (recertification?),
 CSEC2016007,0c3ff6669dcf6968,tp,CSEC2016007,clone (recertification?),
 BSI-DSZ-CC-0639-2010,2f9feaf4121720da,tp,BSI-DSZ-CC-0639-2010,,
 BSI-DSZ-CC-0639-2010,db7627e56b8621b1,fp,KECS-ISIS-0394-2012,number of mentions of component certificate higher than actual,scheme mismatch
-Certification Report 98/94,28bb2a8869397c4c,fp,CERTIFICATION REPORT No. P106,number of mentions of component certificate higher than actual,scheme mismatch
-Certification Report 98/94,28a9fc2137d72ce9,fp,CERTIFICATION REPORT No. P103,number of mentions of component certificate higher than actual,scheme mismatch
+Certification Report 98/94,28bb2a8869397c4c,fp,CRP106,number of mentions of component certificate higher than actual,scheme mismatch
+Certification Report 98/94,28a9fc2137d72ce9,fp,CRP103,number of mentions of component certificate higher than actual,scheme mismatch
diff --git a/data/cert_id_eval/missing_ids.csv b/data/cert_id_eval/missing_ids.csv
index 44d0d8a8..40645785 100644
--- a/data/cert_id_eval/missing_ids.csv
+++ b/data/cert_id_eval/missing_ids.csv
@@ -55,7 +55,7 @@ b43100f226a61f53,FR,,malformed PDF
 1a8ac8fbe5bfcda9,NO,SERTIT-017,404 for report PDF
 3faae83d248da8ba,IT,,cert_id missing
 d5ed93f6586469f6,CA,503-LSS,regex missing
-a45f04b9a5244a51,KR,KECS-CR-09-49,has unicode hyphens 8208
+a45f04b9a5244a51,KR,KECS-ISIS-0189-2009,has unicode hyphens 8208
 243a29b98f34ae9a,SE,CSEC2019009,space in cert_id
 88eb228eeb487597,KR,KECS-ISIS-0223-2010,has unicode hyphens 8208
 739f48568b127f22,CA,,cert_id missing