{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# sec-certs Python API demo\n", "\n", "In this demo, we will:\n", "1. Fetch the fully processed dataset from our web\n", "2. Turn the dataset into a [pandas](https://pandas.pydata.org/) dataframe -- a data structure suitable for further data analysis.\n", "3. Filter the dataset to certificates of our interest\n", "4. Explore various attrributes of a dataset and its individual certificate\n", "5. Learn how to go from a single vulnerability to all certificates that *may suffer* from the vulnerability\n", "6. Plot various characteristics of the dataset\n" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Fetch the fully processed dataset from our web" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# First, import the necessary libraries\n", "from sec_certs.dataset import CCDataset\n", "import pandas as pd" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Downloading CC Dataset: 100%|██████████| 144M/144M [00:14<00:00, 10.6MB/s] \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "The downloaded CCDataset contains 5330 certificates\n" ] } ], "source": [ "# Download the dataset and see how many certificates it contains\n", "dataset = CCDataset.from_web()\n", "print(f\"The downloaded CCDataset contains {len(dataset)} certificates\")" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# # Don't run this! This creates new empty dataset, crawls commoncriteriaportal.org and processed all certificates.\n", "# dset = CCDataset()\n", "# dset.get_certs_from_web()\n", "# dset.process_auxillary_datasets()\n", "# dset.download_all_artifacts()\n", "# dset.convert_all_pdfs()\n", "# dset.analyze_certificates()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Turn the dataset into a [pandas](https://pandas.pydata.org/) dataframe -- a data structure suitable for further data analysis." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "df = dataset.to_pandas()" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Index(['cert_id', 'name', 'status', 'category', 'manufacturer', 'scheme',\n", " 'security_level', 'eal', 'not_valid_before', 'not_valid_after',\n", " 'report_link', 'st_link', 'cert_link', 'manufacturer_web',\n", " 'extracted_versions', 'cpe_matches', 'verified_cpe_matches',\n", " 'related_cves', 'directly_referenced_by', 'indirectly_referenced_by',\n", " 'directly_referencing', 'indirectly_referencing', 'extracted_sars',\n", " 'protection_profiles', 'cert_lab', 'year_from'],\n", " dtype='object')" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# print columns of the dataset\n", "df.columns" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "dgst\n", "8298c7814b3b2860 NaN\n", "9a1c767d358eee50 NaN\n", "979e00ac7d3e229c NaN\n", "a6015339aa4434cf NaN\n", "a89ee15539d006d1 NaN\n", "Name: related_cves, dtype: object" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Select a single column\n", "df_vulns = df.related_cves\n", "df_vulns.head()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "# Save the dataset as json (might not work from browser)\n", "dataset.to_json(\"cc_dataset.json\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Filter the dataset to certificates of our interest" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "# Select only certificates that have some related cves\n", "vulnerable_certificates = df.loc[df.related_cves.notnull()]\n", "\n", "# Select only certificates from BSI\n", "bsi_certs = df.loc[df.cert_lab == \"BSI\"]\n", "\n", "# Select only certificates certified in 2017 or later\n", "certs_2017_and_newer = df.loc[df.not_valid_before.dt.year >= 2017]\n", "\n", "# select only active certificates\n", "active_certificates = df.loc[df.status == \"active\"]\n", "\n", "# Select only certificates that reference some other certificate\n", "reference_rich_certificates = df.loc[df.directly_referencing.notnull()]\n", "\n", "# Select only certificates with EAL6 and higher\n", "eal6_or_more = df.loc[df.eal >= \"EAL6\"]" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
| \n", " | cert_id | \n", "name | \n", "status | \n", "category | \n", "manufacturer | \n", "scheme | \n", "security_level | \n", "eal | \n", "not_valid_before | \n", "not_valid_after | \n", "... | \n", "verified_cpe_matches | \n", "related_cves | \n", "directly_referenced_by | \n", "indirectly_referenced_by | \n", "directly_referencing | \n", "indirectly_referencing | \n", "extracted_sars | \n", "protection_profiles | \n", "cert_lab | \n", "year_from | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dgst | \n", "\n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " |
| ef1fba0df82eb56e | \n", "BSI-DSZ-CC-1096-2021 | \n", "Arbit Data Diode 10 GbE v1.00 | \n", "active | \n", "Boundary Protection Devices and Systems | \n", "Arbit Cyber Defence Systems ApS | \n", "DE | \n", "{EAL7+, ALC_FLR.1} | \n", "EAL7+ | \n", "2021-04-07 | \n", "2026-04-07 | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "{SAR(family='ALC_FLR', level=1), SAR(family='A... | \n", "NaN | \n", "BSI | \n", "2021 | \n", "
| 167ed26aa5bc8193 | \n", "NSCIB-CC-163925-CR | \n", "Fort Fox Hardware Data Diode FFHDD3_1/10 | \n", "active | \n", "Boundary Protection Devices and Systems | \n", "Fox-IT B.V. | \n", "NL | \n", "{ALC_FLR.3, ASE_TSS.2, EAL7+} | \n", "EAL7+ | \n", "2018-07-11 | \n", "2023-07-11 | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "{SAR(family='ASE_CCL', level=1), SAR(family='A... | \n", "NaN | \n", "NaN | \n", "2018 | \n", "
| 6cd7a7a1cffaa67e | \n", "NSCIB-CC-0095534-CR3 | \n", "NXP JCOP 4.7 SE051 | \n", "active | \n", "ICs, Smart Cards and Smart Card-Related Device... | \n", "NXP Semiconductors | \n", "NL | \n", "{ASE_TSS.2, EAL6+, ALC_FLR.1} | \n", "EAL6+ | \n", "2023-03-27 | \n", "2028-03-27 | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "{BSI-DSZ-CC-1136-V3-2022, NSCIB-CC-180212-CR5} | \n", "{BSI-DSZ-CC-1136-2021, BSI-DSZ-CC-1136-V3-2022... | \n", "{SAR(family='ALC_FLR', level=1), SAR(family='A... | \n", "[Java Card Protection Profile - Open Configura... | \n", "NaN | \n", "2023 | \n", "
| 51e4fcdced4053b0 | \n", "ANSSI-CC-2023/14 | \n", "S3D384C/S3D352C/S3D300C/S3D264C/S3D232C/S3K384... | \n", "active | \n", "ICs, Smart Cards and Smart Card-Related Device... | \n", "Samsung Electronics Co., Ltd. | \n", "FR | \n", "{ASE_TSS.2, EAL6+} | \n", "EAL6+ | \n", "2023-03-02 | \n", "2028-03-02 | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "{ANSSI-CC-2021/56} | \n", "{ANSSI-CC-2021/56} | \n", "{SAR(family='ASE_CCL', level=1), SAR(family='A... | \n", "[Security IC Platform Protection Profile with ... | \n", "NaN | \n", "2023 | \n", "
| 1116a6b1b0edbf8d | \n", "NSCIB-CC-66030-CR6 | \n", "NXP Crypto Library V3.1.x on P6021y VB | \n", "active | \n", "ICs, Smart Cards and Smart Card-Related Device... | \n", "NXP Semiconductors Germany GmbH, Business Unit... | \n", "NL | \n", "{ASE_TSS.2, EAL6+, ALC_FLR.1} | \n", "EAL6+ | \n", "2023-02-16 | \n", "2028-02-16 | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "{BSI-DSZ-CC-1072-V5-2022} | \n", "{BSI-DSZ-CC-1072-V2-2019, BSI-DSZ-CC-1072-V5-2... | \n", "{SAR(family='ASE_CCL', level=1), SAR(family='A... | \n", "[Security IC Platform Protection Profile with ... | \n", "NaN | \n", "2023 | \n", "
5 rows × 26 columns
\n", "| \n", " | cert_id | \n", "name | \n", "status | \n", "category | \n", "manufacturer | \n", "scheme | \n", "security_level | \n", "eal | \n", "not_valid_before | \n", "not_valid_after | \n", "... | \n", "directly_referenced_by | \n", "indirectly_referenced_by | \n", "directly_referencing | \n", "indirectly_referencing | \n", "extracted_sars | \n", "protection_profiles | \n", "cert_lab | \n", "year_from | \n", "is_vulnerable_to_roca | \n", "references_roca | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dgst | \n", "\n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " |
| 6a07be907f66f5d1 | \n", "BSI-DSZ-CC-1147-V2-2023 | \n", "MTCOS Pro 2.5 EAC with PACE / P71D352 (N7121) | \n", "active | \n", "ICs, Smart Cards and Smart Card-Related Device... | \n", "MaskTech International GmbH | \n", "DE | \n", "{EAL5+, ALC_DVS.2, AVA_VAN.5} | \n", "EAL5+ | \n", "2023-02-22 | \n", "2028-02-22 | \n", "... | \n", "{BSI-DSZ-CC-1148-V2-2023} | \n", "{BSI-DSZ-CC-1148-V2-2023} | \n", "{BSI-DSZ-CC-1136-V3-2022, BSI-DSZ-CC-1147-2020... | \n", "{BSI-DSZ-CC-0891-2015, BSI-DSZ-CC-1136-V3-2022... | \n", "{SAR(family='ASE_CCL', level=1), SAR(family='A... | \n", "[Machine Readable Travel Document with ICAO Ap... | \n", "BSI | \n", "2023 | \n", "False | \n", "True | \n", "
| 49bcce8117e22c62 | \n", "BSI-DSZ-CC-1148-V2-2023 | \n", "MTCOS Pro 2.5 EAC with PACE / P71D352 (N7121) ... | \n", "active | \n", "ICs, Smart Cards and Smart Card-Related Device... | \n", "MaskTech International GmbH | \n", "DE | \n", "{ALC_DVS.2, EAL4+} | \n", "EAL4+ | \n", "2023-02-22 | \n", "2028-02-22 | \n", "... | \n", "NaN | \n", "NaN | \n", "{BSI-DSZ-CC-1147-V2-2023, BSI-DSZ-CC-1136-V3-2... | \n", "{BSI-DSZ-CC-0891-2015, BSI-DSZ-CC-1136-V3-2022... | \n", "{SAR(family='ASE_CCL', level=1), SAR(family='A... | \n", "[Protection Profile for Machine Readable Trave... | \n", "BSI | \n", "2023 | \n", "False | \n", "True | \n", "
| 01cb2ad202b5df12 | \n", "ANSSI-CC-2022/68 | \n", "ACOS-IDv2.1 eMRTD (A) BAC Configuration(Versio... | \n", "active | \n", "ICs, Smart Cards and Smart Card-Related Device... | \n", "Austria Card plastikkarten und Ausweissysteme ... | \n", "FR | \n", "{ALC_TAT.2, ALC_FLR.1, EAL4+, ALC_CMS.5, ADV_T... | \n", "EAL4+ | \n", "2022-12-15 | \n", "2027-12-15 | \n", "... | \n", "NaN | \n", "NaN | \n", "{ANSSI-CC-2022/08, BSI-DSZ-CC-1110-V5-2022} | \n", "{BSI-DSZ-CC-1110-V4-2021, BSI-DSZ-CC-0945-2017... | \n", "{SAR(family='ALC_FLR', level=1), SAR(family='A... | \n", "[Protection Profile for Machine Readable Trave... | \n", "NaN | \n", "2022 | \n", "False | \n", "True | \n", "
| 37739e3954762bc2 | \n", "ANSSI-CC-2022/64 | \n", "ACOS-IDv2.1 SSCD (A) CB-Comm (Version 2.1 SSCD... | \n", "active | \n", "ICs, Smart Cards and Smart Card-Related Device... | \n", "Austria Card plastikkarten und Ausweissysteme ... | \n", "FR | \n", "{EAL5+, ALC_DVS.2, ALC_FLR.1, AVA_VAN.5} | \n", "EAL5+ | \n", "2022-12-15 | \n", "2027-12-15 | \n", "... | \n", "NaN | \n", "NaN | \n", "{ANSSI-CC-2022/19, BSI-DSZ-CC-1110-V5-2022} | \n", "{ANSSI-CC-2022/19, BSI-DSZ-CC-0945-2017, BSI-D... | \n", "{SAR(family='ALC_FLR', level=1), SAR(family='A... | \n", "[Protection profiles for secure signature crea... | \n", "NaN | \n", "2022 | \n", "False | \n", "True | \n", "
| a4b7ab85878030d7 | \n", "ANSSI-CC-2022/65 | \n", "ACOS-IDv2.1 SSCD (A) CL-TC-Comm (Version 2.1 S... | \n", "active | \n", "ICs, Smart Cards and Smart Card-Related Device... | \n", "Austria Card plastikkarten und Ausweissysteme ... | \n", "FR | \n", "{EAL5+, ALC_DVS.2, ALC_FLR.1, AVA_VAN.5} | \n", "EAL5+ | \n", "2022-12-15 | \n", "2027-12-15 | \n", "... | \n", "NaN | \n", "NaN | \n", "{ANSSI-CC-2022/20, BSI-DSZ-CC-1110-V5-2022} | \n", "{BSI-DSZ-CC-1110-V4-2021, BSI-DSZ-CC-0945-2017... | \n", "{SAR(family='ALC_FLR', level=1), SAR(family='A... | \n", "[Protection profiles for secure signature crea... | \n", "NaN | \n", "2022 | \n", "False | \n", "True | \n", "
5 rows × 28 columns
\n", "| \n", " | cert_id | \n", "name | \n", "status | \n", "category | \n", "manufacturer | \n", "scheme | \n", "security_level | \n", "eal | \n", "not_valid_before | \n", "not_valid_after | \n", "... | \n", "directly_referenced_by | \n", "indirectly_referenced_by | \n", "directly_referencing | \n", "indirectly_referencing | \n", "extracted_sars | \n", "protection_profiles | \n", "cert_lab | \n", "year_from | \n", "is_vulnerable_to_roca | \n", "references_roca | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dgst | \n", "\n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " |
| 350581534e265186 | \n", "ANSSI-CC-2013/55 | \n", "Plateforme jTOP INFv#46 masquée sur composants... | \n", "archived | \n", "ICs, Smart Cards and Smart Card-Related Device... | \n", "Trusted Logic / Infineon | \n", "FR | \n", "{EAL5+, ALC_DVS.2, AVA_VAN.5} | \n", "EAL5+ | \n", "2013-08-07 | \n", "2019-09-01 | \n", "... | \n", "{ANSSI-CC-2015/25, ANSSI-CC-2013/64} | \n", "{ANSSI-CC-2015/25, ANSSI-CC-2013/64} | \n", "{BSI-DSZ-CC-0829-2012} | \n", "{BSI-DSZ-CC-0829-2012, BSI-DSZ-CC-0728-2011, B... | \n", "{SAR(family='ALC_DVS', level=2), SAR(family='A... | \n", "[Java Card™ System Protection Profile Open Con... | \n", "SERMA | \n", "2013 | \n", "False | \n", "True | \n", "
1 rows × 28 columns
\n", "