From 8e7a3d8a172f62e13833f4d033b52e0cd38d66f6 Mon Sep 17 00:00:00 2001
From: Adam Janovsky
Date: Tue, 26 Apr 2022 11:41:58 +0200
Subject: populate docs structure
---
docs/quickstart.md | 3 +++
1 file changed, 3 insertions(+)
create mode 100644 docs/quickstart.md
(limited to 'docs/quickstart.md')
diff --git a/docs/quickstart.md b/docs/quickstart.md
new file mode 100644
index 00000000..b40bab01
--- /dev/null
+++ b/docs/quickstart.md
@@ -0,0 +1,3 @@
+# Quickstart
+
+TBA
\ No newline at end of file
--
cgit v1.3.1
From e6a433e86be235cb4bfc0b80bcef719df42812d7 Mon Sep 17 00:00:00 2001
From: Adam Janovsky
Date: Tue, 26 Apr 2022 16:10:36 +0200
Subject: add simple installation instr. and quickstart
---
CONTRIBUTING.md | 7 +------
README.md | 22 +---------------------
docs/installation.md | 31 ++++++++++++++++++++++++++++++-
docs/quickstart.md | 3 ++-
4 files changed, 34 insertions(+), 29 deletions(-)
(limited to 'docs/quickstart.md')
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 80585c2d..542a39da 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -10,12 +10,7 @@ You contribution is warmly welcomed. You can help by:
## Dependencies
-Our [Dockerfile](https://github.com/crocs-muni/sec-certs/blob/main/docker/Dockerfile) presents all the required dependencies, elaborated below.
-
-- [Java](https://www.java.com/en) is needed to parse tables in FIPS pdf documents, must be available from `PATH`.
-- Some imported libraries have non-trivial dependencies to resolve:
- - [pdftotext](https://github.com/jalan/pdftotext) requires [Poppler](https://poppler.freedesktop.org/) to be installed. We've experienced issues with older versions of Poppler (`0.x`), make sure to install `20.x` version of these libraries.
- - [graphviz](https://pypi.org/project/graphviz/) requires `graphviz` to be on the path
+For complete list of system dependencies, see [docs/installation](seccerts.org/docs/installation).
### Requirements
diff --git a/README.md b/README.md
index 69a036d3..6464900a 100644
--- a/README.md
+++ b/README.md
@@ -15,27 +15,7 @@ This project is developed by the [Centre for Research On Cryptography and Securi
## Installation
-The tool can be pulled as a docker image with
-
-```bash
-docker pull seccerts/sec-certs
-```
-
-Alternatively, it can be installed from PyPi with
-
-```bash
-pip install -U sec-certs
-```
-
-Note, however, that `Python>=3.8` is required and there are some [additional dependencies](https://github.com/crocs-muni/sec-certs/blob/main/CONTRIBUTING.md#dependencies).
-
-The stable release is also published on [GitHub](https://github.com/crocs-muni/sec-certs/releases) from where it can be setup for development with
-
-```bash
-python3 -m venv venv
-source venv/bin/activate
-pip install -e .
-```
+Use Docker with `docker pull seccerts/sec-certs` or just `pip install -U sec-certs`. For more elaborate description, see [docs](seccerts.org/docs/installation)
## Usage (CC)
diff --git a/docs/installation.md b/docs/installation.md
index 0e6c025a..236a9dc9 100644
--- a/docs/installation.md
+++ b/docs/installation.md
@@ -1,3 +1,32 @@
# Installation
-TBA
\ No newline at end of file
+The tool can be pulled as a docker image with
+
+```bash
+docker pull seccerts/sec-certs
+```
+
+Alternatively, it can be installed from PyPi with
+
+```bash
+pip install -U sec-certs
+```
+
+Note, however, that `Python>=3.8` is required and there are some additional dependencies (see below) that are not shipped with the binary distribution.
+
+The stable release is also published on [GitHub](https://github.com/crocs-muni/sec-certs/releases) from where it can be setup for development with
+
+```bash
+python3 -m venv venv
+source venv/bin/activate
+pip install -e .
+```
+
+Alternatively, our Our [Dockerfile](https://github.com/crocs-muni/sec-certs/blob/main/docker/Dockerfile) represents a reproducible way of setting up the environment.
+
+## Dependencies
+
+- [Java](https://www.java.com/en) is needed to parse tables in FIPS pdf documents, must be available from `PATH`.
+- Some imported libraries have non-trivial dependencies to resolve:
+ - [pdftotext](https://github.com/jalan/pdftotext) requires [Poppler](https://poppler.freedesktop.org/) to be installed. We've experienced issues with older versions of Poppler (`0.x`), make sure to install `20.x` version of these libraries.
+ - [graphviz](https://pypi.org/project/graphviz/) requires `graphviz` to be on the path
\ No newline at end of file
diff --git a/docs/quickstart.md b/docs/quickstart.md
index b40bab01..74285025 100644
--- a/docs/quickstart.md
+++ b/docs/quickstart.md
@@ -1,3 +1,4 @@
# Quickstart
-TBA
\ No newline at end of file
+1. Install the latest version with `pip install -U sec-certs` (see [installation](installation.md)).
+2. Use either `cc_cli.py` or `fips_cli.py` for full-fledged processing of the certificate pipeline, or explore the [dataset notebook](notebooks/dataset.ipynb).
--
cgit v1.3.1
From 766164f7534e4d41a1668cc553c51c0ad3c0630f Mon Sep 17 00:00:00 2001
From: Adam Janovsky
Date: Sun, 1 May 2022 21:57:52 +0200
Subject: update notes
---
CONTRIBUTING.md | 2 +-
README.md | 2 +-
cc_cli.py | 4 +-
docs/conf.py | 12 +++---
docs/index.md | 3 +-
docs/quickstart.md | 31 ++++++++++++++-
fips_cli.py | 4 +-
notebooks/examples/common_criteria.ipynb | 65 ++++++++++++++++++++++++++++++++
notebooks/examples/dataset.ipynb | 65 --------------------------------
notebooks/examples/fips.ipynb | 65 ++++++++++++++++++++++++++++++++
10 files changed, 177 insertions(+), 76 deletions(-)
create mode 100644 notebooks/examples/common_criteria.ipynb
delete mode 100644 notebooks/examples/dataset.ipynb
create mode 100644 notebooks/examples/fips.ipynb
(limited to 'docs/quickstart.md')
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 542a39da..803cb340 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -10,7 +10,7 @@ You contribution is warmly welcomed. You can help by:
## Dependencies
-For complete list of system dependencies, see [docs/installation](seccerts.org/docs/installation).
+For complete list of system dependencies, see [docs/installation](https://seccerts.org/docs/installation.html).
### Requirements
diff --git a/README.md b/README.md
index 6464900a..c4b19802 100644
--- a/README.md
+++ b/README.md
@@ -15,7 +15,7 @@ This project is developed by the [Centre for Research On Cryptography and Securi
## Installation
-Use Docker with `docker pull seccerts/sec-certs` or just `pip install -U sec-certs`. For more elaborate description, see [docs](seccerts.org/docs/installation)
+Use Docker with `docker pull seccerts/sec-certs` or just `pip install -U sec-certs`. For more elaborate description, see [docs](https://seccerts.org/docs/installation.html)
## Usage (CC)
diff --git a/cc_cli.py b/cc_cli.py
index 11e1a40e..a9c20fc1 100755
--- a/cc_cli.py
+++ b/cc_cli.py
@@ -23,8 +23,10 @@ logger = logging.getLogger(__name__)
@click.option(
"-o",
"--output",
- type=click.Path(file_okay=False, dir_okay=True, writable=True, readable=True),
+ type=click.Path(file_okay=False, dir_okay=True, writable=True, readable=True, resolve_path=True),
help="Path where the output of the experiment will be stored. May overwrite existing content.",
+ default=Path("./cc_dset/"),
+ show_default=True,
)
@click.option(
"-c",
diff --git a/docs/conf.py b/docs/conf.py
index e6b76999..384e6569 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -18,8 +18,8 @@ from importlib.metadata import version as get_version
# -- Project information -----------------------------------------------------
project = "sec-certs"
-copyright = "2022, Adam Janovsky, Petr Svenda, Jan Jancar, Jiri Michalik, Stanislav Bobon"
-author = "Adam Janovsky, Petr Svenda, Jan Jancar, Jiri Michalik, Stanislav Bobon"
+copyright = "2020-2022, Adam Janovsky, Petr Svenda, Jan Jancar, Jiri Michalik, Stanislav Bobon."
+# author = "Adam Janovsky, Petr Svenda, Jan Jancar, Jiri Michalik, Stanislav Bobon"
# Note thas this inference won't work from Docker: https://github.com/pypa/setuptools_scm/#usage-from-docker
release = ".".join(get_version("sec-certs").split(".")[:3])
@@ -31,6 +31,9 @@ release = ".".join(get_version("sec-certs").split(".")[:3])
# ones.
extensions = ["myst_nb"]
+# Don't exeute notbooks
+nb_execution_mode = "off"
+
# Add any paths that contain templates here, relative to this directory.
templates_path = ["_templates"]
@@ -39,7 +42,6 @@ templates_path = ["_templates"]
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
-
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
@@ -59,7 +61,7 @@ html_theme_options = {
"repository_url": "https://github.com/crocs-muni/sec-certs",
"repository_branch": "issue/195-Docs",
"launch_buttons": {"binderhub_url": "https://mybinder.org"},
+ "use_fullscreen_button": False,
}
-# Don't exeute notbooks
-nb_execution_mode = "off"
+myst_heading_anchors = 3
diff --git a/docs/index.md b/docs/index.md
index 371b917c..8cd9c8ef 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -19,7 +19,8 @@ tutorial.md
```{toctree}
:caption: Notebook examples
-notebooks/examples/dataset.ipynb
+notebooks/examples/common_criteria.ipynb
+notebooks/examples/fips.ipynb
```
```{admonition} Launch notebooks in MyBinder
diff --git a/docs/quickstart.md b/docs/quickstart.md
index 74285025..48f7f1ce 100644
--- a/docs/quickstart.md
+++ b/docs/quickstart.md
@@ -1,4 +1,33 @@
# Quickstart
1. Install the latest version with `pip install -U sec-certs` (see [installation](installation.md)).
-2. Use either `cc_cli.py` or `fips_cli.py` for full-fledged processing of the certificate pipeline, or explore the [dataset notebook](notebooks/dataset.ipynb).
+2. Use
+```python
+dset = CCDataset.from_web_latest()
+```
+
+(Common Criteria) or
+
+```python
+dset = FIPSDataset.from_web_latest()
+```
+
+(FIPS 140) to obtain freshly processed datasets from [seccerts.org](https://seccerts.org).
+
+```{hint}
+You can work with those with the help of the [common criteria notebook](notebooks/examples/common_criteria.ipynb) or [fips notebook](notebooks/examples/fips.ipynb) and even launch them in MyBinder without installing anything. Just use the 🚀 icon (top-right corner).
+```
+
+If you insist on processing the whole certificates pipeline, make sure that you installed all [dependencies](installation.md#dependencies). Then, run
+
+```bash
+cc-certs all
+```
+
+for Common Criteria processing, or
+
+```bash
+fips-certs all
+```
+
+for FIPS 140 processing. This script takes a long time to run (few hours) and will create `./cc_dset` or `./fips_dset` directory. To see all options, call the entrypoint with `--help`.
\ No newline at end of file
diff --git a/fips_cli.py b/fips_cli.py
index 149d2f81..9ebbf6f1 100755
--- a/fips_cli.py
+++ b/fips_cli.py
@@ -26,8 +26,10 @@ logger = logging.getLogger(__name__)
@click.option(
"-o",
"--output",
- type=click.Path(file_okay=False, dir_okay=True, writable=True, readable=True),
+ type=click.Path(file_okay=False, dir_okay=True, writable=True, readable=True, resolve_path=True),
help="Path to the directory where the output of the 'build' or 'new-run' actions will be stored.",
+ default=Path("./fips_dset/"),
+ show_default=True,
)
@click.option(
"-c",
diff --git a/notebooks/examples/common_criteria.ipynb b/notebooks/examples/common_criteria.ipynb
new file mode 100644
index 00000000..4aab6c61
--- /dev/null
+++ b/notebooks/examples/common_criteria.ipynb
@@ -0,0 +1,65 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Common Criteria dataset class\n",
+ "\n",
+ "This notebook illustrates basic functionality with the `CCDataset` class that holds Common Criteria dataset"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sec_certs.dataset.common_criteria import CCDataset"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "4985\n"
+ ]
+ }
+ ],
+ "source": [
+ "dset = CCDataset.from_web_latest()\n",
+ "print(len(dset))"
+ ]
+ }
+ ],
+ "metadata": {
+ "interpreter": {
+ "hash": "6386d1612879d92d026c363e7667e428bc38d86c5a080d58c3d70e7cd43df67d"
+ },
+ "kernelspec": {
+ "display_name": "Python 3.8.1 ('certsvenv': venv)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.1"
+ },
+ "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/notebooks/examples/dataset.ipynb b/notebooks/examples/dataset.ipynb
deleted file mode 100644
index 7f9287aa..00000000
--- a/notebooks/examples/dataset.ipynb
+++ /dev/null
@@ -1,65 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Dataset class examples\n",
- "\n",
- "This notebook illustrates basic functionality with the `CCDataset` class that holds Common Criteria dataset"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "from sec_certs.dataset.common_criteria import CCDataset"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "4985\n"
- ]
- }
- ],
- "source": [
- "dset = CCDataset.from_web_latest()\n",
- "print(len(dset))"
- ]
- }
- ],
- "metadata": {
- "interpreter": {
- "hash": "6386d1612879d92d026c363e7667e428bc38d86c5a080d58c3d70e7cd43df67d"
- },
- "kernelspec": {
- "display_name": "Python 3.8.1 ('certsvenv': venv)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.8.1"
- },
- "orig_nbformat": 4
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/notebooks/examples/fips.ipynb b/notebooks/examples/fips.ipynb
new file mode 100644
index 00000000..ab20ee93
--- /dev/null
+++ b/notebooks/examples/fips.ipynb
@@ -0,0 +1,65 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# FIPS Dataset class\n",
+ "\n",
+ "This notebook illustrates basic functionality with the `FIPSDataset` class that holds FIPS 140 dataset"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sec_certs.dataset.fips import FIPSDataset"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "4985\n"
+ ]
+ }
+ ],
+ "source": [
+ "dset = FIPSDataset.from_web_latest()\n",
+ "print(len(dset))"
+ ]
+ }
+ ],
+ "metadata": {
+ "interpreter": {
+ "hash": "6386d1612879d92d026c363e7667e428bc38d86c5a080d58c3d70e7cd43df67d"
+ },
+ "kernelspec": {
+ "display_name": "Python 3.8.1 ('certsvenv': venv)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.1"
+ },
+ "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
--
cgit v1.3.1
From 99768447af7031d4ceb982ba58f49fcf8d4c4591 Mon Sep 17 00:00:00 2001
From: Adam Janovsky
Date: Fri, 20 May 2022 15:25:28 +0200
Subject: update docs
---
docs/api/model.md | 22 ++++++++++++++++++++++
docs/index.md | 9 +++++----
docs/installation.md | 25 ++++++++++++++++++++-----
docs/quickstart.md | 29 ++++++++++++++++++++++-------
4 files changed, 69 insertions(+), 16 deletions(-)
(limited to 'docs/quickstart.md')
diff --git a/docs/api/model.md b/docs/api/model.md
index 372815c7..406d24be 100644
--- a/docs/api/model.md
+++ b/docs/api/model.md
@@ -1,5 +1,14 @@
# Model package
+```{eval-rst}
+.. automodule:: sec_certs.model
+ :no-members:
+```
+
+```{tip}
+The examples related to this package can be found at [model notebook](./../notebooks/examples/model.ipynb).
+```
+
## CPEClassifier
```{eval-rst}
@@ -18,3 +27,16 @@
## DependencyFinder
+```{eval-rst}
+.. currentmodule:: sec_certs.model
+.. autoclass:: DependencyFinder
+ :members:
+```
+
+## DependencyVulnerabilityFinder
+
+```{eval-rst}
+.. currentmodule:: sec_certs.model
+.. autoclass:: DependencyVulnerabilityFinder
+ :members:
+```
\ No newline at end of file
diff --git a/docs/index.md b/docs/index.md
index 8e0883dc..3f44633b 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -4,6 +4,10 @@ Welcome to the technical documentation of *sec-certs* tool for the data analysis
There are three main parts of this documentation. *User's guide* describes high-level use of our tool. Driven by this knowledge, you can progress to *Notebook examples* that showcase most of the API that we use in the form of Jupyter notebooks. The documentation currently does not have all modules documented with `autodoc`, so for the API reference, you must directly inspect the [sec_certs](https://github.com/crocs-muni/sec-certs/tree/main/sec_certs) module. If you want, you can run the notebooks as they are stored in the [project repository](https://github.com/crocs-muni/sec-certs/tree/main/notebooks). If you are interested in contributing to our project or in other aspects of our development, you can consult the relevant *GitHub artifacts*
+```{admonition} Launch notebooks in MyBinder
+Each of the notebooks can be launched interactively in MyBinder by clicking on 🚀 icon (top-right corner).
+```
+
```{toctree}
:hidden:
:caption: Navigation
@@ -24,6 +28,7 @@ tutorial.md
:caption: Notebook examples
notebooks/examples/common_criteria.ipynb
notebooks/examples/fips.ipynb
+notebooks/examples/model.ipynb
```
```{toctree}
@@ -31,10 +36,6 @@ notebooks/examples/fips.ipynb
api/model.md
```
-```{admonition} Launch notebooks in MyBinder
-Each of the notebooks can be launched interactively in MyBinder by clicking on 🚀 icon (top-right corner).
-```
-
```{toctree}
:maxdepth: 2
:caption: GitHub artifacts
diff --git a/docs/installation.md b/docs/installation.md
index 8d71a170..71e685db 100644
--- a/docs/installation.md
+++ b/docs/installation.md
@@ -1,18 +1,28 @@
# Installation
-The tool can be pulled as a docker image with
+::::{tab-set}
+:::{tab-item} PyPi (pip)
+
+The tool can be installed from PyPi with
```bash
-docker pull seccerts/sec-certs
+pip install -U sec-certs
```
-Alternatively, it can be installed from PyPi with
+Note, that `Python>=3.8` is required.
+
+:::
+
+:::{tab-item} Docker
+
+The tool can be pulled as a docker image with
```bash
-pip install -U sec-certs
+docker pull seccerts/sec-certs
```
-Note, however, that `Python>=3.8` is required and there are some additional dependencies (see below) that are not shipped with the binary distribution.
+:::
+:::{tab-item} Build from sources
The stable release is also published on [GitHub](https://github.com/crocs-muni/sec-certs/releases) from where it can be setup for development with
@@ -24,6 +34,11 @@ pip install -e .
Alternatively, our Our [Dockerfile](https://github.com/crocs-muni/sec-certs/blob/main/Dockerfile) represents a reproducible way of setting up the environment.
+:::
+::::
+
+If you're not using Docker, you must install the dependencies as described below.
+
## Dependencies
- [Java](https://www.java.com/en) is needed to parse tables in FIPS pdf documents, must be available from `PATH`.
diff --git a/docs/quickstart.md b/docs/quickstart.md
index 48f7f1ce..14121212 100644
--- a/docs/quickstart.md
+++ b/docs/quickstart.md
@@ -1,18 +1,29 @@
# Quickstart
+::::{tab-set}
+
+:::{tab-item} Common Criteria
1. Install the latest version with `pip install -U sec-certs` (see [installation](installation.md)).
2. Use
```python
dset = CCDataset.from_web_latest()
```
+to obtain to obtain freshly processed dataset from [seccerts.org](https://seccerts.org).
-(Common Criteria) or
+3. Play with the dataset. See [example notebook](./notebooks/examples/common_criteria.ipynb).
+:::
+:::{tab-item} FIPS 140
+1. Install the latest version with `pip install -U sec-certs` (see [installation](installation.md)).
+2. Use
```python
dset = FIPSDataset.from_web_latest()
```
+to obtain to obtain freshly processed dataset from [seccerts.org](https://seccerts.org).
-(FIPS 140) to obtain freshly processed datasets from [seccerts.org](https://seccerts.org).
+3. Play with the dataset. See [example notebook](./notebooks/examples/fips.ipynb).
+:::
+::::
```{hint}
You can work with those with the help of the [common criteria notebook](notebooks/examples/common_criteria.ipynb) or [fips notebook](notebooks/examples/fips.ipynb) and even launch them in MyBinder without installing anything. Just use the 🚀 icon (top-right corner).
@@ -20,14 +31,18 @@ You can work with those with the help of the [common criteria notebook](notebook
If you insist on processing the whole certificates pipeline, make sure that you installed all [dependencies](installation.md#dependencies). Then, run
+::::{tab-set}
+:::{tab-item} Common Criteria
```bash
-cc-certs all
+$ cc-certs all
```
+:::
-for Common Criteria processing, or
-
+:::{tab-item} FIPS 140
```bash
-fips-certs all
+$ fips-certs new-run
```
+:::
+::::
-for FIPS 140 processing. This script takes a long time to run (few hours) and will create `./cc_dset` or `./fips_dset` directory. To see all options, call the entrypoint with `--help`.
\ No newline at end of file
+This script takes a long time to run (few hours) and will create `./cc_dset` or `./fips_dset` directory. To see all options, call the entrypoint with `--help`.
\ No newline at end of file
--
cgit v1.3.1
From 9db545dbb2b644a01b8e49b4b2506ea03ec36b00 Mon Sep 17 00:00:00 2001
From: Adam Janovsky
Date: Fri, 20 May 2022 17:20:48 +0200
Subject: Docs: add some examples
---
docs/api/dataset.md | 18 +++
docs/api/sample.md | 18 +++
docs/index.md | 19 ++-
docs/quickstart.md | 10 +-
examples/cc_oop_demo.py | 44 -------
notebooks/examples/common_criteria.ipynb | 198 ++++++++++++++++++++++++++++---
notebooks/examples/fips.ipynb | 139 ++++++++++++++++++++--
notebooks/examples/model.ipynb | 76 ++++++++++++
sec_certs/dataset/fips.py | 5 +-
9 files changed, 450 insertions(+), 77 deletions(-)
create mode 100644 docs/api/dataset.md
create mode 100644 docs/api/sample.md
delete mode 100644 examples/cc_oop_demo.py
create mode 100644 notebooks/examples/model.ipynb
(limited to 'docs/quickstart.md')
diff --git a/docs/api/dataset.md b/docs/api/dataset.md
new file mode 100644
index 00000000..ed92b31e
--- /dev/null
+++ b/docs/api/dataset.md
@@ -0,0 +1,18 @@
+# Dataset package
+
+```{eval-rst}
+.. automodule:: sec_certs.dataset
+ :no-members:
+```
+
+```{tip}
+The examples related to this package can be found at [common criteria notebook](./../notebooks/examples/common_criteria.ipynb) and [fips notebook](./../notebooks/examples/fips.ipynb).
+```
+
+## CCDataset
+
+```{eval-rst}
+.. currentmodule:: sec_certs.dataset
+.. autoclass:: CCDataset
+ :members:
+```
\ No newline at end of file
diff --git a/docs/api/sample.md b/docs/api/sample.md
new file mode 100644
index 00000000..29266f90
--- /dev/null
+++ b/docs/api/sample.md
@@ -0,0 +1,18 @@
+# Sample package
+
+```{eval-rst}
+.. automodule:: sec_certs.sample
+ :no-members:
+```
+
+```{tip}
+The examples related to this package can be found at [common criteria notebook](./../notebooks/examples/common_criteria.ipynb) and [fips notebook](./../notebooks/examples/fips.ipynb).
+```
+
+## CommonCriteriaCert
+
+```{eval-rst}
+.. currentmodule:: sec_certs.sample
+.. autoclass:: CommonCriteriaCert
+ :members:
+```
\ No newline at end of file
diff --git a/docs/index.md b/docs/index.md
index 3f44633b..8949debe 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -4,6 +4,13 @@ Welcome to the technical documentation of *sec-certs* tool for the data analysis
There are three main parts of this documentation. *User's guide* describes high-level use of our tool. Driven by this knowledge, you can progress to *Notebook examples* that showcase most of the API that we use in the form of Jupyter notebooks. The documentation currently does not have all modules documented with `autodoc`, so for the API reference, you must directly inspect the [sec_certs](https://github.com/crocs-muni/sec-certs/tree/main/sec_certs) module. If you want, you can run the notebooks as they are stored in the [project repository](https://github.com/crocs-muni/sec-certs/tree/main/notebooks). If you are interested in contributing to our project or in other aspects of our development, you can consult the relevant *GitHub artifacts*
+```{button-ref} quickstart
+:align: center
+:color: primary
+:ref-type: myst
+Show me Quickstart!
+```
+
```{admonition} Launch notebooks in MyBinder
Each of the notebooks can be launched interactively in MyBinder by clicking on 🚀 icon (top-right corner).
```
@@ -11,6 +18,7 @@ Each of the notebooks can be launched interactively in MyBinder by clicking on
```{toctree}
:hidden:
:caption: Navigation
+:maxdepth: 1
Seccerts homepage
Seccerts docs
GitHub repo
@@ -18,7 +26,9 @@ Seccerts PyPi
```
```{toctree}
+:hidden: True
:caption: User's guide
+:maxdepth: 1
installation.md
quickstart.md
tutorial.md
@@ -26,6 +36,8 @@ tutorial.md
```{toctree}
:caption: Notebook examples
+:hidden: True
+:maxdepth: 1
notebooks/examples/common_criteria.ipynb
notebooks/examples/fips.ipynb
notebooks/examples/model.ipynb
@@ -33,11 +45,16 @@ notebooks/examples/model.ipynb
```{toctree}
:caption: API
+:hidden: True
+:maxdepth: 1
+api/sample.md
+api/dataset.md
api/model.md
```
```{toctree}
-:maxdepth: 2
+:maxdepth: 1
+:hidden: True
:caption: GitHub artifacts
readme.md
contributing.md
diff --git a/docs/quickstart.md b/docs/quickstart.md
index 14121212..721e1341 100644
--- a/docs/quickstart.md
+++ b/docs/quickstart.md
@@ -6,6 +6,8 @@
1. Install the latest version with `pip install -U sec-certs` (see [installation](installation.md)).
2. Use
```python
+from sec-certs.dataset import CCDataset
+
dset = CCDataset.from_web_latest()
```
to obtain to obtain freshly processed dataset from [seccerts.org](https://seccerts.org).
@@ -17,6 +19,8 @@ to obtain to obtain freshly processed dataset from [seccerts.org](https://seccer
1. Install the latest version with `pip install -U sec-certs` (see [installation](installation.md)).
2. Use
```python
+from sec-certs.dataset import FIPSDataset
+
dset = FIPSDataset.from_web_latest()
```
to obtain to obtain freshly processed dataset from [seccerts.org](https://seccerts.org).
@@ -45,4 +49,8 @@ $ fips-certs new-run
:::
::::
-This script takes a long time to run (few hours) and will create `./cc_dset` or `./fips_dset` directory. To see all options, call the entrypoint with `--help`.
\ No newline at end of file
+This script takes a long time to run (few hours) and will create `./cc_dset` or `./fips_dset` directory. To see all options, call the entrypoint with `--help`.
+
+:::{hint}
+If you installed the docker image, use `docker run -it sec-certs bash` to run the container interactively.
+:::
diff --git a/examples/cc_oop_demo.py b/examples/cc_oop_demo.py
deleted file mode 100644
index 7876c71f..00000000
--- a/examples/cc_oop_demo.py
+++ /dev/null
@@ -1,44 +0,0 @@
-import logging
-from datetime import datetime
-
-from sec_certs.config.configuration import config
-from sec_certs.dataset import CCDataset
-
-logger = logging.getLogger(__name__)
-
-
-def main():
- file_handler = logging.FileHandler(config.log_filepath)
- stream_handler = logging.StreamHandler()
- formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
- file_handler.setFormatter(formatter)
- stream_handler.setFormatter(formatter)
- logging.basicConfig(level=logging.INFO, handlers=[file_handler, stream_handler])
- start = datetime.now()
-
- # Full processing pipeline of Common Criteria dataset
- dset = CCDataset()
- dset.get_certs_from_web()
- dset.process_protection_profiles()
- dset.download_all_pdfs()
- dset.convert_all_pdfs()
- dset.analyze_certificates() # calls dset._extract_data() and dset._compute_heuristics()
-
- # Other useful API below
-
- # explicitly dump to json
- # dset.to_json(dset.json_path)
-
- # Load dataset from JSON
- # new_dset = CCDataset.from_json("./debug_dataset/cc_full_dataset.json")
- # assert dset == new_dset
-
- # transform to pandas DataFrame
- # df = dset.to_pandas()
-
- end = datetime.now()
- logger.info(f"The computation took {(end-start)} seconds.")
-
-
-if __name__ == "__main__":
- main()
diff --git a/notebooks/examples/common_criteria.ipynb b/notebooks/examples/common_criteria.ipynb
index 4aab6c61..83a56379 100644
--- a/notebooks/examples/common_criteria.ipynb
+++ b/notebooks/examples/common_criteria.ipynb
@@ -4,36 +4,202 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "# Common Criteria dataset class\n",
+ "# Common Criteria example\n",
"\n",
- "This notebook illustrates basic functionality with the `CCDataset` class that holds Common Criteria dataset"
+ "This notebook illustrates basic functionality with the `CCDataset` class that holds Common Criteria dataset and of its sample `CommonCriteriaCert`"
]
},
{
"cell_type": "code",
- "execution_count": 1,
+ "execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
- "from sec_certs.dataset.common_criteria import CCDataset"
+ "from sec_certs.dataset.common_criteria import CCDataset\n",
+ "from sec_certs.dataset.common_criteria import CommonCriteriaCert\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Get fresh dataset snapshot from mirror"
]
},
{
"cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "4985\n"
- ]
- }
- ],
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
"source": [
"dset = CCDataset.from_web_latest()\n",
- "print(len(dset))"
+ "print(len(dset)) # Print number of certificates in the dataset"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Do some basic dataset serialization"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Dump dataset into json and load it back\n",
+ "dset.to_json(\"./cc_dset.json\")\n",
+ "new_dset: CCDataset = CCDataset.from_json(\"./cc_dset.json\")\n",
+ "assert dset == new_dset"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Turn dataset into Pandas DataFrame\n",
+ "df = dset.to_pandas()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Simple dataset manipulation"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 40,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Iterate over certificates in dataset\n",
+ "for cert in dset:\n",
+ " pass\n",
+ "\n",
+ "# Get certificates produced by Infineon manufacturer\n",
+ "infineon_certs = [x for x in dset if \"Infineon\" in x.manufacturer]\n",
+ "df_infineon = df.loc[df.manufacturer.str.contains(\"Infineon\", case=False)]\n",
+ "\n",
+ "# Get certificates with some CVE\n",
+ "vulnerable_certs = [x for x in dset if x.heuristics.related_cves]\n",
+ "df_vulnerable = df.loc[~df.related_cves.isna()]\n",
+ "\n",
+ "# Show CVE ids of some vulnerable certificate\n",
+ "print(f\"{vulnerable_certs[0].heuristics.related_cves=}\")\n",
+ "\n",
+ "# Get certificates from 2015 and newer\n",
+ "df_2015_and_newer = df.loc[df.year_from > 2014]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Plot distribution of years of certification\n",
+ "df.year_from.value_counts().sort_index().plot.line()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Dissect single certificate"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Select a certificate and print some attributes\n",
+ "cert: CommonCriteriaCert = dset[\"bad93fb821395db2\"]\n",
+ "print(f\"{cert.name=}\")\n",
+ "print(f\"{cert.heuristics.cpe_matches=}\")\n",
+ "print(f\"{cert.heuristics.report_references.directly_referencing=}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "vulnerable_certs = [x for x in dset if x.heuristics.related_cves]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Serialize single certificate"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "cert.to_json(\"./cert.json\")\n",
+ "new_cert = cert.from_json(\"./cert.json\")\n",
+ "assert cert == new_cert\n",
+ "\n",
+ "# Serialize as Pandas series\n",
+ "ser = pd.Series(cert.pandas_tuple, index=cert.pandas_columns)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Assign dataset with CPE records and compute vulnerabilities\n",
+ "\n",
+ "*Note*: The data is already computed on dataset obtained with `from_web_latest()`, this is just for illustration. \n",
+ "*Note*: This may likely not run in Binder, as the corresponding `CVEDataset` and `CPEDataset` instances take a lot of memory."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Automatically match CPEs and CVEs\n",
+ "_, cpe_dset, _ = dset.compute_cpe_heuristics()\n",
+ "dset.compute_related_cves()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Create new dataset and fully process it\n",
+ "\n",
+ "*Warning*: It's not good idea to run this from notebook. It may take several hours to finnish. We recommend using `from_web_latest()` or turning this into a Python script."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "dset = CCDataset()\n",
+ "dset.get_certs_from_web()\n",
+ "dset.process_protection_profiles()\n",
+ "dset.download_all_pdfs()\n",
+ "dset.convert_all_pdfs()\n",
+ "dset.analyze_certificates()"
]
}
],
diff --git a/notebooks/examples/fips.ipynb b/notebooks/examples/fips.ipynb
index ab20ee93..28eb2b1a 100644
--- a/notebooks/examples/fips.ipynb
+++ b/notebooks/examples/fips.ipynb
@@ -11,30 +11,143 @@
},
{
"cell_type": "code",
- "execution_count": 1,
+ "execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
- "from sec_certs.dataset.fips import FIPSDataset"
+ "from sec_certs.dataset.fips import FIPSDataset\n",
+ "from sec_certs.sample import FIPSCertificate"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Get fresh dataset snapshot from mirror"
]
},
{
"cell_type": "code",
- "execution_count": 2,
+ "execution_count": null,
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "4985\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
- "dset = FIPSDataset.from_web_latest()\n",
+ "dset: FIPSDataset = FIPSDataset.from_web_latest()\n",
"print(len(dset))"
]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Do some basic dataset serialization"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Dump dataset into json and load it back\n",
+ "dset.to_json(\"./fips_dataset.json\")\n",
+ "new_dset = FIPSDataset.from_json(\"./fips_dataset.json\")\n",
+ "assert dset == new_dset"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Simple dataset manipulation"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Get certificates from a single manufacturer\n",
+ "cisco_certs = [cert for cert in dset if \"Cisco\" in cert.web_scan.vendor]\n",
+ "\n",
+ "# Get certificates with some CVE\n",
+ "vulnerable_certs = [cert for cert in dset if cert.heuristics.related_cves]\n",
+ "\n",
+ "# Show CVE ids of some vulnerable certificate\n",
+ "print(f\"{vulnerable_certs[0].heuristics.related_cves=}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Dissect single certificate"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Select a certificate and print some attributes\n",
+ "cert: FIPSCertificate = dset[\"542cacae1d41132a\"]\n",
+ "\n",
+ "print(f\"{cert.web_scan.module_name=}\")\n",
+ "print(f\"{cert.heuristics.cpe_matches=}\")\n",
+ "print(f\"{cert.web_scan.level=}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Serialize single certificate"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "cert.to_json(\"./cert.json\")\n",
+ "new_cert: FIPSCertificate = FIPSCertificate.from_json(\"./cert.json\")\n",
+ "assert new_cert == cert"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Create new dataset and fully process it\n",
+ "\n",
+ "*Warning*: It's not good idea to run this from notebook. It may take several hours to finnish. We recommend using `from_web_latest()` or turning this into a Python script."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "dset = FIPSDataset()\n",
+ "dset.get_certs_from_web()\n",
+ "dset.convert_all_pdfs()\n",
+ "dset.pdf_scan()\n",
+ "dset.extract_certs_from_tables()\n",
+ "dset.finalize_results()\n",
+ "\n",
+ "# TODO: Resolve https://github.com/crocs-muni/sec-certs/issues/210 and refactor this\n",
+ "# if not no_download_algs:\n",
+ "# aset.get_certs_from_web()\n",
+ "# logger.info(f\"Finished parsing. Have algorithm dataset with {len(aset)} algorithm numbers.\")\n",
+ "# dset.algorithms = aset\n",
+ "\n",
+ "# TODO: Resolve https://github.com/crocs-muni/sec-certs/issues/211 and uncomment\n",
+ "# dset.plot_graphs(show=False)"
+ ]
}
],
"metadata": {
diff --git a/notebooks/examples/model.ipynb b/notebooks/examples/model.ipynb
new file mode 100644
index 00000000..370b13cd
--- /dev/null
+++ b/notebooks/examples/model.ipynb
@@ -0,0 +1,76 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Model\n",
+ "\n",
+ "This notebook illustrates basic functionality with the `model` package that apply complex transformations on certificates.\n",
+ "\n",
+ "*Note*: You probably don't need to use this. Instead, you should use `CCDataset` or `FIPSDataset` classes to handle the transformations for yourself."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sec_certs.dataset.common_criteria import CCDataset\n",
+ "from sec_certs.model import SARTransformer"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "dset: CCDataset = CCDataset.from_web_latest()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## SARTransformer"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "transformer = SARTransformer().fit(dset.certs.values())\n",
+ "extracted_sars = {x.dgst: transformer.transform_single_cert(x) for x in dset}"
+ ]
+ }
+ ],
+ "metadata": {
+ "interpreter": {
+ "hash": "6386d1612879d92d026c363e7667e428bc38d86c5a080d58c3d70e7cd43df67d"
+ },
+ "kernelspec": {
+ "display_name": "Python 3.8.1 ('certsvenv': venv)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.1"
+ },
+ "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/sec_certs/dataset/fips.py b/sec_certs/dataset/fips.py
index 2defd9f1..8cef1520 100644
--- a/sec_certs/dataset/fips.py
+++ b/sec_certs/dataset/fips.py
@@ -254,8 +254,9 @@ class FIPSDataset(Dataset[FIPSCertificate], ComplexSerializableType):
len(dset),
len(dset.algorithms) if dset.algorithms is not None else 0,
)
- logger.info("The dataset does not contain the results of the dependency analysis - calculating them now...")
- dset.finalize_results()
+ # TODO: Fixme, this is really costly
+ # logger.info("The dataset does not contain the results of the dependency analysis - calculating them now...")
+ # dset.finalize_results()
return dset
def set_local_paths(self) -> None:
--
cgit v1.3.1
From f951d8b2d4aa31b9758fe145a5553a56b486f64b Mon Sep 17 00:00:00 2001
From: Adam Janovsky
Date: Mon, 23 May 2022 14:23:38 +0200
Subject: fix typo in quickstart docs
---
docs/quickstart.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
(limited to 'docs/quickstart.md')
diff --git a/docs/quickstart.md b/docs/quickstart.md
index 721e1341..e16a570d 100644
--- a/docs/quickstart.md
+++ b/docs/quickstart.md
@@ -6,7 +6,7 @@
1. Install the latest version with `pip install -U sec-certs` (see [installation](installation.md)).
2. Use
```python
-from sec-certs.dataset import CCDataset
+from sec_certs.dataset import CCDataset
dset = CCDataset.from_web_latest()
```
@@ -19,7 +19,7 @@ to obtain to obtain freshly processed dataset from [seccerts.org](https://seccer
1. Install the latest version with `pip install -U sec-certs` (see [installation](installation.md)).
2. Use
```python
-from sec-certs.dataset import FIPSDataset
+from sec_certs.dataset import FIPSDataset
dset = FIPSDataset.from_web_latest()
```
--
cgit v1.3.1