Update README, add logo, add docs, add test dir.

6 files changed, 222 insertions, 33 deletions

diff --git a/README.md b/README.md
index 22411b98..abdaeea7 100644
--- a/README.md
+++ b/README.md
@@ -1,38 +1,41 @@
-# sec-certs
+# ![](docs/_static/logo.svg)
 
-Analyzer of security certificates (Common Criteria, NIST FIPS140-2...) 
+Tool for analysis of security certificates and their security targets (Common Criteria, NIST FIPS140-2...).
 
-## Usage
+This project is developed by the [Centre for Research On Cryptography and Security](https://crocs.fi.muni.cz) at Faculty of Informatics, Masaryk University.
 
-Steps:
- 1. Run `process_certificates.py` to generate download scripts (download_cc_web.bat)
- 2. Run `download_cc_web.bat` to download important files from Common Criteria website (requires `curl` installed)
- 3. Run `process_certificates.py` to generate download scripts for separate pdf files with certificates (`download_active_certs.bat`, `download_active_updates.bat`...)
- 4. Place all download scripts into folder on disk with at least 5GB free space (yes, there are A LOT of certificates) and run them, wait until download and text extraction is completed 
- 5. Edit process_certificates.py, create new profile (paths_xxx) with correct paths pointing to the place where you dowloaded certificates, set paths_used = paths_xxx variable to your (see paths_20200904 dict for example). Intermediate files generated during the processing will be created inside `current_path/results_yyy` folder where yyy is value set by you at `paths_xxx['id']`. (DEBUG, remove later)
- 6. Run `process_certificates.py` to process CSV, HTML and PDF files. Extracted information is stored in *.json files. `do_extraction` and `do_pairing` variables shall be True to execute this (time consuming) step. Set `do_extraction = False` and `do_pairing = False` to skip processing and read already computed information from `certificate_data_complete.json`. 
- 7. Optional: if info extracted from protection profiles is available, copy `pp_data_complete_processed.json` file into folder with results generated 
- 8. Run `process_certificates.py` with `do_processing = True` to run various heuristics which will create post-processed section `processed` for every certificate (results are stored in `certificate_data_complete_processed.json`).
- 9. Run `process_certificates.py` with `do_analysis = True` to perform analysis of certificates (various graphs, statistics...). If `do_analysis_filtered = True` then same analysis for subsets of certificates is performed.  
- 10. Open, look and enjoy graphs like `num_certs_in_years.png` or `num_certs_eal_in_years.png`. For `certid_graph.dot.pdf` and other large graphs use Chrome to display as Adobe Acrobat Reader will fail to show whole graph. 
- 
-## Downloading again failes which failed to download properly
+## Usage (CC)
 
-Steps:
- 1. Run `search_certificate.py` to generate download script `download_failed_certs.bat` for newly added certificates
- 2. Run `download_failed_certs.bat` to dowload failed/corrupted files again  
- 3. Continue from step 5. 
+The tool requires several Python packages as well as the `pdftotext` binary somewhere on the `PATH`.
+The easiest way to setup the tool is to install it in a virtual environment, e.g.:
+```
+python3 -m venv virt
+. virt/bin/activate
+pip install -e .
+```
 
-## Updating for the newly issued certificates
- 1. Run `search_certificate.py` to generate download scripts (download_cc_web.bat, download_cc_web.sh)
- 2. Run `download_cc_web.bat` to download important files from Common Criteria website (requires `curl` installed)
- 3. Run `search_certificate.py` with do_complete_extraction = False
- 4. 
+The following steps will do a full extraction and analysis of CC certificates:
+
+ 1. Make a directory in which the certificates will be downloaded and processing will take place.
+    The contents of the directory are under control of the tool, and **may be overwritten**!
+ 2. Run `process-certs --fresh --do-download-meta <dir>` to download certificate metadata from the Common Criteria portal.
+ 3. Run `process-certs --fresh --do-extraction-meta <dir>` to extract metadata from the downloaded Common Criteria pages.
+ 4. Run `process-certs --fresh --do-download-certs <dir>` to download the certificate and security target PDF files. This
+    step takes time as there is quite a lot of files. It also takes up a lot of space (around 5GB). It is done in parallel
+    and the number of threads can be changed with the `-t/--threads` switch (the default is 4).
+ 5. Run `process-certs --fresh --do-pdftotext <dir>` to convert the PDF files to text.
+ 6. Run `process-certs --fresh --do-extraction <dir>` to extract information from the certificates and security targets.
+ 7. Run `process-certs --fresh --do-pairing <dir>`.
+ 8. Run `process-certs --fresh --do-processing <dir>` to run various heuristics which will create post-processed section
+   `processed` for every certificate (results are stored in `certificate_data_complete_processed.json`).
+ 9. Run `process-certs --fresh --do-analysis <dir>` to perform analysis of certificates (various graphs, statistics...).
+ 10. Open, look and enjoy graphs like `num_certs_in_years.png` or `num_certs_eal_in_years.png`. For `certid_graph.dot.pdf` 
+     and other large graphs use Chrome to display as Adobe Acrobat Reader will fail to show whole graph. 
 
 
 ## Extending the analysis
 
 The analysis can be extended in several ways:
  1. Additional keywords can be extracted from PDF files (modify `cert_rules.py`)
- 2. Data from `certificate_data_complete.json` can be analyzed in novel way - this is why this project was concieved at first place
- 3. Help to fix problems in data extraction - some PDF files are corrupted, there are many typos even in certificates identificators...
+ 2. Data from `certificate_data_complete.json` can be analyzed in a novel way - this is why this project was concieved in the first place.
+ 3. Help to fix problems in data extraction - some PDF files are corrupted, there are many typos even in certificate IDs...
diff --git a/docs/_static/logo.svg b/docs/_static/logo.svg
new file mode 100644
index 00000000..a6a400be
--- /dev/null
+++ b/docs/_static/logo.svg
@@ -0,0 +1,81 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="188.78629mm"
+   height="68.240051mm"
+   viewBox="0 0 188.78628 68.240052"
+   version="1.1"
+   id="svg8"
+   sodipodi:docname="logo.svg"
+   inkscape:version="1.0.1 (3bc2e813f5, 2020-09-07)">
+  <sodipodi:namedview
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1"
+     objecttolerance="10"
+     gridtolerance="10"
+     guidetolerance="10"
+     inkscape:pageopacity="0"
+     inkscape:pageshadow="2"
+     inkscape:window-width="1920"
+     inkscape:window-height="1023"
+     id="namedview12"
+     showgrid="false"
+     inkscape:zoom="0.94040421"
+     inkscape:cx="281.07984"
+     inkscape:cy="161.94862"
+     inkscape:window-x="0"
+     inkscape:window-y="25"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="layer1" />
+  <defs
+     id="defs2" />
+  <metadata
+     id="metadata5">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     id="layer1"
+     transform="translate(-24.826912,-135.56049)">
+    <path
+       id="rect903"
+       style="opacity:0.998;fill:#000000;stroke:none;stroke-width:4.90715;stroke-linecap:square;paint-order:markers stroke fill"
+       d="M 176.80859 77.863281 L 176.80859 181.24609 L 669.28906 181.24609 L 627.54688 129.48242 L 669.43164 77.863281 L 176.80859 77.863281 z "
+       transform="matrix(0.26458333,0,0,0.26458333,24.826912,135.56049)" />
+    <circle
+       style="opacity:0.998;fill:#000000;stroke-width:0.81783;stroke-linecap:square;paint-order:markers stroke fill"
+       id="path833-3"
+       cx="59.321445"
+       cy="169.68051"
+       r="24.120024" />
+    <path
+       style="opacity:0.998;fill:#ffffff;stroke-width:1.70221;stroke-linecap:square;paint-order:markers stroke fill"
+       id="path835-6"
+       d="m 116.79591,120.92807 -9.93587,24.04724 -12.288555,-22.93431 -24.9958,7.22504 10.268973,-23.90689 -21.233385,-15.037769 25.093756,-6.877093 -1.481798,-25.976824 21.022429,15.331291 19.38561,-17.354802 1.12079,25.994901 25.65526,4.33574 -19.62484,17.083816 12.60598,22.76139 z"
+       transform="matrix(-0.02807995,0.52181971,-0.52181971,-0.02807995,113.47851,118.0225)" />
+    <text
+       xml:space="preserve"
+       style="font-size:25.4px;line-height:1.25;font-family:Metropolis;-inkscape-font-specification:Metropolis;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#ffffff;stroke-width:0.264583"
+       x="137.28783"
+       y="176.96999"
+       id="text877-7"><tspan
+         id="tspan875-5"
+         x="137.28783"
+         y="176.96999"
+         style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:25.4px;font-family:Metropolis;-inkscape-font-specification:Metropolis;fill:#ffffff;stroke-width:0.264583">seccerts</tspan></text>
+  </g>
+</svg>
diff --git a/docs/_static/logo_badge.svg b/docs/_static/logo_badge.svg
new file mode 100644
index 00000000..55b3176d
--- /dev/null
+++ b/docs/_static/logo_badge.svg
@@ -0,0 +1,94 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="71.288811mm"
+   height="71.288811mm"
+   viewBox="0 0 71.28881 71.288811"
+   version="1.1"
+   id="svg1271"
+   inkscape:version="1.0.1 (3bc2e813f5, 2020-09-07)"
+   sodipodi:docname="logo_badge.svg"
+   inkscape:export-filename="/home/johny/dev/sec-certs-page/sec_certs/static/favicon.png"
+   inkscape:export-xdpi="182.42415"
+   inkscape:export-ydpi="182.42415">
+  <defs
+     id="defs1265" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="1.979899"
+     inkscape:cx="229.50586"
+     inkscape:cy="131.30255"
+     inkscape:document-units="mm"
+     inkscape:current-layer="layer1"
+     inkscape:document-rotation="0"
+     showgrid="false"
+     fit-margin-top="0"
+     fit-margin-left="0"
+     fit-margin-right="0"
+     fit-margin-bottom="0"
+     inkscape:window-width="1920"
+     inkscape:window-height="1023"
+     inkscape:window-x="0"
+     inkscape:window-y="25"
+     inkscape:window-maximized="1"
+     inkscape:pagecheckerboard="false" />
+  <metadata
+     id="metadata1268">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Layer 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(-62.821846,-46.556198)">
+    <path
+       id="rect903-9-4"
+       style="opacity:0.998;fill:#000000;stroke:none;stroke-width:2.7116;stroke-linecap:square;paint-order:markers stroke fill"
+       d="M 83.027344 98.328125 L 83.027344 248.53906 L 134.78711 206.80078 L 186.41016 248.6875 L 186.41016 98.328125 L 83.027344 98.328125 z "
+       transform="matrix(0.26458333,0,0,0.26458333,62.821846,46.556198)" />
+    <circle
+       style="opacity:0.998;fill:#000000;stroke-width:0.81783;stroke-linecap:square;paint-order:markers stroke fill"
+       id="path833-3-7-6-5"
+       cx="75.733047"
+       cy="-98.466248"
+       r="24.120024"
+       transform="rotate(90)" />
+    <path
+       sodipodi:type="star"
+       style="opacity:0.998;fill:#ffffff;stroke-width:1.70221;stroke-linecap:square;paint-order:markers stroke fill"
+       id="path835-6-5-2-03"
+       sodipodi:sides="7"
+       sodipodi:cx="104.52818"
+       sodipodi:cy="98.40979"
+       sodipodi:r1="25.643127"
+       sodipodi:r2="46.623867"
+       sodipodi:arg1="1.0719622"
+       sodipodi:arg2="1.5207612"
+       inkscape:flatsided="false"
+       inkscape:rounded="0"
+       inkscape:randomized="0"
+       d="m 116.79591,120.92807 -9.93587,24.04724 -12.288555,-22.93431 -24.9958,7.22504 10.268973,-23.90689 -21.233385,-15.037769 25.093756,-6.877093 -1.481798,-25.976824 21.022429,15.331291 19.38561,-17.354802 1.12079,25.994901 25.65526,4.33574 -19.62484,17.083816 12.60598,22.76139 z"
+       inkscape:transform-center-x="-0.67550987"
+       inkscape:transform-center-y="-1.8662519"
+       transform="matrix(-0.52181972,-0.02807995,0.02807995,-0.52181972,150.12425,129.89008)" />
+  </g>
+</svg>
diff --git a/documentation.md b/documentation.md
deleted file mode 100644
index cb1974f4..00000000
--- a/documentation.md
+++ /dev/null
@@ -1,5 +0,0 @@
-# sec-certs
-
-Analyzer of security certificates (Common Criteria, NIST FIPS140-2...) 
-
-## Usage
diff --git a/setup.py b/setup.py
index d9003923..123159b9 100644
--- a/setup.py
+++ b/setup.py
@@ -3,7 +3,7 @@ from setuptools import setup, find_packages
 
 setup(
     name='sec-certs',
-    author='Petr Svenda',
+    author='Petr Svenda, Stanislav Bobon, Jan Jancar, Adam Janovsky',
     author_email='svenda@fi.muni.cz',
     version='0.0.0',
     packages=find_packages(),
@@ -20,6 +20,7 @@ setup(
         "Intended Audience :: Developers",
         "Intended Audience :: Science/Research"
     ],
+    python_requires=">=3.8",
     install_requires=[
         "PyPDF2",
         "matplotlib",
diff --git a/test/data/certs.csv b/test/data/certs.csv
new file mode 100644
index 00000000..0cfaa57f
--- /dev/null
+++ b/test/data/certs.csv
@@ -0,0 +1,15 @@
+cert,st
+0143a.pdf,0143b.pdf
+ANSSI-CC_2009-62fr.pdf,ANSSI-CC-cible_2009-62en.pdf
+c0478_erpt.pdf,c0478_est.pdf
+bariyer cr.pdf,HVL-BARIYER-ASE-ST-lite_v2.5.pdf
+Aruba NMCv6.4.2.0-1.3_2017_107_CRv1.0.pdf,Aruba NVM Controller OS 6.4.2.0-1.3 FIPS_STv1.0.pdf
+ANSSI-CC-En_2016_45en.pdf,anssi_cible_2016_45_LITE.pdf
+ANSSI-CC-En_2016_52en.pdf,anssi_cible_2016_52_LITE.pdf
+Aselsan_Digital Tachograph-CR.pdf,Aselsan_Digital Tachograph-ST-Lite.pdf
+0519V2a_pdf.pdf,0519V2b_pdf.pdf
+0596V2a_pdf.pdf,0596V2b_pdf.pdf
+KECS-CISS-1003-2020_CR_EN.pdf,KECS-CISS-1003-2020_ST_EN.pdf
+ATES v1.0 Certification Report.pdf,ICT_ATES_ST_v2.9.pdf
+0589a_pdf.pdf,0589b_pdf.pdf
+KECS-CISS-0924-2019_CR_EN.pdf,KECS-CISS-0924-2019_ST_EN.pdf
\ No newline at end of file