Move flaky remote tests into "scheduled" workflow.

15 files changed, 95 insertions, 49 deletions

diff --git a/.github/workflows/cron.yml b/.github/workflows/cron.yml
new file mode 100644
index 00000000..426738a2
--- /dev/null
+++ b/.github/workflows/cron.yml
@@ -0,0 +1,39 @@
+name: Scheduled tests
+on:
+  schedule:
+    - cron: "0 0 * * 3" # Run every Wednesday at midnight
+  workflow_dispatch:
+env:
+  DEBIAN_FRONTEND: noninteractive
+
+jobs:
+  test:
+    runs-on: ubuntu-22.04
+    env:
+      SECCERTS_NVD_API_KEY: ${{ secrets.NVD_API_KEY }}
+    steps:
+      - name: apt-get update
+        run: sudo apt-get update
+      - name: Install Poppler
+        run: sudo apt-get install -y build-essential libpoppler-cpp-dev pkg-config python3-dev
+      - uses: actions/checkout@v4
+      - name: Setup python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.10"
+          cache: "pip"
+          cache-dependency-path: |
+            requirements/test_requirements.txt
+      - name: Install python dependencies
+        run: |
+          pip install -r requirements/test_requirements.txt
+      - name: Install sec-certs
+        run: |
+          pip install -e .
+          python -m spacy download en_core_web_sm
+      - name: Run tests
+        run: pytest --cov=sec_certs -m "remote" tests
+      - name: Code coverage upload
+        uses: codecov/codecov-action@v4
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 5b3353fa..69f55e40 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -31,7 +31,7 @@ jobs:
           pip install -e .
           python -m spacy download en_core_web_sm
       - name: Run tests
-        run: pytest --cov=sec_certs tests
+        run: pytest --cov=sec_certs -m "not remote" tests
       - name: Code coverage upload
         uses: codecov/codecov-action@v4
         with:
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index d0b52a4d..6c339a99 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -1,12 +1,12 @@
 # Contributing
 
-You contribution is warmly welcomed. You can help by:
+Your contribution is warmly welcomed. You can help by:
 
- 0. Spread the word about this project, look at generated processed webpages
- 1. Trying the tool and reporting issues and suggestions for improvement (open Github issue)
- 2. Add new regular expressions to extract relevant information from certificates (update cert_rules.py)
- 3. Perform additional analysis with extracted data (analyze_certificates.py)
- 3. Improve the code (TODO: Follow Github contribution guidelines, ideally contact us first about your plan)
+ 1. Spreading the word about this project, using our website [sec-certs.org](https://sec-certs.org)
+ 2. Trying the tool and reporting issues and suggestions for improvement (open a GitHub issue)
+ 3. Adding new regular expressions to extract relevant information from certificates (update `rules.yaml`)
+ 4. Performing additional analysis with extracted data (let us know about your findings)
+ 5. Improving the code (Follow Github contribution guidelines, ideally contact us first about your plan)
 
 ## Dependencies
 
@@ -30,7 +30,7 @@ Requirements are maintained with [pip-tools](https://github.com/jazzband/pip-too
 - `setuptools-scm` will automatically, upon editable/real install of a package, infer its version and write it to `sec_certs/_version.py`. This file is not indexed as well. See more at [setuptools-scm GitHub](https://github.com/pypa/setuptools_scm)
 - On publishing a release, the tool is automatically published to [PyPi](https://pypi.org/project/sec-certs/) and [DockerHub](https://hub.docker.com/repository/docker/seccerts/sec-certs).
 
-Note on single-sourcing the package version: More can be read [here](https://packaging.python.org/en/latest/guides/single-sourcing-package-version/). The downside of our approach is that `.git` folder and editable/real install is needed to infer the version of the package. Releases can be infered without installing the project.
+Note on single-sourcing the package version: More can be read [here](https://packaging.python.org/en/latest/guides/single-sourcing-package-version/). The downside of our approach is that `.git` folder and editable/real installation is needed to infer the version of the package. Releases can be inferred without installing the project.
 
 ### Currently, the release process is as follows
 
@@ -46,7 +46,7 @@ All commits shall pass the lint pipeline of the following tools:
 - Mypy (see [pyproject.toml](https://github.com/crocs-muni/sec-certs/blob/main/pyproject.toml) for settings)
 - Ruff (see [pyproject.toml](https://github.com/crocs-muni/sec-certs/blob/main/pyproject.toml) for settings)
 
-These tools can be installed via [dev_requirements.txt](https://github.com/crocs-muni/sec-certs/blob/main/requirements/dev_requirements.txt) You can use [pre-commit](https://pre-commit.com/) tool to register git hook that will evalute these checks prior to any commit and abort the commit for you. Note that the pre-commit is not meant to automatically fix the issues, just warn you.
+These tools can be installed via [dev_requirements.txt](https://github.com/crocs-muni/sec-certs/blob/main/requirements/dev_requirements.txt) You can use [pre-commit](https://pre-commit.com/) tool to register git hook that will evaluate these checks prior to any commit and abort the commit for you. Note that the pre-commit is not meant to automatically fix the issues, just warn you.
 
 It should thus suffice to:
 
@@ -56,14 +56,23 @@ pre-commit install &&
 pre-commit run --all-files
 ```
 
-To ivoke the tools manually, you can, in the repository root, use:
+To invoke the tools manually, you can, in the repository root, use:
 - Mypy: `mypy .`
 - Ruff: `ruff ." (or with `--fix` flag to apply fixes)
 - Ruff formatL `ruff format --check .`
 
+## Tests
+
+Tests are run with `pytest`. The tests are located in `tests` folder and are run with `pytest tests`. The tests are also run on every push to the repository with Github Actions.
+There are two custom markers for the tests:
+- `slow` for tests that take longer time to run
+- `remote` for tests that require remote resources and are thus flaky.
+
+To exclude slow tests, use `pytest -m "not slow"`. To exclude remote tests, use `pytest -m "not remote"`. To run only slow tests, use `pytest -m "slow"`. To run only remote tests, use `pytest -m "remote"`.
+
 ## Documentation
 
-Every public method of a module that can be leveraged as an API by user should be documented. The docstrng style should
+Every public method of a module that can be leveraged as an API by user should be documented. The docstring style should
 be `sphinx-oneline`.
 
 The documentation is built using `sphinx` with `mnyst` extension that allows for markdown files. Folder `notebooks/examples` is symbolically linked to `/docs` and its contents will be automatically parsed. These notebooks are supposed to be runnable from Binder.
diff --git a/README.md b/README.md
index e723f5cc..6a44621b 100644
--- a/README.md
+++ b/README.md
@@ -4,8 +4,6 @@
 
 A tool for data scraping and analysis of security certificates from Common Criteria and FIPS 140-2/3 frameworks.
 
-<!-- This project is developed by the [Centre for Research On Cryptography and Security](https://crocs.fi.muni.cz) at Masaryk University, Czech Republic. -->
-
 [![Website](https://img.shields.io/website?down_color=red&down_message=offline&style=flat-square&up_color=SpringGreen&up_message=online&url=https%3A%2F%2Fsec-certs.org)](https://sec-certs.org)
 [![Website](https://img.shields.io/website?label=docs&down_color=red&down_message=offline&style=flat-square&up_color=SpringGreen&up_message=online&url=https%3A%2F%2Fsec-certs.org/docs/index.html)](https://sec-certs.org/docs/index.html)
 [![PyPI](https://img.shields.io/pypi/v/sec-certs?style=flat-square)](https://pypi.org/project/sec-certs/)
diff --git a/pyproject.toml b/pyproject.toml
index f27c9aa6..0cd409a1 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -143,7 +143,8 @@
   exclude = "build/"
 
 [tool.pytest.ini_options]
-  markers = ["slow: marks tests as slow (deselect with '-m \"not slow\"')"]
+  markers = ["slow: marks tests as slow (deselect with '-m \"not slow\"')",
+             "remote: marks tests that require remote resources"]
 
 [tool.coverage.run]
   source = ["src"]
diff --git a/tests/cc/test_cc_dataset.py b/tests/cc/test_cc_dataset.py
index cba06a64..127f4e5b 100644
--- a/tests/cc/test_cc_dataset.py
+++ b/tests/cc/test_cc_dataset.py
@@ -77,13 +77,13 @@ def test_download_and_convert_pdfs(toy_dataset: CCDataset, data_dir: Path):
         )
 
 
-@pytest.mark.slow
-@pytest.mark.xfail(reason="May fail due to network issues.")
+@pytest.mark.remote
 def test_from_web():
     dset = CCDataset.from_web()
     assert len(dset) > 6000
 
 
+@pytest.mark.remote
 def test_archive_fits():
     fsize = helpers.query_file_size(config.cc_latest_full_archive)
     tmpdir = helpers.tempdir_for(fsize)
diff --git a/tests/cc/test_cc_maintenance_updates.py b/tests/cc/test_cc_maintenance_updates.py
index 683c8644..dd470aa3 100644
--- a/tests/cc/test_cc_maintenance_updates.py
+++ b/tests/cc/test_cc_maintenance_updates.py
@@ -77,8 +77,7 @@ def test_to_pandas(mu_dset: CCDatasetMaintenanceUpdates):
     assert set(df.columns) == set(CCMaintenanceUpdate.pandas_columns) - {"dgst"}
 
 
-@pytest.mark.slow
-@pytest.mark.xfail(reason="May fail due to network issues.")
+@pytest.mark.remote
 def test_from_web():
     dset = CCDatasetMaintenanceUpdates.from_web()
     assert dset is not None
diff --git a/tests/cc/test_cc_protection_profiles.py b/tests/cc/test_cc_protection_profiles.py
index 4cf61fd0..4f1bdee7 100644
--- a/tests/cc/test_cc_protection_profiles.py
+++ b/tests/cc/test_cc_protection_profiles.py
@@ -71,14 +71,13 @@ def test_get_certs_from_web(pp_data_dir: Path, toy_pp_dataset: ProtectionProfile
         assert isinstance(cert.web_data.version, str)
 
 
-@pytest.mark.slow
-@pytest.mark.xfail(reason="May fail due to network issues.")
+@pytest.mark.remote
 def test_from_web():
     dset = ProtectionProfileDataset.from_web()
     assert len(dset) > 400
 
 
-@pytest.mark.xfail(reason="May fail due to error on CC server")
+@pytest.mark.remote
 def test_download_html_files():
     with TemporaryDirectory() as tmp_dir:
         dset = ProtectionProfileDataset(root_dir=Path(tmp_dir))
diff --git a/tests/cc/test_cc_schemes.py b/tests/cc/test_cc_schemes.py
index 68599bc5..e14ebf6f 100644
--- a/tests/cc/test_cc_schemes.py
+++ b/tests/cc/test_cc_schemes.py
@@ -2,7 +2,6 @@ from datetime import date
 from urllib.parse import urlparse
 
 import pytest
-from requests import RequestException
 
 import sec_certs.sample.cc_scheme as CCSchemes
 from sec_certs.dataset.auxiliary_dataset_handling import CCSchemeDatasetHandler
@@ -21,7 +20,7 @@ def absolute_urls(results):
     return True
 
 
-@pytest.mark.xfail(reason="May fail due to server errors.", raises=RequestException)
+@pytest.mark.remote
 def test_australia():
     ineval = CCSchemes.get_australia_in_evaluation()
     assert len(ineval) != 0
@@ -33,7 +32,7 @@ def canada_certified():
     return CCSchemes.get_canada_certified()
 
 
-@pytest.mark.xfail(reason="May fail due to server errors.", raises=RequestException)
+@pytest.mark.remote
 @pytest.mark.slow
 def test_canada(canada_certified):
     assert len(canada_certified) != 0
@@ -43,7 +42,7 @@ def test_canada(canada_certified):
     assert absolute_urls(ineval)
 
 
-@pytest.mark.xfail(reason="May fail due to server errors.", raises=RequestException)
+@pytest.mark.remote
 @pytest.mark.slow
 def test_anssi():
     certified = CCSchemes.get_france_certified()
@@ -54,7 +53,7 @@ def test_anssi():
     assert absolute_urls(archived)
 
 
-@pytest.mark.xfail(reason="May fail due to server errors.", raises=RequestException)
+@pytest.mark.remote
 @pytest.mark.slow
 def test_bsi():
     certified = CCSchemes.get_germany_certified()
@@ -62,7 +61,7 @@ def test_bsi():
     assert absolute_urls(certified)
 
 
-@pytest.mark.xfail(reason="May fail due to server errors.", raises=RequestException)
+@pytest.mark.remote
 def test_india():
     certified = CCSchemes.get_india_certified()
     assert len(certified) != 0
@@ -72,7 +71,7 @@ def test_india():
     assert absolute_urls(archived)
 
 
-@pytest.mark.xfail(reason="May fail due to server errors.", raises=RequestException)
+@pytest.mark.remote
 def test_italy():
     certified = CCSchemes.get_italy_certified()
     assert len(certified) != 0
@@ -82,7 +81,7 @@ def test_italy():
     assert absolute_urls(ineval)
 
 
-@pytest.mark.xfail(reason="May fail due to server errors.", raises=RequestException)
+@pytest.mark.remote
 def test_japan():
     certified = CCSchemes.get_japan_certified()
     assert len(certified) != 0
@@ -95,7 +94,7 @@ def test_japan():
     assert absolute_urls(ineval)
 
 
-@pytest.mark.xfail(reason="May fail due to server errors.", raises=RequestException)
+@pytest.mark.remote
 def test_malaysia():
     certified = CCSchemes.get_malaysia_certified()
     assert len(certified) != 0
@@ -108,7 +107,7 @@ def test_malaysia():
     assert absolute_urls(ineval)
 
 
-@pytest.mark.xfail(reason="May fail due to server errors.", raises=RequestException)
+@pytest.mark.remote
 def test_netherlands():
     certified = CCSchemes.get_netherlands_certified()
     assert len(certified) != 0
@@ -118,7 +117,7 @@ def test_netherlands():
     assert absolute_urls(ineval)
 
 
-@pytest.mark.xfail(reason="May fail due to server errors.", raises=RequestException)
+@pytest.mark.remote
 def test_norway():
     certified = CCSchemes.get_norway_certified()
     assert len(certified) != 0
@@ -128,7 +127,7 @@ def test_norway():
     assert absolute_urls(archived)
 
 
-@pytest.mark.xfail(reason="May fail due to server errors.", raises=RequestException)
+@pytest.mark.remote
 def test_korea():
     certified = CCSchemes.get_korea_certified()
     assert len(certified) != 0
@@ -138,7 +137,7 @@ def test_korea():
     assert absolute_urls(archived)
 
 
-@pytest.mark.xfail(reason="May fail due to server errors.", raises=RequestException)
+@pytest.mark.remote
 def test_poland():
     certified = CCSchemes.get_poland_certified()
     assert len(certified) != 0
@@ -148,7 +147,7 @@ def test_poland():
     assert absolute_urls(ineval)
 
 
-@pytest.mark.xfail(reason="May fail due to server errors.", raises=RequestException)
+@pytest.mark.remote
 def test_singapore():
     certified = CCSchemes.get_singapore_certified()
     assert len(certified) != 0
@@ -161,14 +160,14 @@ def test_singapore():
     assert absolute_urls(ineval)
 
 
-@pytest.mark.xfail(reason="May fail due to server errors.", raises=RequestException)
+@pytest.mark.remote
 def test_spain():
     certified = CCSchemes.get_spain_certified()
     assert len(certified) != 0
     assert absolute_urls(certified)
 
 
-@pytest.mark.xfail(reason="May fail due to server errors.", raises=RequestException)
+@pytest.mark.remote
 def test_sweden():
     certified = CCSchemes.get_sweden_certified()
     assert len(certified) != 0
@@ -181,14 +180,14 @@ def test_sweden():
     assert absolute_urls(ineval)
 
 
-@pytest.mark.xfail(reason="May fail due to server errors.", raises=RequestException)
+@pytest.mark.remote
 def test_turkey():
     certified = CCSchemes.get_turkey_certified()
     assert len(certified) != 0
     assert absolute_urls(certified)
 
 
-@pytest.mark.xfail(reason="May fail due to server errors.", raises=RequestException)
+@pytest.mark.remote
 def test_usa():
     certified = CCSchemes.get_usa_certified()
     assert len(certified) != 0
@@ -225,6 +224,7 @@ def test_single_match(cert_one: CCCertificate):
     assert matcher.match(cert_one) > 95
 
 
+@pytest.mark.remote
 def test_matching(toy_dataset: CCDataset, canada_certified):
     matches, scores = CCSchemeMatcher.match_all(canada_certified, "CA", toy_dataset)
     assert len(matches) == 1
diff --git a/tests/fips/test_fips_dataset.py b/tests/fips/test_fips_dataset.py
index a375cbd3..34746842 100644
--- a/tests/fips/test_fips_dataset.py
+++ b/tests/fips/test_fips_dataset.py
@@ -37,12 +37,13 @@ def test_dataset_to_json(toy_dataset: FIPSDataset, data_dir: Path, tmp_path: Pat
     assert data == template_data
 
 
-@pytest.mark.slow
+@pytest.mark.remote
 def test_from_web():
     dset = FIPSDataset.from_web()
     assert len(dset) > 4000
 
 
+@pytest.mark.remote
 def test_archive_fits():
     fsize = helpers.query_file_size(config.fips_latest_full_archive)
     tmpdir = helpers.tempdir_for(fsize)
diff --git a/tests/fips/test_fips_iut.py b/tests/fips/test_fips_iut.py
index 17d15dc8..ede3d13c 100644
--- a/tests/fips/test_fips_iut.py
+++ b/tests/fips/test_fips_iut.py
@@ -7,7 +7,6 @@ from pathlib import Path
 
 import pytest
 import tests.data.fips.iut
-from requests import RequestException
 
 from sec_certs.configuration import config
 from sec_certs.dataset.fips import FIPSDataset
@@ -33,7 +32,7 @@ def test_iut_dataset_from_dumps(data_dir: Path):
     assert len(dset) == 2
 
 
-@pytest.mark.xfail(reason="May fail due to network issues.")
+@pytest.mark.remote
 def test_iut_dataset_from_web():
     assert IUTDataset.from_web()
 
@@ -43,12 +42,13 @@ def test_iut_snapshot_from_dump(data_dump_path: Path):
 
 
 @pytest.mark.parametrize("preferred_source", ["origin", "sec-certs"])
+@pytest.mark.remote
 def test_iut_snapshot_from_web(preferred_source):
     config.preferred_source_remote_datasets = preferred_source
     assert IUTSnapshot.from_web()
 
 
-@pytest.mark.xfail(reason="May fail due to server errors.", raises=RequestException)
+@pytest.mark.remote
 def test_from_nist():
     assert IUTSnapshot.from_nist_web()
 
diff --git a/tests/fips/test_fips_mip.py b/tests/fips/test_fips_mip.py
index b8ac805f..58a11de0 100644
--- a/tests/fips/test_fips_mip.py
+++ b/tests/fips/test_fips_mip.py
@@ -7,7 +7,6 @@ from pathlib import Path
 
 import pytest
 import tests.data.fips.mip
-from requests import RequestException
 
 from sec_certs.configuration import config
 from sec_certs.dataset.fips import FIPSDataset
@@ -33,7 +32,7 @@ def test_mip_dataset_from_dumps(data_dir: Path):
     assert len(dset) == 3
 
 
-@pytest.mark.xfail(reason="May fail due to network issues.")
+@pytest.mark.remote
 def test_mip_flows():
     dset = MIPDataset.from_web()
     assert dset.compute_flows()
@@ -44,12 +43,13 @@ def test_mip_snapshot_from_dump(data_dump_path: Path):
 
 
 @pytest.mark.parametrize("preferred_source", ["sec-certs", "origin"])
+@pytest.mark.remote
 def test_from_web(preferred_source):
     config.preferred_source_remote_datasets = preferred_source
     assert MIPSnapshot.from_web()
 
 
-@pytest.mark.xfail(reason="May fail due to server errors.", raises=RequestException)
+@pytest.mark.remote
 def test_from_nist():
     assert MIPSnapshot.from_nist_web()
 
diff --git a/tests/test_cpe.py b/tests/test_cpe.py
index 637b2cb0..7b24ae59 100644
--- a/tests/test_cpe.py
+++ b/tests/test_cpe.py
@@ -72,7 +72,7 @@ def test_serialization_missing_path():
         dummy_dset.to_json()
 
 
-@pytest.mark.skip(reason="Uses too much bandwidth.")
+@pytest.mark.remote
 def test_cpe_download_from_seccerts():
     cpe_dataset = CPEDataset.from_web()
     assert len(cpe_dataset) > 100000
diff --git a/tests/test_cve.py b/tests/test_cve.py
index 8fb31dde..dcbdda8b 100644
--- a/tests/test_cve.py
+++ b/tests/test_cve.py
@@ -99,7 +99,7 @@ def test_criteria_configuration_expansion(cve_dataset_path: Path, cpe_match_feed
     ]
 
 
-@pytest.mark.skip(reason="Uses too much bandwidth.")
+@pytest.mark.remote
 def test_cve_download_from_seccerts():
     cve_dataset = CVEDataset.from_web()
     assert len(cve_dataset) > 100000
diff --git a/tests/test_nvd_dataset_builder.py b/tests/test_nvd_dataset_builder.py
index ad293604..599e0a28 100644
--- a/tests/test_nvd_dataset_builder.py
+++ b/tests/test_nvd_dataset_builder.py
@@ -20,7 +20,7 @@ def load_test_config():
         config.load_from_yaml(path)
 
 
-@pytest.mark.xfail(reason="May fail due to NVD server errors.")
+@pytest.mark.remote
 @pytest.mark.parametrize(
     "default_dataset, builder_class",
     [