Add comprehensive Copilot instructions for sec-certs repository

Co-authored-by: J08nY <1296726+J08nY@users.noreply.github.com>

1 files changed, 319 insertions, 0 deletions

diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md
new file mode 100644
index 00000000..7631ed77
--- /dev/null
+++ b/.github/copilot-instructions.md
@@ -0,0 +1,319 @@
+# Copilot Instructions for sec-certs
+
+## Repository Overview
+
+**sec-certs** is a Python data scraping and analysis tool for security certificates from Common Criteria (CC) and FIPS 140-2/3 frameworks. The tool processes certification artifacts (PDFs, HTML), extracts data, matches to CVEs/CPEs, and provides datasets for security research.
+
+### Tech Stack
+- **Language**: Python 3.10+ (tested on 3.10, 3.11, 3.12)
+- **Size**: ~75 Python source files (~13.5k LOC), ~36 test files
+- **Package Management**: pip-tools with pinned requirements in `requirements/*.txt`
+- **Key Dependencies**: BeautifulSoup4, pandas, spacy, pdftotext (requires Poppler), pikepdf, pytesseract, scikit-learn, matplotlib, networkx, pydantic
+- **Build System**: setuptools with setuptools-scm for versioning
+- **Testing**: pytest with custom markers (`slow`, `remote`)
+- **Linting**: Ruff (formatter + linter) and MyPy (type checking)
+- **Documentation**: Sphinx with myst-nb, hosted at sec-certs.org
+- **Distribution**: PyPI package and DockerHub image
+
+## Critical Setup Requirements
+
+### System Dependencies (REQUIRED)
+**ALWAYS install these system dependencies before pip packages. Code WILL fail without them:**
+
+```bash
+sudo apt-get update
+sudo apt-get install -y \
+    build-essential \
+    libpoppler-cpp-dev \
+    pkg-config \
+    python3-dev \
+    tesseract-ocr tesseract-ocr-eng tesseract-ocr-deu tesseract-ocr-fra \
+    default-jdk
+```
+
+- **Poppler** (≥20.x): Required by pdftotext library. Older 0.x versions WILL fail.
+- **Tesseract**: Required for OCR of malformed PDFs (with English, French, German data).
+- **Java**: Required to parse tables in FIPS PDF documents. Must be in PATH.
+
+### Python Environment Setup
+
+**The version file `src/sec_certs/_version.py` is auto-generated by setuptools-scm and must NOT be committed.**
+If missing during development, create a temporary version: `echo '__version__ = "dev"' > src/sec_certs/_version.py`
+
+**Standard install (for testing and development):**
+```bash
+# Install test dependencies (includes pytest, coverage, etc.)
+pip install -r requirements/test_requirements.txt
+
+# Install sec-certs in editable mode
+pip install -e .
+
+# ALWAYS download the spacy language model after install
+python -m spacy download en_core_web_sm
+```
+
+**For full development (linting, docs):**
+```bash
+pip install -r requirements/dev_requirements.txt
+pip install -e .
+python -m spacy download en_core_web_sm
+```
+
+**Note on pip-sync**: Do NOT use `pip-sync requirements/all_requirements.txt` in environments with system packages (like GitHub Actions runners). It tries to uninstall system packages and will fail. Use `pip install -r` instead.
+
+## Build, Test, and Validation
+
+### Running Tests
+
+**Basic test run (excludes remote/flaky tests):**
+```bash
+PYTHONPATH=src:$PYTHONPATH pytest tests -m "not remote" -v
+```
+
+**Test with coverage (as in CI):**
+```bash
+pytest --cov=sec_certs -m "not remote" --junitxml=junit.xml tests
+```
+
+**Test markers:**
+- `slow`: Tests that take significant time (run with `-m "slow"` or exclude with `-m "not slow"`)
+- `remote`: Tests requiring remote resources (flaky, run weekly via cron workflow)
+- `xfail`: Known flaky tests due to external server errors
+
+**Typical test runtime**: Fast tests complete in seconds. Full suite (with remote) may take minutes.
+
+### Linting and Code Quality
+
+**ALWAYS run these before committing. CI will fail if they don't pass.**
+
+**Using pre-commit (recommended):**
+```bash
+pip install -r requirements/dev_requirements.txt
+pre-commit install
+pre-commit run --all-files
+```
+
+**Manual linting:**
+```bash
+# Ruff linting (checks code style, imports, complexity)
+ruff check .
+
+# Ruff with auto-fix
+ruff check . --fix
+
+# Ruff formatting check
+ruff format --check .
+
+# Ruff auto-format
+ruff format .
+
+# MyPy type checking
+mypy .
+```
+
+**Linting configuration**: See `pyproject.toml` for Ruff and MyPy settings. Target Python 3.10. Line length: 120. Notebooks (*.ipynb) are excluded from linting.
+
+### Building Documentation
+
+```bash
+cd docs
+make html
+```
+
+Output goes to `docs/_build/html/`. Documentation uses Sphinx with myst-nb for Markdown and Jupyter notebooks.
+
+### Building for Distribution
+
+```bash
+python -m pip install build
+python -m build
+```
+
+This creates source and wheel distributions in `dist/`.
+
+## Project Architecture
+
+### Directory Structure
+
+```
+sec-certs/
+├── src/sec_certs/          # Main package source
+│   ├── dataset/            # Dataset classes (CCDataset, FIPSDataset, etc.)
+│   ├── sample/             # Certificate classes (CCCertificate, FIPSCertificate)
+│   ├── heuristics/         # Heuristic extractors and analyzers
+│   ├── model/              # ML models for matching and NLP
+│   ├── utils/              # Utility functions
+│   ├── serialization/      # JSON schemas and serialization
+│   ├── data/               # Embedded data (annotations, CPEs, etc.)
+│   ├── cli.py              # Click-based CLI entrypoint
+│   ├── configuration.py    # Pydantic config with env var support
+│   ├── rules.yaml          # Regular expressions for cert parsing
+│   └── constants.py        # Constants and enums
+├── tests/                  # Test suite
+│   ├── cc/                 # Common Criteria tests
+│   ├── fips/               # FIPS 140 tests
+│   ├── data/               # Test fixtures and data
+│   └── conftest.py         # Pytest configuration and fixtures
+├── docs/                   # Sphinx documentation source
+├── notebooks/              # Jupyter notebooks (examples, analysis)
+├── requirements/           # Pinned requirements files
+│   ├── requirements.txt    # Core dependencies
+│   ├── dev_requirements.txt    # Dev tools (ruff, mypy, sphinx)
+│   ├── test_requirements.txt   # Test dependencies
+│   ├── nlp_requirements.txt    # Optional NLP dependencies
+│   ├── all_requirements.txt    # All of the above combined
+│   └── compile.sh          # Script to regenerate requirements
+├── pyproject.toml          # Package metadata, build config, tool settings
+├── .pre-commit-config.yaml # Pre-commit hooks configuration
+└── Dockerfile              # Docker image for reproducible environment
+```
+
+### Key Files and Configurations
+
+- **pyproject.toml**: Package definition, dependencies, Ruff/MyPy/pytest config. Single source of truth for dependencies (unpinned).
+- **requirements/*.txt**: Pinned versions generated by `compile.sh`. CI uses these for reproducible builds.
+- **src/sec_certs/rules.yaml**: Regular expressions for extracting data from certificates. Add patterns here.
+- **src/sec_certs/configuration.py**: Runtime configuration using pydantic-settings. Reads from env vars with `SECCERTS_` prefix.
+- **.pre-commit-config.yaml**: Defines pre-commit hooks (ruff, mypy). Versions should match pyproject.toml.
+
+### Main Components
+
+1. **Datasets** (`src/sec_certs/dataset/`):
+   - `CCDataset`, `FIPSDataset`, `ProtectionProfileDataset`: Main dataset classes
+   - `CPEDataset`, `CVEDataset`: Auxiliary datasets from NVD
+   - Load from JSON, web snapshots, or build from scratch
+
+2. **Certificates** (`src/sec_certs/sample/`):
+   - `CCCertificate`, `FIPSCertificate`: Individual certificate representations
+   - Store metadata, extracted text, heuristics, references, CVEs
+
+3. **CLI** (`src/sec_certs/cli.py`):
+   - Entrypoint: `sec-certs {cc|fips|pp} {all|build|download|convert|analyze} [options]`
+   - Actions: `all` (full pipeline), `download` (fetch certs), `convert` (PDFs to text), `analyze` (extract features)
+
+4. **Heuristics** (`src/sec_certs/heuristics/`):
+   - Extract certification metadata (dates, vendors, products, security levels)
+   - CVE/CPE matching and vulnerability analysis
+
+## CI/CD Pipelines
+
+### GitHub Workflows (`.github/workflows/`)
+
+1. **tests.yml** (runs on every push):
+   - Tests on Python 3.10, 3.11, 3.12 (Ubuntu 22.04)
+   - Installs system deps, test_requirements.txt, spacy model
+   - Runs: `pytest --cov=sec_certs -m "not remote" tests`
+   - Uploads coverage to Codecov
+
+2. **pre-commit.yml** (runs on every push):
+   - Runs pre-commit hooks (Ruff, MyPy) on all files
+   - Fails if linting issues found
+
+3. **docs.yml** (runs on push, release):
+   - Builds Sphinx docs with `cd docs && make html`
+   - Uploads to sec-certs.org on main branch or tag push
+
+4. **release.yml** (triggered by GitHub release):
+   - Builds package with `python -m build`
+   - Publishes to PyPI
+   - Builds multi-arch Docker image (amd64, arm64) and pushes to DockerHub
+
+5. **cron.yml** (weekly, Wednesday midnight):
+   - Runs remote/flaky tests with `-m "remote"`
+   - Continue on error (expected to be flaky)
+
+### To Replicate CI Locally
+
+```bash
+# Replicate tests.yml
+sudo apt-get update
+sudo apt-get install -y build-essential libpoppler-cpp-dev pkg-config python3-dev
+pip install -r requirements/test_requirements.txt
+pip install -e .
+python -m spacy download en_core_web_sm
+pytest --cov=sec_certs -m "not remote" tests
+
+# Replicate pre-commit.yml
+pre-commit run --all-files
+```
+
+## Common Workflows
+
+### Adding a New Feature
+
+1. Create branch from `main` (only stable branch for PRs)
+2. Make minimal code changes
+3. Add tests in appropriate `tests/` subdirectory
+4. Run linters: `pre-commit run --all-files` or `ruff check . && mypy .`
+5. Run tests: `pytest tests -m "not remote" -v`
+6. Update docs if public API changed
+7. Commit and push (CI will validate)
+
+### Updating Dependencies
+
+```bash
+# Edit pyproject.toml to add/update dependency
+# Regenerate pinned requirements
+cd requirements
+./compile.sh
+# Commit both pyproject.toml and requirements/*.txt changes
+```
+
+### Working with Datasets
+
+**Loading pre-processed datasets (recommended):**
+```python
+from sec_certs.dataset.cc import CCDataset
+dset = CCDataset.from_web()  # Downloads from sec-certs.org
+```
+
+**Processing from scratch (requires full setup, takes hours):**
+```bash
+sec-certs cc all -o ./dataset
+```
+
+**Note**: Full processing requires NVD API key (set `SECCERTS_NVD_API_KEY` env var) or config change to fetch from sec-certs.org. See docs/user_guide.md.
+
+## Common Pitfalls and Gotchas
+
+1. **Missing `_version.py`**: Auto-generated by setuptools-scm. Create manually for dev: `echo '__version__ = "dev"' > src/sec_certs/_version.py`
+
+2. **Poppler version**: Ensure Poppler ≥20.x. Version 0.x will cause pdftotext failures.
+
+3. **Spacy model**: ALWAYS run `python -m spacy download en_core_web_sm` after install. Code will fail without it.
+
+4. **Java in PATH**: Required for FIPS table parsing. Verify with `java -version`.
+
+5. **pip-sync on GitHub Actions**: Don't use it with system packages. Use `pip install -r requirements/*.txt` instead.
+
+6. **Test markers**: Exclude flaky remote tests with `-m "not remote"` for stable local testing.
+
+7. **Import from src**: When running without install, set `PYTHONPATH=src:$PYTHONPATH` to import sec_certs modules.
+
+8. **Default dataset location**: CLI creates `./dataset` by default. Add to .gitignore if working locally.
+
+9. **Pre-commit hook behavior**: Pre-commit hooks warn about issues but don't auto-fix. Run `ruff check . --fix` to apply fixes.
+
+10. **Long-running commands**: Full dataset processing (`sec-certs cc all`) takes hours. Use pre-processed datasets from web for analysis.
+
+## Additional Resources
+
+- **README.md**: Quick start, installation, basic usage examples
+- **CONTRIBUTING.md**: Detailed contribution guidelines, release process, dependency management
+- **docs/installation.md**: System dependencies, multiple install methods
+- **docs/quickstart.md**: Quick usage examples for CC and FIPS datasets
+- **docs/user_guide.md**: Advanced topics (NVD datasets, reference context inference)
+- **notebooks/examples/**: Jupyter notebooks demonstrating dataset analysis
+- **Website**: https://sec-certs.org (dataset downloads, interactive docs)
+- **Documentation**: https://sec-certs.org/docs
+
+## Trust These Instructions
+
+These instructions have been validated by examining repository structure, workflows, documentation, and testing commands. When working on this repository:
+
+1. **Trust these build/test commands** - they are verified to work
+2. **Follow the setup order** (system deps → pip deps → spacy model → install)
+3. **Only search/explore if** these instructions are incomplete or incorrect
+4. **Refer to these instructions first** before trying alternative approaches
+
+If you encounter issues not covered here, check CONTRIBUTING.md and docs/ before extensive exploration.