Merge pull request #491 from crocs-muni/490_semantic_search

Semantic search demo

8 files changed, 600 insertions, 151 deletions

diff --git a/notebooks/cc/vector_search.md b/notebooks/cc/vector_search.md
new file mode 100644
index 00000000..fd1ce628
--- /dev/null
+++ b/notebooks/cc/vector_search.md
@@ -0,0 +1,106 @@
+# Certificate Vector Search Demo
+
+This is a single-file Python script that demonstrates how to process and search through certificate documents using vector embeddings. It’s not a full-blown project but rather a proof-of-concept for semantic search over certificate datasets. It’s designed to be simple and accurate, though not highly scalable out of the box.
+
+---
+
+## What It Does
+
+1. **Processes Certificate Documents**:
+   - Takes a dataset of certificates (loaded from a JSON file).
+   - Extracts text from associated `.txt` files (e.g., reports, security targets, certificates).
+   - Splits the text into smaller, overlapping chunks (512 tokens max, with 128 tokens of overlap) to handle large documents.
+
+2. **Generates Embeddings**:
+   - Uses the `sentence-transformers/all-MiniLM-L6-v2` model to convert each text chunk into a vector embedding. It's a small model that can be run on the CPU.
+   - Stores these embeddings in a SQLite database with vector support (thanks to `sqlite-vec`).
+
+3. **Makes It Searchable**:
+   - You can query the database with a text string, and it’ll find the most semantically similar chunks of text from the certificates. The query is embedded with the aforementioned `sentence-transformers` model.
+   - Results include metadata (like certificate name, source type, etc.) and the actual text of the chunk.
+
+4. **Provides a Web Interface**:
+   - A simple HTTP server lets you interact with the system via a browser.
+   - Submit a query, and it’ll show you the top results with snippets of the most relevant text.
+---
+
+## How It Works
+
+### Dataset Processing
+- The dataset is validated to ensure it contains the required files (`dataset.json` and `.txt` files).
+- Text is extracted from the `.txt` files, split into chunks, and converted into embeddings.
+
+### Database Setup
+The SQLite database has three main tables:
+- **`cert_chunks`**: Stores the vector embeddings, along with metadata like the certificate digest and chunk index.
+- **`metadata`**: Stores certificate-level info (name, manufacturer, validity dates, etc.).
+- **`chunk_texts`**: Stores the actual text of each chunk for easy retrieval.
+
+The database is initialized with vector support using the `sqlite-vec` extension, which enables similarity searches.
+
+### Querying
+- When you submit a query, the system:
+  1. Converts the query text into an embedding.
+  2. Searches the database for chunks with the closest embeddings.
+  3. Groups chunks by unique document (digest x document type: report, security target) combination.
+  4. Ranks results using a weighted score (combining the closest match and average similarity).
+  5. Returns the top `k` results, including metadata and the text of the most relevant chunk.
+
+### Web Interface
+- A basic HTTP server runs locally, serving a simple HTML page where you can enter queries and see results.
+- The user query is embedded on the back-end and a similarity search is performed.
+- Results are displayed with the certificate name, source type, similarity score, and a snippet of the text.
+
+---
+
+## Scalability Notes
+
+This demo uses [brute-force](https://github.com/asg017/sqlite-vec/issues/172#issuecomment-2608754427) linear search for simplicity and accuracy. While this works fine for small to medium datasets, it’s not scalable for large datasets. For better performance, you could integrate:
+
+- **Approximate Nearest Neighbor (ANN) Search**: Plugins supporting approaches like [FAISS](https://github.com/maylad31/vector_sqlite) or [HNSW](https://github.com/nmslib/hnswlib) can speed up retrieval for large datasets, at the cost of some accuracy.
+- **Vectorlite**: A lightweight SQLite [extension](https://github.com/1yefuwang1/vectorlite) for fast vector search, seems most reasonable as an upgrade.
+
+---
+
+## Mixed Search
+In the future, it might be desirable to combine semantic search (vector embeddings) with keyword-based search (BM25 or TF-IDF) for hybrid retrieval.
+
+
+## How to Use It
+
+1. **Install Dependencies**:
+   ```bash
+   pip install sec_certs sentence-transformers nltk sqlite-vec
+   ```
+
+2. **Run the Script**:
+   ```bash
+   python vector_search.py --data-path /path/to/dataset --db-path /path/to/database.sqlite --force-rebuild --port 8080
+   ```
+   - `--data-path`: Path to the dataset directory (should contain `dataset.json` and the `certs` folder).
+   - `--db-path`: Path to the SQLite database file.
+   - `--force-rebuild`: Rebuild the database from scratch (optional).
+   - `--port`: Port to run the HTTP server on (default is 8000).
+
+3. **Search**:
+   - Open your browser and go to `http://localhost:8000`.
+   - Enter a query, and see the results!
+
+---
+
+
+## Code Overview
+
+The script is a single Python file with the following key components:
+- **`CertProcessor`**: Handles dataset processing, embedding generation, and database storage.
+- **`query_similar_chunks`**: Queries the database for similar chunks and returns results with metadata.
+- **`Server`**: A simple HTTP server that serves the web interface and handles search queries.
+
+---
+
+## Dependencies
+
+- `sentence-transformers`: For generating text embeddings.
+- `nltk`: For tokenizing text into words.
+- `sqlite-vec`: For enabling vector operations in SQLite.
+- `sec_certs`: For loading and processing the certificate dataset.
diff --git a/notebooks/cc/vector_search.py b/notebooks/cc/vector_search.py
new file mode 100644
index 00000000..ae0445a4
--- /dev/null
+++ b/notebooks/cc/vector_search.py
@@ -0,0 +1,388 @@
+import argparse
+import sqlite3
+import struct
+import sys
+import urllib.parse
+from http.server import BaseHTTPRequestHandler, HTTPServer
+from pathlib import Path
+
+import nltk
+import sqlite_vec
+from nltk.tokenize import word_tokenize
+from sentence_transformers import SentenceTransformer
+from tqdm import tqdm
+
+from sec_certs.dataset import CCDataset
+from sec_certs.sample import CCCertificate
+
+# initialize NLP resources - one-time downloads
+nltk.download("punkt")
+nltk.download("punkt_tab")
+
+TOKEN_LIMIT = 512
+OVERLAP = 128
+MODEL_NAME = "sentence-transformers/all-MiniLM-L6-v2"
+
+
+def serialize_f32(vector: list[float]) -> bytes:
+    """Serialize float list to bytes for sqlite_vec"""
+    return struct.pack(f"{len(vector)}f", *vector)
+
+
+class CertProcessor:
+    def __init__(self, model=None, fresh_db=False):
+        if model is not None:
+            self.model = model
+        else:
+            self.model = SentenceTransformer(MODEL_NAME)
+        self.db = self.init_db(fresh_db)
+        self.cursor = self.db.cursor()
+
+    def init_db(self, fresh_db) -> sqlite3.Connection:
+        """Initialize database with vector extension"""
+        conn = sqlite3.connect(DB_PATH)
+        conn.enable_load_extension(True)
+        sqlite_vec.load(conn)
+        conn.enable_load_extension(False)
+
+        if fresh_db is True:
+            conn.execute("DROP TABLE IF EXISTS cert_chunks;")
+            conn.execute("DROP TABLE IF EXISTS metadata;")
+            conn.execute("DROP TABLE IF EXISTS chunk_texts;")
+
+            conn.execute("""
+                CREATE VIRTUAL TABLE cert_chunks USING vec0(
+                    embedding float[384],
+                    dgst TEXT,
+                    chunk_index INTEGER,
+                    source_type TEXT,
+                )
+            """)
+            conn.execute("""
+                CREATE TABLE metadata (
+                    dgst TEXT PRIMARY KEY,
+                    name TEXT,
+                    category TEXT,
+                    manufacturer TEXT,
+                    security_level TEXT,
+                    valid_from TEXT,
+                    valid_to TEXT,
+                    report_link TEXT,
+                    st_link TEXT,
+                    cert_link TEXT,
+                    manufacturer_web TEXT
+                )
+            """)
+
+            conn.execute("""
+                CREATE TABLE chunk_texts (
+                    dgst TEXT,
+                    source_type TEXT,
+                    chunk_index INTEGER,
+                    text TEXT,
+                    PRIMARY KEY (dgst, source_type, chunk_index)
+                )
+            """)
+
+            # create indices for metadata
+            conn.execute("CREATE INDEX idx_chunk_texts_dgst ON chunk_texts(dgst);")
+            conn.execute("CREATE INDEX idx_chunk_texts_source ON chunk_texts(source_type);")
+
+        return conn
+
+    def store_metadata(self, meta: dict) -> None:
+        """Store certificate metadata in the metadata table"""
+        params = {
+            "dgst": meta["dgst"],
+            "name": meta.get("name", "Unknown"),
+            "category": meta.get("category", "unknown"),
+            "manufacturer": meta.get("manufacturer", "Unknown"),
+            "security_level": ",".join(meta.get("security_level", [])),
+            "valid_from": meta["not_valid_before"].isoformat(),
+            "valid_to": meta["not_valid_after"].isoformat(),
+            "report_link": meta.get("report_link", ""),
+            "st_link": meta.get("st_link", ""),
+            "cert_link": meta.get("cert_link", ""),
+            "manufacturer_web": meta.get("manufacturer_web", ""),
+        }
+        self.cursor.execute(
+            """
+            INSERT INTO metadata
+            (dgst, name, category, manufacturer, security_level,
+             valid_from, valid_to, report_link, st_link, cert_link, manufacturer_web)
+            VALUES (:dgst, :name, :category, :manufacturer, :security_level,
+                    :valid_from, :valid_to, :report_link, :st_link, :cert_link, :manufacturer_web)
+        """,
+            params,
+        )
+
+    def chunk_text(self, text: str) -> list[str]:
+        """Split text into overlapping chunks"""
+        words = word_tokenize(text)
+        return [" ".join(words[i : i + TOKEN_LIMIT]) for i in range(0, len(words), TOKEN_LIMIT - OVERLAP)]
+
+    def process_cert(self, cert: CCCertificate) -> None:
+        """Process a single certificate with all its documents"""
+        meta = cert.to_dict()
+        file_paths = {
+            "report": cert.state.report._txt_path,
+            "targets": cert.state.st._txt_path,
+            "cert": cert.state.cert._txt_path,
+        }
+
+        for doc_type, path in file_paths.items():
+            if path is None or not path.exists():
+                continue
+
+            with path.open(encoding="utf-8") as f:
+                text = f.read()
+
+            chunks = self.chunk_text(text)
+            self.store_chunks(chunks, meta, doc_type)
+
+    def store_chunks(self, chunks: list[str], meta: dict, doc_type: str) -> None:
+        """Store chunks with metadata in database"""
+        for idx, chunk in enumerate(tqdm(chunks, desc=f"Chunking {doc_type}", leave=False)):
+            # generate embedding
+            embedding = self.model.encode(chunk).flatten().tolist()
+
+            # insert into cert_chunks table
+            self.cursor.execute(
+                """
+                INSERT INTO cert_chunks
+                (embedding, dgst, chunk_index, source_type)
+                VALUES (?, ?, ?, ?)
+            """,
+                [serialize_f32(embedding), meta["dgst"], idx, doc_type],
+            )
+
+            self.cursor.execute(
+                """
+                INSERT INTO chunk_texts
+                (dgst, source_type, chunk_index, text)
+                VALUES (?, ?, ?, ?)
+            """,
+                [meta["dgst"], doc_type, idx, chunk],
+            )
+
+        # store metadata if not already present
+        self.cursor.execute("SELECT dgst FROM metadata WHERE dgst = ?", [meta["dgst"]])
+        if not self.cursor.fetchone():
+            self.store_metadata(meta)
+
+    def process_all(self, dataset) -> None:
+        """Process entire dataset"""
+        for cert in tqdm(dataset, desc="Processing certificates", unit="cert"):
+            self.process_cert(cert)
+        self.db.commit()
+
+    def close(self) -> None:
+        """Close the database connection"""
+        self.cursor.close()
+        self.db.close()
+
+
+def query_similar_chunks(
+    query_text: str, processor: CertProcessor, k: int = 5
+) -> list[tuple[str, str, str, str, str, str]]:
+    """
+    Query the database for similar chunks and include metadata.
+
+    Args:
+        query_embedding: The embedding of the query text.
+        processor: An instance of CertProcessor.
+        k: The number of results to return.
+
+    Returns:
+        A list of tuples containing (dgst, name, chunk_index, source_type, distance).
+    """
+    query_embedding = processor.model.encode(query_text).flatten().tolist()
+    serialized_query = serialize_f32(query_embedding)
+
+    # fetch similar chunks and join with metadata
+    query = """
+        SELECT
+        m.dgst,
+        m.name,
+        c.source_type,
+        0.7 * MIN(c.distance) + 0.3 * AVG(c.distance) AS weighted_score,
+        first_value(c.chunk_index) OVER (
+            PARTITION BY m.dgst, c.source_type
+            ORDER BY c.distance ASC
+        ) AS best_chunk_index
+        FROM cert_chunks c
+        JOIN metadata m ON c.dgst = m.dgst
+        WHERE c.embedding MATCH ? AND k=2000
+        GROUP BY
+        m.dgst, c.source_type
+        ORDER BY
+        weighted_score ASC
+        LIMIT ?
+        """
+
+    doc_results = processor.db.execute(query, [serialized_query, k]).fetchall()
+
+    # fetch the text for each best chunk
+    results_with_text = []
+    for dgst, name, source_type, weighted_score, best_chunk_idx in doc_results:
+        # query to get the text chunk
+        text_query = """
+            SELECT text
+            FROM chunk_texts
+            WHERE dgst = ? AND source_type = ? AND chunk_index = ?
+        """
+        text_result = processor.db.execute(text_query, [dgst, source_type, best_chunk_idx]).fetchone()
+        chunk_text = text_result[0] if text_result else "Text not found"
+
+        # add text chunk to result
+        results_with_text.append((dgst, name, source_type, weighted_score, best_chunk_idx, chunk_text))
+
+    return results_with_text
+
+
+source_map = {"targets": "Security target", "report": "Certification report", "cert": "Certificate"}
+
+
+class Server(BaseHTTPRequestHandler):
+    def do_GET(self) -> None:
+        self.send_response(200)
+        self.send_header("Content-type", "text/html; charset=utf-8")
+        self.end_headers()
+
+        # parse query
+        query = ""
+        results = []
+        if "?" in self.path:
+            query = urllib.parse.parse_qs(self.path.split("?")[1]).get("q", [""])[0]
+            if query:
+                results = query_similar_chunks(query, processor, k=5)
+
+        # Build response
+        html = f"""
+        <!DOCTYPE html>
+        <html>
+        <head>
+            <meta charset="UTF-8">
+            <title>Cert Search</title>
+            <style>
+                body {{ font-family: sans-serif; max-width: 900px; margin: 0 auto; padding: 20px; }}
+                .result {{ border: 1px solid #ddd; margin: 15px 0; padding: 15px; border-radius: 5px; }}
+            </style>
+        </head>
+        <body>
+            <h1>Certificate Vector Search</h1>
+            <form method="get">
+                <input type="text" name="q" placeholder="Search query..." value="{query}">
+                <input type="submit" value="Search">
+            </form>
+        """
+
+        if results:
+            html += "<h2>Results</h2>"
+            for dgst, name, source_type, score, chunk_idx, text in results:
+                html += f"""
+                <div class="result">
+                    <h3>{name}</h3>
+                    <p>Source: {source_map.get(source_type, "Unknown")} | Score: {score:.4f}</p>
+                    <p>{text[:500]}...</p>
+                </div>
+                """
+
+        html += "</body></html>"
+        self.wfile.write(html.encode())
+
+
+def check_dataset_path(path_dset: Path) -> bool:
+    """Verify that the dataset path exists and contains dataset.json."""
+    if not path_dset.exists():
+        print(f"Error: Dataset path '{path_dset}' does not exist")
+        sys.exit(1)
+
+    if not (path_dset / "dataset.json").exists():
+        print(f"Error: dataset.json not found in '{path_dset}'")
+        sys.exit(1)
+
+    return True
+
+
+def check_certificate_count(dset: CCDataset) -> int:
+    """Check and report the number of certificates in the dataset."""
+    cert_count = len(list(dset.certs))
+    print(f"Dataset contains {cert_count} certificates")
+    if cert_count == 0:
+        print("Error: Dataset is empty")
+        sys.exit(1)
+
+    return cert_count
+
+
+def check_txt_files(path_dset: Path) -> int:
+    """Verify that text files exist in the expected directories."""
+    txt_paths = [path_dset / "certs/certificates/txt", path_dset / "certs/reports/txt", path_dset / "certs/targets/txt"]
+
+    txt_counts = []
+    for path in txt_paths:
+        if path.exists():
+            count = len(list(path.glob("*.txt")))
+            txt_counts.append(count)
+            print(f"Found {count} .txt files in {path}")
+        else:
+            print(f"Warning: Path {path} does not exist")
+            txt_counts.append(0)
+
+    total = sum(txt_counts)
+    if total == 0:
+        print("Error: No .txt files found in any of the expected directories")
+        sys.exit(1)
+
+    return total
+
+
+def manage_database(db_path: Path, model: SentenceTransformer, force_rebuild: bool, dset: CCDataset) -> bool:
+    """Create or rebuild the database if needed."""
+    if force_rebuild or not db_path.exists():
+        print(f"{'Rebuilding' if force_rebuild else 'Creating'} database at {db_path}")
+        if db_path.exists() and force_rebuild:
+            db_path.unlink()
+
+        processor = CertProcessor(model, fresh_db=True)
+        processor.process_all(dset)
+        processor.close()
+        print("Database initialization complete.")
+        return True
+    return False
+
+
+# pip install sec_certs sentence-transformers nltk sqlite-vec
+# Usage example: python vector_search.py --data-path ../../cc_data_gemini --db-path vector_db.sqlite --force-rebuild --port 8080
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="Certificate Vector Search")
+    parser.add_argument("--data-path", type=str, default="../../cc_data_gemini", help="Path to the dataset directory")
+    parser.add_argument("--db-path", type=str, default="vector_db.sqlite", help="Path to the SQLite database file")
+    parser.add_argument("--force-rebuild", action="store_true", help="Force database rebuild even if it exists")
+    parser.add_argument("--port", type=int, default=8000, help="Port to run the server on")
+    args = parser.parse_args()
+
+    path_dset = Path(args.data_path)
+    DB_PATH = Path(args.db_path)
+
+    check_dataset_path(path_dset)
+
+    dset = CCDataset.from_json(path_dset / "dataset.json")
+
+    cert_count = check_certificate_count(dset)
+    txt_count = check_txt_files(path_dset)
+
+    model = SentenceTransformer(MODEL_NAME)
+
+    db_rebuilt = manage_database(DB_PATH, model, args.force_rebuild, dset)
+
+    processor = CertProcessor(model, fresh_db=False)
+
+    server = HTTPServer(("localhost", 8000), Server)
+    print("Server started at http://localhost:8000")
+    try:
+        server.serve_forever()
+    except KeyboardInterrupt:
+        processor.close()
+        print("Server stopped.")
diff --git a/pyproject.toml b/pyproject.toml
index 2f7c7db9..aafd0575 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -92,6 +92,9 @@
       "umap-learn[plot]",
       "plotly",
       "scikit-learn",
+      "sentence-transformers",
+      "nltk",
+      "sqlite-vec",
       "openai",
       "pyarrow"
     ]
diff --git a/requirements/all_requirements.txt b/requirements/all_requirements.txt
index d2a1b04a..c153ce6f 100644
--- a/requirements/all_requirements.txt
+++ b/requirements/all_requirements.txt
@@ -2,7 +2,7 @@ accelerate==1.4.0
     # via sentence-transformers
 accessible-pygments==0.0.5
     # via pydata-sphinx-theme
-aiohappyeyeballs==2.4.6
+aiohappyeyeballs==2.5.0
     # via aiohttp
 aiohttp==3.11.13
     # via
@@ -12,7 +12,7 @@ aiosignal==1.3.2
     # via aiohttp
 alabaster==1.0.0
     # via sphinx
-alembic==1.14.1
+alembic==1.15.1
     # via optuna
 annotated-types==0.7.0
     # via pydantic
@@ -20,6 +20,8 @@ anyio==4.8.0
     # via
     #   httpx
     #   openai
+appnope==0.1.4
+    # via ipykernel
 asttokens==3.0.0
     # via stack-data
 async-timeout==5.0.1
@@ -70,10 +72,11 @@ charset-normalizer==3.4.1
 click==8.1.8
     # via
     #   jupyter-cache
+    #   nltk
     #   pip-tools
     #   sec-certs (../pyproject.toml)
     #   typer
-cloudpathlib==0.20.0
+cloudpathlib==0.21.0
     # via weasel
 colorcet==3.1.0
     # via
@@ -98,7 +101,7 @@ coverage[toml]==7.6.12
     # via
     #   pytest-cov
     #   sec-certs (../pyproject.toml)
-cryptography==44.0.1
+cryptography==44.0.2
     # via pypdf
 cycler==0.12.1
     # via matplotlib
@@ -117,7 +120,7 @@ datashader==0.17.0
     # via umap-learn
 dateparser==1.2.1
     # via sec-certs (../pyproject.toml)
-debugpy==1.8.12
+debugpy==1.8.13
     # via ipykernel
 decorator==5.2.1
     # via ipython
@@ -173,8 +176,6 @@ gprof2dot==2024.6.6
     # via pytest-profiling
 graphviz==0.20.3
     # via catboost
-greenlet==3.1.1
-    # via sqlalchemy
 h11==0.14.0
     # via httpcore
 holoviews==1.20.1
@@ -185,7 +186,7 @@ httpcore==1.0.7
     # via httpx
 httpx==0.28.1
     # via openai
-huggingface-hub==0.29.1
+huggingface-hub==0.29.2
     # via
     #   accelerate
     #   datasets
@@ -194,7 +195,7 @@ huggingface-hub==0.29.1
     #   setfit
     #   tokenizers
     #   transformers
-identify==2.6.8
+identify==2.6.9
     # via pre-commit
 idna==3.10
     # via
@@ -216,7 +217,7 @@ ipykernel==6.29.5
     # via
     #   myst-nb
     #   sec-certs (../pyproject.toml)
-ipython==8.32.0
+ipython==8.34.0
     # via
     #   ipykernel
     #   ipywidgets
@@ -237,6 +238,7 @@ jiter==0.8.2
     # via openai
 joblib==1.4.2
     # via
+    #   nltk
     #   pynndescent
     #   scikit-learn
 jsonschema==4.23.0
@@ -293,7 +295,7 @@ markupsafe==3.0.2
     # via
     #   jinja2
     #   mako
-matplotlib==3.10.0
+matplotlib==3.10.1
     # via
     #   catboost
     #   pysankeybeta
@@ -337,7 +339,7 @@ myst-nb==1.2.0
     # via sec-certs (../pyproject.toml)
 myst-parser==4.0.1
     # via myst-nb
-narwhals==1.28.0
+narwhals==1.29.1
     # via plotly
 nbclient==0.10.2
     # via
@@ -355,6 +357,8 @@ networkx==3.4.2
     #   scikit-image
     #   sec-certs (../pyproject.toml)
     #   torch
+nltk==3.9.1
+    # via sec-certs (../pyproject.toml)
 nodeenv==1.9.1
     # via pre-commit
 numba==0.61.0
@@ -391,41 +395,7 @@ numpy==1.26.4
     #   transformers
     #   umap-learn
     #   xarray
-nvidia-cublas-cu12==12.4.5.8
-    # via
-    #   nvidia-cudnn-cu12
-    #   nvidia-cusolver-cu12
-    #   torch
-nvidia-cuda-cupti-cu12==12.4.127
-    # via torch
-nvidia-cuda-nvrtc-cu12==12.4.127
-    # via torch
-nvidia-cuda-runtime-cu12==12.4.127
-    # via torch
-nvidia-cudnn-cu12==9.1.0.70
-    # via torch
-nvidia-cufft-cu12==11.2.1.3
-    # via torch
-nvidia-curand-cu12==10.3.5.147
-    # via torch
-nvidia-cusolver-cu12==11.6.1.9
-    # via torch
-nvidia-cusparse-cu12==12.3.1.170
-    # via
-    #   nvidia-cusolver-cu12
-    #   torch
-nvidia-cusparselt-cu12==0.6.2
-    # via torch
-nvidia-nccl-cu12==2.21.5
-    # via torch
-nvidia-nvjitlink-cu12==12.4.127
-    # via
-    #   nvidia-cusolver-cu12
-    #   nvidia-cusparse-cu12
-    #   torch
-nvidia-nvtx-cu12==12.4.127
-    # via torch
-openai==1.64.0
+openai==1.65.5
     # via sec-certs (../pyproject.toml)
 optuna==4.2.1
     # via sec-certs (../pyproject.toml)
@@ -556,7 +526,7 @@ pydantic==2.10.6
     #   weasel
 pydantic-core==2.27.2
     # via pydantic
-pydantic-settings==2.8.0
+pydantic-settings==2.8.1
     # via sec-certs (../pyproject.toml)
 pydata-sphinx-theme==0.15.4
     # via sphinx-book-theme
@@ -571,7 +541,7 @@ pynndescent==0.5.13
     # via umap-learn
 pyparsing==3.2.1
     # via matplotlib
-pypdf[crypto]==5.3.0
+pypdf[crypto]==5.3.1
     # via sec-certs (../pyproject.toml)
 pyproject-hooks==1.2.0
     # via
@@ -581,7 +551,7 @@ pysankeybeta==1.4.2
     # via sec-certs (../pyproject.toml)
 pytesseract==0.3.13
     # via sec-certs (../pyproject.toml)
-pytest==8.3.4
+pytest==8.3.5
     # via
     #   pytest-cov
     #   pytest-monitor
@@ -627,7 +597,7 @@ pyzmq==26.2.1
     # via
     #   ipykernel
     #   jupyter-client
-rapidfuzz==3.12.1
+rapidfuzz==3.12.2
     # via sec-certs (../pyproject.toml)
 referencing==0.36.2
     # via
@@ -636,6 +606,7 @@ referencing==0.36.2
 regex==2024.11.6
     # via
     #   dateparser
+    #   nltk
     #   transformers
 requests==2.32.3
     # via
@@ -687,7 +658,9 @@ seaborn==0.13.2
     #   sec-certs (../pyproject.toml)
     #   umap-learn
 sentence-transformers[train]==3.4.1
-    # via setfit
+    # via
+    #   sec-certs (../pyproject.toml)
+    #   setfit
 setfit==1.1.1
     # via sec-certs (../pyproject.toml)
 setuptools-scm==8.2.0
@@ -748,6 +721,8 @@ sqlalchemy==2.0.38
     #   alembic
     #   jupyter-cache
     #   optuna
+sqlite-vec==0.1.6
+    # via sec-certs (../pyproject.toml)
 srsly==2.5.1
     # via
     #   confection
@@ -795,6 +770,7 @@ tqdm==4.67.1
     #   datasets
     #   evaluate
     #   huggingface-hub
+    #   nltk
     #   openai
     #   optuna
     #   panel
@@ -818,9 +794,7 @@ transformers==4.49.0
     # via
     #   sentence-transformers
     #   setfit
-triton==3.2.0
-    # via torch
-typer==0.15.1
+typer==0.15.2
     # via
     #   spacy
     #   weasel
@@ -828,7 +802,7 @@ types-python-dateutil==2.9.0.20241206
     # via sec-certs (../pyproject.toml)
 types-pyyaml==6.0.12.20241230
     # via sec-certs (../pyproject.toml)
-types-requests==2.32.0.20241016
+types-requests==2.32.0.20250306
     # via sec-certs (../pyproject.toml)
 typing-extensions==4.12.2
     # via
@@ -854,7 +828,7 @@ typing-extensions==4.12.2
     #   typer
 tzdata==2025.1
     # via pandas
-tzlocal==5.3
+tzlocal==5.3.1
     # via dateparser
 uc-micro-py==1.0.3
     # via linkify-it-py
@@ -864,7 +838,7 @@ urllib3==2.3.0
     # via
     #   requests
     #   types-requests
-virtualenv==20.29.2
+virtualenv==20.29.3
     # via pre-commit
 wasabi==1.1.3
     # via
diff --git a/requirements/dev_requirements.txt b/requirements/dev_requirements.txt
index c0992df2..de594cb0 100644
--- a/requirements/dev_requirements.txt
+++ b/requirements/dev_requirements.txt
@@ -1,6 +1,6 @@
 accessible-pygments==0.0.5
     # via pydata-sphinx-theme
-aiohappyeyeballs==2.4.6
+aiohappyeyeballs==2.5.0
     # via aiohttp
 aiohttp==3.11.13
     # via
@@ -12,6 +12,8 @@ alabaster==1.0.0
     # via sphinx
 annotated-types==0.7.0
     # via pydantic
+appnope==0.1.4
+    # via ipykernel
 asttokens==3.0.0
     # via stack-data
 async-timeout==5.0.1
@@ -53,7 +55,7 @@ click==8.1.8
     #   pip-tools
     #   sec-certs (../pyproject.toml)
     #   typer
-cloudpathlib==0.20.0
+cloudpathlib==0.21.0
     # via weasel
 comm==0.2.2
     # via
@@ -67,7 +69,7 @@ contourpy==1.3.1
     # via matplotlib
 coverage[toml]==7.6.12
     # via pytest-cov
-cryptography==44.0.1
+cryptography==44.0.2
     # via pypdf
 cycler==0.12.1
     # via matplotlib
@@ -80,7 +82,7 @@ datasets==3.3.2
     # via sec-certs (../pyproject.toml)
 dateparser==1.2.1
     # via sec-certs (../pyproject.toml)
-debugpy==1.8.12
+debugpy==1.8.13
     # via ipykernel
 decorator==5.2.1
     # via ipython
@@ -124,13 +126,11 @@ fsspec[http]==2024.12.0
     #   huggingface-hub
 gprof2dot==2024.6.6
     # via pytest-profiling
-greenlet==3.1.1
-    # via sqlalchemy
 html5lib==1.1
     # via sec-certs (../pyproject.toml)
-huggingface-hub==0.29.1
+huggingface-hub==0.29.2
     # via datasets
-identify==2.6.8
+identify==2.6.9
     # via pre-commit
 idna==3.10
     # via
@@ -148,7 +148,7 @@ ipykernel==6.29.5
     # via
     #   myst-nb
     #   sec-certs (../pyproject.toml)
-ipython==8.32.0
+ipython==8.34.0
     # via
     #   ipykernel
     #   ipywidgets
@@ -204,7 +204,7 @@ markdown-it-py==3.0.0
     #   rich
 markupsafe==3.0.2
     # via jinja2
-matplotlib==3.10.0
+matplotlib==3.10.1
     # via
     #   pysankeybeta
     #   seaborn
@@ -351,7 +351,7 @@ pydantic==2.10.6
     #   weasel
 pydantic-core==2.27.2
     # via pydantic
-pydantic-settings==2.8.0
+pydantic-settings==2.8.1
     # via sec-certs (../pyproject.toml)
 pydata-sphinx-theme==0.15.4
     # via sphinx-book-theme
@@ -364,7 +364,7 @@ pygments==2.19.1
     #   sphinx
 pyparsing==3.2.1
     # via matplotlib
-pypdf[crypto]==5.3.0
+pypdf[crypto]==5.3.1
     # via sec-certs (../pyproject.toml)
 pyproject-hooks==1.2.0
     # via
@@ -374,7 +374,7 @@ pysankeybeta==1.4.2
     # via sec-certs (../pyproject.toml)
 pytesseract==0.3.13
     # via sec-certs (../pyproject.toml)
-pytest==8.3.4
+pytest==8.3.5
     # via
     #   pytest-cov
     #   pytest-monitor
@@ -412,7 +412,7 @@ pyzmq==26.2.1
     # via
     #   ipykernel
     #   jupyter-client
-rapidfuzz==3.12.1
+rapidfuzz==3.12.2
     # via sec-certs (../pyproject.toml)
 referencing==0.36.2
     # via
@@ -543,7 +543,7 @@ traitlets==5.14.3
     #   matplotlib-inline
     #   nbclient
     #   nbformat
-typer==0.15.1
+typer==0.15.2
     # via
     #   spacy
     #   weasel
@@ -551,7 +551,7 @@ types-python-dateutil==2.9.0.20241206
     # via sec-certs (../pyproject.toml)
 types-pyyaml==6.0.12.20241230
     # via sec-certs (../pyproject.toml)
-types-requests==2.32.0.20241016
+types-requests==2.32.0.20250306
     # via sec-certs (../pyproject.toml)
 typing-extensions==4.12.2
     # via
@@ -572,13 +572,13 @@ typing-extensions==4.12.2
     #   typer
 tzdata==2025.1
     # via pandas
-tzlocal==5.3
+tzlocal==5.3.1
     # via dateparser
 urllib3==2.3.0
     # via
     #   requests
     #   types-requests
-virtualenv==20.29.2
+virtualenv==20.29.3
     # via pre-commit
 wasabi==1.1.3
     # via
diff --git a/requirements/nlp_requirements.txt b/requirements/nlp_requirements.txt
index 1d5e273d..541a26b8 100644
--- a/requirements/nlp_requirements.txt
+++ b/requirements/nlp_requirements.txt
@@ -1,6 +1,6 @@
 accelerate==1.4.0
     # via sentence-transformers
-aiohappyeyeballs==2.4.6
+aiohappyeyeballs==2.5.0
     # via aiohttp
 aiohttp==3.11.13
     # via
@@ -8,7 +8,7 @@ aiohttp==3.11.13
     #   fsspec
 aiosignal==1.3.2
     # via aiohttp
-alembic==1.14.1
+alembic==1.15.1
     # via optuna
 annotated-types==0.7.0
     # via pydantic
@@ -16,6 +16,8 @@ anyio==4.8.0
     # via
     #   httpx
     #   openai
+appnope==0.1.4
+    # via ipykernel
 asttokens==3.0.0
     # via stack-data
 async-timeout==5.0.1
@@ -54,9 +56,10 @@ charset-normalizer==3.4.1
     # via requests
 click==8.1.8
     # via
+    #   nltk
     #   sec-certs (../pyproject.toml)
     #   typer
-cloudpathlib==0.20.0
+cloudpathlib==0.21.0
     # via weasel
 colorcet==3.1.0
     # via
@@ -77,7 +80,7 @@ contourpy==1.3.1
     # via
     #   bokeh
     #   matplotlib
-cryptography==44.0.1
+cryptography==44.0.2
     # via pypdf
 cycler==0.12.1
     # via matplotlib
@@ -95,7 +98,7 @@ datashader==0.17.0
     # via umap-learn
 dateparser==1.2.1
     # via sec-certs (../pyproject.toml)
-debugpy==1.8.12
+debugpy==1.8.13
     # via ipykernel
 decorator==5.2.1
     # via ipython
@@ -138,8 +141,6 @@ fsspec[http]==2024.12.0
     #   torch
 graphviz==0.20.3
     # via catboost
-greenlet==3.1.1
-    # via sqlalchemy
 h11==0.14.0
     # via httpcore
 holoviews==1.20.1
@@ -150,7 +151,7 @@ httpcore==1.0.7
     # via httpx
 httpx==0.28.1
     # via openai
-huggingface-hub==0.29.1
+huggingface-hub==0.29.2
     # via
     #   accelerate
     #   datasets
@@ -169,7 +170,7 @@ imageio==2.37.0
     # via scikit-image
 ipykernel==6.29.5
     # via sec-certs (../pyproject.toml)
-ipython==8.32.0
+ipython==8.34.0
     # via
     #   ipykernel
     #   ipywidgets
@@ -186,6 +187,7 @@ jiter==0.8.2
     # via openai
 joblib==1.4.2
     # via
+    #   nltk
     #   pynndescent
     #   scikit-learn
 jsonschema==4.23.0
@@ -233,7 +235,7 @@ markupsafe==3.0.2
     # via
     #   jinja2
     #   mako
-matplotlib==3.10.0
+matplotlib==3.10.1
     # via
     #   catboost
     #   pysankeybeta
@@ -265,7 +267,7 @@ murmurhash==1.0.12
     #   preshed
     #   spacy
     #   thinc
-narwhals==1.28.0
+narwhals==1.29.1
     # via plotly
 nest-asyncio==1.6.0
     # via ipykernel
@@ -274,6 +276,8 @@ networkx==3.4.2
     #   scikit-image
     #   sec-certs (../pyproject.toml)
     #   torch
+nltk==3.9.1
+    # via sec-certs (../pyproject.toml)
 numba==0.61.0
     # via
     #   datashader
@@ -308,41 +312,7 @@ numpy==1.26.4
     #   transformers
     #   umap-learn
     #   xarray
-nvidia-cublas-cu12==12.4.5.8
-    # via
-    #   nvidia-cudnn-cu12
-    #   nvidia-cusolver-cu12
-    #   torch
-nvidia-cuda-cupti-cu12==12.4.127
-    # via torch
-nvidia-cuda-nvrtc-cu12==12.4.127
-    # via torch
-nvidia-cuda-runtime-cu12==12.4.127
-    # via torch
-nvidia-cudnn-cu12==9.1.0.70
-    # via torch
-nvidia-cufft-cu12==11.2.1.3
-    # via torch
-nvidia-curand-cu12==10.3.5.147
-    # via torch
-nvidia-cusolver-cu12==11.6.1.9
-    # via torch
-nvidia-cusparse-cu12==12.3.1.170
-    # via
-    #   nvidia-cusolver-cu12
-    #   torch
-nvidia-cusparselt-cu12==0.6.2
-    # via torch
-nvidia-nccl-cu12==2.21.5
-    # via torch
-nvidia-nvjitlink-cu12==12.4.127
-    # via
-    #   nvidia-cusolver-cu12
-    #   nvidia-cusparse-cu12
-    #   torch
-nvidia-nvtx-cu12==12.4.127
-    # via torch
-openai==1.64.0
+openai==1.65.5
     # via sec-certs (../pyproject.toml)
 optuna==4.2.1
     # via sec-certs (../pyproject.toml)
@@ -459,7 +429,7 @@ pydantic==2.10.6
     #   weasel
 pydantic-core==2.27.2
     # via pydantic
-pydantic-settings==2.8.0
+pydantic-settings==2.8.1
     # via sec-certs (../pyproject.toml)
 pygments==2.19.1
     # via
@@ -469,7 +439,7 @@ pynndescent==0.5.13
     # via umap-learn
 pyparsing==3.2.1
     # via matplotlib
-pypdf[crypto]==5.3.0
+pypdf[crypto]==5.3.1
     # via sec-certs (../pyproject.toml)
 pysankeybeta==1.4.2
     # via sec-certs (../pyproject.toml)
@@ -505,7 +475,7 @@ pyzmq==26.2.1
     # via
     #   ipykernel
     #   jupyter-client
-rapidfuzz==3.12.1
+rapidfuzz==3.12.2
     # via sec-certs (../pyproject.toml)
 referencing==0.36.2
     # via
@@ -514,6 +484,7 @@ referencing==0.36.2
 regex==2024.11.6
     # via
     #   dateparser
+    #   nltk
     #   transformers
 requests==2.32.3
     # via
@@ -561,7 +532,9 @@ seaborn==0.13.2
     #   sec-certs (../pyproject.toml)
     #   umap-learn
 sentence-transformers[train]==3.4.1
-    # via setfit
+    # via
+    #   sec-certs (../pyproject.toml)
+    #   setfit
 setfit==1.1.1
     # via sec-certs (../pyproject.toml)
 setuptools-scm==8.2.0
@@ -591,6 +564,8 @@ sqlalchemy==2.0.38
     # via
     #   alembic
     #   optuna
+sqlite-vec==0.1.6
+    # via sec-certs (../pyproject.toml)
 srsly==2.5.1
     # via
     #   confection
@@ -629,6 +604,7 @@ tqdm==4.67.1
     #   datasets
     #   evaluate
     #   huggingface-hub
+    #   nltk
     #   openai
     #   optuna
     #   panel
@@ -650,9 +626,7 @@ transformers==4.49.0
     # via
     #   sentence-transformers
     #   setfit
-triton==3.2.0
-    # via torch
-typer==0.15.1
+typer==0.15.2
     # via
     #   spacy
     #   weasel
@@ -677,7 +651,7 @@ typing-extensions==4.12.2
     #   typer
 tzdata==2025.1
     # via pandas
-tzlocal==5.3
+tzlocal==5.3.1
     # via dateparser
 uc-micro-py==1.0.3
     # via linkify-it-py
diff --git a/requirements/requirements.txt b/requirements/requirements.txt
index fa7cced3..09ea258d 100644
--- a/requirements/requirements.txt
+++ b/requirements/requirements.txt
@@ -1,5 +1,7 @@
 annotated-types==0.7.0
     # via pydantic
+appnope==0.1.4
+    # via ipykernel
 asttokens==3.0.0
     # via stack-data
 attrs==25.1.0
@@ -25,7 +27,7 @@ click==8.1.8
     # via
     #   sec-certs (../pyproject.toml)
     #   typer
-cloudpathlib==0.20.0
+cloudpathlib==0.21.0
     # via weasel
 comm==0.2.2
     # via
@@ -37,7 +39,7 @@ confection==0.1.5
     #   weasel
 contourpy==1.3.1
     # via matplotlib
-cryptography==44.0.1
+cryptography==44.0.2
     # via pypdf
 cycler==0.12.1
     # via matplotlib
@@ -48,7 +50,7 @@ cymem==2.0.11
     #   thinc
 dateparser==1.2.1
     # via sec-certs (../pyproject.toml)
-debugpy==1.8.12
+debugpy==1.8.13
     # via ipykernel
 decorator==5.2.1
     # via ipython
@@ -68,7 +70,7 @@ idna==3.10
     # via requests
 ipykernel==6.29.5
     # via sec-certs (../pyproject.toml)
-ipython==8.32.0
+ipython==8.34.0
     # via
     #   ipykernel
     #   ipywidgets
@@ -108,7 +110,7 @@ markdown-it-py==3.0.0
     # via rich
 markupsafe==3.0.2
     # via jinja2
-matplotlib==3.10.0
+matplotlib==3.10.1
     # via
     #   pysankeybeta
     #   seaborn
@@ -202,7 +204,7 @@ pydantic==2.10.6
     #   weasel
 pydantic-core==2.27.2
     # via pydantic
-pydantic-settings==2.8.0
+pydantic-settings==2.8.1
     # via sec-certs (../pyproject.toml)
 pygments==2.19.1
     # via
@@ -210,7 +212,7 @@ pygments==2.19.1
     #   rich
 pyparsing==3.2.1
     # via matplotlib
-pypdf[crypto]==5.3.0
+pypdf[crypto]==5.3.1
     # via sec-certs (../pyproject.toml)
 pysankeybeta==1.4.2
     # via sec-certs (../pyproject.toml)
@@ -235,7 +237,7 @@ pyzmq==26.2.1
     # via
     #   ipykernel
     #   jupyter-client
-rapidfuzz==3.12.1
+rapidfuzz==3.12.2
     # via sec-certs (../pyproject.toml)
 referencing==0.36.2
     # via
@@ -315,7 +317,7 @@ traitlets==5.14.3
     #   jupyter-client
     #   jupyter-core
     #   matplotlib-inline
-typer==0.15.1
+typer==0.15.2
     # via
     #   spacy
     #   weasel
@@ -332,7 +334,7 @@ typing-extensions==4.12.2
     #   typer
 tzdata==2025.1
     # via pandas
-tzlocal==5.3
+tzlocal==5.3.1
     # via dateparser
 urllib3==2.3.0
     # via requests
diff --git a/requirements/test_requirements.txt b/requirements/test_requirements.txt
index b9bdc30d..a4ea19d5 100644
--- a/requirements/test_requirements.txt
+++ b/requirements/test_requirements.txt
@@ -1,5 +1,7 @@
 annotated-types==0.7.0
     # via pydantic
+appnope==0.1.4
+    # via ipykernel
 asttokens==3.0.0
     # via stack-data
 attrs==25.1.0
@@ -25,7 +27,7 @@ click==8.1.8
     # via
     #   sec-certs (../pyproject.toml)
     #   typer
-cloudpathlib==0.20.0
+cloudpathlib==0.21.0
     # via weasel
 comm==0.2.2
     # via
@@ -41,7 +43,7 @@ coverage[toml]==7.6.12
     # via
     #   pytest-cov
     #   sec-certs (../pyproject.toml)
-cryptography==44.0.1
+cryptography==44.0.2
     # via pypdf
 cycler==0.12.1
     # via matplotlib
@@ -52,7 +54,7 @@ cymem==2.0.11
     #   thinc
 dateparser==1.2.1
     # via sec-certs (../pyproject.toml)
-debugpy==1.8.12
+debugpy==1.8.13
     # via ipykernel
 decorator==5.2.1
     # via ipython
@@ -76,7 +78,7 @@ iniconfig==2.0.0
     # via pytest
 ipykernel==6.29.5
     # via sec-certs (../pyproject.toml)
-ipython==8.32.0
+ipython==8.34.0
     # via
     #   ipykernel
     #   ipywidgets
@@ -116,7 +118,7 @@ markdown-it-py==3.0.0
     # via rich
 markupsafe==3.0.2
     # via jinja2
-matplotlib==3.10.0
+matplotlib==3.10.1
     # via
     #   pysankeybeta
     #   seaborn
@@ -213,7 +215,7 @@ pydantic==2.10.6
     #   weasel
 pydantic-core==2.27.2
     # via pydantic
-pydantic-settings==2.8.0
+pydantic-settings==2.8.1
     # via sec-certs (../pyproject.toml)
 pygments==2.19.1
     # via
@@ -221,13 +223,13 @@ pygments==2.19.1
     #   rich
 pyparsing==3.2.1
     # via matplotlib
-pypdf[crypto]==5.3.0
+pypdf[crypto]==5.3.1
     # via sec-certs (../pyproject.toml)
 pysankeybeta==1.4.2
     # via sec-certs (../pyproject.toml)
 pytesseract==0.3.13
     # via sec-certs (../pyproject.toml)
-pytest==8.3.4
+pytest==8.3.5
     # via
     #   pytest-cov
     #   sec-certs (../pyproject.toml)
@@ -252,7 +254,7 @@ pyzmq==26.2.1
     # via
     #   ipykernel
     #   jupyter-client
-rapidfuzz==3.12.1
+rapidfuzz==3.12.2
     # via sec-certs (../pyproject.toml)
 referencing==0.36.2
     # via
@@ -335,7 +337,7 @@ traitlets==5.14.3
     #   jupyter-client
     #   jupyter-core
     #   matplotlib-inline
-typer==0.15.1
+typer==0.15.2
     # via
     #   spacy
     #   weasel
@@ -352,7 +354,7 @@ typing-extensions==4.12.2
     #   typer
 tzdata==2025.1
     # via pandas
-tzlocal==5.3
+tzlocal==5.3.1
     # via dateparser
 urllib3==2.3.0
     # via requests