AEGIS-DocIntel

Status: Alpha / Experimental —

An experimental document-structuring research platform for pre-LLM context compilation, with one component validated and decoupled for immediate production usage:

Three real, independently-tested document-processing primitives — PII redaction, submodular context packing, and spatial reading-order extraction — extracted from this platform because they are the pieces with the strongest real-world validation (see real benchmark results from Phase 9 testing, not marketing copy):

# Not yet published to PyPI — install from source:
git clone https://github.com/SahilKhutey/AEGIS-DocIntel.git
pip install ./AEGIS-DocIntel/aegis-docprep

• Dependencies: Minimal (numpy>=1.26.0 only). Zero deep learning or GPU dependencies required.
• PII Detection & Redaction: Regex and tokenization-based detection of US SSNs, Luhn-validated credit-card numbers, phone numbers (US, international E.164, local 7-digit), email addresses, API keys/tokens, and Title-Case person names. Treat it as a first-pass filter with human review, not a compliance guarantee — see limitations.
• Submodular Context Packing: Greedy knapsack optimization with provable $(1 - 1/e)/2 \approx 0.316$ information coverage guarantee under strict LLM token ceilings.
• Spatial Reading-Order Recovery: Directed topological DAG sorting over 2D bounding boxes resolving multi-column layout interleaving without heavy neural visual models.
• Framework Integrations: Ready-to-use LangChain Document Transformers and LlamaIndex Node Postprocessors.

See the aegis-docprep package documentation and docs/docprep.md for quickstart guides.

A 16-mathematical-domain research system for pre-LLM document structuring. Real, substantial mathematical implementations exist for most core domains (see the honest layer-by-layer status, sourced from the project's own internal Extended Monograph's Appendix E). Several domains remain research targets rather than production features — see STATUS.md for exactly which is which.

The platform centers on the formal 10-tuple Master State:

$$\mathcal{D} = (P, S, G, R, F, M, T, X, H, E)$$

implemented as a strictly-typed immutable schema in src/core/master_state.py.

  Raw Document (PDF, DOCX, Scans)
               │
               ▼
  ┌─────────────────────────────────────────────────────────────┐
  │              AEGIS Master State Extraction Pipeline         │
  │                                                             │
  │  1. Spatial Bounding Boxes (P)  ──► PyMuPDF extraction      │
  │  2. Reading-Order DAG (G)       ──► Topological sort        │
  │  3. Table Matrices (T)          ──► PDFPlumber bbox mapping │
  │  4. PII Redaction Layer (R)     ──► Deterministic masks     │
  │  5. Submodular Chunks (E)       ──► Knapsack budget solver  │
  └─────────────────────────────────────────────────────────────┘
               │
               ▼
  Master State D = (P, S, G, R, F, M, T, X, H, E)
               │
               ▼
  Token-Bounded, Non-Redundant Context Injection for LLMs

No external pilots have been completed yet. Earlier revisions of this README described pilot case studies with specific metrics; they could not be corroborated (the repository has no pilot issues) and were removed — see STATUS.md. Pilot results will be published here only when each one links to a public GitHub issue.

Want to try it and tell us what breaks? Start with aegis-docprep, then open an issue using the pilot feedback template. Invitation draft: docs/pilot/recruitment_announcement.md.

Corpus: 62 real-world PDFs (provenance). Reproduce with python scripts/run_benchmark.py; raw per-document results are in production/performance-report/.

AEGIS-DocIntel includes a versioned evaluation framework in evaluation/ testing document extraction fidelity, context packing invariants, PII redaction precision/recall, and retrieval grounding:

Run the unified evaluation suite locally:

python evaluation/scripts/run_all.py --fail-under-gates

Per the project's own internal Appendix E status matrix (Phase 5 finding):

• Hardened Today: Reading-order graph, elastic chunker, knapsack solvers, spectral embedder, tabular compiler, PII redaction, submodular context packing — these are real, tested, and what aegis-docprep packages.
• Real but Partially Mock: Semantic (LayoutLM-class) encoding — works when sentence-transformers is installed, falls back to clearly-flagged deterministic feature vectors otherwise (MasterState.semantic_status = 'mock_fallback').
• Research Targets, Not Yet Production Features: Persistent-homology-based hierarchy (topology's $H$ layer), full distributed production multi-cluster topology, end-to-end benchmark evaluation at scale. These are documented as targets because they are active research areas — not because we claim they already work in production.

Follow the tiered installation structure to avoid downloading unnecessary dependencies:

Only needs numpy. Not yet on PyPI, so install from a clone of this repo:

git clone https://github.com/SahilKhutey/AEGIS-DocIntel.git
cd AEGIS-DocIntel
pip install ./aegis-docprep

git clone https://github.com/SahilKhutey/AEGIS-DocIntel.git
cd AEGIS-DocIntel
python -m venv venv
# Linux / macOS:
source venv/bin/activate
# Windows:
venv\Scripts\activate

# Install core runtime
pip install -r requirements-core.txt

These exact commands are run on a fresh virtual environment by CI on every push (scripts/cold_start_check.sh), so this path cannot silently break:

pip install -r requirements-core.txt -r requirements-dev.txt
pytest tests/

Supported Python: 3.12 (canonical MVP runtime, validated in CI PR gate) and 3.13 (experimental / compatibility sweep in scheduled maintenance). Dependencies are defined once in pyproject.toml; the requirements-*.txt files are generated from it (python scripts/check_dependency_sync.py --write).

OCR tests additionally use pytesseract; real OCR needs the tesseract system binary.

• Deep Learning / Embeddings: pip install -r requirements-ml.txt (or pip install -e ".[ml]")
• Interactive Web UI Dashboard: pip install -r requirements-ui.txt (or pip install -e ".[ui]")

streamlit run dashboard/app.py

Full documentation is available via Material for MkDocs:

# Build and preview local documentation
pip install mkdocs mkdocs-material
mkdocs serve

Browse the documentation sections:

• Status & 16-Phase Journey
• aegis-docprep Guide
• Architecture & Master State
• Tenant Isolation & Multi-Tenancy
• Data Governance & Lifecycle Policies
• Pluggable Adapter Contracts
• Enterprise Security Threat Model
• Phase P4 Execution Report
• Deployment & Operational Runbooks
• Performance & Benchmark Plan
• Installation Guide
• REST API Reference
• Compliance Gap Analysis
• Security Policy & Audit
• Support Policy
• Contributing Guidelines

• aegis-docprep: Licensed under the permissive Apache License 2.0.
• Full AEGIS-DocIntel Platform: Available under evaluation and research terms per LICENSE.