Initial commit — phase 1 + 2 of artenschutz-digest concept
Phase 1: router skeleton + Qdrant read path - FastAPI app with /health, /search/generic, /ingest endpoints - Qdrant reader (hybrid dense+sparse w/ RRF, plus payload scroll) - BGE-M3 TEI client for query-side embedding - memory-api client proxying writes to /memory/store (m2-memory untouched) - Pydantic Fact + Exemplar payloads, discriminated by `kind` - Dockerfile + docker-compose joining the coolify Docker network Phase 2: authoritative-source ingest pipeline - PDF text extraction + paragraph-aware (§ N) and size chunkers - Loaders for bnatschg, mhbasp, biotopwertliste, state-<slug> - CLI: artenschutz-ingest --source <name> [--states ...] [--dry-run] - Helper script to fetch BNatSchG from gesetze-im-internet.de - End-to-end smoke script (scripts/smoke.sh) for Coolify validation 22/22 unit tests pass. Real-world dry-runs verified against mhbasp Anhang 4, biotopwertlisteNEU and Berlin Kartierstandards PDFs from GST-DATA. See COOLIFY-DEPLOY.md for staging deploy + smoke procedure.
This commit is contained in:
commit
6da874b2e6
46 changed files with 1969 additions and 0 deletions
22
.env.example
Normal file
22
.env.example
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
# Qdrant + BGE-M3 — same services agent.memory.system uses.
|
||||
# In the Coolify Docker network these resolve by container name.
|
||||
# For local dev pointed at a remote stack, use container IPs from
|
||||
# docker inspect or memory-api's resolved endpoints.
|
||||
QDRANT_URL=http://memory-qdrant:6333
|
||||
QDRANT_COLLECTION=agent_memory
|
||||
BGE_TEI_URL=http://memory-embeddings:8000
|
||||
|
||||
# memory-api is only used for the WRITE path (/ingest → /memory/store).
|
||||
# Reads go directly to Qdrant.
|
||||
MEMORY_API_URL=http://memory-api:8000
|
||||
|
||||
# agent_id partition in Qdrant. Default scopes by client; override per deploy.
|
||||
# Examples:
|
||||
# gruenstifter — production client
|
||||
# gruenstifter_staging — same client, staging env (cleanly deletable)
|
||||
ARTENSCHUTZ_AGENT_ID=gruenstifter
|
||||
|
||||
# Service binding
|
||||
ROUTER_HOST=0.0.0.0
|
||||
ROUTER_PORT=8080
|
||||
LOG_LEVEL=INFO
|
||||
19
.gitignore
vendored
Normal file
19
.gitignore
vendored
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
.pytest_cache/
|
||||
.ruff_cache/
|
||||
.venv/
|
||||
venv/
|
||||
.env
|
||||
*.egg-info/
|
||||
build/
|
||||
dist/
|
||||
|
||||
# Local source dumps (real authoritative sources are large; keep .gitkeep markers)
|
||||
sources/**/*.pdf
|
||||
sources/**/*.docx
|
||||
sources/**/*.txt
|
||||
sources/**/*.html
|
||||
!sources/**/.gitkeep
|
||||
!sources/**/README.md
|
||||
128
COOLIFY-DEPLOY.md
Normal file
128
COOLIFY-DEPLOY.md
Normal file
|
|
@ -0,0 +1,128 @@
|
|||
# Coolify staging deploy — artenschutz-router
|
||||
|
||||
Goal: stand up the router on Coolify staging, in the same Docker network as
|
||||
`agent.memory.system`, so it can reach `memory-qdrant`, `memory-embeddings`,
|
||||
and `memory-api` by container name. Then run an end-to-end smoke ingest +
|
||||
search against real data.
|
||||
|
||||
## Pre-flight
|
||||
|
||||
You should already have on staging Coolify:
|
||||
|
||||
- A running `agent.memory.system` (memory-api + memory-qdrant + memory-embeddings)
|
||||
- The Docker network name those services live on (probably `coolify` — check
|
||||
`docker network ls` on the staging host)
|
||||
|
||||
Note down:
|
||||
|
||||
| Setting | Value (example) |
|
||||
|---|---|
|
||||
| Coolify project | `gruenstifter-staging` |
|
||||
| Docker network | `coolify` |
|
||||
| memory-api container name | `memory-api-<id>` |
|
||||
| memory-qdrant container name | `memory-qdrant-<id>` |
|
||||
| memory-embeddings container name | `memory-embeddings-<id>` |
|
||||
|
||||
> The *full* container names include Coolify's stack hash suffix. The
|
||||
> compose file uses the short service names (`memory-api`, etc.) which work
|
||||
> when the router is on the same Coolify-managed network — Coolify creates
|
||||
> service-name aliases. If they don't resolve, fall back to container IPs
|
||||
> via `docker inspect`.
|
||||
|
||||
## Path A — Push to forgejo, let Coolify auto-deploy (preferred)
|
||||
|
||||
```bash
|
||||
cd /home/m2/artenschutz-router
|
||||
git init -b main
|
||||
git add .
|
||||
git commit -m "Initial commit — phase 1 + 2"
|
||||
git remote add origin <forgejo-url>:gruenstifter/artenschutz-router.git
|
||||
git push -u origin main
|
||||
```
|
||||
|
||||
In Coolify staging:
|
||||
|
||||
1. New Resource → **Application** → **Public Git Repository** (or Private,
|
||||
add the SSH key)
|
||||
2. Source: the forgejo URL above
|
||||
3. Build Pack: **Dockerfile**
|
||||
4. Set environment variables:
|
||||
```
|
||||
QDRANT_URL=http://memory-qdrant:6333
|
||||
QDRANT_COLLECTION=agent_memory
|
||||
BGE_TEI_URL=http://memory-embeddings:8000
|
||||
MEMORY_API_URL=http://memory-api:8000
|
||||
ARTENSCHUTZ_AGENT_ID=gruenstifter_staging
|
||||
ROUTER_PORT=8080
|
||||
LOG_LEVEL=INFO
|
||||
```
|
||||
`gruenstifter_staging` keeps smoke data cleanly separable from any future
|
||||
production write.
|
||||
5. Network: join the same network as `agent.memory.system` (usually `coolify`).
|
||||
6. Port: 8080 (no public exposure needed for smoke — internal-only is fine).
|
||||
7. Deploy.
|
||||
|
||||
## Path B — Docker Compose on the Coolify host
|
||||
|
||||
If you'd rather not push to forgejo yet:
|
||||
|
||||
```bash
|
||||
# On the Coolify host (or via SSH):
|
||||
scp -r /home/m2/artenschutz-router coolify-host:/opt/coolify-apps/
|
||||
ssh coolify-host
|
||||
cd /opt/coolify-apps/artenschutz-router
|
||||
cp .env.example .env
|
||||
# Edit .env to set ARTENSCHUTZ_AGENT_ID=gruenstifter_staging
|
||||
docker compose build
|
||||
docker compose up -d
|
||||
docker compose logs -f artenschutz-router
|
||||
```
|
||||
|
||||
This bypasses Coolify's UI but uses the same `coolify` external network the
|
||||
memory stack joins.
|
||||
|
||||
## Verify it's up
|
||||
|
||||
From any shell on the Coolify host:
|
||||
|
||||
```bash
|
||||
docker exec artenschutz-router curl -sS http://localhost:8080/health
|
||||
# expect: {"status":"ok","qdrant":true,"bge":true,"memory_api":true}
|
||||
```
|
||||
|
||||
`status: "degraded"` with one of the three `false` means that service isn't
|
||||
reachable from inside the router's network namespace — fix container names /
|
||||
network attachment before continuing.
|
||||
|
||||
## End-to-end smoke (uses the test data already symlinked in `sources/`)
|
||||
|
||||
```bash
|
||||
# 1. Fetch BNatSchG into the container's sources/bnatschg/ if not already.
|
||||
docker exec artenschutz-router python scripts/fetch_bnatschg.py
|
||||
|
||||
# 2. Run the smoke (ingests small batches + queries them back).
|
||||
docker exec artenschutz-router bash scripts/smoke.sh
|
||||
```
|
||||
|
||||
The smoke script ingests ~5 records per source, then runs three searches:
|
||||
|
||||
- Hybrid query `"Fledermäuse Kartierung"` filtered to `fact_type=method`
|
||||
- Hybrid query `"besonders geschützte Arten"` filtered to `fact_type=legal`
|
||||
- Payload-only scroll on `scope.states=BE` to verify state filtering
|
||||
|
||||
Expected: each search returns ≥1 hit. If hybrid returns 0 hits but scroll
|
||||
works, BGE-M3 embedding is failing — check the logs.
|
||||
|
||||
## Cleanup (after smoke)
|
||||
|
||||
Drop the staging agent_id from Qdrant:
|
||||
|
||||
```bash
|
||||
docker exec memory-qdrant-<id> curl -X POST \
|
||||
http://localhost:6333/collections/agent_memory/points/delete \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"filter":{"must":[{"key":"agent_id","match":{"value":"gruenstifter_staging"}}]}}'
|
||||
```
|
||||
|
||||
This wipes every point we wrote under the staging agent_id. Nothing else is
|
||||
affected because partition isolation is by `agent_id`.
|
||||
25
Dockerfile
Normal file
25
Dockerfile
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
FROM python:3.11-slim
|
||||
|
||||
ENV PYTHONUNBUFFERED=1 \
|
||||
PYTHONDONTWRITEBYTECODE=1 \
|
||||
PIP_NO_CACHE_DIR=1
|
||||
|
||||
# curl: for smoke.sh + /health probes.
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY pyproject.toml ./
|
||||
COPY src ./src
|
||||
COPY scripts ./scripts
|
||||
COPY sources ./sources
|
||||
|
||||
RUN pip install --upgrade pip && pip install .
|
||||
|
||||
# sources/ is bind-mountable so an operator can drop files (esp. bnatschg.txt)
|
||||
# without a rebuild.
|
||||
VOLUME ["/app/sources"]
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
CMD ["uvicorn", "artenschutz_router.main:app", "--host", "0.0.0.0", "--port", "8080"]
|
||||
67
README.md
Normal file
67
README.md
Normal file
|
|
@ -0,0 +1,67 @@
|
|||
# artenschutz-router
|
||||
|
||||
Sidecar service for Artenschutzgutachten domain queries against
|
||||
`agent.memory.system`. **m2-memory stays untouched.**
|
||||
|
||||
- **Writes** proxy memory-api `/memory/store` — the existing endpoint accepts
|
||||
arbitrary `metadata: dict`, so our Fact / Exemplar payloads slot in.
|
||||
- **Reads** go directly to Qdrant for rich payload-filtered hybrid queries
|
||||
(dense + sparse via the existing BGE-M3 TEI service).
|
||||
|
||||
See `.planning/notes/artenschutz-digest-concept.md` in `GST-DATA` for the
|
||||
draft concept that birthed this.
|
||||
|
||||
## Endpoints
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|---|---|---|
|
||||
| `GET` | `/health` | Live-probe Qdrant, BGE-M3, memory-api |
|
||||
| `POST` | `/search/generic` | Hybrid search (or scroll if no `query`); arbitrary payload filters |
|
||||
| `POST` | `/ingest` | Push a Fact or Exemplar record |
|
||||
|
||||
Domain-specific endpoints (`/facts/search`, `/exemplar/search`,
|
||||
`/mitigation/search`) are deferred to a later phase — they will be thin
|
||||
wrappers over `/search/generic`.
|
||||
|
||||
## Quick start (local dev)
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# Edit .env to point QDRANT_URL / BGE_TEI_URL / MEMORY_API_URL at your stack
|
||||
docker compose up --build
|
||||
curl http://localhost:8080/health
|
||||
```
|
||||
|
||||
If you're targeting the production Coolify memory stack, leave the default
|
||||
container-name URLs and ensure the `coolify` Docker network is joined.
|
||||
|
||||
## Running ingest
|
||||
|
||||
```bash
|
||||
# After you've placed authoritative source files into sources/...
|
||||
artenschutz-ingest --source bnatschg --router-url http://localhost:8080
|
||||
artenschutz-ingest --source mhbasp --router-url http://localhost:8080
|
||||
artenschutz-ingest --source biotopwertliste --router-url http://localhost:8080
|
||||
artenschutz-ingest --source state-berlin --router-url http://localhost:8080
|
||||
```
|
||||
|
||||
See `sources/README.md` for which files each source expects.
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
pip install -e .[dev]
|
||||
pytest
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
m2-gpt agents
|
||||
↓
|
||||
artenschutz-router (this repo)
|
||||
├─ /ingest ─→ memory-api /memory/store (unchanged)
|
||||
└─ /search/generic ─→ Qdrant /points/query + BGE-M3 TEI /embed[_sparse]
|
||||
↓
|
||||
agent_memory collection (Qdrant)
|
||||
```
|
||||
24
docker-compose.yml
Normal file
24
docker-compose.yml
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
# Local dev compose. Joins the existing Coolify "coolify" network so it can
|
||||
# resolve memory-qdrant / memory-embeddings / memory-api by container name.
|
||||
# If those services aren't reachable, override the *_URL env vars in .env.
|
||||
|
||||
services:
|
||||
artenschutz-router:
|
||||
build: .
|
||||
container_name: artenschutz-router
|
||||
env_file:
|
||||
- .env
|
||||
ports:
|
||||
- "8080:8080"
|
||||
networks:
|
||||
- coolify
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/health')"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
networks:
|
||||
coolify:
|
||||
external: true
|
||||
42
pyproject.toml
Normal file
42
pyproject.toml
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
[project]
|
||||
name = "artenschutz-router"
|
||||
version = "0.1.0"
|
||||
description = "Sidecar router for Artenschutz domain queries against agent.memory.system"
|
||||
requires-python = ">=3.11"
|
||||
readme = "README.md"
|
||||
|
||||
dependencies = [
|
||||
"fastapi>=0.110",
|
||||
"uvicorn[standard]>=0.27",
|
||||
"pydantic>=2.6",
|
||||
"pydantic-settings>=2.2",
|
||||
"httpx>=0.27",
|
||||
"qdrant-client>=1.9",
|
||||
"pypdf>=4.0",
|
||||
"python-dateutil>=2.9",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=8.0",
|
||||
"pytest-asyncio>=0.23",
|
||||
"ruff>=0.4",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
artenschutz-ingest = "artenschutz_router.ingest.cli:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["setuptools>=68"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
asyncio_mode = "auto"
|
||||
testpaths = ["tests"]
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 100
|
||||
target-version = "py311"
|
||||
112
scripts/fetch_bnatschg.py
Executable file
112
scripts/fetch_bnatschg.py
Executable file
|
|
@ -0,0 +1,112 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Fetch the consolidated text of BNatSchG §44 and §45b from
|
||||
gesetze-im-internet.de and emit a plain-text file suitable for the
|
||||
bnatschg ingest loader.
|
||||
|
||||
Usage:
|
||||
python scripts/fetch_bnatschg.py [--out sources/bnatschg/bnatschg.txt]
|
||||
|
||||
The output uses one heading line per section ("§ N ..." or "Anlage N ...")
|
||||
exactly as the bnatschg loader's paragraph splitter expects.
|
||||
|
||||
Best-effort HTML scrape. If gesetze-im-internet.de changes layout, paste
|
||||
text manually instead (see sources/README.md).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import sys
|
||||
from html.parser import HTMLParser
|
||||
from pathlib import Path
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
# §s most relevant for Artenschutzgutachten. Add more as needed.
|
||||
SECTIONS = [
|
||||
("§ 44 Vorschriften für besonders geschützte und bestimmte andere Tier- und Pflanzenarten",
|
||||
"https://www.gesetze-im-internet.de/bnatschg_2009/__44.html"),
|
||||
("§ 45 Ausnahmen; Ermächtigung zum Erlass von Rechtsverordnungen",
|
||||
"https://www.gesetze-im-internet.de/bnatschg_2009/__45.html"),
|
||||
("§ 45b Schutz wild lebender Vogelarten bei der Errichtung und dem Betrieb von Windenergieanlagen an Land",
|
||||
"https://www.gesetze-im-internet.de/bnatschg_2009/__45b.html"),
|
||||
("Anlage 1 (zu § 45b Absatz 1 bis 5) Liste der bei der Errichtung und dem Betrieb von Windenergieanlagen an Land kollisionsgefährdeten Brutvogelarten",
|
||||
"https://www.gesetze-im-internet.de/bnatschg_2009/anlage_1.html"),
|
||||
]
|
||||
|
||||
|
||||
class TextExtractor(HTMLParser):
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self._buf: list[str] = []
|
||||
self._skip = 0 # depth of <script>/<style>
|
||||
|
||||
def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
|
||||
if tag in {"script", "style"}:
|
||||
self._skip += 1
|
||||
elif tag in {"br", "p", "div", "li", "tr"}:
|
||||
self._buf.append("\n")
|
||||
|
||||
def handle_endtag(self, tag: str) -> None:
|
||||
if tag in {"script", "style"} and self._skip:
|
||||
self._skip -= 1
|
||||
elif tag in {"p", "div", "li", "tr"}:
|
||||
self._buf.append("\n")
|
||||
|
||||
def handle_data(self, data: str) -> None:
|
||||
if not self._skip:
|
||||
self._buf.append(data)
|
||||
|
||||
def value(self) -> str:
|
||||
raw = "".join(self._buf)
|
||||
# Collapse excess blank lines while preserving paragraph breaks.
|
||||
raw = re.sub(r"[ \t]+", " ", raw)
|
||||
raw = re.sub(r"\n{3,}", "\n\n", raw)
|
||||
return raw.strip()
|
||||
|
||||
|
||||
def fetch(url: str, timeout: float = 30.0) -> str:
|
||||
req = Request(url, headers={"User-Agent": "artenschutz-router/0.1 (+fetch_bnatschg)"})
|
||||
with urlopen(req, timeout=timeout) as resp:
|
||||
raw_bytes = resp.read()
|
||||
# gesetze-im-internet.de serves ISO-8859-1; sniff from header if present.
|
||||
charset = resp.headers.get_content_charset() or "iso-8859-1"
|
||||
return raw_bytes.decode(charset, errors="replace")
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(prog="fetch_bnatschg")
|
||||
parser.add_argument(
|
||||
"--out",
|
||||
type=Path,
|
||||
default=Path("sources/bnatschg/bnatschg.txt"),
|
||||
help="Output path (default: %(default)s)",
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
chunks: list[str] = []
|
||||
for heading, url in SECTIONS:
|
||||
print(f" fetching {heading.split('(')[0].strip()} ...", file=sys.stderr)
|
||||
try:
|
||||
html = fetch(url)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(f" FAILED ({exc}) — skipping", file=sys.stderr)
|
||||
continue
|
||||
extractor = TextExtractor()
|
||||
extractor.feed(html)
|
||||
body = extractor.value()
|
||||
# Anchor each section with the expected heading on its own line so the
|
||||
# paragraph chunker recognises it.
|
||||
chunks.append(f"{heading}\n\n{body}\n")
|
||||
|
||||
if not chunks:
|
||||
print("No sections fetched — refusing to write empty output", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
args.out.write_text("\n\n".join(chunks), encoding="utf-8")
|
||||
print(f"Wrote {args.out} ({sum(len(c) for c in chunks):,} chars)", file=sys.stderr)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
72
scripts/smoke.sh
Executable file
72
scripts/smoke.sh
Executable file
|
|
@ -0,0 +1,72 @@
|
|||
#!/usr/bin/env bash
|
||||
# End-to-end smoke for artenschutz-router against the live memory stack.
|
||||
# Runs from inside the container; ROUTER_URL defaults to localhost:8080.
|
||||
set -euo pipefail
|
||||
|
||||
ROUTER_URL="${ROUTER_URL:-http://localhost:8080}"
|
||||
|
||||
step() { printf '\n\033[1;36m== %s ==\033[0m\n' "$*"; }
|
||||
pass() { printf '\033[1;32m ✓ %s\033[0m\n' "$*"; }
|
||||
fail() { printf '\033[1;31m ✗ %s\033[0m\n' "$*"; exit 1; }
|
||||
|
||||
step "1/5 health"
|
||||
health=$(curl -sS "$ROUTER_URL/health")
|
||||
echo " $health"
|
||||
echo "$health" | grep -q '"status":"ok"' || fail "health is not ok — fix services first"
|
||||
pass "router can reach Qdrant + BGE-M3 + memory-api"
|
||||
|
||||
ingest_source() {
|
||||
local source="$1" ; shift
|
||||
local limit="${1:-5}"
|
||||
step "ingest --source $source --limit $limit"
|
||||
python -m artenschutz_router.ingest.cli \
|
||||
--source "$source" \
|
||||
--router-url "$ROUTER_URL" \
|
||||
--limit "$limit" "$@" \
|
||||
2>&1 | tail -5
|
||||
}
|
||||
|
||||
step "2/5 fetch BNatSchG if missing"
|
||||
if [ ! -s sources/bnatschg/bnatschg.txt ]; then
|
||||
python scripts/fetch_bnatschg.py
|
||||
fi
|
||||
|
||||
step "3/5 ingest small batches"
|
||||
ingest_source bnatschg 5
|
||||
ingest_source mhbasp 5
|
||||
ingest_source biotopwertliste 3 --states BY
|
||||
ingest_source state-berlin 5
|
||||
|
||||
step "4/5 search — hybrid, fact_type=method"
|
||||
result=$(curl -sS -X POST "$ROUTER_URL/search/generic" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"query":"Fledermäuse Kartierung","metadata_filter":{"fact_type":"method"},"limit":3}')
|
||||
hits=$(echo "$result" | python -c "import sys,json; print(len(json.load(sys.stdin)['hits']))")
|
||||
echo " hits: $hits"
|
||||
[ "$hits" -ge 1 ] || fail "hybrid search returned 0 hits — check BGE-M3"
|
||||
pass "hybrid retrieval works"
|
||||
|
||||
step "4b/5 search — hybrid, fact_type=legal"
|
||||
result=$(curl -sS -X POST "$ROUTER_URL/search/generic" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"query":"besonders geschützte Arten","metadata_filter":{"fact_type":"legal"},"limit":3}')
|
||||
hits=$(echo "$result" | python -c "import sys,json; print(len(json.load(sys.stdin)['hits']))")
|
||||
echo " hits: $hits"
|
||||
[ "$hits" -ge 1 ] || fail "legal hybrid search returned 0 hits"
|
||||
pass "BNatSchG indexed and queryable"
|
||||
|
||||
step "5/5 scroll — scope.states contains BE"
|
||||
result=$(curl -sS -X POST "$ROUTER_URL/search/generic" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"metadata_filter":{"scope.states":["BE"]},"limit":3}')
|
||||
hits=$(echo "$result" | python -c "import sys,json; print(len(json.load(sys.stdin)['hits']))")
|
||||
echo " hits: $hits"
|
||||
[ "$hits" -ge 1 ] || fail "scope.states=BE scroll returned 0 hits — Berlin ingest missed?"
|
||||
pass "payload filtering on nested keys works"
|
||||
|
||||
printf '\n\033[1;32mALL SMOKE STEPS PASSED\033[0m\n'
|
||||
echo
|
||||
echo "To clean up the staging data later:"
|
||||
echo " curl -X POST <qdrant>:6333/collections/agent_memory/points/delete \\"
|
||||
echo " -H 'Content-Type: application/json' \\"
|
||||
echo " -d '{\"filter\":{\"must\":[{\"key\":\"agent_id\",\"match\":{\"value\":\"gruenstifter_staging\"}}]}}'"
|
||||
22
sources/README.md
Normal file
22
sources/README.md
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
# Authoritative source files
|
||||
|
||||
The ingest pipeline reads from this tree. Files are git-ignored (large + may
|
||||
have licensing constraints) — keep originals here on each operator's machine.
|
||||
|
||||
| Subdir | Expected files | Notes |
|
||||
|---|---|---|
|
||||
| `bnatschg/` | `bnatschg.txt` (current consolidated text of §44, §45b, Anlagen) | Fetch from <https://www.gesetze-im-internet.de/bnatschg_2009/> — paste as UTF-8 plain text. One `§ N ...` per heading line. Cross-reference: a `.webloc` link to <https://www.buzer.de/Anlage_1_BNatSchG.htm> already lives in `GST-DATA/xx_GS_VORLAGEN_FachGA/`. |
|
||||
| `mhbasp/` | `mhbasp_anhang4.pdf` | "Methodische Hinweise und Behandlungsempfehlungen ASP" Anhang 4 — species × method matrix |
|
||||
| `biotopwertliste/` | `biotopwertliste.pdf` | A biotope value list — **always state-specific**. Pass `--states BE` (or BB, BY, …) when ingesting. State-specific lists are also welcome under `states/<slug>/`; the generic slot here is for one-off cases. |
|
||||
| `states/berlin/` | One or more state-specific PDFs (Kartierstandards, methodische Hinweise) | First state implemented as proof-of-concept |
|
||||
| `states/brandenburg/`<br>`states/bayern/`<br>`states/baden_wuerttemberg/`<br>`states/sachsen/` | TODO | Add as ingest support is implemented per state |
|
||||
|
||||
A copy of `mhbasp_anhang4_artspezifisch geeignete kartiermethoden.pdf` already
|
||||
lives under `GST-DATA/xx_GS_VORLAGEN_FachGA/`. Symlinking is fine:
|
||||
|
||||
```bash
|
||||
ln -s "/home/m2/clients/SDJS/GST-DATA/xx_GS_VORLAGEN_FachGA/mhbasp_anhang4_artspezifisch geeignete kartiermethoden.pdf" \
|
||||
sources/mhbasp/mhbasp_anhang4.pdf
|
||||
ln -s "/home/m2/clients/SDJS/GST-DATA/xx_GS_VORLAGEN_FachGA/biotopwertlisteNEU.pdf" \
|
||||
sources/biotopwertliste/biotopwertliste.pdf
|
||||
```
|
||||
0
sources/biotopwertliste/.gitkeep
Normal file
0
sources/biotopwertliste/.gitkeep
Normal file
0
sources/bnatschg/.gitkeep
Normal file
0
sources/bnatschg/.gitkeep
Normal file
0
sources/mhbasp/.gitkeep
Normal file
0
sources/mhbasp/.gitkeep
Normal file
0
sources/states/baden_wuerttemberg/.gitkeep
Normal file
0
sources/states/baden_wuerttemberg/.gitkeep
Normal file
0
sources/states/bayern/.gitkeep
Normal file
0
sources/states/bayern/.gitkeep
Normal file
0
sources/states/berlin/.gitkeep
Normal file
0
sources/states/berlin/.gitkeep
Normal file
0
sources/states/brandenburg/.gitkeep
Normal file
0
sources/states/brandenburg/.gitkeep
Normal file
0
sources/states/sachsen/.gitkeep
Normal file
0
sources/states/sachsen/.gitkeep
Normal file
3
src/artenschutz_router/__init__.py
Normal file
3
src/artenschutz_router/__init__.py
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
"""Artenschutz-Router — sidecar service for Artenschutz domain queries."""
|
||||
|
||||
__version__ = "0.1.0"
|
||||
1
src/artenschutz_router/clients/__init__.py
Normal file
1
src/artenschutz_router/clients/__init__.py
Normal file
|
|
@ -0,0 +1 @@
|
|||
"""External service clients: Qdrant (direct), BGE-M3 TEI, memory-api."""
|
||||
74
src/artenschutz_router/clients/bge.py
Normal file
74
src/artenschutz_router/clients/bge.py
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class BGEClient:
|
||||
"""Thin async client for the BGE-M3 TEI server.
|
||||
|
||||
Only exposes what the router needs for query-side embedding: dense vector
|
||||
and sparse vector. ColBERT is left to memory-api's own pipeline if ever
|
||||
required.
|
||||
"""
|
||||
|
||||
def __init__(self, base_url: str, timeout_s: float = 30.0) -> None:
|
||||
self._base_url = base_url.rstrip("/")
|
||||
self._client = httpx.AsyncClient(base_url=self._base_url, timeout=timeout_s)
|
||||
|
||||
async def close(self) -> None:
|
||||
await self._client.aclose()
|
||||
|
||||
async def embed_dense(self, text: str) -> list[float]:
|
||||
resp = await self._client.post("/embed", json={"inputs": text})
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
# TEI returns list[list[float]] for batched, list[float] for single.
|
||||
if isinstance(data, list) and data and isinstance(data[0], list):
|
||||
return data[0]
|
||||
return data # type: ignore[return-value]
|
||||
|
||||
async def embed_sparse(self, text: str) -> dict[int, float]:
|
||||
resp = await self._client.post("/embed_sparse", json={"inputs": text})
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
# TEI returns either {"sparse": [{"index": int, "value": float}, ...]}
|
||||
# or directly a list of those, depending on version.
|
||||
sparse_list = data[0] if isinstance(data, list) and data and isinstance(data[0], list) else data
|
||||
out: dict[int, float] = {}
|
||||
for item in sparse_list:
|
||||
if isinstance(item, dict) and "index" in item and "value" in item:
|
||||
out[int(item["index"])] = float(item["value"])
|
||||
return out
|
||||
|
||||
async def embed_hybrid(self, text: str) -> tuple[list[float], dict[int, float]]:
|
||||
dense = await self.embed_dense(text)
|
||||
sparse = await self.embed_sparse(text)
|
||||
return dense, sparse
|
||||
|
||||
|
||||
def build_qdrant_filter(metadata_filter: dict[str, Any] | None) -> dict[str, Any] | None:
|
||||
"""Translate a flat `{key: value | [values]}` dict into a Qdrant `must` filter.
|
||||
|
||||
None values are skipped. List values become `match: {any: [...]}` clauses.
|
||||
Scalar values become `match: {value: x}` clauses.
|
||||
Supports nested keys via dotted form, e.g. `scope.states`.
|
||||
"""
|
||||
if not metadata_filter:
|
||||
return None
|
||||
|
||||
must: list[dict[str, Any]] = []
|
||||
for key, value in metadata_filter.items():
|
||||
if value is None:
|
||||
continue
|
||||
if isinstance(value, list):
|
||||
must.append({"key": key, "match": {"any": value}})
|
||||
else:
|
||||
must.append({"key": key, "match": {"value": value}})
|
||||
if not must:
|
||||
return None
|
||||
return {"must": must}
|
||||
57
src/artenschutz_router/clients/memory_api.py
Normal file
57
src/artenschutz_router/clients/memory_api.py
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class MemoryApiClient:
|
||||
"""Write-path proxy to agent.memory.system's /memory/store endpoint.
|
||||
|
||||
The router does NOT touch m2-memory source code. The fact/exemplar
|
||||
payload travels as the free-form `metadata` dict that /memory/store
|
||||
already accepts.
|
||||
"""
|
||||
|
||||
def __init__(self, base_url: str, timeout_s: float = 30.0) -> None:
|
||||
self._base_url = base_url.rstrip("/")
|
||||
self._client = httpx.AsyncClient(base_url=self._base_url, timeout=timeout_s)
|
||||
|
||||
async def close(self) -> None:
|
||||
await self._client.aclose()
|
||||
|
||||
async def store(
|
||||
self,
|
||||
*,
|
||||
content: str,
|
||||
agent_id: str,
|
||||
memory_type: str = "fact",
|
||||
importance: float = 0.5,
|
||||
source: str = "document",
|
||||
entities: list[str] | None = None,
|
||||
language: str = "de",
|
||||
metadata: dict[str, Any] | None = None,
|
||||
) -> str:
|
||||
body = {
|
||||
"content": content,
|
||||
"agent_id": agent_id,
|
||||
"memory_type": memory_type,
|
||||
"importance": importance,
|
||||
"source": source,
|
||||
"entities": entities or [],
|
||||
"language": language,
|
||||
"metadata": metadata or {},
|
||||
}
|
||||
resp = await self._client.post("/memory/store", json=body)
|
||||
resp.raise_for_status()
|
||||
return resp.json()["id"]
|
||||
|
||||
async def health(self) -> bool:
|
||||
try:
|
||||
resp = await self._client.get("/health")
|
||||
return resp.status_code == 200
|
||||
except httpx.HTTPError:
|
||||
return False
|
||||
112
src/artenschutz_router/clients/qdrant.py
Normal file
112
src/artenschutz_router/clients/qdrant.py
Normal file
|
|
@ -0,0 +1,112 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from qdrant_client import AsyncQdrantClient
|
||||
from qdrant_client.http import models as qm
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class QdrantReader:
|
||||
"""Read-only Qdrant wrapper for hybrid + payload-filter queries.
|
||||
|
||||
Writes are NOT exposed here on purpose — they go through memory-api
|
||||
so we stay aligned with whatever ingest pipeline / dedup / vector
|
||||
schema the canonical memory service enforces.
|
||||
"""
|
||||
|
||||
def __init__(self, url: str, collection: str) -> None:
|
||||
self._client = AsyncQdrantClient(url=url, check_compatibility=False)
|
||||
self._collection = collection
|
||||
|
||||
async def close(self) -> None:
|
||||
await self._client.close()
|
||||
|
||||
def _to_filter(self, raw: dict[str, Any] | None) -> qm.Filter | None:
|
||||
if not raw:
|
||||
return None
|
||||
must = raw.get("must", [])
|
||||
if not must:
|
||||
return None
|
||||
conditions: list[qm.FieldCondition] = []
|
||||
for clause in must:
|
||||
key = clause["key"]
|
||||
match = clause["match"]
|
||||
if "any" in match:
|
||||
conditions.append(
|
||||
qm.FieldCondition(key=key, match=qm.MatchAny(any=match["any"]))
|
||||
)
|
||||
else:
|
||||
conditions.append(
|
||||
qm.FieldCondition(key=key, match=qm.MatchValue(value=match["value"]))
|
||||
)
|
||||
return qm.Filter(must=conditions)
|
||||
|
||||
async def hybrid_query(
|
||||
self,
|
||||
*,
|
||||
agent_id: str,
|
||||
dense: list[float],
|
||||
sparse: dict[int, float],
|
||||
metadata_filter: dict[str, Any] | None,
|
||||
limit: int = 10,
|
||||
prefetch_limit: int = 50,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Dense + sparse hybrid with RRF fusion, scoped to agent_id."""
|
||||
|
||||
combined: dict[str, Any] = {"must": [{"key": "agent_id", "match": {"value": agent_id}}]}
|
||||
if metadata_filter and "must" in metadata_filter:
|
||||
combined["must"].extend(metadata_filter["must"])
|
||||
qfilter = self._to_filter(combined)
|
||||
|
||||
sparse_vec = qm.SparseVector(
|
||||
indices=list(sparse.keys()),
|
||||
values=list(sparse.values()),
|
||||
)
|
||||
|
||||
result = await self._client.query_points(
|
||||
collection_name=self._collection,
|
||||
prefetch=[
|
||||
qm.Prefetch(query=dense, using="dense", limit=prefetch_limit, filter=qfilter),
|
||||
qm.Prefetch(query=sparse_vec, using="sparse", limit=prefetch_limit, filter=qfilter),
|
||||
],
|
||||
query=qm.FusionQuery(fusion=qm.Fusion.RRF),
|
||||
query_filter=qfilter,
|
||||
limit=limit,
|
||||
with_payload=True,
|
||||
with_vectors=False,
|
||||
)
|
||||
return [self._format_point(p) for p in result.points]
|
||||
|
||||
async def scroll(
|
||||
self,
|
||||
*,
|
||||
agent_id: str,
|
||||
metadata_filter: dict[str, Any] | None,
|
||||
limit: int = 100,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Pure payload-filter retrieval (no vector ranking)."""
|
||||
|
||||
combined: dict[str, Any] = {"must": [{"key": "agent_id", "match": {"value": agent_id}}]}
|
||||
if metadata_filter and "must" in metadata_filter:
|
||||
combined["must"].extend(metadata_filter["must"])
|
||||
qfilter = self._to_filter(combined)
|
||||
|
||||
points, _next = await self._client.scroll(
|
||||
collection_name=self._collection,
|
||||
scroll_filter=qfilter,
|
||||
limit=limit,
|
||||
with_payload=True,
|
||||
with_vectors=False,
|
||||
)
|
||||
return [self._format_point(p) for p in points]
|
||||
|
||||
@staticmethod
|
||||
def _format_point(p: Any) -> dict[str, Any]:
|
||||
return {
|
||||
"id": str(p.id),
|
||||
"score": getattr(p, "score", None),
|
||||
"payload": p.payload or {},
|
||||
}
|
||||
29
src/artenschutz_router/config.py
Normal file
29
src/artenschutz_router/config.py
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from functools import lru_cache
|
||||
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore")
|
||||
|
||||
qdrant_url: str = "http://memory-qdrant:6333"
|
||||
qdrant_collection: str = "agent_memory"
|
||||
bge_tei_url: str = "http://memory-embeddings:8000"
|
||||
memory_api_url: str = "http://memory-api:8000"
|
||||
|
||||
# Default agent_id is the client name — keeps the partition tenant-scoped
|
||||
# rather than domain-scoped. Multi-client deployments override per-instance.
|
||||
artenschutz_agent_id: str = "gruenstifter"
|
||||
|
||||
router_host: str = "0.0.0.0"
|
||||
router_port: int = 8080
|
||||
log_level: str = "INFO"
|
||||
|
||||
request_timeout_s: float = 30.0
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def get_settings() -> Settings:
|
||||
return Settings()
|
||||
1
src/artenschutz_router/ingest/__init__.py
Normal file
1
src/artenschutz_router/ingest/__init__.py
Normal file
|
|
@ -0,0 +1 @@
|
|||
"""Ingest pipelines for authoritative sources and (later) past Gutachten."""
|
||||
10
src/artenschutz_router/ingest/authoritative/__init__.py
Normal file
10
src/artenschutz_router/ingest/authoritative/__init__.py
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
"""Loaders for authoritative legal/methodological sources.
|
||||
|
||||
Each loader takes a Path to its source file(s) and yields `IngestRecord`
|
||||
instances. The CLI consumes those records and POSTs them to the router's
|
||||
`/ingest` endpoint.
|
||||
"""
|
||||
|
||||
from .base import IngestRecord
|
||||
|
||||
__all__ = ["IngestRecord"]
|
||||
30
src/artenschutz_router/ingest/authoritative/base.py
Normal file
30
src/artenschutz_router/ingest/authoritative/base.py
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from ...models import FactPayload
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class IngestRecord:
|
||||
"""One record ready to be POSTed to /ingest.
|
||||
|
||||
`content` is the embedded text. `payload` carries the structured
|
||||
metadata that lands in Qdrant payload.
|
||||
"""
|
||||
|
||||
content: str
|
||||
payload: FactPayload
|
||||
importance: float = 0.5
|
||||
language: str = "de"
|
||||
|
||||
def to_request_body(self, agent_id: str | None = None) -> dict:
|
||||
body = {
|
||||
"content": self.content,
|
||||
"payload": self.payload.model_dump(mode="json"),
|
||||
"importance": self.importance,
|
||||
"language": self.language,
|
||||
}
|
||||
if agent_id is not None:
|
||||
body["agent_id"] = agent_id
|
||||
return body
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
|
||||
from ...models import FactPayload, FactScope, SourceInfo
|
||||
from ..chunker import chunk_by_size
|
||||
from ..pdf_text import extract_pdf_pages
|
||||
from .base import IngestRecord
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_PATH = Path("sources/biotopwertliste/biotopwertliste.pdf")
|
||||
|
||||
# NO default states — every Bundesland publishes its own Biotopwertliste
|
||||
# with different codes (Berlin's, BayKompV in Bayern, etc.). Operator must
|
||||
# tag the file with the right state(s) via the --states CLI flag, or place
|
||||
# state-specific lists under sources/states/<slug>/ for the state_docs loader.
|
||||
DEFAULT_STATES: list[str] = []
|
||||
|
||||
|
||||
def load(path: Path = DEFAULT_PATH, *, states: list[str] | None = None) -> Iterator[IngestRecord]:
|
||||
"""Yield biotope-code records from a Biotopwertliste PDF.
|
||||
|
||||
`states` should be set when the file is state-specific. Leave empty only
|
||||
when the document is genuinely federal/cross-state — most Biotopwertlisten
|
||||
are NOT.
|
||||
"""
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(
|
||||
f"Biotopwertliste source not found at {path}. "
|
||||
"See sources/README.md."
|
||||
)
|
||||
|
||||
pages = extract_pdf_pages(path)
|
||||
if not any(pages):
|
||||
logger.warning("No text extracted from %s", path)
|
||||
return
|
||||
|
||||
scope_states = states if states is not None else DEFAULT_STATES
|
||||
|
||||
for page_idx, page_text in enumerate(pages):
|
||||
if not page_text.strip():
|
||||
continue
|
||||
for chunk in chunk_by_size(page_text, target_chars=1800, overlap_chars=100):
|
||||
payload = FactPayload(
|
||||
fact_type="biotope",
|
||||
source=SourceInfo(type="authoritative", doc="biotopwertliste"),
|
||||
scope=FactScope(states=list(scope_states)),
|
||||
citation_anchor=f"Biotopwertliste · Seite {page_idx + 1}",
|
||||
)
|
||||
yield IngestRecord(content=chunk.text, payload=payload, importance=0.75)
|
||||
58
src/artenschutz_router/ingest/authoritative/bnatschg.py
Normal file
58
src/artenschutz_router/ingest/authoritative/bnatschg.py
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from collections.abc import Iterator
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
|
||||
from ...models import FactPayload, FactScope, SourceInfo
|
||||
from ..chunker import chunk_by_paragraph_marker
|
||||
from .base import IngestRecord
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_PATH = Path("sources/bnatschg/bnatschg.txt")
|
||||
|
||||
|
||||
def load(path: Path = DEFAULT_PATH, *, valid_from: date | None = None) -> Iterator[IngestRecord]:
|
||||
"""Yield FactPayload records from a plain-text BNatSchG dump.
|
||||
|
||||
Expected format: UTF-8 text with `§ N ...` headings (and optionally
|
||||
`Anlage N ...`). Sections in between become the chunk body.
|
||||
Citations are anchored to the heading line.
|
||||
|
||||
BNatSchG is **federal** legislation — `scope.states` stays empty so
|
||||
every state's query matches.
|
||||
"""
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(
|
||||
f"BNatSchG source not found at {path}. "
|
||||
"See sources/README.md for how to populate it."
|
||||
)
|
||||
|
||||
text = path.read_text(encoding="utf-8")
|
||||
chunks = chunk_by_paragraph_marker(text)
|
||||
if not chunks:
|
||||
logger.warning("No § markers detected in %s — emitting nothing", path)
|
||||
return
|
||||
|
||||
for chunk in chunks:
|
||||
anchor = _normalise_anchor(chunk.anchor) if chunk.anchor else None
|
||||
payload = FactPayload(
|
||||
fact_type="legal",
|
||||
source=SourceInfo(type="authoritative", doc="BNatSchG"),
|
||||
scope=FactScope(),
|
||||
valid_from=valid_from,
|
||||
citation_anchor=anchor,
|
||||
)
|
||||
yield IngestRecord(content=chunk.text, payload=payload, importance=0.9)
|
||||
|
||||
|
||||
_ANCHOR_NORMALISE_RE = re.compile(r"\s+")
|
||||
|
||||
|
||||
def _normalise_anchor(raw: str) -> str:
|
||||
"""Trim and collapse whitespace; keep just the heading line."""
|
||||
line = raw.splitlines()[0] if raw else ""
|
||||
return _ANCHOR_NORMALISE_RE.sub(" ", line).strip()
|
||||
49
src/artenschutz_router/ingest/authoritative/mhbasp.py
Normal file
49
src/artenschutz_router/ingest/authoritative/mhbasp.py
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
|
||||
from ...models import FactPayload, FactScope, SourceInfo
|
||||
from ..chunker import chunk_by_size
|
||||
from ..pdf_text import extract_pdf_pages
|
||||
from .base import IngestRecord
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_PATH = Path("sources/mhbasp/mhbasp_anhang4.pdf")
|
||||
|
||||
|
||||
def load(path: Path = DEFAULT_PATH) -> Iterator[IngestRecord]:
|
||||
"""Yield method-fact records from mhbasp Anhang 4.
|
||||
|
||||
Anhang 4 documents species-specific survey methods ("artspezifisch
|
||||
geeignete Kartiermethoden"). PDF text extraction is noisy and the
|
||||
document is table-heavy; v1 ingests page-level chunks with size-bound
|
||||
splits inside long pages. A later phase can replace this with a
|
||||
proper table parser.
|
||||
|
||||
Like BNatSchG, mhbasp is federal — `scope.states` stays empty.
|
||||
"""
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(
|
||||
f"mhbasp source not found at {path}. "
|
||||
"See sources/README.md (a symlink into GST-DATA is the quickest path)."
|
||||
)
|
||||
|
||||
pages = extract_pdf_pages(path)
|
||||
if not any(pages):
|
||||
logger.warning("No text extracted from %s", path)
|
||||
return
|
||||
|
||||
for page_idx, page_text in enumerate(pages):
|
||||
if not page_text.strip():
|
||||
continue
|
||||
for chunk in chunk_by_size(page_text, target_chars=1500, overlap_chars=150):
|
||||
payload = FactPayload(
|
||||
fact_type="method",
|
||||
source=SourceInfo(type="authoritative", doc="mhbasp_anhang4"),
|
||||
scope=FactScope(),
|
||||
citation_anchor=f"mhbasp Anhang 4 · Seite {page_idx + 1}",
|
||||
)
|
||||
yield IngestRecord(content=chunk.text, payload=payload, importance=0.85)
|
||||
54
src/artenschutz_router/ingest/authoritative/state_docs.py
Normal file
54
src/artenschutz_router/ingest/authoritative/state_docs.py
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
|
||||
from ...models import FactPayload, FactScope, SourceInfo
|
||||
from ..chunker import chunk_by_size
|
||||
from ..pdf_text import extract_pdf_pages
|
||||
from .base import IngestRecord
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Map of state slug → (Bundesland code, source-dir name).
|
||||
STATES: dict[str, tuple[str, str]] = {
|
||||
"berlin": ("BE", "berlin"),
|
||||
"brandenburg": ("BB", "brandenburg"),
|
||||
"bayern": ("BY", "bayern"),
|
||||
"baden_wuerttemberg": ("BW", "baden_wuerttemberg"),
|
||||
"sachsen": ("SN", "sachsen"),
|
||||
}
|
||||
|
||||
|
||||
def load(state_slug: str, sources_root: Path = Path("sources/states")) -> Iterator[IngestRecord]:
|
||||
"""Yield method-fact records from all PDFs under sources/states/<slug>/."""
|
||||
if state_slug not in STATES:
|
||||
raise ValueError(
|
||||
f"Unknown state '{state_slug}'. Known: {', '.join(sorted(STATES))}"
|
||||
)
|
||||
state_code, dirname = STATES[state_slug]
|
||||
state_dir = sources_root / dirname
|
||||
if not state_dir.exists():
|
||||
raise FileNotFoundError(f"State source dir not found: {state_dir}")
|
||||
|
||||
pdfs = sorted(state_dir.glob("*.pdf"))
|
||||
if not pdfs:
|
||||
logger.warning("No PDFs under %s — nothing to ingest for state %s", state_dir, state_slug)
|
||||
return
|
||||
|
||||
for pdf in pdfs:
|
||||
logger.info("Ingesting %s for state %s", pdf.name, state_code)
|
||||
pages = extract_pdf_pages(pdf)
|
||||
for page_idx, page_text in enumerate(pages):
|
||||
if not page_text.strip():
|
||||
continue
|
||||
for chunk in chunk_by_size(page_text, target_chars=1500, overlap_chars=150):
|
||||
payload = FactPayload(
|
||||
fact_type="method",
|
||||
source=SourceInfo(type="authoritative", doc=pdf.stem),
|
||||
scope=FactScope(states=[state_code]),
|
||||
citation_anchor=f"{pdf.name} · Seite {page_idx + 1}",
|
||||
)
|
||||
yield IngestRecord(content=chunk.text, payload=payload, importance=0.8)
|
||||
93
src/artenschutz_router/ingest/chunker.py
Normal file
93
src/artenschutz_router/ingest/chunker.py
Normal file
|
|
@ -0,0 +1,93 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Iterator
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Chunk:
|
||||
"""A chunk of text plus a heading anchor (e.g. '§ 44' or section title)."""
|
||||
|
||||
anchor: str | None
|
||||
text: str
|
||||
|
||||
|
||||
_PARAGRAPH_RE = re.compile(r"^[ \t]*§[ \t]*\d+\w*", re.MULTILINE)
|
||||
_ANLAGE_RE = re.compile(r"^[ \t]*Anlage[ \t]+\d+", re.MULTILINE)
|
||||
_HEADING_RE = re.compile(r"^(\d+(\.\d+)*\s+\S.*|[A-ZÄÖÜ][^\n]{0,80})$", re.MULTILINE)
|
||||
|
||||
|
||||
def chunk_by_paragraph_marker(text: str) -> list[Chunk]:
|
||||
"""Split a legal text by `§ N` (and `Anlage N`) markers.
|
||||
|
||||
Each chunk starts at a marker line and runs until the next marker.
|
||||
The marker line itself becomes the chunk anchor.
|
||||
"""
|
||||
# Find all marker positions (paragraph + anlage).
|
||||
starts: list[tuple[int, str]] = []
|
||||
for m in _PARAGRAPH_RE.finditer(text):
|
||||
starts.append((m.start(), _first_line(text, m.start())))
|
||||
for m in _ANLAGE_RE.finditer(text):
|
||||
starts.append((m.start(), _first_line(text, m.start())))
|
||||
starts.sort()
|
||||
|
||||
if not starts:
|
||||
return [Chunk(anchor=None, text=text.strip())] if text.strip() else []
|
||||
|
||||
chunks: list[Chunk] = []
|
||||
for idx, (pos, anchor) in enumerate(starts):
|
||||
end = starts[idx + 1][0] if idx + 1 < len(starts) else len(text)
|
||||
body = text[pos:end].strip()
|
||||
if body:
|
||||
chunks.append(Chunk(anchor=anchor.strip(), text=body))
|
||||
return chunks
|
||||
|
||||
|
||||
def chunk_by_size(text: str, *, target_chars: int = 1500, overlap_chars: int = 200) -> list[Chunk]:
|
||||
"""Sliding-window chunker for noisy text (PDF extracts). Breaks on
|
||||
paragraph boundaries when possible."""
|
||||
text = text.strip()
|
||||
if not text:
|
||||
return []
|
||||
|
||||
paragraphs = re.split(r"\n\s*\n", text)
|
||||
chunks: list[Chunk] = []
|
||||
buf: list[str] = []
|
||||
buf_len = 0
|
||||
|
||||
def flush():
|
||||
if buf:
|
||||
chunks.append(Chunk(anchor=None, text="\n\n".join(buf).strip()))
|
||||
|
||||
for para in paragraphs:
|
||||
para = para.strip()
|
||||
if not para:
|
||||
continue
|
||||
if buf_len + len(para) + 2 > target_chars and buf:
|
||||
flush()
|
||||
# carry overlap
|
||||
if overlap_chars > 0 and buf:
|
||||
tail = buf[-1][-overlap_chars:]
|
||||
buf = [tail]
|
||||
buf_len = len(tail)
|
||||
else:
|
||||
buf = []
|
||||
buf_len = 0
|
||||
buf.append(para)
|
||||
buf_len += len(para) + 2
|
||||
|
||||
flush()
|
||||
return chunks
|
||||
|
||||
|
||||
def iter_section_chunks(text: str, *, target_chars: int = 1500) -> Iterator[Chunk]:
|
||||
"""Generator wrapper for size-based chunking."""
|
||||
yield from chunk_by_size(text, target_chars=target_chars)
|
||||
|
||||
|
||||
def _first_line(text: str, pos: int) -> str:
|
||||
end = text.find("\n", pos)
|
||||
if end == -1:
|
||||
end = min(len(text), pos + 120)
|
||||
return text[pos:end]
|
||||
136
src/artenschutz_router/ingest/cli.py
Normal file
136
src/artenschutz_router/ingest/cli.py
Normal file
|
|
@ -0,0 +1,136 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
|
||||
from .authoritative import IngestRecord
|
||||
from .authoritative import bnatschg as bnatschg_loader
|
||||
from .authoritative import biotopwertliste as biotop_loader
|
||||
from .authoritative import mhbasp as mhbasp_loader
|
||||
from .authoritative import state_docs as state_loader
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
_SOURCE_HELP = (
|
||||
"Source slug. One of: bnatschg, mhbasp, biotopwertliste, state-<slug> "
|
||||
f"(states: {', '.join(sorted(state_loader.STATES))})."
|
||||
)
|
||||
|
||||
|
||||
def _load_records(
|
||||
source: str, sources_root: Path, *, states: list[str] | None = None
|
||||
) -> Iterator[IngestRecord]:
|
||||
if source == "bnatschg":
|
||||
yield from bnatschg_loader.load(sources_root / "bnatschg" / "bnatschg.txt")
|
||||
return
|
||||
if source == "mhbasp":
|
||||
yield from mhbasp_loader.load(sources_root / "mhbasp" / "mhbasp_anhang4.pdf")
|
||||
return
|
||||
if source == "biotopwertliste":
|
||||
yield from biotop_loader.load(
|
||||
sources_root / "biotopwertliste" / "biotopwertliste.pdf", states=states
|
||||
)
|
||||
return
|
||||
if source.startswith("state-"):
|
||||
slug = source[len("state-") :]
|
||||
yield from state_loader.load(slug, sources_root=sources_root / "states")
|
||||
return
|
||||
raise SystemExit(f"Unknown --source '{source}'. {_SOURCE_HELP}")
|
||||
|
||||
|
||||
def _post_record(client: httpx.Client, router_url: str, record: IngestRecord) -> str:
|
||||
resp = client.post(
|
||||
f"{router_url.rstrip('/')}/ingest",
|
||||
json=record.to_request_body(),
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()["id"]
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="artenschutz-ingest",
|
||||
description="Ingest authoritative Artenschutz sources via the router /ingest endpoint.",
|
||||
)
|
||||
parser.add_argument("--source", required=True, help=_SOURCE_HELP)
|
||||
parser.add_argument(
|
||||
"--router-url",
|
||||
default="http://localhost:8080",
|
||||
help="Base URL of the running artenschutz-router (default: %(default)s)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--sources-root",
|
||||
default="sources",
|
||||
type=Path,
|
||||
help="Root dir holding source files (default: %(default)s)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dry-run",
|
||||
action="store_true",
|
||||
help="Print records as JSON, don't POST to the router.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--limit", type=int, default=0, help="Stop after N records (0 = no limit)."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--states",
|
||||
nargs="*",
|
||||
default=None,
|
||||
help="Override `scope.states` for sources that need a state tag (currently: biotopwertliste).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--log-level", default="INFO", help="Logging level (default: %(default)s)"
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
logging.basicConfig(
|
||||
level=getattr(logging, args.log_level.upper(), logging.INFO),
|
||||
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
|
||||
)
|
||||
|
||||
records = _load_records(args.source, args.sources_root, states=args.states)
|
||||
if args.limit > 0:
|
||||
records = _take(records, args.limit)
|
||||
|
||||
if args.dry_run:
|
||||
count = 0
|
||||
for r in records:
|
||||
print(json.dumps(r.to_request_body(), ensure_ascii=False))
|
||||
count += 1
|
||||
logger.info("Dry-run produced %d records", count)
|
||||
return 0
|
||||
|
||||
posted = 0
|
||||
errors = 0
|
||||
with httpx.Client(timeout=60.0) as client:
|
||||
for record in records:
|
||||
try:
|
||||
memory_id = _post_record(client, args.router_url, record)
|
||||
logger.debug("Stored %s (anchor=%s)", memory_id, record.payload.citation_anchor)
|
||||
posted += 1
|
||||
except httpx.HTTPError as exc:
|
||||
logger.error("POST failed for record: %s", exc)
|
||||
errors += 1
|
||||
if errors > 5 and posted == 0:
|
||||
logger.error("Aborting after %d consecutive failures with no successes", errors)
|
||||
return 2
|
||||
logger.info("Posted %d records (errors: %d)", posted, errors)
|
||||
return 0 if errors == 0 else 1
|
||||
|
||||
|
||||
def _take(iterable, n: int):
|
||||
for i, x in enumerate(iterable):
|
||||
if i >= n:
|
||||
return
|
||||
yield x
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
27
src/artenschutz_router/ingest/pdf_text.py
Normal file
27
src/artenschutz_router/ingest/pdf_text.py
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
from pypdf import PdfReader
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def extract_pdf_pages(path: Path) -> list[str]:
|
||||
"""Return page texts. Empty strings for pages where extraction failed."""
|
||||
reader = PdfReader(str(path))
|
||||
pages: list[str] = []
|
||||
for i, page in enumerate(reader.pages):
|
||||
try:
|
||||
text = page.extract_text() or ""
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("Page %d of %s failed text extraction: %s", i + 1, path.name, exc)
|
||||
text = ""
|
||||
pages.append(text.strip())
|
||||
return pages
|
||||
|
||||
|
||||
def extract_pdf_text(path: Path) -> str:
|
||||
"""Concatenated page text with double-newline separators."""
|
||||
return "\n\n".join(p for p in extract_pdf_pages(path) if p)
|
||||
93
src/artenschutz_router/main.py
Normal file
93
src/artenschutz_router/main.py
Normal file
|
|
@ -0,0 +1,93 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI
|
||||
|
||||
from .clients.bge import BGEClient
|
||||
from .clients.memory_api import MemoryApiClient
|
||||
from .clients.qdrant import QdrantReader
|
||||
from .config import get_settings
|
||||
from .models import HealthResponse
|
||||
from .routes import ingest as ingest_routes
|
||||
from .routes import search as search_routes
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
settings = get_settings()
|
||||
logging.basicConfig(
|
||||
level=getattr(logging, settings.log_level.upper(), logging.INFO),
|
||||
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
logger.info("Starting artenschutz-router")
|
||||
logger.info(" Qdrant: %s (collection: %s)", settings.qdrant_url, settings.qdrant_collection)
|
||||
logger.info(" BGE TEI: %s", settings.bge_tei_url)
|
||||
logger.info(" memory-api: %s", settings.memory_api_url)
|
||||
logger.info(" agent_id: %s", settings.artenschutz_agent_id)
|
||||
|
||||
qdrant = QdrantReader(url=settings.qdrant_url, collection=settings.qdrant_collection)
|
||||
bge = BGEClient(base_url=settings.bge_tei_url, timeout_s=settings.request_timeout_s)
|
||||
memory_api = MemoryApiClient(base_url=settings.memory_api_url, timeout_s=settings.request_timeout_s)
|
||||
|
||||
app.state.settings = settings
|
||||
app.state.qdrant = qdrant
|
||||
app.state.bge = bge
|
||||
app.state.memory_api = memory_api
|
||||
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
await qdrant.close()
|
||||
await bge.close()
|
||||
await memory_api.close()
|
||||
|
||||
|
||||
app = FastAPI(
|
||||
title="artenschutz-router",
|
||||
description=(
|
||||
"Sidecar router for Artenschutz domain queries against agent.memory.system. "
|
||||
"Writes proxy memory-api /memory/store; reads go directly to Qdrant."
|
||||
),
|
||||
version="0.1.0",
|
||||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
|
||||
app.include_router(search_routes.router)
|
||||
app.include_router(ingest_routes.router)
|
||||
|
||||
|
||||
@app.get("/health", response_model=HealthResponse, tags=["health"])
|
||||
async def health() -> HealthResponse:
|
||||
"""Live-probes Qdrant, BGE-M3 TEI, memory-api. `degraded` if any fail."""
|
||||
qdrant_ok = True
|
||||
bge_ok = True
|
||||
memory_api_ok = False
|
||||
|
||||
try:
|
||||
await app.state.qdrant._client.get_collections() # noqa: SLF001
|
||||
except Exception:
|
||||
qdrant_ok = False
|
||||
|
||||
try:
|
||||
# TEI exposes /health; falling back to /info on older versions.
|
||||
resp = await app.state.bge._client.get("/health") # noqa: SLF001
|
||||
bge_ok = resp.status_code == 200
|
||||
except Exception:
|
||||
bge_ok = False
|
||||
|
||||
try:
|
||||
memory_api_ok = await app.state.memory_api.health()
|
||||
except Exception:
|
||||
memory_api_ok = False
|
||||
|
||||
overall = "ok" if all([qdrant_ok, bge_ok, memory_api_ok]) else "degraded"
|
||||
return HealthResponse(
|
||||
status=overall,
|
||||
qdrant=qdrant_ok,
|
||||
bge=bge_ok,
|
||||
memory_api=memory_api_ok,
|
||||
)
|
||||
142
src/artenschutz_router/models.py
Normal file
142
src/artenschutz_router/models.py
Normal file
|
|
@ -0,0 +1,142 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from datetime import date
|
||||
from typing import Annotated, Any, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Payload schemas — what travels in Qdrant payload via the `metadata` field
|
||||
# on memory-api /memory/store.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class SourceInfo(BaseModel):
|
||||
"""Where a memory came from."""
|
||||
|
||||
type: Literal["authoritative", "extracted"]
|
||||
doc: str = Field(..., description="Short slug — e.g. 'BNatSchG', 'mhbasp_anhang4', or project_slug")
|
||||
year: int | None = None
|
||||
tier: Literal["A", "B", "C"] | None = None
|
||||
|
||||
|
||||
class FactScope(BaseModel):
|
||||
"""Filterable scope dimensions for facts."""
|
||||
|
||||
states: list[str] = Field(default_factory=list, description="DE Bundesland codes, e.g. ['BE', 'BB']")
|
||||
anlagen_typ: list[str] = Field(default_factory=list, description="['WEA', 'PV', 'Neubau', 'Eingriff', ...]")
|
||||
species: list[str] = Field(default_factory=list, description="GBIF/Faunistik IDs or scientific names")
|
||||
impact_types: list[str] = Field(default_factory=list, description="['lichtemission', 'rotor', 'bauzeit', ...]")
|
||||
|
||||
|
||||
FactType = Literal["legal", "method", "biotope", "mitigation", "verdict"]
|
||||
|
||||
|
||||
class FactPayload(BaseModel):
|
||||
"""Structured fact record (lives in Qdrant payload)."""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
kind: Literal["fact"] = "fact"
|
||||
fact_type: FactType
|
||||
source: SourceInfo
|
||||
scope: FactScope = Field(default_factory=FactScope)
|
||||
valid_from: date | None = None
|
||||
valid_until: date | None = None
|
||||
accepted_by_behoerde: bool | None = None
|
||||
supersedes: str | None = None
|
||||
citation_anchor: str | None = Field(
|
||||
default=None, description="Human-readable anchor, e.g. 'BNatSchG §44(1) Nr. 1'"
|
||||
)
|
||||
|
||||
|
||||
ReportType = Literal["AFB", "FachGA", "MBKS", "ASB", "LBP", "KONZ", "Other"]
|
||||
|
||||
|
||||
class ExemplarPayload(BaseModel):
|
||||
"""Prose chunk from a past Gutachten (lives in Qdrant payload)."""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
kind: Literal["exemplar"] = "exemplar"
|
||||
project_slug: str
|
||||
report_type: ReportType
|
||||
state: str | None = None
|
||||
anlagen_typ: str | None = None
|
||||
year: int | None = None
|
||||
tier: Literal["A", "B", "C"]
|
||||
section: str | None = Field(default=None, description="Canonical section name from template")
|
||||
species_mentioned: list[str] = Field(default_factory=list)
|
||||
source_path: str | None = None
|
||||
is_draft: bool = False
|
||||
|
||||
|
||||
Payload = Annotated[FactPayload | ExemplarPayload, Field(discriminator="kind")]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# API request / response models
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class SearchRequest(BaseModel):
|
||||
"""Generic search — hybrid (dense+sparse) when `query` provided,
|
||||
payload-filter scroll otherwise."""
|
||||
|
||||
query: str | None = None
|
||||
metadata_filter: dict[str, Any] | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Flat dict of payload key → value or [values]. "
|
||||
"Dotted keys reach into nested payload (e.g. 'scope.states')."
|
||||
),
|
||||
)
|
||||
memory_types: list[str] | None = None
|
||||
limit: int = 10
|
||||
prefetch_limit: int = 50
|
||||
agent_id: str | None = Field(
|
||||
default=None,
|
||||
description="Overrides the configured default — leave unset to use the router's agent_id.",
|
||||
)
|
||||
|
||||
|
||||
class SearchHit(BaseModel):
|
||||
id: str
|
||||
score: float | None
|
||||
payload: dict[str, Any]
|
||||
|
||||
|
||||
class SearchResponse(BaseModel):
|
||||
strategy: Literal["hybrid", "scroll"]
|
||||
hits: list[SearchHit]
|
||||
|
||||
|
||||
class IngestRequest(BaseModel):
|
||||
"""Push a single fact or exemplar into the memory store.
|
||||
|
||||
The router rewrites the body to memory-api's /memory/store contract:
|
||||
- `content` becomes the embedded text
|
||||
- `payload` becomes the `metadata` dict
|
||||
- `agent_id` defaults to the router's configured value
|
||||
"""
|
||||
|
||||
content: str
|
||||
payload: Payload
|
||||
importance: float = 0.5
|
||||
entities: list[str] = Field(default_factory=list)
|
||||
language: str = "de"
|
||||
agent_id: str | None = None
|
||||
# MemoryType in m2-memory: working|episodic|semantic|fact|research|other.
|
||||
# Facts → 'fact'; exemplars → 'semantic'.
|
||||
memory_type: str | None = None
|
||||
|
||||
|
||||
class IngestResponse(BaseModel):
|
||||
id: str
|
||||
|
||||
|
||||
class HealthResponse(BaseModel):
|
||||
status: Literal["ok", "degraded"]
|
||||
qdrant: bool
|
||||
bge: bool
|
||||
memory_api: bool
|
||||
1
src/artenschutz_router/routes/__init__.py
Normal file
1
src/artenschutz_router/routes/__init__.py
Normal file
|
|
@ -0,0 +1 @@
|
|||
"""HTTP routes."""
|
||||
43
src/artenschutz_router/routes/ingest.py
Normal file
43
src/artenschutz_router/routes/ingest.py
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
|
||||
from ..models import FactPayload, IngestRequest, IngestResponse
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/ingest", tags=["ingest"])
|
||||
|
||||
|
||||
@router.post("", response_model=IngestResponse)
|
||||
async def ingest(req: IngestRequest, request: Request) -> IngestResponse:
|
||||
"""Push a single Fact or Exemplar record into agent.memory.system.
|
||||
|
||||
Translates to memory-api's /memory/store contract — see clients/memory_api.py.
|
||||
"""
|
||||
settings = request.app.state.settings
|
||||
memory_api = request.app.state.memory_api
|
||||
|
||||
agent_id = req.agent_id or settings.artenschutz_agent_id
|
||||
memory_type = req.memory_type or ("fact" if isinstance(req.payload, FactPayload) else "semantic")
|
||||
|
||||
payload_dict = req.payload.model_dump(mode="json")
|
||||
|
||||
try:
|
||||
memory_id = await memory_api.store(
|
||||
content=req.content,
|
||||
agent_id=agent_id,
|
||||
memory_type=memory_type,
|
||||
importance=req.importance,
|
||||
source="document",
|
||||
entities=req.entities,
|
||||
language=req.language,
|
||||
metadata=payload_dict,
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.exception("memory-api /store failed")
|
||||
raise HTTPException(502, f"memory-api store failed: {exc}") from exc
|
||||
|
||||
return IngestResponse(id=memory_id)
|
||||
62
src/artenschutz_router/routes/search.py
Normal file
62
src/artenschutz_router/routes/search.py
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
|
||||
from ..clients.bge import build_qdrant_filter
|
||||
from ..models import SearchHit, SearchRequest, SearchResponse
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/search", tags=["search"])
|
||||
|
||||
|
||||
@router.post("/generic", response_model=SearchResponse)
|
||||
async def search_generic(req: SearchRequest, request: Request) -> SearchResponse:
|
||||
"""Hybrid retrieval when `query` is provided; payload-filter scroll otherwise.
|
||||
|
||||
The router scopes every query to its configured `agent_id` (overridable per
|
||||
request). Memory-type filtering is applied as an additional payload `must`
|
||||
clause when supplied.
|
||||
"""
|
||||
settings = request.app.state.settings
|
||||
qdrant = request.app.state.qdrant
|
||||
bge = request.app.state.bge
|
||||
|
||||
agent_id = req.agent_id or settings.artenschutz_agent_id
|
||||
|
||||
raw_filter = build_qdrant_filter(req.metadata_filter)
|
||||
if req.memory_types:
|
||||
extra = {"key": "memory_type", "match": {"any": req.memory_types}}
|
||||
raw_filter = {"must": (raw_filter or {}).get("must", []) + [extra]}
|
||||
|
||||
if req.query:
|
||||
try:
|
||||
dense, sparse = await bge.embed_hybrid(req.query)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.exception("BGE embedding failed")
|
||||
raise HTTPException(502, f"BGE embedding failed: {exc}") from exc
|
||||
|
||||
hits = await qdrant.hybrid_query(
|
||||
agent_id=agent_id,
|
||||
dense=dense,
|
||||
sparse=sparse,
|
||||
metadata_filter=raw_filter,
|
||||
limit=req.limit,
|
||||
prefetch_limit=req.prefetch_limit,
|
||||
)
|
||||
return SearchResponse(
|
||||
strategy="hybrid",
|
||||
hits=[SearchHit(**h) for h in hits],
|
||||
)
|
||||
|
||||
hits = await qdrant.scroll(
|
||||
agent_id=agent_id,
|
||||
metadata_filter=raw_filter,
|
||||
limit=req.limit,
|
||||
)
|
||||
return SearchResponse(
|
||||
strategy="scroll",
|
||||
hits=[SearchHit(**h) for h in hits],
|
||||
)
|
||||
0
tests/__init__.py
Normal file
0
tests/__init__.py
Normal file
30
tests/test_app.py
Normal file
30
tests/test_app.py
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from artenschutz_router.main import app
|
||||
|
||||
|
||||
def test_app_constructs():
|
||||
"""Smoke test: app object exists, routes registered."""
|
||||
paths = {route.path for route in app.routes}
|
||||
assert "/health" in paths
|
||||
assert "/search/generic" in paths
|
||||
assert "/ingest" in paths
|
||||
|
||||
|
||||
def test_openapi_schema_renders():
|
||||
# Force lifespan to skip (we don't have Qdrant/BGE running in unit tests).
|
||||
# We can still call openapi.json which doesn't touch the lifespan-managed
|
||||
# clients.
|
||||
with TestClient(app, raise_server_exceptions=False) as client:
|
||||
# Disable lifespan by short-circuiting via app.state — not strictly
|
||||
# necessary for openapi but keeps the test fast.
|
||||
resp = client.get("/openapi.json")
|
||||
# If lifespan failed connecting to deps, app may still serve openapi.
|
||||
assert resp.status_code == 200
|
||||
spec = resp.json()
|
||||
assert spec["info"]["title"] == "artenschutz-router"
|
||||
assert "/search/generic" in spec["paths"]
|
||||
assert "/ingest" in spec["paths"]
|
||||
assert "/health" in spec["paths"]
|
||||
58
tests/test_bnatschg_loader.py
Normal file
58
tests/test_bnatschg_loader.py
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from artenschutz_router.ingest.authoritative import bnatschg as loader
|
||||
|
||||
|
||||
SAMPLE_BNATSCHG = """§ 44 Vorschriften für besonders geschützte und bestimmte andere Tier- und Pflanzenarten
|
||||
|
||||
(1) Es ist verboten,
|
||||
1. wild lebenden Tieren der besonders geschützten Arten nachzustellen,
|
||||
sie zu fangen, zu verletzen oder zu töten oder ihre Entwicklungsformen aus
|
||||
der Natur zu entnehmen, zu beschädigen oder zu zerstören.
|
||||
|
||||
(5) Für nach § 15 Absatz 1 unvermeidbare Beeinträchtigungen …
|
||||
|
||||
§ 45b Schutz wild lebender Vögel
|
||||
|
||||
(1) Beim Betrieb von Windenergieanlagen sind die in Anlage 1 …
|
||||
"""
|
||||
|
||||
|
||||
def test_loads_records_from_synthetic_text(tmp_path: Path):
|
||||
path = tmp_path / "bnatschg.txt"
|
||||
path.write_text(SAMPLE_BNATSCHG, encoding="utf-8")
|
||||
|
||||
records = list(loader.load(path, valid_from=date(2024, 1, 1)))
|
||||
assert len(records) == 2
|
||||
anchors = [r.payload.citation_anchor for r in records]
|
||||
assert any(a and a.startswith("§ 44") for a in anchors)
|
||||
assert any(a and a.startswith("§ 45b") for a in anchors)
|
||||
for r in records:
|
||||
assert r.payload.fact_type == "legal"
|
||||
assert r.payload.source.doc == "BNatSchG"
|
||||
assert r.payload.source.type == "authoritative"
|
||||
assert r.payload.scope.states == []
|
||||
assert r.payload.valid_from == date(2024, 1, 1)
|
||||
assert r.importance > 0
|
||||
|
||||
|
||||
def test_missing_file_raises(tmp_path: Path):
|
||||
path = tmp_path / "nope.txt"
|
||||
with pytest.raises(FileNotFoundError):
|
||||
list(loader.load(path))
|
||||
|
||||
|
||||
def test_request_body_shape(tmp_path: Path):
|
||||
path = tmp_path / "bnatschg.txt"
|
||||
path.write_text(SAMPLE_BNATSCHG, encoding="utf-8")
|
||||
record = next(loader.load(path))
|
||||
body = record.to_request_body()
|
||||
assert body["content"]
|
||||
assert body["payload"]["kind"] == "fact"
|
||||
assert body["payload"]["fact_type"] == "legal"
|
||||
assert "agent_id" not in body # left to the router unless overridden
|
||||
50
tests/test_chunker.py
Normal file
50
tests/test_chunker.py
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from artenschutz_router.ingest.chunker import chunk_by_paragraph_marker, chunk_by_size
|
||||
|
||||
|
||||
def test_paragraph_chunker_splits_at_section_markers():
|
||||
text = """Vor § 44.
|
||||
|
||||
§ 44 Vorschriften für besonders geschützte und bestimmte andere Tier- und Pflanzenarten
|
||||
|
||||
(1) Es ist verboten,
|
||||
1. wild lebenden Tieren der besonders geschützten Arten nachzustellen, sie zu fangen, …
|
||||
|
||||
§ 45b Schutz wild lebender Vögel
|
||||
|
||||
(1) Beim Betrieb von Windenergieanlagen …
|
||||
|
||||
Anlage 1 (zu § 54 Absatz 4)
|
||||
|
||||
Bestimmte besonders geschützte Arten …
|
||||
"""
|
||||
chunks = chunk_by_paragraph_marker(text)
|
||||
anchors = [c.anchor for c in chunks]
|
||||
assert any(a and a.startswith("§ 44") for a in anchors)
|
||||
assert any(a and a.startswith("§ 45b") for a in anchors)
|
||||
assert any(a and a.startswith("Anlage 1") for a in anchors)
|
||||
# Pre-§44 prefix should NOT become its own chunk (we anchor on markers).
|
||||
assert "Vor § 44" not in chunks[0].text
|
||||
|
||||
|
||||
def test_paragraph_chunker_returns_single_when_no_markers():
|
||||
text = "Plain prose without any paragraph markers, just regular text."
|
||||
chunks = chunk_by_paragraph_marker(text)
|
||||
assert len(chunks) == 1
|
||||
assert chunks[0].anchor is None
|
||||
|
||||
|
||||
def test_size_chunker_respects_target_size():
|
||||
paragraphs = ["abc def ghi jkl"] * 200
|
||||
text = "\n\n".join(paragraphs)
|
||||
chunks = chunk_by_size(text, target_chars=200, overlap_chars=20)
|
||||
assert len(chunks) > 1
|
||||
for c in chunks:
|
||||
# Overlap may push slightly over; tolerate 2x.
|
||||
assert len(c.text) < 600
|
||||
|
||||
|
||||
def test_size_chunker_handles_empty():
|
||||
assert chunk_by_size("") == []
|
||||
assert chunk_by_size(" \n ") == []
|
||||
49
tests/test_cli.py
Normal file
49
tests/test_cli.py
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from io import StringIO
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from artenschutz_router.ingest import cli
|
||||
|
||||
|
||||
SAMPLE_BNATSCHG = """§ 44 Vorschriften
|
||||
|
||||
(1) Es ist verboten …
|
||||
|
||||
§ 45b Schutz wild lebender Vögel
|
||||
|
||||
(1) Beim Betrieb von WEA …
|
||||
"""
|
||||
|
||||
|
||||
def test_cli_dry_run_emits_json(tmp_path: Path, capsys: pytest.CaptureFixture):
|
||||
sources_root = tmp_path / "sources"
|
||||
bn_dir = sources_root / "bnatschg"
|
||||
bn_dir.mkdir(parents=True)
|
||||
(bn_dir / "bnatschg.txt").write_text(SAMPLE_BNATSCHG, encoding="utf-8")
|
||||
|
||||
rc = cli.main(
|
||||
[
|
||||
"--source",
|
||||
"bnatschg",
|
||||
"--sources-root",
|
||||
str(sources_root),
|
||||
"--dry-run",
|
||||
]
|
||||
)
|
||||
assert rc == 0
|
||||
out = capsys.readouterr().out.strip().splitlines()
|
||||
assert len(out) == 2
|
||||
for line in out:
|
||||
record = json.loads(line)
|
||||
assert record["payload"]["kind"] == "fact"
|
||||
assert record["payload"]["fact_type"] == "legal"
|
||||
assert record["content"]
|
||||
|
||||
|
||||
def test_cli_rejects_unknown_source(tmp_path: Path):
|
||||
with pytest.raises(SystemExit):
|
||||
cli.main(["--source", "nonsense", "--sources-root", str(tmp_path), "--dry-run"])
|
||||
30
tests/test_filter_builder.py
Normal file
30
tests/test_filter_builder.py
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from artenschutz_router.clients.bge import build_qdrant_filter
|
||||
|
||||
|
||||
def test_returns_none_for_empty():
|
||||
assert build_qdrant_filter(None) is None
|
||||
assert build_qdrant_filter({}) is None
|
||||
|
||||
|
||||
def test_scalar_match_value():
|
||||
f = build_qdrant_filter({"kind": "fact"})
|
||||
assert f == {"must": [{"key": "kind", "match": {"value": "fact"}}]}
|
||||
|
||||
|
||||
def test_list_match_any():
|
||||
f = build_qdrant_filter({"fact_type": ["legal", "mitigation"]})
|
||||
assert f == {"must": [{"key": "fact_type", "match": {"any": ["legal", "mitigation"]}}]}
|
||||
|
||||
|
||||
def test_drops_none_values():
|
||||
f = build_qdrant_filter({"a": "x", "b": None, "c": [1, 2]})
|
||||
must = f["must"]
|
||||
keys = {clause["key"] for clause in must}
|
||||
assert keys == {"a", "c"}
|
||||
|
||||
|
||||
def test_nested_dotted_keys_pass_through():
|
||||
f = build_qdrant_filter({"scope.states": ["BE", "BB"]})
|
||||
assert f["must"][0]["key"] == "scope.states"
|
||||
91
tests/test_models.py
Normal file
91
tests/test_models.py
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from datetime import date
|
||||
|
||||
import pytest
|
||||
from pydantic import ValidationError
|
||||
|
||||
from artenschutz_router.models import (
|
||||
ExemplarPayload,
|
||||
FactPayload,
|
||||
FactScope,
|
||||
IngestRequest,
|
||||
SearchRequest,
|
||||
SourceInfo,
|
||||
)
|
||||
|
||||
|
||||
def test_fact_payload_minimal():
|
||||
p = FactPayload(
|
||||
fact_type="legal",
|
||||
source=SourceInfo(type="authoritative", doc="BNatSchG"),
|
||||
)
|
||||
assert p.kind == "fact"
|
||||
assert p.scope.states == []
|
||||
assert p.valid_until is None
|
||||
|
||||
|
||||
def test_fact_payload_rich():
|
||||
p = FactPayload(
|
||||
fact_type="mitigation",
|
||||
source=SourceInfo(type="extracted", doc="GS_GA_BER_Foo_FachGA", year=2024, tier="A"),
|
||||
scope=FactScope(
|
||||
states=["BE"],
|
||||
anlagen_typ=["Neubau"],
|
||||
species=["passer_domesticus"],
|
||||
impact_types=["bauzeit"],
|
||||
),
|
||||
valid_from=date(2023, 4, 1),
|
||||
accepted_by_behoerde=True,
|
||||
citation_anchor="§44(1) Nr. 1",
|
||||
)
|
||||
dumped = p.model_dump(mode="json")
|
||||
assert dumped["valid_from"] == "2023-04-01"
|
||||
assert dumped["accepted_by_behoerde"] is True
|
||||
|
||||
|
||||
def test_fact_payload_rejects_unknown_fact_type():
|
||||
with pytest.raises(ValidationError):
|
||||
FactPayload(fact_type="bogus", source=SourceInfo(type="authoritative", doc="x"))
|
||||
|
||||
|
||||
def test_exemplar_payload_minimal():
|
||||
p = ExemplarPayload(
|
||||
project_slug="GS_GA_BER_Covivio_Hasenheide_94_FachGA",
|
||||
report_type="FachGA",
|
||||
tier="A",
|
||||
)
|
||||
assert p.kind == "exemplar"
|
||||
assert p.is_draft is False
|
||||
assert p.species_mentioned == []
|
||||
|
||||
|
||||
def test_ingest_request_discriminates_payload():
|
||||
fact_req = IngestRequest(
|
||||
content="§44 (1) BNatSchG …",
|
||||
payload={
|
||||
"kind": "fact",
|
||||
"fact_type": "legal",
|
||||
"source": {"type": "authoritative", "doc": "BNatSchG"},
|
||||
},
|
||||
)
|
||||
assert isinstance(fact_req.payload, FactPayload)
|
||||
|
||||
exemplar_req = IngestRequest(
|
||||
content="Im Untersuchungsraum wurden …",
|
||||
payload={
|
||||
"kind": "exemplar",
|
||||
"project_slug": "GS_GA_BER_Foo",
|
||||
"report_type": "FachGA",
|
||||
"tier": "A",
|
||||
},
|
||||
)
|
||||
assert isinstance(exemplar_req.payload, ExemplarPayload)
|
||||
|
||||
|
||||
def test_search_request_defaults():
|
||||
s = SearchRequest()
|
||||
assert s.query is None
|
||||
assert s.limit == 10
|
||||
assert s.prefetch_limit == 50
|
||||
assert s.metadata_filter is None
|
||||
Loading…
Reference in a new issue