Initial commit — phase 1 + 2 of artenschutz-digest concept

Phase 1: router skeleton + Qdrant read path
- FastAPI app with /health, /search/generic, /ingest endpoints
- Qdrant reader (hybrid dense+sparse w/ RRF, plus payload scroll)
- BGE-M3 TEI client for query-side embedding
- memory-api client proxying writes to /memory/store (m2-memory untouched)
- Pydantic Fact + Exemplar payloads, discriminated by `kind`
- Dockerfile + docker-compose joining the coolify Docker network

Phase 2: authoritative-source ingest pipeline
- PDF text extraction + paragraph-aware (§ N) and size chunkers
- Loaders for bnatschg, mhbasp, biotopwertliste, state-<slug>
- CLI: artenschutz-ingest --source <name> [--states ...] [--dry-run]
- Helper script to fetch BNatSchG from gesetze-im-internet.de
- End-to-end smoke script (scripts/smoke.sh) for Coolify validation

22/22 unit tests pass. Real-world dry-runs verified against mhbasp Anhang 4,
biotopwertlisteNEU and Berlin Kartierstandards PDFs from GST-DATA.

See COOLIFY-DEPLOY.md for staging deploy + smoke procedure.
This commit is contained in:
m2 (AI Agent) 2026-05-14 16:36:00 +02:00
commit 6da874b2e6
46 changed files with 1969 additions and 0 deletions

22
.env.example Normal file
View file

@ -0,0 +1,22 @@
# Qdrant + BGE-M3 — same services agent.memory.system uses.
# In the Coolify Docker network these resolve by container name.
# For local dev pointed at a remote stack, use container IPs from
# docker inspect or memory-api's resolved endpoints.
QDRANT_URL=http://memory-qdrant:6333
QDRANT_COLLECTION=agent_memory
BGE_TEI_URL=http://memory-embeddings:8000
# memory-api is only used for the WRITE path (/ingest → /memory/store).
# Reads go directly to Qdrant.
MEMORY_API_URL=http://memory-api:8000
# agent_id partition in Qdrant. Default scopes by client; override per deploy.
# Examples:
# gruenstifter — production client
# gruenstifter_staging — same client, staging env (cleanly deletable)
ARTENSCHUTZ_AGENT_ID=gruenstifter
# Service binding
ROUTER_HOST=0.0.0.0
ROUTER_PORT=8080
LOG_LEVEL=INFO

19
.gitignore vendored Normal file
View file

@ -0,0 +1,19 @@
__pycache__/
*.pyc
*.pyo
.pytest_cache/
.ruff_cache/
.venv/
venv/
.env
*.egg-info/
build/
dist/
# Local source dumps (real authoritative sources are large; keep .gitkeep markers)
sources/**/*.pdf
sources/**/*.docx
sources/**/*.txt
sources/**/*.html
!sources/**/.gitkeep
!sources/**/README.md

128
COOLIFY-DEPLOY.md Normal file
View file

@ -0,0 +1,128 @@
# Coolify staging deploy — artenschutz-router
Goal: stand up the router on Coolify staging, in the same Docker network as
`agent.memory.system`, so it can reach `memory-qdrant`, `memory-embeddings`,
and `memory-api` by container name. Then run an end-to-end smoke ingest +
search against real data.
## Pre-flight
You should already have on staging Coolify:
- A running `agent.memory.system` (memory-api + memory-qdrant + memory-embeddings)
- The Docker network name those services live on (probably `coolify` — check
`docker network ls` on the staging host)
Note down:
| Setting | Value (example) |
|---|---|
| Coolify project | `gruenstifter-staging` |
| Docker network | `coolify` |
| memory-api container name | `memory-api-<id>` |
| memory-qdrant container name | `memory-qdrant-<id>` |
| memory-embeddings container name | `memory-embeddings-<id>` |
> The *full* container names include Coolify's stack hash suffix. The
> compose file uses the short service names (`memory-api`, etc.) which work
> when the router is on the same Coolify-managed network — Coolify creates
> service-name aliases. If they don't resolve, fall back to container IPs
> via `docker inspect`.
## Path A — Push to forgejo, let Coolify auto-deploy (preferred)
```bash
cd /home/m2/artenschutz-router
git init -b main
git add .
git commit -m "Initial commit — phase 1 + 2"
git remote add origin <forgejo-url>:gruenstifter/artenschutz-router.git
git push -u origin main
```
In Coolify staging:
1. New Resource → **Application****Public Git Repository** (or Private,
add the SSH key)
2. Source: the forgejo URL above
3. Build Pack: **Dockerfile**
4. Set environment variables:
```
QDRANT_URL=http://memory-qdrant:6333
QDRANT_COLLECTION=agent_memory
BGE_TEI_URL=http://memory-embeddings:8000
MEMORY_API_URL=http://memory-api:8000
ARTENSCHUTZ_AGENT_ID=gruenstifter_staging
ROUTER_PORT=8080
LOG_LEVEL=INFO
```
`gruenstifter_staging` keeps smoke data cleanly separable from any future
production write.
5. Network: join the same network as `agent.memory.system` (usually `coolify`).
6. Port: 8080 (no public exposure needed for smoke — internal-only is fine).
7. Deploy.
## Path B — Docker Compose on the Coolify host
If you'd rather not push to forgejo yet:
```bash
# On the Coolify host (or via SSH):
scp -r /home/m2/artenschutz-router coolify-host:/opt/coolify-apps/
ssh coolify-host
cd /opt/coolify-apps/artenschutz-router
cp .env.example .env
# Edit .env to set ARTENSCHUTZ_AGENT_ID=gruenstifter_staging
docker compose build
docker compose up -d
docker compose logs -f artenschutz-router
```
This bypasses Coolify's UI but uses the same `coolify` external network the
memory stack joins.
## Verify it's up
From any shell on the Coolify host:
```bash
docker exec artenschutz-router curl -sS http://localhost:8080/health
# expect: {"status":"ok","qdrant":true,"bge":true,"memory_api":true}
```
`status: "degraded"` with one of the three `false` means that service isn't
reachable from inside the router's network namespace — fix container names /
network attachment before continuing.
## End-to-end smoke (uses the test data already symlinked in `sources/`)
```bash
# 1. Fetch BNatSchG into the container's sources/bnatschg/ if not already.
docker exec artenschutz-router python scripts/fetch_bnatschg.py
# 2. Run the smoke (ingests small batches + queries them back).
docker exec artenschutz-router bash scripts/smoke.sh
```
The smoke script ingests ~5 records per source, then runs three searches:
- Hybrid query `"Fledermäuse Kartierung"` filtered to `fact_type=method`
- Hybrid query `"besonders geschützte Arten"` filtered to `fact_type=legal`
- Payload-only scroll on `scope.states=BE` to verify state filtering
Expected: each search returns ≥1 hit. If hybrid returns 0 hits but scroll
works, BGE-M3 embedding is failing — check the logs.
## Cleanup (after smoke)
Drop the staging agent_id from Qdrant:
```bash
docker exec memory-qdrant-<id> curl -X POST \
http://localhost:6333/collections/agent_memory/points/delete \
-H 'Content-Type: application/json' \
-d '{"filter":{"must":[{"key":"agent_id","match":{"value":"gruenstifter_staging"}}]}}'
```
This wipes every point we wrote under the staging agent_id. Nothing else is
affected because partition isolation is by `agent_id`.

25
Dockerfile Normal file
View file

@ -0,0 +1,25 @@
FROM python:3.11-slim
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PIP_NO_CACHE_DIR=1
# curl: for smoke.sh + /health probes.
RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY pyproject.toml ./
COPY src ./src
COPY scripts ./scripts
COPY sources ./sources
RUN pip install --upgrade pip && pip install .
# sources/ is bind-mountable so an operator can drop files (esp. bnatschg.txt)
# without a rebuild.
VOLUME ["/app/sources"]
EXPOSE 8080
CMD ["uvicorn", "artenschutz_router.main:app", "--host", "0.0.0.0", "--port", "8080"]

67
README.md Normal file
View file

@ -0,0 +1,67 @@
# artenschutz-router
Sidecar service for Artenschutzgutachten domain queries against
`agent.memory.system`. **m2-memory stays untouched.**
- **Writes** proxy memory-api `/memory/store` — the existing endpoint accepts
arbitrary `metadata: dict`, so our Fact / Exemplar payloads slot in.
- **Reads** go directly to Qdrant for rich payload-filtered hybrid queries
(dense + sparse via the existing BGE-M3 TEI service).
See `.planning/notes/artenschutz-digest-concept.md` in `GST-DATA` for the
draft concept that birthed this.
## Endpoints
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/health` | Live-probe Qdrant, BGE-M3, memory-api |
| `POST` | `/search/generic` | Hybrid search (or scroll if no `query`); arbitrary payload filters |
| `POST` | `/ingest` | Push a Fact or Exemplar record |
Domain-specific endpoints (`/facts/search`, `/exemplar/search`,
`/mitigation/search`) are deferred to a later phase — they will be thin
wrappers over `/search/generic`.
## Quick start (local dev)
```bash
cp .env.example .env
# Edit .env to point QDRANT_URL / BGE_TEI_URL / MEMORY_API_URL at your stack
docker compose up --build
curl http://localhost:8080/health
```
If you're targeting the production Coolify memory stack, leave the default
container-name URLs and ensure the `coolify` Docker network is joined.
## Running ingest
```bash
# After you've placed authoritative source files into sources/...
artenschutz-ingest --source bnatschg --router-url http://localhost:8080
artenschutz-ingest --source mhbasp --router-url http://localhost:8080
artenschutz-ingest --source biotopwertliste --router-url http://localhost:8080
artenschutz-ingest --source state-berlin --router-url http://localhost:8080
```
See `sources/README.md` for which files each source expects.
## Tests
```bash
pip install -e .[dev]
pytest
```
## Architecture
```
m2-gpt agents
artenschutz-router (this repo)
├─ /ingest ─→ memory-api /memory/store (unchanged)
└─ /search/generic ─→ Qdrant /points/query + BGE-M3 TEI /embed[_sparse]
agent_memory collection (Qdrant)
```

24
docker-compose.yml Normal file
View file

@ -0,0 +1,24 @@
# Local dev compose. Joins the existing Coolify "coolify" network so it can
# resolve memory-qdrant / memory-embeddings / memory-api by container name.
# If those services aren't reachable, override the *_URL env vars in .env.
services:
artenschutz-router:
build: .
container_name: artenschutz-router
env_file:
- .env
ports:
- "8080:8080"
networks:
- coolify
restart: unless-stopped
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/health')"]
interval: 30s
timeout: 5s
retries: 3
networks:
coolify:
external: true

42
pyproject.toml Normal file
View file

@ -0,0 +1,42 @@
[project]
name = "artenschutz-router"
version = "0.1.0"
description = "Sidecar router for Artenschutz domain queries against agent.memory.system"
requires-python = ">=3.11"
readme = "README.md"
dependencies = [
"fastapi>=0.110",
"uvicorn[standard]>=0.27",
"pydantic>=2.6",
"pydantic-settings>=2.2",
"httpx>=0.27",
"qdrant-client>=1.9",
"pypdf>=4.0",
"python-dateutil>=2.9",
]
[project.optional-dependencies]
dev = [
"pytest>=8.0",
"pytest-asyncio>=0.23",
"ruff>=0.4",
]
[project.scripts]
artenschutz-ingest = "artenschutz_router.ingest.cli:main"
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
[tool.setuptools.packages.find]
where = ["src"]
[tool.pytest.ini_options]
asyncio_mode = "auto"
testpaths = ["tests"]
[tool.ruff]
line-length = 100
target-version = "py311"

112
scripts/fetch_bnatschg.py Executable file
View file

@ -0,0 +1,112 @@
#!/usr/bin/env python3
"""Fetch the consolidated text of BNatSchG §44 and §45b from
gesetze-im-internet.de and emit a plain-text file suitable for the
bnatschg ingest loader.
Usage:
python scripts/fetch_bnatschg.py [--out sources/bnatschg/bnatschg.txt]
The output uses one heading line per section ("§ N ..." or "Anlage N ...")
exactly as the bnatschg loader's paragraph splitter expects.
Best-effort HTML scrape. If gesetze-im-internet.de changes layout, paste
text manually instead (see sources/README.md).
"""
from __future__ import annotations
import argparse
import re
import sys
from html.parser import HTMLParser
from pathlib import Path
from urllib.request import Request, urlopen
# §s most relevant for Artenschutzgutachten. Add more as needed.
SECTIONS = [
("§ 44 Vorschriften für besonders geschützte und bestimmte andere Tier- und Pflanzenarten",
"https://www.gesetze-im-internet.de/bnatschg_2009/__44.html"),
("§ 45 Ausnahmen; Ermächtigung zum Erlass von Rechtsverordnungen",
"https://www.gesetze-im-internet.de/bnatschg_2009/__45.html"),
("§ 45b Schutz wild lebender Vogelarten bei der Errichtung und dem Betrieb von Windenergieanlagen an Land",
"https://www.gesetze-im-internet.de/bnatschg_2009/__45b.html"),
("Anlage 1 (zu § 45b Absatz 1 bis 5) Liste der bei der Errichtung und dem Betrieb von Windenergieanlagen an Land kollisionsgefährdeten Brutvogelarten",
"https://www.gesetze-im-internet.de/bnatschg_2009/anlage_1.html"),
]
class TextExtractor(HTMLParser):
def __init__(self) -> None:
super().__init__()
self._buf: list[str] = []
self._skip = 0 # depth of <script>/<style>
def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
if tag in {"script", "style"}:
self._skip += 1
elif tag in {"br", "p", "div", "li", "tr"}:
self._buf.append("\n")
def handle_endtag(self, tag: str) -> None:
if tag in {"script", "style"} and self._skip:
self._skip -= 1
elif tag in {"p", "div", "li", "tr"}:
self._buf.append("\n")
def handle_data(self, data: str) -> None:
if not self._skip:
self._buf.append(data)
def value(self) -> str:
raw = "".join(self._buf)
# Collapse excess blank lines while preserving paragraph breaks.
raw = re.sub(r"[ \t]+", " ", raw)
raw = re.sub(r"\n{3,}", "\n\n", raw)
return raw.strip()
def fetch(url: str, timeout: float = 30.0) -> str:
req = Request(url, headers={"User-Agent": "artenschutz-router/0.1 (+fetch_bnatschg)"})
with urlopen(req, timeout=timeout) as resp:
raw_bytes = resp.read()
# gesetze-im-internet.de serves ISO-8859-1; sniff from header if present.
charset = resp.headers.get_content_charset() or "iso-8859-1"
return raw_bytes.decode(charset, errors="replace")
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(prog="fetch_bnatschg")
parser.add_argument(
"--out",
type=Path,
default=Path("sources/bnatschg/bnatschg.txt"),
help="Output path (default: %(default)s)",
)
args = parser.parse_args(argv)
args.out.parent.mkdir(parents=True, exist_ok=True)
chunks: list[str] = []
for heading, url in SECTIONS:
print(f" fetching {heading.split('(')[0].strip()} ...", file=sys.stderr)
try:
html = fetch(url)
except Exception as exc: # noqa: BLE001
print(f" FAILED ({exc}) — skipping", file=sys.stderr)
continue
extractor = TextExtractor()
extractor.feed(html)
body = extractor.value()
# Anchor each section with the expected heading on its own line so the
# paragraph chunker recognises it.
chunks.append(f"{heading}\n\n{body}\n")
if not chunks:
print("No sections fetched — refusing to write empty output", file=sys.stderr)
return 2
args.out.write_text("\n\n".join(chunks), encoding="utf-8")
print(f"Wrote {args.out} ({sum(len(c) for c in chunks):,} chars)", file=sys.stderr)
return 0
if __name__ == "__main__":
sys.exit(main())

72
scripts/smoke.sh Executable file
View file

@ -0,0 +1,72 @@
#!/usr/bin/env bash
# End-to-end smoke for artenschutz-router against the live memory stack.
# Runs from inside the container; ROUTER_URL defaults to localhost:8080.
set -euo pipefail
ROUTER_URL="${ROUTER_URL:-http://localhost:8080}"
step() { printf '\n\033[1;36m== %s ==\033[0m\n' "$*"; }
pass() { printf '\033[1;32m ✓ %s\033[0m\n' "$*"; }
fail() { printf '\033[1;31m ✗ %s\033[0m\n' "$*"; exit 1; }
step "1/5 health"
health=$(curl -sS "$ROUTER_URL/health")
echo " $health"
echo "$health" | grep -q '"status":"ok"' || fail "health is not ok — fix services first"
pass "router can reach Qdrant + BGE-M3 + memory-api"
ingest_source() {
local source="$1" ; shift
local limit="${1:-5}"
step "ingest --source $source --limit $limit"
python -m artenschutz_router.ingest.cli \
--source "$source" \
--router-url "$ROUTER_URL" \
--limit "$limit" "$@" \
2>&1 | tail -5
}
step "2/5 fetch BNatSchG if missing"
if [ ! -s sources/bnatschg/bnatschg.txt ]; then
python scripts/fetch_bnatschg.py
fi
step "3/5 ingest small batches"
ingest_source bnatschg 5
ingest_source mhbasp 5
ingest_source biotopwertliste 3 --states BY
ingest_source state-berlin 5
step "4/5 search — hybrid, fact_type=method"
result=$(curl -sS -X POST "$ROUTER_URL/search/generic" \
-H 'Content-Type: application/json' \
-d '{"query":"Fledermäuse Kartierung","metadata_filter":{"fact_type":"method"},"limit":3}')
hits=$(echo "$result" | python -c "import sys,json; print(len(json.load(sys.stdin)['hits']))")
echo " hits: $hits"
[ "$hits" -ge 1 ] || fail "hybrid search returned 0 hits — check BGE-M3"
pass "hybrid retrieval works"
step "4b/5 search — hybrid, fact_type=legal"
result=$(curl -sS -X POST "$ROUTER_URL/search/generic" \
-H 'Content-Type: application/json' \
-d '{"query":"besonders geschützte Arten","metadata_filter":{"fact_type":"legal"},"limit":3}')
hits=$(echo "$result" | python -c "import sys,json; print(len(json.load(sys.stdin)['hits']))")
echo " hits: $hits"
[ "$hits" -ge 1 ] || fail "legal hybrid search returned 0 hits"
pass "BNatSchG indexed and queryable"
step "5/5 scroll — scope.states contains BE"
result=$(curl -sS -X POST "$ROUTER_URL/search/generic" \
-H 'Content-Type: application/json' \
-d '{"metadata_filter":{"scope.states":["BE"]},"limit":3}')
hits=$(echo "$result" | python -c "import sys,json; print(len(json.load(sys.stdin)['hits']))")
echo " hits: $hits"
[ "$hits" -ge 1 ] || fail "scope.states=BE scroll returned 0 hits — Berlin ingest missed?"
pass "payload filtering on nested keys works"
printf '\n\033[1;32mALL SMOKE STEPS PASSED\033[0m\n'
echo
echo "To clean up the staging data later:"
echo " curl -X POST <qdrant>:6333/collections/agent_memory/points/delete \\"
echo " -H 'Content-Type: application/json' \\"
echo " -d '{\"filter\":{\"must\":[{\"key\":\"agent_id\",\"match\":{\"value\":\"gruenstifter_staging\"}}]}}'"

22
sources/README.md Normal file
View file

@ -0,0 +1,22 @@
# Authoritative source files
The ingest pipeline reads from this tree. Files are git-ignored (large + may
have licensing constraints) — keep originals here on each operator's machine.
| Subdir | Expected files | Notes |
|---|---|---|
| `bnatschg/` | `bnatschg.txt` (current consolidated text of §44, §45b, Anlagen) | Fetch from <https://www.gesetze-im-internet.de/bnatschg_2009/> — paste as UTF-8 plain text. One `§ N ...` per heading line. Cross-reference: a `.webloc` link to <https://www.buzer.de/Anlage_1_BNatSchG.htm> already lives in `GST-DATA/xx_GS_VORLAGEN_FachGA/`. |
| `mhbasp/` | `mhbasp_anhang4.pdf` | "Methodische Hinweise und Behandlungsempfehlungen ASP" Anhang 4 — species × method matrix |
| `biotopwertliste/` | `biotopwertliste.pdf` | A biotope value list — **always state-specific**. Pass `--states BE` (or BB, BY, …) when ingesting. State-specific lists are also welcome under `states/<slug>/`; the generic slot here is for one-off cases. |
| `states/berlin/` | One or more state-specific PDFs (Kartierstandards, methodische Hinweise) | First state implemented as proof-of-concept |
| `states/brandenburg/`<br>`states/bayern/`<br>`states/baden_wuerttemberg/`<br>`states/sachsen/` | TODO | Add as ingest support is implemented per state |
A copy of `mhbasp_anhang4_artspezifisch geeignete kartiermethoden.pdf` already
lives under `GST-DATA/xx_GS_VORLAGEN_FachGA/`. Symlinking is fine:
```bash
ln -s "/home/m2/clients/SDJS/GST-DATA/xx_GS_VORLAGEN_FachGA/mhbasp_anhang4_artspezifisch geeignete kartiermethoden.pdf" \
sources/mhbasp/mhbasp_anhang4.pdf
ln -s "/home/m2/clients/SDJS/GST-DATA/xx_GS_VORLAGEN_FachGA/biotopwertlisteNEU.pdf" \
sources/biotopwertliste/biotopwertliste.pdf
```

View file

View file

0
sources/mhbasp/.gitkeep Normal file
View file

View file

View file

View file

View file

View file

@ -0,0 +1,3 @@
"""Artenschutz-Router — sidecar service for Artenschutz domain queries."""
__version__ = "0.1.0"

View file

@ -0,0 +1 @@
"""External service clients: Qdrant (direct), BGE-M3 TEI, memory-api."""

View file

@ -0,0 +1,74 @@
from __future__ import annotations
import logging
from typing import Any
import httpx
logger = logging.getLogger(__name__)
class BGEClient:
"""Thin async client for the BGE-M3 TEI server.
Only exposes what the router needs for query-side embedding: dense vector
and sparse vector. ColBERT is left to memory-api's own pipeline if ever
required.
"""
def __init__(self, base_url: str, timeout_s: float = 30.0) -> None:
self._base_url = base_url.rstrip("/")
self._client = httpx.AsyncClient(base_url=self._base_url, timeout=timeout_s)
async def close(self) -> None:
await self._client.aclose()
async def embed_dense(self, text: str) -> list[float]:
resp = await self._client.post("/embed", json={"inputs": text})
resp.raise_for_status()
data = resp.json()
# TEI returns list[list[float]] for batched, list[float] for single.
if isinstance(data, list) and data and isinstance(data[0], list):
return data[0]
return data # type: ignore[return-value]
async def embed_sparse(self, text: str) -> dict[int, float]:
resp = await self._client.post("/embed_sparse", json={"inputs": text})
resp.raise_for_status()
data = resp.json()
# TEI returns either {"sparse": [{"index": int, "value": float}, ...]}
# or directly a list of those, depending on version.
sparse_list = data[0] if isinstance(data, list) and data and isinstance(data[0], list) else data
out: dict[int, float] = {}
for item in sparse_list:
if isinstance(item, dict) and "index" in item and "value" in item:
out[int(item["index"])] = float(item["value"])
return out
async def embed_hybrid(self, text: str) -> tuple[list[float], dict[int, float]]:
dense = await self.embed_dense(text)
sparse = await self.embed_sparse(text)
return dense, sparse
def build_qdrant_filter(metadata_filter: dict[str, Any] | None) -> dict[str, Any] | None:
"""Translate a flat `{key: value | [values]}` dict into a Qdrant `must` filter.
None values are skipped. List values become `match: {any: [...]}` clauses.
Scalar values become `match: {value: x}` clauses.
Supports nested keys via dotted form, e.g. `scope.states`.
"""
if not metadata_filter:
return None
must: list[dict[str, Any]] = []
for key, value in metadata_filter.items():
if value is None:
continue
if isinstance(value, list):
must.append({"key": key, "match": {"any": value}})
else:
must.append({"key": key, "match": {"value": value}})
if not must:
return None
return {"must": must}

View file

@ -0,0 +1,57 @@
from __future__ import annotations
import logging
from typing import Any
import httpx
logger = logging.getLogger(__name__)
class MemoryApiClient:
"""Write-path proxy to agent.memory.system's /memory/store endpoint.
The router does NOT touch m2-memory source code. The fact/exemplar
payload travels as the free-form `metadata` dict that /memory/store
already accepts.
"""
def __init__(self, base_url: str, timeout_s: float = 30.0) -> None:
self._base_url = base_url.rstrip("/")
self._client = httpx.AsyncClient(base_url=self._base_url, timeout=timeout_s)
async def close(self) -> None:
await self._client.aclose()
async def store(
self,
*,
content: str,
agent_id: str,
memory_type: str = "fact",
importance: float = 0.5,
source: str = "document",
entities: list[str] | None = None,
language: str = "de",
metadata: dict[str, Any] | None = None,
) -> str:
body = {
"content": content,
"agent_id": agent_id,
"memory_type": memory_type,
"importance": importance,
"source": source,
"entities": entities or [],
"language": language,
"metadata": metadata or {},
}
resp = await self._client.post("/memory/store", json=body)
resp.raise_for_status()
return resp.json()["id"]
async def health(self) -> bool:
try:
resp = await self._client.get("/health")
return resp.status_code == 200
except httpx.HTTPError:
return False

View file

@ -0,0 +1,112 @@
from __future__ import annotations
import logging
from typing import Any
from qdrant_client import AsyncQdrantClient
from qdrant_client.http import models as qm
logger = logging.getLogger(__name__)
class QdrantReader:
"""Read-only Qdrant wrapper for hybrid + payload-filter queries.
Writes are NOT exposed here on purpose they go through memory-api
so we stay aligned with whatever ingest pipeline / dedup / vector
schema the canonical memory service enforces.
"""
def __init__(self, url: str, collection: str) -> None:
self._client = AsyncQdrantClient(url=url, check_compatibility=False)
self._collection = collection
async def close(self) -> None:
await self._client.close()
def _to_filter(self, raw: dict[str, Any] | None) -> qm.Filter | None:
if not raw:
return None
must = raw.get("must", [])
if not must:
return None
conditions: list[qm.FieldCondition] = []
for clause in must:
key = clause["key"]
match = clause["match"]
if "any" in match:
conditions.append(
qm.FieldCondition(key=key, match=qm.MatchAny(any=match["any"]))
)
else:
conditions.append(
qm.FieldCondition(key=key, match=qm.MatchValue(value=match["value"]))
)
return qm.Filter(must=conditions)
async def hybrid_query(
self,
*,
agent_id: str,
dense: list[float],
sparse: dict[int, float],
metadata_filter: dict[str, Any] | None,
limit: int = 10,
prefetch_limit: int = 50,
) -> list[dict[str, Any]]:
"""Dense + sparse hybrid with RRF fusion, scoped to agent_id."""
combined: dict[str, Any] = {"must": [{"key": "agent_id", "match": {"value": agent_id}}]}
if metadata_filter and "must" in metadata_filter:
combined["must"].extend(metadata_filter["must"])
qfilter = self._to_filter(combined)
sparse_vec = qm.SparseVector(
indices=list(sparse.keys()),
values=list(sparse.values()),
)
result = await self._client.query_points(
collection_name=self._collection,
prefetch=[
qm.Prefetch(query=dense, using="dense", limit=prefetch_limit, filter=qfilter),
qm.Prefetch(query=sparse_vec, using="sparse", limit=prefetch_limit, filter=qfilter),
],
query=qm.FusionQuery(fusion=qm.Fusion.RRF),
query_filter=qfilter,
limit=limit,
with_payload=True,
with_vectors=False,
)
return [self._format_point(p) for p in result.points]
async def scroll(
self,
*,
agent_id: str,
metadata_filter: dict[str, Any] | None,
limit: int = 100,
) -> list[dict[str, Any]]:
"""Pure payload-filter retrieval (no vector ranking)."""
combined: dict[str, Any] = {"must": [{"key": "agent_id", "match": {"value": agent_id}}]}
if metadata_filter and "must" in metadata_filter:
combined["must"].extend(metadata_filter["must"])
qfilter = self._to_filter(combined)
points, _next = await self._client.scroll(
collection_name=self._collection,
scroll_filter=qfilter,
limit=limit,
with_payload=True,
with_vectors=False,
)
return [self._format_point(p) for p in points]
@staticmethod
def _format_point(p: Any) -> dict[str, Any]:
return {
"id": str(p.id),
"score": getattr(p, "score", None),
"payload": p.payload or {},
}

View file

@ -0,0 +1,29 @@
from __future__ import annotations
from functools import lru_cache
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore")
qdrant_url: str = "http://memory-qdrant:6333"
qdrant_collection: str = "agent_memory"
bge_tei_url: str = "http://memory-embeddings:8000"
memory_api_url: str = "http://memory-api:8000"
# Default agent_id is the client name — keeps the partition tenant-scoped
# rather than domain-scoped. Multi-client deployments override per-instance.
artenschutz_agent_id: str = "gruenstifter"
router_host: str = "0.0.0.0"
router_port: int = 8080
log_level: str = "INFO"
request_timeout_s: float = 30.0
@lru_cache(maxsize=1)
def get_settings() -> Settings:
return Settings()

View file

@ -0,0 +1 @@
"""Ingest pipelines for authoritative sources and (later) past Gutachten."""

View file

@ -0,0 +1,10 @@
"""Loaders for authoritative legal/methodological sources.
Each loader takes a Path to its source file(s) and yields `IngestRecord`
instances. The CLI consumes those records and POSTs them to the router's
`/ingest` endpoint.
"""
from .base import IngestRecord
__all__ = ["IngestRecord"]

View file

@ -0,0 +1,30 @@
from __future__ import annotations
from dataclasses import dataclass
from ...models import FactPayload
@dataclass(frozen=True)
class IngestRecord:
"""One record ready to be POSTed to /ingest.
`content` is the embedded text. `payload` carries the structured
metadata that lands in Qdrant payload.
"""
content: str
payload: FactPayload
importance: float = 0.5
language: str = "de"
def to_request_body(self, agent_id: str | None = None) -> dict:
body = {
"content": self.content,
"payload": self.payload.model_dump(mode="json"),
"importance": self.importance,
"language": self.language,
}
if agent_id is not None:
body["agent_id"] = agent_id
return body

View file

@ -0,0 +1,53 @@
from __future__ import annotations
import logging
from collections.abc import Iterator
from pathlib import Path
from ...models import FactPayload, FactScope, SourceInfo
from ..chunker import chunk_by_size
from ..pdf_text import extract_pdf_pages
from .base import IngestRecord
logger = logging.getLogger(__name__)
DEFAULT_PATH = Path("sources/biotopwertliste/biotopwertliste.pdf")
# NO default states — every Bundesland publishes its own Biotopwertliste
# with different codes (Berlin's, BayKompV in Bayern, etc.). Operator must
# tag the file with the right state(s) via the --states CLI flag, or place
# state-specific lists under sources/states/<slug>/ for the state_docs loader.
DEFAULT_STATES: list[str] = []
def load(path: Path = DEFAULT_PATH, *, states: list[str] | None = None) -> Iterator[IngestRecord]:
"""Yield biotope-code records from a Biotopwertliste PDF.
`states` should be set when the file is state-specific. Leave empty only
when the document is genuinely federal/cross-state most Biotopwertlisten
are NOT.
"""
if not path.exists():
raise FileNotFoundError(
f"Biotopwertliste source not found at {path}. "
"See sources/README.md."
)
pages = extract_pdf_pages(path)
if not any(pages):
logger.warning("No text extracted from %s", path)
return
scope_states = states if states is not None else DEFAULT_STATES
for page_idx, page_text in enumerate(pages):
if not page_text.strip():
continue
for chunk in chunk_by_size(page_text, target_chars=1800, overlap_chars=100):
payload = FactPayload(
fact_type="biotope",
source=SourceInfo(type="authoritative", doc="biotopwertliste"),
scope=FactScope(states=list(scope_states)),
citation_anchor=f"Biotopwertliste · Seite {page_idx + 1}",
)
yield IngestRecord(content=chunk.text, payload=payload, importance=0.75)

View file

@ -0,0 +1,58 @@
from __future__ import annotations
import logging
import re
from collections.abc import Iterator
from datetime import date
from pathlib import Path
from ...models import FactPayload, FactScope, SourceInfo
from ..chunker import chunk_by_paragraph_marker
from .base import IngestRecord
logger = logging.getLogger(__name__)
DEFAULT_PATH = Path("sources/bnatschg/bnatschg.txt")
def load(path: Path = DEFAULT_PATH, *, valid_from: date | None = None) -> Iterator[IngestRecord]:
"""Yield FactPayload records from a plain-text BNatSchG dump.
Expected format: UTF-8 text with `§ N ...` headings (and optionally
`Anlage N ...`). Sections in between become the chunk body.
Citations are anchored to the heading line.
BNatSchG is **federal** legislation `scope.states` stays empty so
every state's query matches.
"""
if not path.exists():
raise FileNotFoundError(
f"BNatSchG source not found at {path}. "
"See sources/README.md for how to populate it."
)
text = path.read_text(encoding="utf-8")
chunks = chunk_by_paragraph_marker(text)
if not chunks:
logger.warning("No § markers detected in %s — emitting nothing", path)
return
for chunk in chunks:
anchor = _normalise_anchor(chunk.anchor) if chunk.anchor else None
payload = FactPayload(
fact_type="legal",
source=SourceInfo(type="authoritative", doc="BNatSchG"),
scope=FactScope(),
valid_from=valid_from,
citation_anchor=anchor,
)
yield IngestRecord(content=chunk.text, payload=payload, importance=0.9)
_ANCHOR_NORMALISE_RE = re.compile(r"\s+")
def _normalise_anchor(raw: str) -> str:
"""Trim and collapse whitespace; keep just the heading line."""
line = raw.splitlines()[0] if raw else ""
return _ANCHOR_NORMALISE_RE.sub(" ", line).strip()

View file

@ -0,0 +1,49 @@
from __future__ import annotations
import logging
from collections.abc import Iterator
from pathlib import Path
from ...models import FactPayload, FactScope, SourceInfo
from ..chunker import chunk_by_size
from ..pdf_text import extract_pdf_pages
from .base import IngestRecord
logger = logging.getLogger(__name__)
DEFAULT_PATH = Path("sources/mhbasp/mhbasp_anhang4.pdf")
def load(path: Path = DEFAULT_PATH) -> Iterator[IngestRecord]:
"""Yield method-fact records from mhbasp Anhang 4.
Anhang 4 documents species-specific survey methods ("artspezifisch
geeignete Kartiermethoden"). PDF text extraction is noisy and the
document is table-heavy; v1 ingests page-level chunks with size-bound
splits inside long pages. A later phase can replace this with a
proper table parser.
Like BNatSchG, mhbasp is federal `scope.states` stays empty.
"""
if not path.exists():
raise FileNotFoundError(
f"mhbasp source not found at {path}. "
"See sources/README.md (a symlink into GST-DATA is the quickest path)."
)
pages = extract_pdf_pages(path)
if not any(pages):
logger.warning("No text extracted from %s", path)
return
for page_idx, page_text in enumerate(pages):
if not page_text.strip():
continue
for chunk in chunk_by_size(page_text, target_chars=1500, overlap_chars=150):
payload = FactPayload(
fact_type="method",
source=SourceInfo(type="authoritative", doc="mhbasp_anhang4"),
scope=FactScope(),
citation_anchor=f"mhbasp Anhang 4 · Seite {page_idx + 1}",
)
yield IngestRecord(content=chunk.text, payload=payload, importance=0.85)

View file

@ -0,0 +1,54 @@
from __future__ import annotations
import logging
from collections.abc import Iterator
from pathlib import Path
from ...models import FactPayload, FactScope, SourceInfo
from ..chunker import chunk_by_size
from ..pdf_text import extract_pdf_pages
from .base import IngestRecord
logger = logging.getLogger(__name__)
# Map of state slug → (Bundesland code, source-dir name).
STATES: dict[str, tuple[str, str]] = {
"berlin": ("BE", "berlin"),
"brandenburg": ("BB", "brandenburg"),
"bayern": ("BY", "bayern"),
"baden_wuerttemberg": ("BW", "baden_wuerttemberg"),
"sachsen": ("SN", "sachsen"),
}
def load(state_slug: str, sources_root: Path = Path("sources/states")) -> Iterator[IngestRecord]:
"""Yield method-fact records from all PDFs under sources/states/<slug>/."""
if state_slug not in STATES:
raise ValueError(
f"Unknown state '{state_slug}'. Known: {', '.join(sorted(STATES))}"
)
state_code, dirname = STATES[state_slug]
state_dir = sources_root / dirname
if not state_dir.exists():
raise FileNotFoundError(f"State source dir not found: {state_dir}")
pdfs = sorted(state_dir.glob("*.pdf"))
if not pdfs:
logger.warning("No PDFs under %s — nothing to ingest for state %s", state_dir, state_slug)
return
for pdf in pdfs:
logger.info("Ingesting %s for state %s", pdf.name, state_code)
pages = extract_pdf_pages(pdf)
for page_idx, page_text in enumerate(pages):
if not page_text.strip():
continue
for chunk in chunk_by_size(page_text, target_chars=1500, overlap_chars=150):
payload = FactPayload(
fact_type="method",
source=SourceInfo(type="authoritative", doc=pdf.stem),
scope=FactScope(states=[state_code]),
citation_anchor=f"{pdf.name} · Seite {page_idx + 1}",
)
yield IngestRecord(content=chunk.text, payload=payload, importance=0.8)

View file

@ -0,0 +1,93 @@
from __future__ import annotations
import re
from collections.abc import Iterator
from dataclasses import dataclass
@dataclass(frozen=True)
class Chunk:
"""A chunk of text plus a heading anchor (e.g. '§ 44' or section title)."""
anchor: str | None
text: str
_PARAGRAPH_RE = re.compile(r"^[ \t]*§[ \t]*\d+\w*", re.MULTILINE)
_ANLAGE_RE = re.compile(r"^[ \t]*Anlage[ \t]+\d+", re.MULTILINE)
_HEADING_RE = re.compile(r"^(\d+(\.\d+)*\s+\S.*|[A-ZÄÖÜ][^\n]{0,80})$", re.MULTILINE)
def chunk_by_paragraph_marker(text: str) -> list[Chunk]:
"""Split a legal text by `§ N` (and `Anlage N`) markers.
Each chunk starts at a marker line and runs until the next marker.
The marker line itself becomes the chunk anchor.
"""
# Find all marker positions (paragraph + anlage).
starts: list[tuple[int, str]] = []
for m in _PARAGRAPH_RE.finditer(text):
starts.append((m.start(), _first_line(text, m.start())))
for m in _ANLAGE_RE.finditer(text):
starts.append((m.start(), _first_line(text, m.start())))
starts.sort()
if not starts:
return [Chunk(anchor=None, text=text.strip())] if text.strip() else []
chunks: list[Chunk] = []
for idx, (pos, anchor) in enumerate(starts):
end = starts[idx + 1][0] if idx + 1 < len(starts) else len(text)
body = text[pos:end].strip()
if body:
chunks.append(Chunk(anchor=anchor.strip(), text=body))
return chunks
def chunk_by_size(text: str, *, target_chars: int = 1500, overlap_chars: int = 200) -> list[Chunk]:
"""Sliding-window chunker for noisy text (PDF extracts). Breaks on
paragraph boundaries when possible."""
text = text.strip()
if not text:
return []
paragraphs = re.split(r"\n\s*\n", text)
chunks: list[Chunk] = []
buf: list[str] = []
buf_len = 0
def flush():
if buf:
chunks.append(Chunk(anchor=None, text="\n\n".join(buf).strip()))
for para in paragraphs:
para = para.strip()
if not para:
continue
if buf_len + len(para) + 2 > target_chars and buf:
flush()
# carry overlap
if overlap_chars > 0 and buf:
tail = buf[-1][-overlap_chars:]
buf = [tail]
buf_len = len(tail)
else:
buf = []
buf_len = 0
buf.append(para)
buf_len += len(para) + 2
flush()
return chunks
def iter_section_chunks(text: str, *, target_chars: int = 1500) -> Iterator[Chunk]:
"""Generator wrapper for size-based chunking."""
yield from chunk_by_size(text, target_chars=target_chars)
def _first_line(text: str, pos: int) -> str:
end = text.find("\n", pos)
if end == -1:
end = min(len(text), pos + 120)
return text[pos:end]

View file

@ -0,0 +1,136 @@
from __future__ import annotations
import argparse
import json
import logging
import sys
from collections.abc import Iterator
from pathlib import Path
import httpx
from .authoritative import IngestRecord
from .authoritative import bnatschg as bnatschg_loader
from .authoritative import biotopwertliste as biotop_loader
from .authoritative import mhbasp as mhbasp_loader
from .authoritative import state_docs as state_loader
logger = logging.getLogger(__name__)
_SOURCE_HELP = (
"Source slug. One of: bnatschg, mhbasp, biotopwertliste, state-<slug> "
f"(states: {', '.join(sorted(state_loader.STATES))})."
)
def _load_records(
source: str, sources_root: Path, *, states: list[str] | None = None
) -> Iterator[IngestRecord]:
if source == "bnatschg":
yield from bnatschg_loader.load(sources_root / "bnatschg" / "bnatschg.txt")
return
if source == "mhbasp":
yield from mhbasp_loader.load(sources_root / "mhbasp" / "mhbasp_anhang4.pdf")
return
if source == "biotopwertliste":
yield from biotop_loader.load(
sources_root / "biotopwertliste" / "biotopwertliste.pdf", states=states
)
return
if source.startswith("state-"):
slug = source[len("state-") :]
yield from state_loader.load(slug, sources_root=sources_root / "states")
return
raise SystemExit(f"Unknown --source '{source}'. {_SOURCE_HELP}")
def _post_record(client: httpx.Client, router_url: str, record: IngestRecord) -> str:
resp = client.post(
f"{router_url.rstrip('/')}/ingest",
json=record.to_request_body(),
)
resp.raise_for_status()
return resp.json()["id"]
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="artenschutz-ingest",
description="Ingest authoritative Artenschutz sources via the router /ingest endpoint.",
)
parser.add_argument("--source", required=True, help=_SOURCE_HELP)
parser.add_argument(
"--router-url",
default="http://localhost:8080",
help="Base URL of the running artenschutz-router (default: %(default)s)",
)
parser.add_argument(
"--sources-root",
default="sources",
type=Path,
help="Root dir holding source files (default: %(default)s)",
)
parser.add_argument(
"--dry-run",
action="store_true",
help="Print records as JSON, don't POST to the router.",
)
parser.add_argument(
"--limit", type=int, default=0, help="Stop after N records (0 = no limit)."
)
parser.add_argument(
"--states",
nargs="*",
default=None,
help="Override `scope.states` for sources that need a state tag (currently: biotopwertliste).",
)
parser.add_argument(
"--log-level", default="INFO", help="Logging level (default: %(default)s)"
)
args = parser.parse_args(argv)
logging.basicConfig(
level=getattr(logging, args.log_level.upper(), logging.INFO),
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
)
records = _load_records(args.source, args.sources_root, states=args.states)
if args.limit > 0:
records = _take(records, args.limit)
if args.dry_run:
count = 0
for r in records:
print(json.dumps(r.to_request_body(), ensure_ascii=False))
count += 1
logger.info("Dry-run produced %d records", count)
return 0
posted = 0
errors = 0
with httpx.Client(timeout=60.0) as client:
for record in records:
try:
memory_id = _post_record(client, args.router_url, record)
logger.debug("Stored %s (anchor=%s)", memory_id, record.payload.citation_anchor)
posted += 1
except httpx.HTTPError as exc:
logger.error("POST failed for record: %s", exc)
errors += 1
if errors > 5 and posted == 0:
logger.error("Aborting after %d consecutive failures with no successes", errors)
return 2
logger.info("Posted %d records (errors: %d)", posted, errors)
return 0 if errors == 0 else 1
def _take(iterable, n: int):
for i, x in enumerate(iterable):
if i >= n:
return
yield x
if __name__ == "__main__":
sys.exit(main())

View file

@ -0,0 +1,27 @@
from __future__ import annotations
import logging
from pathlib import Path
from pypdf import PdfReader
logger = logging.getLogger(__name__)
def extract_pdf_pages(path: Path) -> list[str]:
"""Return page texts. Empty strings for pages where extraction failed."""
reader = PdfReader(str(path))
pages: list[str] = []
for i, page in enumerate(reader.pages):
try:
text = page.extract_text() or ""
except Exception as exc: # noqa: BLE001
logger.warning("Page %d of %s failed text extraction: %s", i + 1, path.name, exc)
text = ""
pages.append(text.strip())
return pages
def extract_pdf_text(path: Path) -> str:
"""Concatenated page text with double-newline separators."""
return "\n\n".join(p for p in extract_pdf_pages(path) if p)

View file

@ -0,0 +1,93 @@
from __future__ import annotations
import logging
from contextlib import asynccontextmanager
from fastapi import FastAPI
from .clients.bge import BGEClient
from .clients.memory_api import MemoryApiClient
from .clients.qdrant import QdrantReader
from .config import get_settings
from .models import HealthResponse
from .routes import ingest as ingest_routes
from .routes import search as search_routes
@asynccontextmanager
async def lifespan(app: FastAPI):
settings = get_settings()
logging.basicConfig(
level=getattr(logging, settings.log_level.upper(), logging.INFO),
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
)
logger = logging.getLogger(__name__)
logger.info("Starting artenschutz-router")
logger.info(" Qdrant: %s (collection: %s)", settings.qdrant_url, settings.qdrant_collection)
logger.info(" BGE TEI: %s", settings.bge_tei_url)
logger.info(" memory-api: %s", settings.memory_api_url)
logger.info(" agent_id: %s", settings.artenschutz_agent_id)
qdrant = QdrantReader(url=settings.qdrant_url, collection=settings.qdrant_collection)
bge = BGEClient(base_url=settings.bge_tei_url, timeout_s=settings.request_timeout_s)
memory_api = MemoryApiClient(base_url=settings.memory_api_url, timeout_s=settings.request_timeout_s)
app.state.settings = settings
app.state.qdrant = qdrant
app.state.bge = bge
app.state.memory_api = memory_api
try:
yield
finally:
await qdrant.close()
await bge.close()
await memory_api.close()
app = FastAPI(
title="artenschutz-router",
description=(
"Sidecar router for Artenschutz domain queries against agent.memory.system. "
"Writes proxy memory-api /memory/store; reads go directly to Qdrant."
),
version="0.1.0",
lifespan=lifespan,
)
app.include_router(search_routes.router)
app.include_router(ingest_routes.router)
@app.get("/health", response_model=HealthResponse, tags=["health"])
async def health() -> HealthResponse:
"""Live-probes Qdrant, BGE-M3 TEI, memory-api. `degraded` if any fail."""
qdrant_ok = True
bge_ok = True
memory_api_ok = False
try:
await app.state.qdrant._client.get_collections() # noqa: SLF001
except Exception:
qdrant_ok = False
try:
# TEI exposes /health; falling back to /info on older versions.
resp = await app.state.bge._client.get("/health") # noqa: SLF001
bge_ok = resp.status_code == 200
except Exception:
bge_ok = False
try:
memory_api_ok = await app.state.memory_api.health()
except Exception:
memory_api_ok = False
overall = "ok" if all([qdrant_ok, bge_ok, memory_api_ok]) else "degraded"
return HealthResponse(
status=overall,
qdrant=qdrant_ok,
bge=bge_ok,
memory_api=memory_api_ok,
)

View file

@ -0,0 +1,142 @@
from __future__ import annotations
from datetime import date
from typing import Annotated, Any, Literal
from pydantic import BaseModel, ConfigDict, Field
# ---------------------------------------------------------------------------
# Payload schemas — what travels in Qdrant payload via the `metadata` field
# on memory-api /memory/store.
# ---------------------------------------------------------------------------
class SourceInfo(BaseModel):
"""Where a memory came from."""
type: Literal["authoritative", "extracted"]
doc: str = Field(..., description="Short slug — e.g. 'BNatSchG', 'mhbasp_anhang4', or project_slug")
year: int | None = None
tier: Literal["A", "B", "C"] | None = None
class FactScope(BaseModel):
"""Filterable scope dimensions for facts."""
states: list[str] = Field(default_factory=list, description="DE Bundesland codes, e.g. ['BE', 'BB']")
anlagen_typ: list[str] = Field(default_factory=list, description="['WEA', 'PV', 'Neubau', 'Eingriff', ...]")
species: list[str] = Field(default_factory=list, description="GBIF/Faunistik IDs or scientific names")
impact_types: list[str] = Field(default_factory=list, description="['lichtemission', 'rotor', 'bauzeit', ...]")
FactType = Literal["legal", "method", "biotope", "mitigation", "verdict"]
class FactPayload(BaseModel):
"""Structured fact record (lives in Qdrant payload)."""
model_config = ConfigDict(extra="forbid")
kind: Literal["fact"] = "fact"
fact_type: FactType
source: SourceInfo
scope: FactScope = Field(default_factory=FactScope)
valid_from: date | None = None
valid_until: date | None = None
accepted_by_behoerde: bool | None = None
supersedes: str | None = None
citation_anchor: str | None = Field(
default=None, description="Human-readable anchor, e.g. 'BNatSchG §44(1) Nr. 1'"
)
ReportType = Literal["AFB", "FachGA", "MBKS", "ASB", "LBP", "KONZ", "Other"]
class ExemplarPayload(BaseModel):
"""Prose chunk from a past Gutachten (lives in Qdrant payload)."""
model_config = ConfigDict(extra="forbid")
kind: Literal["exemplar"] = "exemplar"
project_slug: str
report_type: ReportType
state: str | None = None
anlagen_typ: str | None = None
year: int | None = None
tier: Literal["A", "B", "C"]
section: str | None = Field(default=None, description="Canonical section name from template")
species_mentioned: list[str] = Field(default_factory=list)
source_path: str | None = None
is_draft: bool = False
Payload = Annotated[FactPayload | ExemplarPayload, Field(discriminator="kind")]
# ---------------------------------------------------------------------------
# API request / response models
# ---------------------------------------------------------------------------
class SearchRequest(BaseModel):
"""Generic search — hybrid (dense+sparse) when `query` provided,
payload-filter scroll otherwise."""
query: str | None = None
metadata_filter: dict[str, Any] | None = Field(
default=None,
description=(
"Flat dict of payload key → value or [values]. "
"Dotted keys reach into nested payload (e.g. 'scope.states')."
),
)
memory_types: list[str] | None = None
limit: int = 10
prefetch_limit: int = 50
agent_id: str | None = Field(
default=None,
description="Overrides the configured default — leave unset to use the router's agent_id.",
)
class SearchHit(BaseModel):
id: str
score: float | None
payload: dict[str, Any]
class SearchResponse(BaseModel):
strategy: Literal["hybrid", "scroll"]
hits: list[SearchHit]
class IngestRequest(BaseModel):
"""Push a single fact or exemplar into the memory store.
The router rewrites the body to memory-api's /memory/store contract:
- `content` becomes the embedded text
- `payload` becomes the `metadata` dict
- `agent_id` defaults to the router's configured value
"""
content: str
payload: Payload
importance: float = 0.5
entities: list[str] = Field(default_factory=list)
language: str = "de"
agent_id: str | None = None
# MemoryType in m2-memory: working|episodic|semantic|fact|research|other.
# Facts → 'fact'; exemplars → 'semantic'.
memory_type: str | None = None
class IngestResponse(BaseModel):
id: str
class HealthResponse(BaseModel):
status: Literal["ok", "degraded"]
qdrant: bool
bge: bool
memory_api: bool

View file

@ -0,0 +1 @@
"""HTTP routes."""

View file

@ -0,0 +1,43 @@
from __future__ import annotations
import logging
from fastapi import APIRouter, HTTPException, Request
from ..models import FactPayload, IngestRequest, IngestResponse
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/ingest", tags=["ingest"])
@router.post("", response_model=IngestResponse)
async def ingest(req: IngestRequest, request: Request) -> IngestResponse:
"""Push a single Fact or Exemplar record into agent.memory.system.
Translates to memory-api's /memory/store contract — see clients/memory_api.py.
"""
settings = request.app.state.settings
memory_api = request.app.state.memory_api
agent_id = req.agent_id or settings.artenschutz_agent_id
memory_type = req.memory_type or ("fact" if isinstance(req.payload, FactPayload) else "semantic")
payload_dict = req.payload.model_dump(mode="json")
try:
memory_id = await memory_api.store(
content=req.content,
agent_id=agent_id,
memory_type=memory_type,
importance=req.importance,
source="document",
entities=req.entities,
language=req.language,
metadata=payload_dict,
)
except Exception as exc: # noqa: BLE001
logger.exception("memory-api /store failed")
raise HTTPException(502, f"memory-api store failed: {exc}") from exc
return IngestResponse(id=memory_id)

View file

@ -0,0 +1,62 @@
from __future__ import annotations
import logging
from fastapi import APIRouter, HTTPException, Request
from ..clients.bge import build_qdrant_filter
from ..models import SearchHit, SearchRequest, SearchResponse
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/search", tags=["search"])
@router.post("/generic", response_model=SearchResponse)
async def search_generic(req: SearchRequest, request: Request) -> SearchResponse:
"""Hybrid retrieval when `query` is provided; payload-filter scroll otherwise.
The router scopes every query to its configured `agent_id` (overridable per
request). Memory-type filtering is applied as an additional payload `must`
clause when supplied.
"""
settings = request.app.state.settings
qdrant = request.app.state.qdrant
bge = request.app.state.bge
agent_id = req.agent_id or settings.artenschutz_agent_id
raw_filter = build_qdrant_filter(req.metadata_filter)
if req.memory_types:
extra = {"key": "memory_type", "match": {"any": req.memory_types}}
raw_filter = {"must": (raw_filter or {}).get("must", []) + [extra]}
if req.query:
try:
dense, sparse = await bge.embed_hybrid(req.query)
except Exception as exc: # noqa: BLE001
logger.exception("BGE embedding failed")
raise HTTPException(502, f"BGE embedding failed: {exc}") from exc
hits = await qdrant.hybrid_query(
agent_id=agent_id,
dense=dense,
sparse=sparse,
metadata_filter=raw_filter,
limit=req.limit,
prefetch_limit=req.prefetch_limit,
)
return SearchResponse(
strategy="hybrid",
hits=[SearchHit(**h) for h in hits],
)
hits = await qdrant.scroll(
agent_id=agent_id,
metadata_filter=raw_filter,
limit=req.limit,
)
return SearchResponse(
strategy="scroll",
hits=[SearchHit(**h) for h in hits],
)

0
tests/__init__.py Normal file
View file

30
tests/test_app.py Normal file
View file

@ -0,0 +1,30 @@
from __future__ import annotations
from fastapi.testclient import TestClient
from artenschutz_router.main import app
def test_app_constructs():
"""Smoke test: app object exists, routes registered."""
paths = {route.path for route in app.routes}
assert "/health" in paths
assert "/search/generic" in paths
assert "/ingest" in paths
def test_openapi_schema_renders():
# Force lifespan to skip (we don't have Qdrant/BGE running in unit tests).
# We can still call openapi.json which doesn't touch the lifespan-managed
# clients.
with TestClient(app, raise_server_exceptions=False) as client:
# Disable lifespan by short-circuiting via app.state — not strictly
# necessary for openapi but keeps the test fast.
resp = client.get("/openapi.json")
# If lifespan failed connecting to deps, app may still serve openapi.
assert resp.status_code == 200
spec = resp.json()
assert spec["info"]["title"] == "artenschutz-router"
assert "/search/generic" in spec["paths"]
assert "/ingest" in spec["paths"]
assert "/health" in spec["paths"]

View file

@ -0,0 +1,58 @@
from __future__ import annotations
from datetime import date
from pathlib import Path
import pytest
from artenschutz_router.ingest.authoritative import bnatschg as loader
SAMPLE_BNATSCHG = """§ 44 Vorschriften für besonders geschützte und bestimmte andere Tier- und Pflanzenarten
(1) Es ist verboten,
1. wild lebenden Tieren der besonders geschützten Arten nachzustellen,
sie zu fangen, zu verletzen oder zu töten oder ihre Entwicklungsformen aus
der Natur zu entnehmen, zu beschädigen oder zu zerstören.
(5) Für nach § 15 Absatz 1 unvermeidbare Beeinträchtigungen
§ 45b Schutz wild lebender Vögel
(1) Beim Betrieb von Windenergieanlagen sind die in Anlage 1
"""
def test_loads_records_from_synthetic_text(tmp_path: Path):
path = tmp_path / "bnatschg.txt"
path.write_text(SAMPLE_BNATSCHG, encoding="utf-8")
records = list(loader.load(path, valid_from=date(2024, 1, 1)))
assert len(records) == 2
anchors = [r.payload.citation_anchor for r in records]
assert any(a and a.startswith("§ 44") for a in anchors)
assert any(a and a.startswith("§ 45b") for a in anchors)
for r in records:
assert r.payload.fact_type == "legal"
assert r.payload.source.doc == "BNatSchG"
assert r.payload.source.type == "authoritative"
assert r.payload.scope.states == []
assert r.payload.valid_from == date(2024, 1, 1)
assert r.importance > 0
def test_missing_file_raises(tmp_path: Path):
path = tmp_path / "nope.txt"
with pytest.raises(FileNotFoundError):
list(loader.load(path))
def test_request_body_shape(tmp_path: Path):
path = tmp_path / "bnatschg.txt"
path.write_text(SAMPLE_BNATSCHG, encoding="utf-8")
record = next(loader.load(path))
body = record.to_request_body()
assert body["content"]
assert body["payload"]["kind"] == "fact"
assert body["payload"]["fact_type"] == "legal"
assert "agent_id" not in body # left to the router unless overridden

50
tests/test_chunker.py Normal file
View file

@ -0,0 +1,50 @@
from __future__ import annotations
from artenschutz_router.ingest.chunker import chunk_by_paragraph_marker, chunk_by_size
def test_paragraph_chunker_splits_at_section_markers():
text = """Vor § 44.
§ 44 Vorschriften für besonders geschützte und bestimmte andere Tier- und Pflanzenarten
(1) Es ist verboten,
1. wild lebenden Tieren der besonders geschützten Arten nachzustellen, sie zu fangen,
§ 45b Schutz wild lebender Vögel
(1) Beim Betrieb von Windenergieanlagen
Anlage 1 (zu § 54 Absatz 4)
Bestimmte besonders geschützte Arten
"""
chunks = chunk_by_paragraph_marker(text)
anchors = [c.anchor for c in chunks]
assert any(a and a.startswith("§ 44") for a in anchors)
assert any(a and a.startswith("§ 45b") for a in anchors)
assert any(a and a.startswith("Anlage 1") for a in anchors)
# Pre-§44 prefix should NOT become its own chunk (we anchor on markers).
assert "Vor § 44" not in chunks[0].text
def test_paragraph_chunker_returns_single_when_no_markers():
text = "Plain prose without any paragraph markers, just regular text."
chunks = chunk_by_paragraph_marker(text)
assert len(chunks) == 1
assert chunks[0].anchor is None
def test_size_chunker_respects_target_size():
paragraphs = ["abc def ghi jkl"] * 200
text = "\n\n".join(paragraphs)
chunks = chunk_by_size(text, target_chars=200, overlap_chars=20)
assert len(chunks) > 1
for c in chunks:
# Overlap may push slightly over; tolerate 2x.
assert len(c.text) < 600
def test_size_chunker_handles_empty():
assert chunk_by_size("") == []
assert chunk_by_size(" \n ") == []

49
tests/test_cli.py Normal file
View file

@ -0,0 +1,49 @@
from __future__ import annotations
import json
from io import StringIO
from pathlib import Path
import pytest
from artenschutz_router.ingest import cli
SAMPLE_BNATSCHG = """§ 44 Vorschriften
(1) Es ist verboten
§ 45b Schutz wild lebender Vögel
(1) Beim Betrieb von WEA
"""
def test_cli_dry_run_emits_json(tmp_path: Path, capsys: pytest.CaptureFixture):
sources_root = tmp_path / "sources"
bn_dir = sources_root / "bnatschg"
bn_dir.mkdir(parents=True)
(bn_dir / "bnatschg.txt").write_text(SAMPLE_BNATSCHG, encoding="utf-8")
rc = cli.main(
[
"--source",
"bnatschg",
"--sources-root",
str(sources_root),
"--dry-run",
]
)
assert rc == 0
out = capsys.readouterr().out.strip().splitlines()
assert len(out) == 2
for line in out:
record = json.loads(line)
assert record["payload"]["kind"] == "fact"
assert record["payload"]["fact_type"] == "legal"
assert record["content"]
def test_cli_rejects_unknown_source(tmp_path: Path):
with pytest.raises(SystemExit):
cli.main(["--source", "nonsense", "--sources-root", str(tmp_path), "--dry-run"])

View file

@ -0,0 +1,30 @@
from __future__ import annotations
from artenschutz_router.clients.bge import build_qdrant_filter
def test_returns_none_for_empty():
assert build_qdrant_filter(None) is None
assert build_qdrant_filter({}) is None
def test_scalar_match_value():
f = build_qdrant_filter({"kind": "fact"})
assert f == {"must": [{"key": "kind", "match": {"value": "fact"}}]}
def test_list_match_any():
f = build_qdrant_filter({"fact_type": ["legal", "mitigation"]})
assert f == {"must": [{"key": "fact_type", "match": {"any": ["legal", "mitigation"]}}]}
def test_drops_none_values():
f = build_qdrant_filter({"a": "x", "b": None, "c": [1, 2]})
must = f["must"]
keys = {clause["key"] for clause in must}
assert keys == {"a", "c"}
def test_nested_dotted_keys_pass_through():
f = build_qdrant_filter({"scope.states": ["BE", "BB"]})
assert f["must"][0]["key"] == "scope.states"

91
tests/test_models.py Normal file
View file

@ -0,0 +1,91 @@
from __future__ import annotations
from datetime import date
import pytest
from pydantic import ValidationError
from artenschutz_router.models import (
ExemplarPayload,
FactPayload,
FactScope,
IngestRequest,
SearchRequest,
SourceInfo,
)
def test_fact_payload_minimal():
p = FactPayload(
fact_type="legal",
source=SourceInfo(type="authoritative", doc="BNatSchG"),
)
assert p.kind == "fact"
assert p.scope.states == []
assert p.valid_until is None
def test_fact_payload_rich():
p = FactPayload(
fact_type="mitigation",
source=SourceInfo(type="extracted", doc="GS_GA_BER_Foo_FachGA", year=2024, tier="A"),
scope=FactScope(
states=["BE"],
anlagen_typ=["Neubau"],
species=["passer_domesticus"],
impact_types=["bauzeit"],
),
valid_from=date(2023, 4, 1),
accepted_by_behoerde=True,
citation_anchor="§44(1) Nr. 1",
)
dumped = p.model_dump(mode="json")
assert dumped["valid_from"] == "2023-04-01"
assert dumped["accepted_by_behoerde"] is True
def test_fact_payload_rejects_unknown_fact_type():
with pytest.raises(ValidationError):
FactPayload(fact_type="bogus", source=SourceInfo(type="authoritative", doc="x"))
def test_exemplar_payload_minimal():
p = ExemplarPayload(
project_slug="GS_GA_BER_Covivio_Hasenheide_94_FachGA",
report_type="FachGA",
tier="A",
)
assert p.kind == "exemplar"
assert p.is_draft is False
assert p.species_mentioned == []
def test_ingest_request_discriminates_payload():
fact_req = IngestRequest(
content="§44 (1) BNatSchG …",
payload={
"kind": "fact",
"fact_type": "legal",
"source": {"type": "authoritative", "doc": "BNatSchG"},
},
)
assert isinstance(fact_req.payload, FactPayload)
exemplar_req = IngestRequest(
content="Im Untersuchungsraum wurden …",
payload={
"kind": "exemplar",
"project_slug": "GS_GA_BER_Foo",
"report_type": "FachGA",
"tier": "A",
},
)
assert isinstance(exemplar_req.payload, ExemplarPayload)
def test_search_request_defaults():
s = SearchRequest()
assert s.query is None
assert s.limit == 10
assert s.prefetch_limit == 50
assert s.metadata_filter is None