m2 (AI Agent)
6da874b2e6
Phase 1: router skeleton + Qdrant read path - FastAPI app with /health, /search/generic, /ingest endpoints - Qdrant reader (hybrid dense+sparse w/ RRF, plus payload scroll) - BGE-M3 TEI client for query-side embedding - memory-api client proxying writes to /memory/store (m2-memory untouched) - Pydantic Fact + Exemplar payloads, discriminated by `kind` - Dockerfile + docker-compose joining the coolify Docker network Phase 2: authoritative-source ingest pipeline - PDF text extraction + paragraph-aware (§ N) and size chunkers - Loaders for bnatschg, mhbasp, biotopwertliste, state-<slug> - CLI: artenschutz-ingest --source <name> [--states ...] [--dry-run] - Helper script to fetch BNatSchG from gesetze-im-internet.de - End-to-end smoke script (scripts/smoke.sh) for Coolify validation 22/22 unit tests pass. Real-world dry-runs verified against mhbasp Anhang 4, biotopwertlisteNEU and Berlin Kartierstandards PDFs from GST-DATA. See COOLIFY-DEPLOY.md for staging deploy + smoke procedure.
128 lines
4.3 KiB
Markdown
128 lines
4.3 KiB
Markdown
# Coolify staging deploy — artenschutz-router
|
|
|
|
Goal: stand up the router on Coolify staging, in the same Docker network as
|
|
`agent.memory.system`, so it can reach `memory-qdrant`, `memory-embeddings`,
|
|
and `memory-api` by container name. Then run an end-to-end smoke ingest +
|
|
search against real data.
|
|
|
|
## Pre-flight
|
|
|
|
You should already have on staging Coolify:
|
|
|
|
- A running `agent.memory.system` (memory-api + memory-qdrant + memory-embeddings)
|
|
- The Docker network name those services live on (probably `coolify` — check
|
|
`docker network ls` on the staging host)
|
|
|
|
Note down:
|
|
|
|
| Setting | Value (example) |
|
|
|---|---|
|
|
| Coolify project | `gruenstifter-staging` |
|
|
| Docker network | `coolify` |
|
|
| memory-api container name | `memory-api-<id>` |
|
|
| memory-qdrant container name | `memory-qdrant-<id>` |
|
|
| memory-embeddings container name | `memory-embeddings-<id>` |
|
|
|
|
> The *full* container names include Coolify's stack hash suffix. The
|
|
> compose file uses the short service names (`memory-api`, etc.) which work
|
|
> when the router is on the same Coolify-managed network — Coolify creates
|
|
> service-name aliases. If they don't resolve, fall back to container IPs
|
|
> via `docker inspect`.
|
|
|
|
## Path A — Push to forgejo, let Coolify auto-deploy (preferred)
|
|
|
|
```bash
|
|
cd /home/m2/artenschutz-router
|
|
git init -b main
|
|
git add .
|
|
git commit -m "Initial commit — phase 1 + 2"
|
|
git remote add origin <forgejo-url>:gruenstifter/artenschutz-router.git
|
|
git push -u origin main
|
|
```
|
|
|
|
In Coolify staging:
|
|
|
|
1. New Resource → **Application** → **Public Git Repository** (or Private,
|
|
add the SSH key)
|
|
2. Source: the forgejo URL above
|
|
3. Build Pack: **Dockerfile**
|
|
4. Set environment variables:
|
|
```
|
|
QDRANT_URL=http://memory-qdrant:6333
|
|
QDRANT_COLLECTION=agent_memory
|
|
BGE_TEI_URL=http://memory-embeddings:8000
|
|
MEMORY_API_URL=http://memory-api:8000
|
|
ARTENSCHUTZ_AGENT_ID=gruenstifter_staging
|
|
ROUTER_PORT=8080
|
|
LOG_LEVEL=INFO
|
|
```
|
|
`gruenstifter_staging` keeps smoke data cleanly separable from any future
|
|
production write.
|
|
5. Network: join the same network as `agent.memory.system` (usually `coolify`).
|
|
6. Port: 8080 (no public exposure needed for smoke — internal-only is fine).
|
|
7. Deploy.
|
|
|
|
## Path B — Docker Compose on the Coolify host
|
|
|
|
If you'd rather not push to forgejo yet:
|
|
|
|
```bash
|
|
# On the Coolify host (or via SSH):
|
|
scp -r /home/m2/artenschutz-router coolify-host:/opt/coolify-apps/
|
|
ssh coolify-host
|
|
cd /opt/coolify-apps/artenschutz-router
|
|
cp .env.example .env
|
|
# Edit .env to set ARTENSCHUTZ_AGENT_ID=gruenstifter_staging
|
|
docker compose build
|
|
docker compose up -d
|
|
docker compose logs -f artenschutz-router
|
|
```
|
|
|
|
This bypasses Coolify's UI but uses the same `coolify` external network the
|
|
memory stack joins.
|
|
|
|
## Verify it's up
|
|
|
|
From any shell on the Coolify host:
|
|
|
|
```bash
|
|
docker exec artenschutz-router curl -sS http://localhost:8080/health
|
|
# expect: {"status":"ok","qdrant":true,"bge":true,"memory_api":true}
|
|
```
|
|
|
|
`status: "degraded"` with one of the three `false` means that service isn't
|
|
reachable from inside the router's network namespace — fix container names /
|
|
network attachment before continuing.
|
|
|
|
## End-to-end smoke (uses the test data already symlinked in `sources/`)
|
|
|
|
```bash
|
|
# 1. Fetch BNatSchG into the container's sources/bnatschg/ if not already.
|
|
docker exec artenschutz-router python scripts/fetch_bnatschg.py
|
|
|
|
# 2. Run the smoke (ingests small batches + queries them back).
|
|
docker exec artenschutz-router bash scripts/smoke.sh
|
|
```
|
|
|
|
The smoke script ingests ~5 records per source, then runs three searches:
|
|
|
|
- Hybrid query `"Fledermäuse Kartierung"` filtered to `fact_type=method`
|
|
- Hybrid query `"besonders geschützte Arten"` filtered to `fact_type=legal`
|
|
- Payload-only scroll on `scope.states=BE` to verify state filtering
|
|
|
|
Expected: each search returns ≥1 hit. If hybrid returns 0 hits but scroll
|
|
works, BGE-M3 embedding is failing — check the logs.
|
|
|
|
## Cleanup (after smoke)
|
|
|
|
Drop the staging agent_id from Qdrant:
|
|
|
|
```bash
|
|
docker exec memory-qdrant-<id> curl -X POST \
|
|
http://localhost:6333/collections/agent_memory/points/delete \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{"filter":{"must":[{"key":"agent_id","match":{"value":"gruenstifter_staging"}}]}}'
|
|
```
|
|
|
|
This wipes every point we wrote under the staging agent_id. Nothing else is
|
|
affected because partition isolation is by `agent_id`.
|