artenschutz-router/COOLIFY-DEPLOY.md
m2 (AI Agent) 6da874b2e6 Initial commit — phase 1 + 2 of artenschutz-digest concept
Phase 1: router skeleton + Qdrant read path
- FastAPI app with /health, /search/generic, /ingest endpoints
- Qdrant reader (hybrid dense+sparse w/ RRF, plus payload scroll)
- BGE-M3 TEI client for query-side embedding
- memory-api client proxying writes to /memory/store (m2-memory untouched)
- Pydantic Fact + Exemplar payloads, discriminated by `kind`
- Dockerfile + docker-compose joining the coolify Docker network

Phase 2: authoritative-source ingest pipeline
- PDF text extraction + paragraph-aware (§ N) and size chunkers
- Loaders for bnatschg, mhbasp, biotopwertliste, state-<slug>
- CLI: artenschutz-ingest --source <name> [--states ...] [--dry-run]
- Helper script to fetch BNatSchG from gesetze-im-internet.de
- End-to-end smoke script (scripts/smoke.sh) for Coolify validation

22/22 unit tests pass. Real-world dry-runs verified against mhbasp Anhang 4,
biotopwertlisteNEU and Berlin Kartierstandards PDFs from GST-DATA.

See COOLIFY-DEPLOY.md for staging deploy + smoke procedure.
2026-05-14 16:36:00 +02:00

4.3 KiB

Coolify staging deploy — artenschutz-router

Goal: stand up the router on Coolify staging, in the same Docker network as agent.memory.system, so it can reach memory-qdrant, memory-embeddings, and memory-api by container name. Then run an end-to-end smoke ingest + search against real data.

Pre-flight

You should already have on staging Coolify:

  • A running agent.memory.system (memory-api + memory-qdrant + memory-embeddings)
  • The Docker network name those services live on (probably coolify — check docker network ls on the staging host)

Note down:

Setting Value (example)
Coolify project gruenstifter-staging
Docker network coolify
memory-api container name memory-api-<id>
memory-qdrant container name memory-qdrant-<id>
memory-embeddings container name memory-embeddings-<id>

The full container names include Coolify's stack hash suffix. The compose file uses the short service names (memory-api, etc.) which work when the router is on the same Coolify-managed network — Coolify creates service-name aliases. If they don't resolve, fall back to container IPs via docker inspect.

Path A — Push to forgejo, let Coolify auto-deploy (preferred)

cd /home/m2/artenschutz-router
git init -b main
git add .
git commit -m "Initial commit — phase 1 + 2"
git remote add origin <forgejo-url>:gruenstifter/artenschutz-router.git
git push -u origin main

In Coolify staging:

  1. New Resource → ApplicationPublic Git Repository (or Private, add the SSH key)
  2. Source: the forgejo URL above
  3. Build Pack: Dockerfile
  4. Set environment variables:
    QDRANT_URL=http://memory-qdrant:6333
    QDRANT_COLLECTION=agent_memory
    BGE_TEI_URL=http://memory-embeddings:8000
    MEMORY_API_URL=http://memory-api:8000
    ARTENSCHUTZ_AGENT_ID=gruenstifter_staging
    ROUTER_PORT=8080
    LOG_LEVEL=INFO
    
    gruenstifter_staging keeps smoke data cleanly separable from any future production write.
  5. Network: join the same network as agent.memory.system (usually coolify).
  6. Port: 8080 (no public exposure needed for smoke — internal-only is fine).
  7. Deploy.

Path B — Docker Compose on the Coolify host

If you'd rather not push to forgejo yet:

# On the Coolify host (or via SSH):
scp -r /home/m2/artenschutz-router  coolify-host:/opt/coolify-apps/
ssh coolify-host
cd /opt/coolify-apps/artenschutz-router
cp .env.example .env
# Edit .env to set ARTENSCHUTZ_AGENT_ID=gruenstifter_staging
docker compose build
docker compose up -d
docker compose logs -f artenschutz-router

This bypasses Coolify's UI but uses the same coolify external network the memory stack joins.

Verify it's up

From any shell on the Coolify host:

docker exec artenschutz-router curl -sS http://localhost:8080/health
# expect: {"status":"ok","qdrant":true,"bge":true,"memory_api":true}

status: "degraded" with one of the three false means that service isn't reachable from inside the router's network namespace — fix container names / network attachment before continuing.

End-to-end smoke (uses the test data already symlinked in sources/)

# 1. Fetch BNatSchG into the container's sources/bnatschg/ if not already.
docker exec artenschutz-router python scripts/fetch_bnatschg.py

# 2. Run the smoke (ingests small batches + queries them back).
docker exec artenschutz-router bash scripts/smoke.sh

The smoke script ingests ~5 records per source, then runs three searches:

  • Hybrid query "Fledermäuse Kartierung" filtered to fact_type=method
  • Hybrid query "besonders geschützte Arten" filtered to fact_type=legal
  • Payload-only scroll on scope.states=BE to verify state filtering

Expected: each search returns ≥1 hit. If hybrid returns 0 hits but scroll works, BGE-M3 embedding is failing — check the logs.

Cleanup (after smoke)

Drop the staging agent_id from Qdrant:

docker exec memory-qdrant-<id> curl -X POST \
  http://localhost:6333/collections/agent_memory/points/delete \
  -H 'Content-Type: application/json' \
  -d '{"filter":{"must":[{"key":"agent_id","match":{"value":"gruenstifter_staging"}}]}}'

This wipes every point we wrote under the staging agent_id. Nothing else is affected because partition isolation is by agent_id.