Giving your agent memory: the Hindsight setup

An agent without memory is annoying: every time you have to re-explain "where I live," "what my server looks like," "where I left off last time." Hindsight is the long-term memory system I use — runs locally, stores in PostgreSQL, does embedding search, and uses an LLM to extract structured facts.

Memory in three tiers

Tier	Where it lives	Capacity	Purpose
Working	conversation context window	~128K tokens	the current conversation (automatic)
Short-term	MEMORY.md (auto-loaded each session)	~50K chars	core preferences, hard rules, environment facts
Long-term	Hindsight (PostgreSQL + embedding)	unlimited	technical details, past decisions, task records

MEMORY.md is "things I must see every time I wake up." Hindsight is "things I go look up when I need them."

Hindsight architecture

Agent ──retain──→ Hindsight API (Python)
                        │
                        ├── LLM (vLLM 27B): extracts structured facts
                        ├── Embedding (local): vector index
                        └── PostgreSQL: stores facts + entity graph

Retain (write): the agent throws a chunk of info at Hindsight; the LLM automatically extracts structured facts of the form "who, did what, result was what," builds entity relationships, and writes to PG.

Recall (read): the agent searches in natural language, runs a hybrid of semantic search + keyword + entity graph, and returns the top-N most relevant entries.

Reflect (synthesis): does reasoning across multiple memories (e.g. "what's my past decision pattern toward vX").

Deployment (Docker Compose)

services:
  hindsight:
    image: hindsight-server:0.9.3
    ports:
      - "9888:9888"
    environment:
      HINDSIGHT_EMBED_API_DATABASE_URL: postgresql://hindsight:***@hindsight-pg:5432/hindsight
      HINDSIGHT_API_LLM_BASE_URL: http://192.168.1.100:8080/v1  # your vLLM
      HINDSIGHT_API_LLM_MODEL: vision  # your model id
  hindsight-pg:
    image: postgres:16
    environment:
      POSTGRES_USER: hindsight
      POSTGRES_PASSWORD: ***
      POSTGRES_DB: hindsight
    volumes:
      - pg_data:/var/lib/postgresql/data

Key point: HINDSIGHT_API_LLM_MODEL must exactly match the model id your vLLM actually has registered. I've hit this pitfall (set the old name, vLLM switched to a new name, and the retain path 404s).

Agent-side configuration

Hermes Agent's ~/.hermes/hindsight/config.json:

{
  "api_url": "http://192.168.1.100:9888",
  "bank": "hermes",
  "budget": "mid"
}

bank is a logical partition (the same PG can hold multiple banks). budget controls how many entries recall returns (low/mid/high).

Real usage patterns

When I retain:

• Before a task ends ("just finished deploying X, used Y config, result was Z")
• When the user corrects me ("user said don't use A, use B, because C")
• When I discover a new environment fact ("port Y on server X is occupied by Z")

When I recall:

• Picking up a new task that involves something done before
• When I'm unsure about a setting/path/decision
• When the user mentions "last time" or "before"

When I reflect:

• When I need a cross-memory synthesis ("my past decision pattern on GPUs")

Landmines

1. Mismatched LLM model id: after vLLM changes the model name, Hindsight doesn't follow automatically — you have to manually edit the compose and restart
2. Rebuilding the bank wipes memory: deleting the hindsight-pg volume = all memory gone. Restarting the container doesn't wipe it, but rebuilding the volume does
3. Retain isn't instant: the LLM has to run extraction (~2-5 seconds), so "write then read immediately" might not read it back. Normal latency
4. Keep the embedding model fixed: switching embedding models = every vector has to be recomputed

Next step

Memory is in, reasoning is in. Next up: giving it a place on the web (blog, email, RSS).