# ButlerBrain Semantic Memory Skill This skill gives your AI assistant semantic memory via ButlerBrain, a Brain-as-a-Service platform that stores and retrieves your notes, thoughts, web pages, calendar events, and documents. Connected via MCP over curl, your AI can search your knowledge base, save new memories, manage your calendar, and crawl web pages across Discord, Signal, WhatsApp, Telegram, or any platform OpenClaw supports. ## Configuration Set these environment variables before using this skill: ``` BUTLERBRAIN_API_KEY=your_api_key_here BUTLERBRAIN_BRAIN_NAME={brain_name} BUTLERBRAIN_ENDPOINT=https://api.butlerbrain.ai/v1/{brain_name}/mcp ``` ## API Access All ButlerBrain tools are called via HTTP POST to your brain's MCP endpoint: ``` https://api.butlerbrain.ai/v1/{brain_name}/mcp ``` Authenticate with your API key in the `x-api-key` header. --- ## Private visibility ButlerBrain supports marking individual items as private. Private items are visible only to the owner who saved them. Other family members or co-owners in the same brain cannot see them in search results. ### When to mark items private Watch for these user intents: - "Save this privately" - "Don't share this with [other family member]" - "Keep this between us" - "Just for me" / "private note" - Anything about medical appointments, gifts for family members, personal finances, or therapy - Calendar events the user describes as "doctor," "therapy," or that they explicitly flag as sensitive When the user's intent is clear, add `private: true` to the tool call. When it's ambiguous, ask: "Would you like to save this privately so only you can see it?" ### How to mark items private All write tools accept an optional `private` parameter: - `save_thought(text=..., private=true)` - `add_event(title=..., private=true, ...)` - `crawl_and_save(url=..., private=true)` - `ingest_pdf(s3_key=..., private=true)` Default is `false` (shared with all owners in the brain). Pass `true` to mark the item private to the current owner only. ### Obsidian vault files For users with Obsidian syncing to ButlerBrain, vault files are marked private in two ways (the user controls this in Obsidian, not you): 1. Placing the file under a folder named exactly `private` at any level of the path (e.g. `Health/private/medical.md`). Folder names like `Private Notes` or `MyPrivate` do **not** match. 2. Adding `private: true` to the file's frontmatter: ``` --- private: true --- # Note content... ``` Either trigger is sufficient. If a user asks how to make Obsidian notes private, recommend the frontmatter approach. It keeps their folder structure intact. ### Identity vs. Search Filters The `owner` and `requesting_owner` parameters are different: - `owner` is a **SEARCH FILTER**. It narrows results to content authored by that owner. It does **NOT** grant access to private items. Example use: "find Chris's grocery lists" means `search_brain(query="grocery", owner="chris")`. - `requesting_owner` is an **IDENTITY CLAIM**. Set this to the human user's identity if you (the AI) are configured for a specific user. Private items owned by this identity become visible. Example: if you are Lily's personal AI, always pass `requesting_owner="lily"` on every read call. If you are a shared AI agent with no specific user identity (e.g., a family agent that multiple humans chat with): do **NOT** pass `requesting_owner`. You will see only shared items. This is by design. It prevents accidentally exposing one family member's private content to another family member in the same chat session. If you are configured for a specific user: always pass `requesting_owner` as that user's owner identifier on every read call. Never guess based on conversational context. Use your configured identity only. ### Searching with private visibility The `search_brain` tool returns: - All shared items matching the query - If `requesting_owner` is set: the current owner's own private items matching the query - If `requesting_owner` is set: a `hidden_private_count` field indicating how many OTHER owners' private items matched but were withheld If `requesting_owner` is unset, `hidden_private_count` is **omitted entirely**. You made no identity claim, so there is nothing withheld "from you" to surface. If `hidden_private_count > 0`, do NOT treat this as an error or missing data. It is working as designed: other family members have private items matching the query and you are correctly not seeing them. You can mention this to the user if relevant: "I found 3 shared results. Other family members have 2 private items matching this query that I can't access." ### The trust model ButlerBrain operates on a "shared brain = shared trust" model. Owners in the same brain can see each other's shared content by default. That is the family-brain value proposition. Private is an explicit opt-in for individual sensitive items. For true isolation, the user should create a separate brain, not rely on private flags alone. Private enforcement is a social-contract boundary, not a cryptographic one. The backend trusts the AI client's `requesting_owner` claim. A well-behaved client configured for a specific user always claims that user's identity. A shared client without per-user identity always leaves `requesting_owner` unset and sees only shared content. --- ### search_brain: Search stored memory Search semantically across your brain. Use when someone asks about something they've stored, asks "what do I know about X", or wants to find notes, documents, or web pages on a topic. **Table options:** `vault` (Obsidian notes), `web` (crawled pages), `docs` (documents and imported conversations), `thoughts` (saved thoughts), `all` (search everywhere, the default; includes calendar events). For date-range calendar questions, prefer `get_events`. **Parameters:** - `limit` (optional): max results, default 10. The parameter is named `limit`, not `top_k`. - `owner` (optional, content filter): narrows results to content authored by that owner. It does NOT grant access to private items, and you should never infer it from names mentioned in the query. Omit it for general retrieval. - `requesting_owner` (optional, identity claim): your configured user's identity. When set, that owner's private items become visible. See the "Identity vs. Search Filters" section above. - `sort` (optional): `"relevance"` (default) returns the best semantic matches first. `"recency"` returns the same matches ordered most-recent-first by the content's original date (for example the original conversation date for imported chats), falling back to the date it was added. Use `"recency"` when the user asks for their latest or most recent items. ```bash curl -s https://api.butlerbrain.ai/v1/{brain_name}/mcp \ -H "x-api-key: your_api_key_here" \ -H "Content-Type: application/json" \ -d '{ "method": "tools/call", "params": { "name": "search_brain", "arguments": { "query": "SEARCH_QUERY_HERE", "table": "all", "requesting_owner": "alice", "limit": 10 } } }' ``` --- ### get_context: Broader context on a topic Use when the user wants broader background or context on a topic across all sources: not a narrow search, but a gathered view of what the brain knows. **Parameters:** same `owner` (content filter) / `requesting_owner` (identity claim) split as `search_brain`, plus the same optional `sort` and `limit`. ```bash curl -s https://api.butlerbrain.ai/v1/{brain_name}/mcp \ -H "x-api-key: your_api_key_here" \ -H "Content-Type: application/json" \ -d '{ "method": "tools/call", "params": { "name": "get_context", "arguments": { "topic": "TOPIC_HERE", "requesting_owner": "alice", "limit": 10 } } }' ``` --- ### get_person: Look up a person Search all notes, thoughts, web pages, documents, and events for mentions of a person. Uses both semantic and literal name matching and groups results by source type with relevance scoring. **Parameters:** `name` (required); `owner` (optional content filter); `requesting_owner` (optional identity claim, same semantics as `search_brain`); `limit` (optional, default 5 per table). ```bash curl -s https://api.butlerbrain.ai/v1/{brain_name}/mcp \ -H "x-api-key: your_api_key_here" \ -H "Content-Type: application/json" \ -d '{ "method": "tools/call", "params": { "name": "get_person", "arguments": { "name": "PERSON_NAME", "requesting_owner": "alice", "limit": 5 } } }' ``` --- ### save_thought: Save a note or thought `save_thought(text, owner, tags?, private?)` Use when someone says "remember this", "save this", "note that", or wants to store a piece of information. Both `text` and `owner` are required. Pass `private: true` to make the thought visible only to the calling `owner` (default: shared with all owners in the brain). **Save documents verbatim by default.** When the user wants a pasted or attached document remembered, save the full text word for word with `save_thought`. Do not condense, paraphrase, or summarize unless the user explicitly asks for a summary. Always include a provenance note naming the source document (for example `"(from attached PDF: )"`). Short thoughts come back with `{"stored": true, ...}` once the chunks are written. Document-sized pastes come back with `{"status": "queued", "chunks": N, ...}` and are processed in the background. Confirm the save to the user and let them know the content will be searchable shortly. ```bash curl -s https://api.butlerbrain.ai/v1/{brain_name}/mcp \ -H "x-api-key: your_api_key_here" \ -H "Content-Type: application/json" \ -d '{ "method": "tools/call", "params": { "name": "save_thought", "arguments": { "text": "TEXT_TO_SAVE_HERE", "owner": "alice", "tags": [ "tag1", "tag2" ] } } }' ``` --- ### crawl_and_save: Save a web page or URL `crawl_and_save(url, owner, tags?, private?)` Use when someone shares a URL and wants it saved, or says "save this article / page / link". Both `url` and `owner` are required. The crawler fetches and embeds the full page content. This call is async: the page is saved in the background. Pass `private: true` to make the crawled page visible only to the calling `owner` (default: shared with all owners in the brain). **What the crawler can read.** It reads public web pages: articles, blog posts, docs, public product pages. It cannot read pages that show a sign-in wall, an app/workspace screen, or an AI-chat share link (for example a ChatGPT or Claude `share` or project link): those return only the sign-in chrome, so nothing is saved and you get an error back. When that happens, tell the user to import the source or upload it as a file instead. ```bash curl -s https://api.butlerbrain.ai/v1/{brain_name}/mcp \ -H "x-api-key: your_api_key_here" \ -H "Content-Type: application/json" \ -d '{ "method": "tools/call", "params": { "name": "crawl_and_save", "arguments": { "url": "URL_HERE", "owner": "alice", "tags": [ "article" ] } } }' ``` --- ### ingest_pdf: Process an uploaded PDF `ingest_pdf(s3_key, owner, title?, url?, tags?, private?)` Use when someone has uploaded a PDF and wants it indexed. `s3_key` is the storage key returned by `get_upload_url`; `owner` is required, and a descriptive `title` helps retrieval. This call is async. Pass `private: true` to make the extracted document visible only to the calling `owner` (default: shared with all owners in the brain). **Born-digital vs scanned PDFs.** `ingest_pdf` reads the PDF's text layer. It works on born-digital PDFs (invoices, statements, exported reports, anything with selectable text) and is the default for normal PDFs: fast and free. It cannot read scanned or image-only PDFs (photos or scans of paper, signed contracts, notarized closing packages): extraction returns empty, the ingest is rejected, and the owner gets a "scanned document, no readable text" email. Do not send these to `ingest_pdf`. For scanned PDFs, read the document directly (you are multimodal) and save the full extracted text verbatim via `save_thought`, including a provenance note like `"(transcribed from scanned PDF: )"`. Summarize only when the user explicitly asks for a summary. This routes text extraction through your own reading at no extra cost and makes the document searchable. How to tell: selectable text means born-digital, use `ingest_pdf`; pages are images you can only read visually means scanned, transcribe and `save_thought`. For mixed PDFs (some born-digital pages, some scanned inserts), transcribe the whole document and use `save_thought`. A partial `ingest_pdf` would only capture the text-layer pages and silently drop the rest. **Chat attachments are not uploads.** Files attached in the chat exist only in the chat environment. Never pass a chat attachment's file path (for example a sandbox or mounted path) as `s3_key` or as any other tool argument: the brain's storage cannot see that path, so the call will be rejected and nothing is saved. Instead, read the attachment directly and save its full text with `save_thought`. ```bash curl -s https://api.butlerbrain.ai/v1/{brain_name}/mcp \ -H "x-api-key: your_api_key_here" \ -H "Content-Type: application/json" \ -d '{ "method": "tools/call", "params": { "name": "ingest_pdf", "arguments": { "s3_key": "tenants/{brain_name}/uploads/alice/document.pdf", "title": "Document Title", "owner": "alice", "tags": [ "pdf" ] } } }' ``` --- ### add_event: Add a calendar event `add_event(title, start_time, owner, end_time?, location?, recurrence?, tags?, notes?, private?)` Use when someone wants to schedule something, mentions a meeting, appointment, or event with a date/time. `title`, `start_time`, and `owner` are required. Pass `private: true` to hide the event from family-wide calendar queries (e.g. mom's appointments). It will be visible only to the calling `owner` (default: shared with all owners in the brain). ```bash curl -s https://api.butlerbrain.ai/v1/{brain_name}/mcp \ -H "x-api-key: your_api_key_here" \ -H "Content-Type: application/json" \ -d '{ "method": "tools/call", "params": { "name": "add_event", "arguments": { "title": "EVENT_TITLE", "start_time": "2025-06-15T14:00:00", "end_time": "2025-06-15T15:00:00", "owner": "alice", "location": "LOCATION_OPTIONAL", "recurrence": null, "tags": [], "notes": "NOTES_OPTIONAL" } } }' ``` Recurrence examples: `"daily"`, `"weekly:mon,wed,fri"`, `"monthly:15"`, `"yearly:03-15"`. --- ### get_events: Query calendar events Use when someone asks what's on their calendar, what they have today or this week, or about upcoming events. **Parameters:** `owner` (optional content filter) and `requesting_owner` (optional identity claim) follow the same split as `search_brain`. If you are configured for a specific user, always pass `requesting_owner` so private events owned by them are visible. ```bash curl -s https://api.butlerbrain.ai/v1/{brain_name}/mcp \ -H "x-api-key: your_api_key_here" \ -H "Content-Type: application/json" \ -d '{ "method": "tools/call", "params": { "name": "get_events", "arguments": { "date": "2025-06-15", "range": "week", "requesting_owner": "alice" } } }' ``` Range options: `"day"`, `"week"`, `"month"`. --- ### delete_event: Remove a calendar event Use when someone wants to cancel or delete a scheduled event. For recurring events, this removes all future instances. ```bash curl -s https://api.butlerbrain.ai/v1/{brain_name}/mcp \ -H "x-api-key: your_api_key_here" \ -H "Content-Type: application/json" \ -d '{ "method": "tools/call", "params": { "name": "delete_event", "arguments": { "event_id": "EVENT_UUID_HERE" } } }' ``` --- ### get_upload_url: Get a pre-signed upload URL The upload pipeline is for AI clients that can perform the upload step themselves. The flow has three steps, and you (the AI client) perform all of them: call `get_upload_url` to receive a temporary upload URL (valid for 5 minutes) plus the `s3_key`, upload the file's bytes to that URL with an HTTP PUT, then call `ingest_pdf` with the `s3_key` to index it. If you cannot perform the PUT yourself, do not use this pipeline: read the document directly and save its full text with `save_thought` instead. **Parameters:** `filename` (required); `owner` (required); `content_type` (optional, default `application/pdf`). ```bash curl -s https://api.butlerbrain.ai/v1/{brain_name}/mcp \ -H "x-api-key: your_api_key_here" \ -H "Content-Type: application/json" \ -d '{ "method": "tools/call", "params": { "name": "get_upload_url", "arguments": { "filename": "document.pdf", "owner": "alice", "content_type": "application/pdf" } } }' ``` --- ## URL Detection When a user sends a message containing a URL, use judgment about what to do: - **Always acknowledge** the URL so the user knows you saw it. - **Offer to save** if the URL looks like an article, blog post, documentation page, or research paper. - **Auto-save** if the user explicitly asked you to save or remember the link. - **Skip saving** for memes, social media posts, short video links, login pages, or one-time-use links. When a URL is worth saving, call `crawl_and_save`. The crawl is async. Confirm to the user that it is being saved in the background. **Examples:** - "Check out this article: [url]" means offer to save it - "Save this for me: [url]" means auto-save with `crawl_and_save` - "Look at this meme: [url]" means skip saving, just respond --- ## Search Result Presentation When returning search results, use these indicators to show where content came from: | Indicator | Source | |-----------|--------| | 📝 | Note or Obsidian vault (`vault`) | | 🌐 | Web page (`web`) | | 📄 | Document or imported conversation (`docs`) | | 💭 | Saved thought (`thoughts`) | | 📅 | Calendar event (`calendar`) | Format results as a brief summary with the source indicator. If results are sparse or below confidence threshold, say so honestly rather than fabricating context. --- ## Owner Routing If multiple people share this brain, map each person's messaging identity to their owner name so saves and searches are attributed correctly. Update the IDs below with your own. You can find Discord user IDs by enabling Developer Mode in Discord settings. Signal UUIDs appear in OpenClaw logs. ``` # Discord user ID → owner name DISCORD_USER_111111111111111111=alice DISCORD_USER_222222222222222222=bob # Signal UUID → owner name SIGNAL_UUID_aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee=alice SIGNAL_UUID_ffffffff-0000-1111-2222-333333333333=bob # WhatsApp / iMessage / Telegram: use display name or phone mapping WHATSAPP_alice@example.com=alice WHATSAPP_bob@example.com=bob # Shared/group content: use "shared" as the owner GROUP_CHANNEL_DEFAULT_OWNER=shared ``` **Rules for shared content:** - Messages in a group channel default to owner `"shared"` unless a specific member is identified. - DMs from a known identity always use that person's owner name. - If identity is unknown, use `"shared"` rather than guessing. - When searching in a group context, omit `owner` to search across all members. --- ## Voice Note Handling When a voice message is transcribed and passed to you: 1. Respond to the spoken request normally. 2. If the content is worth remembering (a note, task, insight, or decision), also call `save_thought` with the transcript and a brief synthesized summary appended. 3. Tag voice saves with `["voice"]` plus any relevant topic tags. --- ## Tagging Include 2 to 4 tags whenever calling `save_thought` or `crawl_and_save`. Tags improve retrieval and help with filtering later. **Common tag categories:** | Category | Example tags | |----------|-------------| | Household | `household`, `home`, `repairs`, `supplies` | | Calendar | `calendar`, `appointments`, `schedule` | | Work | `work`, `meetings`, `projects`, `decisions` | | Projects | `project-[name]`, `research`, `planning` | | School | `school`, `homework`, `courses`, `exams` | | Activities | `sports`, `fitness`, `hobbies`, `travel` | | Family | `family`, `kids`, `parenting` | | Health | `health`, `medical`, `wellness` | | Finance | `finance`, `budget`, `bills`, `taxes` | | People | `person`, `[name]`, `relationships`, `contact`, `preferences` | | Food | `food`, `recipes`, `restaurants`, `groceries` | | Ideas | `ideas`, `brainstorm`, `inspiration` | | Learning | `learning`, `books`, `articles`, `notes` | | Web | `web`, `article`, `reference`, `link` | This list is not exhaustive. Use whatever tags best describe the content. Err on the side of more specific over more generic. --- ## Saving People & Relationships When someone mentions a person with context worth remembering (preferences, relationships, contact info, important details), use `save_thought` with person-specific tags. Don't ask "should I save this?". If it is clearly personal context about someone, just save it. **Tag format:** Always include `"person"` as the first tag, then the person's lowercase name, then descriptive tags. **Examples:** | User says | save_thought text | tags | |---|---|---| | "John loves sushi" | `John loves sushi` | `["person", "john", "food", "preferences"]` | | "Sarah's birthday is March 15" | `Sarah's birthday is March 15` | `["person", "sarah", "birthday"]` | | "My boss Dave prefers email over Slack" | `Dave prefers email over Slack for communication` | `["person", "dave", "work", "communication"]` | | "Mom's new address is 123 Oak St" | `Mom's new address is 123 Oak St` | `["person", "mom", "contact", "address"]` | **Retrieving person info:** Use `get_person` to look up everything the brain knows about someone. It searches across ALL tables (thoughts, vault, web, docs, and calendar) for mentions of that person, using both semantic and literal name matching. The combination of `save_thought` with person tags for storage and `get_person` for retrieval gives you a living address book that grows naturally from conversation. Over time, asking "what do we know about Sarah?" returns her birthday, preferences, work details, and anything else that's been saved. --- ## When to Search the Brain Search automatically (without being asked) when: - The user asks about something they've mentioned before - The user asks "do I have anything on X" or "what do I know about X" - Context from past conversations would meaningfully improve your answer - The user references a person, project, or topic by name that might be in the brain Search when asked for: - Notes, documents, or articles on a topic - Past decisions or reasoning - Calendar: "what do I have today/this week/this month" - Information about a person: "what do we know about [name]" --- ## When to Save to the Brain **Save:** - Anything the user explicitly says to save, note, or remember - Shared URLs when the content is substantive (article, doc, reference) - Decisions, conclusions, or outcomes of a conversation - Grocery lists, task lists, or to-dos when asked **Don't save:** - Casual chitchat or greetings - Content the user is just sharing for a reaction (memes, jokes) - Redundant information already in the brain (check first if unsure) - Login pages, one-time links, or ephemeral URLs - The AI's own responses (save user content, not your output) --- ## Conversational Memory Patterns **Grocery list:** "Add milk and eggs to the grocery list" means `save_thought` with `["food", "groceries"]` **Appointments:** "I have a dentist appointment Thursday at 2pm" means `add_event` with title, start_time, owner **Project notes:** "Note that we decided to use Postgres for the new API" means `save_thought` with `["work", "decisions", "project-api"]` **Family info:** "Alice is allergic to peanuts" means `save_thought` with owner `"shared"`, tags `["family", "health"]` **Saved articles:** User shares article URL means `crawl_and_save`, confirm saving in background **Research follow-up:** User asks about a topic discussed last week means `search_brain` first, answer with retrieved context --- ## Decision Guide | User says... | Use tool | |---|---| | "Search for / find / look up / what do I know about X" | `search_brain` | | "Remember / save / note / store this" | `save_thought` | | "Save this link / article / page / URL" | `crawl_and_save` | | "Upload / index this PDF" (born-digital, has selectable text) | `get_upload_url`, HTTP PUT the file, then `ingest_pdf` | | "Save this document" (pasted into or attached to the chat) | Read it, then `save_thought` with the full text verbatim | | "Index this scanned PDF / photo of a document" (no text layer) | Read it directly, then `save_thought` with the full transcription | | "Schedule / add to calendar / I have a meeting" | `add_event` | | "What's on my calendar / what do I have today/this week" | `get_events` | | "Cancel / delete [event]" | `delete_event` | | "What do you know about X / give me context on X" | `get_context` | | "What do we know about [person] / look up [name]" | `get_person` | --- ## Installation **Quick install (OpenClaw CLI):** ```bash openclaw skill install butlerbrain ``` **Manual install:** Copy this file to your OpenClaw skills directory (e.g., `~/openclaw/skills/butlerbrain/SKILL.md`), then set the `BUTLERBRAIN_API_KEY` and `BUTLERBRAIN_BRAIN_NAME` environment variables. Get your API key and brain name from your [ButlerBrain dashboard](https://butlerbrain.ai/dashboard).