How retrieval works

Memory.Search(query, about: …, limit: …) — the pipeline behind the call: index small, match on meaning plus exact tokens, order by closeness then recency, return whole

What happens between `Memory.Search("…")` and the list you get back — indexing, matching, ranking and the two narrowings — written out as one pipeline. You never call any of it; this is here so the behaviour is explainable rather than mysterious, and so you can tell a bad result from a wrong expectation.

Summary#

Memory.Search("…") is one call, and behind it are four decisions: what got indexed, what the query is matched against, how candidates are ordered, and what is returned. None of it is yours to configure — the settings are platform-wide and chosen by measurement, not by an app — but all of it is yours to understand, because the difference between "retrieval is bad" and "I asked for the wrong thing" is usually visible from here.

The short version: a [Searchable] field is cut into small pieces, the small piece is what gets matched, and the whole field is what comes back.

Signature#

The only surface is the one you already have. Everything below describes what it does.

// You write this…
List<SearchHit> hits = Memory.Search("when do they want the report?", about: [customer], limit: 5);

// …and the pipeline decides the rest. There is no ranking knob on the call, deliberately: what a good
// ranking is turns out to be a question about the ENGINE, measured once, not a question an app can answer.

Description#

1 · What gets indexed — small pieces, not whole fields#

A [Searchable] field is split into short pieces when it is saved. Each piece is indexed on its own, and the whole field is kept alongside them as the thing an answer resolves to.

entity Customer {
  [MaxLength(120)] string Name;
  [Searchable(Memory)] string Notes;   // ← split into short pieces on save; the whole text kept as the answer
}

The reason is that one column was doing two jobs that pull in opposite directions. To be FOUND, text must be short and specific. A long note is stored as the average of everything it says, so a question about one sentence inside it lands nowhere near the middle. To be USEFUL, text must be long and complete — a matched fragment on its own is a fact with its context cut off.

Splitting the two is what lets both be true: match the small piece, answer with the whole thing. It is also why you never see a fragment in a SearchHit — the piece that matched is an index entry, not a result.

⚠ This is not a size setting you should reach for. Pieces are small — around a sentence — and there is deliberately no overlap between them, because the whole field is always one step away and repeating text between neighbours only makes neighbouring pieces harder to tell apart.

2 · What the query is matched against#

Two things run, and the second usually stays quiet.

Meaning. Your query is turned into a vector and compared against the indexed pieces. This is what makes "when do they want the report?" find "the monthly numbers go out as a spreadsheet" — no words in common.

Exact tokens. If your query contains something a vector cannot represent — an order number, a product code, an SKU — that token is also matched literally, and weighted heavily. If it contains no such token, this half does not run at all.

Memory.Search("did we ever sort out the packaging fault on PROD-4471?");
// → meaning finds "packaging fault … outer carton"
// → and `PROD-4471` is matched exactly, because a vector cannot represent an opaque code

Memory.Search("who do we chase about an unpaid invoice?");
// → meaning only. There is no distinctive token here, so the exact-match half stays silent.

⚑ The silence is the feature. Matching words is only useful when the words are distinctive. On an ordinary question — where the wording is by definition not the answer's wording — a literal matcher returns confident nonsense, and mixing confident nonsense into a good signal makes it worse. So it fires on the tokens it is for and abstains on everything else.

You can still supply your own term with keyword: when you know one. That is trusted less than an extracted token, not more: you chose a word, the extractor recognised a shape.

3 · How candidates are ordered#

Closeness comes first, then two adjustments.

Age. Between two pieces that match equally well, the more recent one wins. This is what makes a corrected fact beat the thing it corrected — nothing else can separate them, since they are about the same subject and phrased alike. The adjustment is gentle and it saturates: last week against last year is a real difference, ten years against eleven is not.

// Remembered in March:     "invoices are settled by bank transfer"
// Remembered in November:  "they moved to card payments in the autumn"
Memory.Search("how do they pay?");   // → the November memory, though both match equally well

⚠ Age is a tie-breaker, not a sort. A much better older match still beats a barely-relevant newer one. If you want strict recency you are asking for an ordinary query with an OrderBy, not for search.

Being linked. A memory reached by following a stated Link (see related:) is included, but no single linked record may flood the result — it contributes its best piece, not its whole file. Being related is a reason to be considered; it is not a reason to be believed.

⚠ And the top answer belongs to a record you NAMED. A linked memory can take every position below the first, and it will when it matches better — but the lead result is about something in your about: list whenever there is one to give. This is what keeps related: from changing the answer to a question you asked about one record: widening the search should add what else is worth reading, not replace what you asked for.

// Even if the sister company's note is a WORD-FOR-WORD match and this customer's is only close,
// the customer's own memory leads — and the sister company's follows it, marked with the link that reached it.
Memory.Search("who signs off an urgent credit?", about: [customer], related: 1);

4 · What is returned, and what is filtered#

You get whole answers, deduplicated: one SearchHit per underlying memory however many of its pieces matched, and the text is the complete field rather than the fragment that won.

Two narrowings apply, and they are not the same kind of thing:

// YOURS — a narrowing you asked for.
Memory.Search("payment terms", about: [customer]);        // only memory about this record
Memory.Search("payment terms", of: [Invoice], limit: 5);  // only these types, only five answers

// NOT YOURS — and not visible in the call at all.
// Every result is already restricted to memory anchored to rows you may read. A memory is reachable
// only through the record it is about, so if you cannot read the record you cannot reach its memory.

⚑ Search is not a way around your app's security, and there is nothing to remember about that. It is not a filter applied to results afterwards — memory is unreachable in the first place unless the record it belongs to is readable by the caller. That holds for every path, including a link hop, which checks both ends before it travels.

Examples#

The pipeline in one place, as it applies to a single call:

using Osyrin.Memory;

entity Customer {
  [MaxLength(120)] string Name;
  [Searchable(Memory)] string Notes;    // 1. split into short pieces on save; whole text kept as the answer
}

List<SearchHit> WhatDoTheyWant(Customer customer) {
  // 2. matched on MEANING — this query shares no words with "the monthly numbers go out as a spreadsheet",
  //    and the exact-token half stays silent because there is no code or order number to match literally
  // 3. ordered by closeness, then nudged by recency, with a cap on what any one linked record contributes
  // 4. returned whole and deduplicated — one hit per memory, full text, never the fragment that matched;
  //    `about:` is the narrowing you asked for, and readability is the one you did not have to
  return Memory.Search("how should we send them their month-end numbers?", about: [customer], limit: 5);
}

⚠ What the numbers on a SearchHit mean. Distance is how far the matched piece was from your query — lower is closer — and it carries the adjustments above, so it is a ranking figure and not a pure similarity. Score is relevance, higher first. Neither is a probability and neither is comparable between two different queries.