DATA
6de0bf9f5b
Vendored deer-flow upstream (bytedance/deer-flow) plus prompt-injection hardening: - New deerflow.security package: content_delimiter, html_cleaner, sanitizer (8 layers — invisible chars, control chars, symbols, NFC, PUA, tag chars, horizontal whitespace collapse with newline/tab preservation, length cap) - New deerflow.community.searx package: web_search, web_fetch, image_search backed by a private SearX instance, every external string sanitized and wrapped in <<<EXTERNAL_UNTRUSTED_CONTENT>>> delimiters - All native community web providers (ddg_search, tavily, exa, firecrawl, jina_ai, infoquest, image_search) replaced with hard-fail stubs that raise NativeWebToolDisabledError at import time, so a misconfigured tool.use path fails loud rather than silently falling back to unsanitized output - Native client back-doors (jina_client.py, infoquest_client.py) stubbed too - Native-tool tests quarantined under tests/_disabled_native/ (collect_ignore_glob via local conftest.py) - Sanitizer Layer 7 fix: only collapse horizontal whitespace, preserve newlines and tabs so list/table structure survives - Hardened runtime config.yaml references only the searx-backed tools - Factory overlay (backend/) kept in sync with deer-flow tree as a reference / source See HARDENING.md for the full audit trail and verification steps.
39 lines
1.3 KiB
Markdown
39 lines
1.3 KiB
Markdown
# Memory System Improvements - Summary
|
|
|
|
## Sync Note (2026-03-10)
|
|
|
|
This summary is synchronized with the `main` branch implementation.
|
|
TF-IDF/context-aware retrieval is **planned**, not merged yet.
|
|
|
|
## Implemented
|
|
|
|
- Accurate token counting with `tiktoken` in memory injection.
|
|
- Facts are injected into `<memory>` prompt content.
|
|
- Facts are ordered by confidence and bounded by `max_injection_tokens`.
|
|
|
|
## Planned (Not Yet Merged)
|
|
|
|
- TF-IDF cosine similarity recall based on recent conversation context.
|
|
- `current_context` parameter for `format_memory_for_injection`.
|
|
- Weighted ranking (`similarity` + `confidence`).
|
|
- Runtime extraction/injection flow for context-aware fact selection.
|
|
|
|
## Why This Sync Was Needed
|
|
|
|
Earlier docs described TF-IDF behavior as already implemented, which did not match code in `main`.
|
|
This mismatch is tracked in issue `#1059`.
|
|
|
|
## Current API Shape
|
|
|
|
```python
|
|
def format_memory_for_injection(memory_data: dict[str, Any], max_tokens: int = 2000) -> str:
|
|
```
|
|
|
|
No `current_context` argument is currently available in `main`.
|
|
|
|
## Verification Pointers
|
|
|
|
- Implementation: `packages/harness/deerflow/agents/memory/prompt.py`
|
|
- Prompt assembly: `packages/harness/deerflow/agents/lead_agent/prompt.py`
|
|
- Regression tests: `backend/tests/test_memory_prompt_injection.py`
|