Initial commit: hardened DeerFlow factory
Vendored deer-flow upstream (bytedance/deer-flow) plus prompt-injection hardening: - New deerflow.security package: content_delimiter, html_cleaner, sanitizer (8 layers — invisible chars, control chars, symbols, NFC, PUA, tag chars, horizontal whitespace collapse with newline/tab preservation, length cap) - New deerflow.community.searx package: web_search, web_fetch, image_search backed by a private SearX instance, every external string sanitized and wrapped in <<<EXTERNAL_UNTRUSTED_CONTENT>>> delimiters - All native community web providers (ddg_search, tavily, exa, firecrawl, jina_ai, infoquest, image_search) replaced with hard-fail stubs that raise NativeWebToolDisabledError at import time, so a misconfigured tool.use path fails loud rather than silently falling back to unsanitized output - Native client back-doors (jina_client.py, infoquest_client.py) stubbed too - Native-tool tests quarantined under tests/_disabled_native/ (collect_ignore_glob via local conftest.py) - Sanitizer Layer 7 fix: only collapse horizontal whitespace, preserve newlines and tabs so list/table structure survives - Hardened runtime config.yaml references only the searx-backed tools - Factory overlay (backend/) kept in sync with deer-flow tree as a reference / source See HARDENING.md for the full audit trail and verification steps.
This commit is contained in:
38
deer-flow/backend/docs/MEMORY_IMPROVEMENTS_SUMMARY.md
Normal file
38
deer-flow/backend/docs/MEMORY_IMPROVEMENTS_SUMMARY.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# Memory System Improvements - Summary
|
||||
|
||||
## Sync Note (2026-03-10)
|
||||
|
||||
This summary is synchronized with the `main` branch implementation.
|
||||
TF-IDF/context-aware retrieval is **planned**, not merged yet.
|
||||
|
||||
## Implemented
|
||||
|
||||
- Accurate token counting with `tiktoken` in memory injection.
|
||||
- Facts are injected into `<memory>` prompt content.
|
||||
- Facts are ordered by confidence and bounded by `max_injection_tokens`.
|
||||
|
||||
## Planned (Not Yet Merged)
|
||||
|
||||
- TF-IDF cosine similarity recall based on recent conversation context.
|
||||
- `current_context` parameter for `format_memory_for_injection`.
|
||||
- Weighted ranking (`similarity` + `confidence`).
|
||||
- Runtime extraction/injection flow for context-aware fact selection.
|
||||
|
||||
## Why This Sync Was Needed
|
||||
|
||||
Earlier docs described TF-IDF behavior as already implemented, which did not match code in `main`.
|
||||
This mismatch is tracked in issue `#1059`.
|
||||
|
||||
## Current API Shape
|
||||
|
||||
```python
|
||||
def format_memory_for_injection(memory_data: dict[str, Any], max_tokens: int = 2000) -> str:
|
||||
```
|
||||
|
||||
No `current_context` argument is currently available in `main`.
|
||||
|
||||
## Verification Pointers
|
||||
|
||||
- Implementation: `packages/harness/deerflow/agents/memory/prompt.py`
|
||||
- Prompt assembly: `packages/harness/deerflow/agents/lead_agent/prompt.py`
|
||||
- Regression tests: `backend/tests/test_memory_prompt_injection.py`
|
||||
Reference in New Issue
Block a user