deerflow-factory/DEERFLOW_PROMPT_INJECTION_PROTECTION_PLAN.md

# DeerFlow Prompt Injection Protection Integration Plan

**Based on OpenClaw Hardened Scripts Analysis**  
**Date:** 2026-04-11  
**Source Reference:** `~/.openclaw/workspace-websearch/searx-scripts/` and `~/.openclaw/workspace-websearch/fetch-scripts/`

---

## Executive Summary

This document outlines the integration of OpenClaw-style prompt injection hardening into DeerFlow's web search and web fetch tools. The OpenClaw implementation demonstrates a **defense-in-depth** approach with multiple sanitization layers and clear content delimitation.

**Current State:** DeerFlow has NO prompt injection protection for web search/fetch results.  
**Target State:** Multi-layer sanitization with content delimiters and hardened script execution.

---

## 1. Analysis of OpenClaw Protection Layers

### 1.1 Content Delimiter Pattern (CRITICAL)

OpenClaw wraps external content with explicit markers:

```
<<<EXTERNAL_UNTRUSTED_CONTENT>>>
{sanitized_search_results}
<<<END_EXTERNAL_UNTRUSTED_CONTENT>>>
```

**Benefit:** LLM can semantically distinguish between system instructions and untrusted external data.

### 1.2 Unicode Attack Surface Reduction

| Category | Characters | Purpose |
|----------|-----------|---------|
| Zero-width | `\u200b-\u200f`, `\u2060-\u2064` | Steganography, hidden payloads |
| BOM/Format | `\ufeff`, `\ufffe` | Byte-order confusion |
| Control | `\u00ad`, `\u034f` | Soft hyphen, grapheme joiner |
| Private Use | `\uE000-\uF8FF` | Custom glyph substitution attacks |
| Tag Characters | `\uE0000-\uE007F` | Unicode tag sequences |

### 1.3 HTML Threat Reduction

Removed elements: `<script>`, `<style>`, `<noscript>`, `<header>`, `<footer>`, `<nav>`, `<aside>`

### 1.4 Length Limiting

- Search results: 500 chars per field
- Fetch content: 10,000 chars (configurable)

---

## 2. Integration Architecture

### 2.1 High-Level Flow

```
┌─────────────────────────────────────────────────────────────┐
│                    BEFORE (Current)                          │
│                                                               │
│  Web Search/Fetch → Provider → JSON.dumps → LLM Prompt      │
│                      (NO SANITIZATION)                        │
└─────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────┐
│                    AFTER (Target)                            │
│                                                               │
│  Web Search/Fetch → Provider → Sanitizer → Delimited → LLM  │
│                              ↓                                │
│                    Multi-layer hardening                    │
└─────────────────────────────────────────────────────────────┘
```

### 2.2 Component Placement

```
deerflow/
└── security/
    ├── __init__.py
    ├── sanitizer.py          # Core sanitization logic
    ├── content_delimiter.py  # Delimiter wrapping
    └── html_cleaner.py       # HTML stripping for web_fetch
└── community/
    └── searx/                # New hardened SearX provider
        ├── __init__.py
        ├── tools.py
        └── client.py
```

---

## 3. Implementation Plan

### Phase 1: Core Sanitization Module (Priority: HIGH)

**File:** `backend/packages/harness/deerflow/security/sanitizer.py`

```python
"""Prompt injection hardening sanitizer based on OpenClaw patterns."""

import re
import unicodedata
from typing import Optional


class PromptInjectionSanitizer:
    """Sanitizes external content for safe LLM consumption."""
    
    # Zero-width and invisible characters (OpenClaw pattern)
    INVISIBLE_CHARS = [
        '\u200b', '\u200c', '\u200d', '\u200e', '\u200f',  # Zero-width spaces
        '\u2060', '\u2061', '\u2062', '\u2063', '\u2064',  # Word joiners
        '\ufeff', '\ufffe',                                  # BOM
        '\u00ad',                                           # Soft hyphen
        '\u034f',                                           # Combining grapheme
        '\u061c',                                           # Arabic letter mark
        '\u115f', '\u1160',                                  # Hangul fillers
        '\u17b4', '\u17b5',                                  # Khmer vowels
        '\u180e',                                           # Mongolian separator
        '\u3164',                                           # Hangul filler
        '\uffa0',                                           # Halfwidth Hangul
    ]
    
    def sanitize(self, text: str, max_length: Optional[int] = None) -> str:
        """Apply all sanitization layers.
        
        Args:
            text: Raw text to sanitize
            max_length: Optional maximum length (with ellipsis)
            
        Returns:
            Sanitized text safe for LLM prompts
        """
        if not text:
            return ''
        
        # Layer 1: Remove invisible/zero-width characters
        text = self._remove_invisible(text)
        
        # Layer 2: Remove control characters (except \n, \t)
        text = self._remove_control_chars(text)
        
        # Layer 3: Remove symbols (So, Sk categories)
        text = self._remove_symbols(text)
        
        # Layer 4: Normalize Unicode (NFC)
        text = unicodedata.normalize('NFC', text)
        
        # Layer 5: Remove Private Use Area
        text = self._remove_pua(text)
        
        # Layer 6: Remove tag characters
        text = self._remove_tag_chars(text)
        
        # Layer 7: Collapse whitespace
        text = re.sub(r'\s+', ' ', text).strip()
        
        # Layer 8: Length limiting
        if max_length and len(text) > max_length:
            text = text[:max_length-3] + '...'
        
        return text
    
    def _remove_invisible(self, text: str) -> str:
        for char in self.INVISIBLE_CHARS:
            text = text.replace(char, '')
        return text
    
    def _remove_control_chars(self, text: str) -> str:
        return ''.join(c for c in text 
                      if unicodedata.category(c) != 'Cc' or c in '\n\t')
    
    def _remove_symbols(self, text: str) -> str:
        return ''.join(c for c in text 
                      if unicodedata.category(c) not in ('So', 'Sk'))
    
    def _remove_pua(self, text: str) -> str:
        return ''.join(c for c in text 
                      if not (0xE000 <= ord(c) <= 0xF8FF 
                             or 0xF0000 <= ord(c) <= 0x10FFFF))
    
    def _remove_tag_chars(self, text: str) -> str:
        return ''.join(c for c in text 
                      if not (0xE0000 <= ord(c) <= 0xE007F))


# Global instance
sanitizer = PromptInjectionSanitizer()
```

### Phase 2: Content Delimiter (Priority: HIGH)

**File:** `backend/packages/harness/deerflow/security/content_delimiter.py`

```python
"""Content delimiter wrapper for safe LLM prompt integration."""

from typing import Union
import json


# OpenClaw-style delimiters
START_DELIMITER = "<<<EXTERNAL_UNTRUSTED_CONTENT>>>"
END_DELIMITER = "<<<END_EXTERNAL_UNTRUSTED_CONTENT>>>"


def wrap_untrusted_content(content: Union[str, dict, list]) -> str:
    """Wrap external content with safety delimiters.
    
    This creates a semantic boundary between system instructions
    and untrusted external data, helping prevent prompt injection.
    
    Args:
        content: Raw content (string, dict, or list)
        
    Returns:
        Delimited string for LLM consumption
    """
    if isinstance(content, (dict, list)):
        text = json.dumps(content, indent=2, ensure_ascii=False)
    else:
        text = str(content)
    
    return f"{START_DELIMITER}\n{text}\n{END_DELIMITER}"


def unwrap_trusted_content(delimited: str) -> str:
    """Extract content from delimiters (for testing/debugging).
    
    Args:
        delimited: Content wrapped in delimiters
        
    Returns:
        Raw content string
    """
    lines = delimited.split('\n')
    if lines[0] == START_DELIMITER and lines[-1] == END_DELIMITER:
        return '\n'.join(lines[1:-1])
    return delimited
```

### Phase 3: HTML Cleaner for Web Fetch (Priority: HIGH)

**File:** `backend/packages/harness/deerflow/security/html_cleaner.py`

```python
"""HTML-to-text extraction with security-focused stripping."""

from html.parser import HTMLParser
from typing import Optional


class SecureTextExtractor(HTMLParser):
    """Extract visible text while stripping potentially dangerous elements.
    
    Based on OpenClaw's fetch.sh implementation.
    """
    
    DANGEROUS_TAGS = {
        'script', 'style', 'noscript', 
        'header', 'footer', 'nav', 'aside',
        'iframe', 'object', 'embed', 'form',
    }
    
    def __init__(self):
        super().__init__()
        self.text = []
        self.skip_depth = 0
    
    def handle_starttag(self, tag, attrs):
        if tag in self.DANGEROUS_TAGS:
            self.skip_depth += 1
    
    def handle_endtag(self, tag):
        if tag in self.DANGEROUS_TAGS and self.skip_depth > 0:
            self.skip_depth -= 1
    
    def handle_data(self, data):
        if self.skip_depth == 0:
            self.text.append(data)
    
    def get_text(self) -> str:
        return ' '.join(self.text)


def extract_secure_text(html: str, max_chars: Optional[int] = None) -> str:
    """Extract clean text from HTML.
    
    Args:
        html: Raw HTML content
        max_chars: Optional maximum length
        
    Returns:
        Clean text without dangerous elements
    """
    extractor = SecureTextExtractor()
    extractor.feed(html)
    text = extractor.get_text()
    
    # Collapse whitespace
    import re
    text = re.sub(r'[ \t]+', ' ', text)
    text = re.sub(r'\n{3,}', '\n\n', text)
    text = text.strip()
    
    if max_chars and len(text) > max_chars:
        text = text[:max_chars-3] + '...'
    
    return text
```

### Phase 4: Hardened SearX Provider (Priority: HIGH)

**File:** `backend/packages/harness/deerflow/community/searx/tools.py`

```python
"""Hardened SearX web search and fetch tools."""

import json
import os
from urllib.parse import quote

import httpx
from langchain.tools import tool

from deerflow.config import get_app_config
from deerflow.security.content_delimiter import wrap_untrusted_content
from deerflow.security.sanitizer import sanitizer
from deerflow.security.html_cleaner import extract_secure_text


def _get_searx_config() -> dict:
    """Get SearX configuration from app config."""
    config = get_app_config().get_tool_config("web_search")
    return {
        "url": config.model_extra.get("searx_url", "http://localhost:8888"),
        "max_results": config.model_extra.get("max_results", 10),
    }


@tool("web_search", parse_docstring=True)
def web_search_tool(query: str, max_results: int = 10) -> str:
    """Search the web using hardened SearX instance.
    
    All results are sanitized against prompt injection attacks.
    
    Args:
        query: Search keywords
        max_results: Maximum results to return (default 10)
    """
    cfg = _get_searx_config()
    searx_url = cfg["url"]
    
    # URL-safe encoding
    encoded_query = quote(query)
    
    try:
        response = httpx.get(
            f"{searx_url}/search",
            params={
                "q": encoded_query,
                "format": "json",
                "max_results": min(max_results, cfg["max_results"]),
            },
            timeout=30.0
        )
        response.raise_for_status()
        data = response.json()
    except Exception as e:
        return json.dumps({"error": f"Search failed: {e}"})
    
    # Sanitize and limit results
    results = []
    for r in data.get("results", [])[:max_results]:
        results.append({
            "title": sanitizer.sanitize(r.get("title", "")),
            "url": r.get("url", ""),  # Keep URL intact
            "content": sanitizer.sanitize(r.get("content", ""), max_length=500),
        })
    
    output = {
        "query": query,
        "total_results": len(results),
        "results": results,
    }
    
    # Wrap with security delimiters
    return wrap_untrusted_content(output)


@tool("web_fetch", parse_docstring=True)
async def web_fetch_tool(url: str, max_chars: int = 10000) -> str:
    """Fetch web page content with security hardening.
    
    Dangerous HTML elements are stripped and content is sanitized.
    
    Args:
        url: URL to fetch
        max_chars: Maximum characters to return (default 10000)
    """
    try:
        async with httpx.AsyncClient() as client:
            response = await client.get(url, timeout=30.0)
            response.raise_for_status()
            html = response.text
    except Exception as e:
        return wrap_untrusted_content({"error": f"Fetch failed: {e}"})
    
    # Extract text and sanitize
    raw_text = extract_secure_text(html)
    clean_text = sanitizer.sanitize(raw_text, max_length=max_chars)
    
    # Wrap with security delimiters
    return wrap_untrusted_content(clean_text)
```

### Phase 5: Configuration Schema Update (Priority: MEDIUM)

**Update:** `config.example.yaml`

```yaml
# ============================================================================
# Security-First Web Search Configuration
# ============================================================================
# Hardened SearX provider with prompt injection protection

tools:
  # Hardened SearX web search (RECOMMENDED for private instances)
  - name: web_search
    group: web
    use: deerflow.community.searx.tools:web_search_tool
    searx_url: http://your-searx-instance:8888
    max_results: 10
  
  # Hardened web fetch
  - name: web_fetch
    group: web
    use: deerflow.community.searx.tools:web_fetch_tool
```

---

## 4. Testing Strategy

### 4.1 Unit Tests

**File:** `backend/tests/test_security_sanitizer.py`

```python
"""Tests for prompt injection sanitizer."""

import pytest
from deerflow.security.sanitizer import PromptInjectionSanitizer


class TestPromptInjectionSanitizer:
    """Test cases based on OpenClaw patterns."""
    
    def test_removes_zero_width_spaces(self):
        """Zero-width characters are common steganography vectors."""
        sanitizer = PromptInjectionSanitizer()
        text = "Hello\u200bWorld\u200c"  # ZWSP and ZWNJ
        result = sanitizer.sanitize(text)
        assert "\u200b" not in result
        assert "\u200c" not in result
        assert result == "HelloWorld"
    
    def test_removes_control_chars(self):
        """Control chars can disrupt prompt parsing."""
        sanitizer = PromptInjectionSanitizer()
        text = "Hello\x00World\x01Test"
        result = sanitizer.sanitize(text)
        assert "\x00" not in result
        assert "\x01" not in result
        assert "Hello" in result
    
    def test_preserves_newlines_and_tabs(self):
        """Structural characters should be preserved."""
        sanitizer = PromptInjectionSanitizer()
        text = "Line1\nLine2\tTabbed"
        result = sanitizer.sanitize(text)
        assert "\n" in result
        assert "\t" in result
    
    def test_truncates_long_content(self):
        """Length limiting prevents context overflow."""
        sanitizer = PromptInjectionSanitizer()
        text = "A" * 1000
        result = sanitizer.sanitize(text, max_length=100)
        assert len(result) == 100
        assert result.endswith("...")
    
    def test_handles_pua_characters(self):
        """Private Use Area chars can encode hidden data."""
        sanitizer = PromptInjectionSanitizer()
        text = "Hello\uE000World"  # PUA start
        result = sanitizer.sanitize(text)
        assert "\uE000" not in result


class TestContentDelimiter:
    """Test delimiter wrapping."""
    
    def test_wraps_dict_content(self):
        from deerflow.security.content_delimiter import wrap_untrusted_content
        
        content = {"title": "Test", "url": "http://example.com"}
        result = wrap_untrusted_content(content)
        
        assert "<<<EXTERNAL_UNTRUSTED_CONTENT>>>" in result
        assert "<<<END_EXTERNAL_UNTRUSTED_CONTENT>>>" in result
        assert "Test" in result
    
    def test_wraps_string_content(self):
        from deerflow.security.content_delimiter import wrap_untrusted_content
        
        content = "Raw text from web"
        result = wrap_untrusted_content(content)
        
        assert "<<<EXTERNAL_UNTRUSTED_CONTENT>>>" in result
        assert "Raw text from web" in result
```

### 4.2 Integration Tests

```python
"""Integration tests for hardened web tools."""

import pytest


class TestHardenedSearxSearch:
    """Test hardened SearX search against prompt injection."""
    
    def test_search_results_are_delimited(self):
        """Results must be wrapped in security delimiters."""
        from deerflow.community.searx.tools import web_search_tool
        
        result = web_search_tool("test query")
        
        assert "<<<EXTERNAL_UNTRUSTED_CONTENT>>>" in result
        assert "<<<END_EXTERNAL_UNTRUSTED_CONTENT>>>" in result
    
    def test_malicious_content_is_sanitized(self):
        """Malicious payloads in search results are neutralized."""
        # This would require mocking the SearX response
        pass


class TestHardenedWebFetch:
    """Test hardened web fetch against XSS/prompt injection."""
    
    def test_html_scripts_are_removed(self):
        """Script tags must be stripped."""
        from deerflow.security.html_cleaner import extract_secure_text
        
        html = "<p>Hello</p><script>alert('xss')</script><p>World</p>"
        result = extract_secure_text(html)
        
        assert "script" not in result.lower()
        assert "alert" not in result
        assert "Hello" in result
        assert "World" in result
```

---

## 5. Deployment Plan

### Step 1: Add Security Module
```bash
cd /home/data/deerflow-factory/deer-flow/backend/packages/harness/deerflow
mkdir -p security
# Create sanitizer.py, content_delimiter.py, html_cleaner.py
```

### Step 2: Add SearX Provider
```bash
mkdir -p community/searx
# Create __init__.py, tools.py
```

### Step 3: Update Dependencies
```bash
# Verify httpx is available (should be via langchain)
uv pip show httpx
```

### Step 4: Configuration
1. Copy `config.example.yaml` to `config.yaml`
2. Replace `web_search` and `web_fetch` tools with hardened SearX versions
3. Set `searx_url` to your private instance

### Step 5: Testing
```bash
cd backend
uv run python -m pytest tests/test_security_sanitizer.py -v
uv run python -m pytest tests/test_searx_tools.py -v
```

---

## 6. Migration Guide for Existing Deployments

### From DuckDuckGo/Tavily to Hardened SearX

1. **Backup current config:**
   ```bash
   cp config.yaml config.yaml.pre-security.bak
   ```

2. **Update tools section:**
   ```yaml
   # OLD (remove or comment)
   # - name: web_search
   #   group: web
   #   use: deerflow.community.ddg_search.tools:web_search_tool
   
   # NEW (add)
   - name: web_search
     group: web
     use: deerflow.community.searx.tools:web_search_tool
     searx_url: http://your-searx:8888
     max_results: 10
   ```

3. **Restart services:**
   ```bash
   make docker-restart
   # or
   make dev-restart
   ```

---

## 7. Verification Checklist

- [ ] Sanitizer unit tests pass
- [ ] Content delimiter tests pass
- [ ] HTML cleaner tests pass
- [ ] SearX search integration tests pass
- [ ] SearX fetch integration tests pass
- [ ] Malicious payload test: zero-width characters removed
- [ ] Malicious payload test: control characters removed
- [ ] Malicious payload test: script tags stripped
- [ ] Content delimiters present in output
- [ ] Private SearX instance responds correctly
- [ ] Configuration migration documented

---

## 8. References

### OpenClaw Sources
- `~/.openclaw/workspace-websearch/searx-scripts/search.sh`
- `~/.openclaw/workspace-websearch/fetch-scripts/fetch.sh`
- `~/.openclaw/workspace-websearch/AGENTS.md`
- `~/.openclaw/workspace-websearch/SOUL.md`

### OWASP Resources
- OWASP Top 10 for LLM Applications: LLM01 (Prompt Injection)
- OWASP LLM Threats: https://genai.owasp.org/llm-top-10/

### DeerFlow Integration Points
- `deerflow/community/ddg_search/tools.py` (reference)
- `deerflow/community/jina_ai/tools.py` (reference)
- `deerflow/guardrails/` (existing security framework)

---

## 9. Summary

This integration plan brings OpenClaw's battle-tested prompt injection hardening to DeerFlow through:

1. **Content Delimiters**: Clear semantic boundary markers
2. **Unicode Sanitization**: Removal of zero-width and invisible characters
3. **HTML Threat Reduction**: Stripping of dangerous elements
4. **Length Limiting**: Context overflow protection
5. **Clean Architecture**: Reusable security module

**Estimated Effort:** 2-3 days for full implementation and testing  
**Risk Level:** LOW (additive changes, existing tools remain available)  
**Security Impact:** HIGH (eliminates major prompt injection vector)