DATA/deerflow-factory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: hardened DeerFlow factory

Browse Source 
Vendored deer-flow upstream (bytedance/deer-flow) plus prompt-injection
hardening:

- New deerflow.security package: content_delimiter, html_cleaner,
  sanitizer (8 layers — invisible chars, control chars, symbols, NFC,
  PUA, tag chars, horizontal whitespace collapse with newline/tab
  preservation, length cap)
- New deerflow.community.searx package: web_search, web_fetch,
  image_search backed by a private SearX instance, every external
  string sanitized and wrapped in <<<EXTERNAL_UNTRUSTED_CONTENT>>>
  delimiters
- All native community web providers (ddg_search, tavily, exa,
  firecrawl, jina_ai, infoquest, image_search) replaced with hard-fail
  stubs that raise NativeWebToolDisabledError at import time, so a
  misconfigured tool.use path fails loud rather than silently falling
  back to unsanitized output
- Native client back-doors (jina_client.py, infoquest_client.py)
  stubbed too
- Native-tool tests quarantined under tests/_disabled_native/
  (collect_ignore_glob via local conftest.py)
- Sanitizer Layer 7 fix: only collapse horizontal whitespace, preserve
  newlines and tabs so list/table structure survives
- Hardened runtime config.yaml references only the searx-backed tools
- Factory overlay (backend/) kept in sync with deer-flow tree as a
  reference / source

See HARDENING.md for the full audit trail and verification steps.
This commit is contained in:
DATA
2026-04-12 14:23:57 +02:00
commit 6de0bf9f5b
889 changed files with 173052 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

452
deer-flow/.agent/skills/smoke-test/references/SOP.md Normal file
View File

@@ -0,0 +1,452 @@
# DeerFlow Smoke Test Standard Operating Procedure (SOP)
This document describes the detailed operating steps for each phase of the DeerFlow smoke test.
## Phase 1: Code Update Check
### 1.1 Confirm Current Directory
**Objective**: Verify that the current working directory is the DeerFlow project root.
**Steps**:
1. Run `pwd` to view the current working directory
2. Check whether the directory contains the following files/directories:
- `Makefile`
- `backend/`
- `frontend/`
- `config.example.yaml`
**Success Criteria**: The current directory contains all of the files/directories listed above.
---
### 1.2 Check Git Status
**Objective**: Check whether there are uncommitted changes.
**Steps**:
1. Run `git status`
2. Check whether the output includes "Changes not staged for commit" or "Untracked files"
**Notes**:
- If there are uncommitted changes, recommend that the user commit or stash them first to avoid conflicts while pulling
- If the user confirms that they want to continue, this step can be skipped
---
### 1.3 Pull the Latest Code
**Objective**: Fetch the latest code updates.
**Steps**:
1. Run `git fetch origin main`
2. Run `git pull origin main`
**Success Criteria**:
- The commands succeed without errors
- The output shows "Already up to date" or indicates that new commits were pulled successfully
---
### 1.4 Confirm Code Update
**Objective**: Verify that the latest code was pulled successfully.
**Steps**:
1. Run `git log -1 --oneline` to view the latest commit
2. Record the commit hash and message
---
## Phase 2: Deployment Mode Selection and Environment Check
### 2.1 Choose Deployment Mode
**Objective**: Decide whether to use local mode or Docker mode.
**Decision Flow**:
1. Prefer local mode first to avoid network-related issues
2. If the user explicitly requests Docker, use Docker
3. If Docker network issues occur, switch to local mode automatically
---
### 2.2 Local Mode Environment Check
**Objective**: Verify that local development environment dependencies are satisfied.
#### 2.2.1 Check Node.js Version
**Steps**:
1. If nvm is used, run `nvm use 22` to switch to Node 22+
2. Run `node --version`
**Success Criteria**: Version >= 22.x
**Failure Handling**:
- If the version is too low, ask the user to install/switch Node.js with nvm:
```bash
nvm install 22
nvm use 22
```
- Or install it from the official website: https://nodejs.org/
---
#### 2.2.2 Check pnpm
**Steps**:
1. Run `pnpm --version`
**Success Criteria**: The command returns pnpm version information.
**Failure Handling**:
- If pnpm is not installed, ask the user to install it with `npm install -g pnpm`
---
#### 2.2.3 Check uv
**Steps**:
1. Run `uv --version`
**Success Criteria**: The command returns uv version information.
**Failure Handling**:
- If uv is not installed, ask the user to install uv
---
#### 2.2.4 Check nginx
**Steps**:
1. Run `nginx -v`
**Success Criteria**: The command returns nginx version information.
**Failure Handling**:
- macOS: install with Homebrew using `brew install nginx`
- Linux: install using the system package manager
---
#### 2.2.5 Check Required Ports
**Steps**:
1. Run the following commands to check ports:
```bash
lsof -i :2026 # Main port
lsof -i :3000 # Frontend
lsof -i :8001 # Gateway
lsof -i :2024 # LangGraph
```
**Success Criteria**: All ports are free, or they are occupied only by DeerFlow-related processes.
**Failure Handling**:
- If a port is occupied, ask the user to stop the related process
---
### 2.3 Docker Mode Environment Check (If Docker Is Selected)
#### 2.3.1 Check Whether Docker Is Installed
**Steps**:
1. Run `docker --version`
**Success Criteria**: The command returns Docker version information, such as "Docker version 24.x.x".
---
#### 2.3.2 Check Docker Daemon Status
**Steps**:
1. Run `docker info`
**Success Criteria**: The command runs successfully and shows Docker system information.
**Failure Handling**:
- If it fails, ask the user to start Docker Desktop or the Docker service
---
#### 2.3.3 Check Docker Compose Availability
**Steps**:
1. Run `docker compose version`
**Success Criteria**: The command returns Docker Compose version information.
---
#### 2.3.4 Check Required Ports
**Steps**:
1. Run `lsof -i :2026` (macOS/Linux) or `netstat -ano | findstr :2026` (Windows)
**Success Criteria**: Port 2026 is free, or it is occupied only by a DeerFlow-related process.
**Failure Handling**:
- If the port is occupied by another process, ask the user to stop that process or change the configuration
---
## Phase 3: Configuration Preparation
### 3.1 Check config.yaml
**Steps**:
1. Check whether `config.yaml` exists
2. If it does not exist, run `make config`
3. If it already exists, consider running `make config-upgrade` to merge new fields
**Validation**:
- Check whether at least one model is configured in config.yaml
- Check whether the model configuration references the correct environment variables
---
### 3.2 Check the .env File
**Steps**:
1. Check whether the `.env` file exists
2. If it does not exist, copy it from `.env.example`
3. Check whether the following environment variables are configured:
- `OPENAI_API_KEY` (or other model API keys)
- Other required settings
---
## Phase 4: Deployment Execution
### 4.1 Local Mode Deployment
#### 4.1.1 Check Dependencies
**Steps**:
1. Run `make check`
**Description**: This command validates all required tools (Node.js 22+, pnpm, uv, nginx).
---
#### 4.1.2 Install Dependencies
**Steps**:
1. Run `make install`
**Description**: This command installs both backend and frontend dependencies.
**Notes**:
- This step may take some time
- If network issues cause failures, try using a closer or mirrored package registry
---
#### 4.1.3 (Optional) Pre-pull the Sandbox Image
**Steps**:
1. If Docker / Container sandbox is used, run `make setup-sandbox`
**Description**: This step is optional and not needed for local sandbox mode.
---
#### 4.1.4 Start Services
**Steps**:
1. Run `make dev-daemon` (background mode)
**Description**: This command starts all services (LangGraph, Gateway, Frontend, Nginx).
**Notes**:
- `make dev` runs in the foreground and stops with Ctrl+C
- `make dev-daemon` runs in the background
- Use `make stop` to stop services
---
#### 4.1.5 Wait for Services to Start
**Steps**:
1. Wait 90-120 seconds for all services to start completely
2. You can monitor startup progress by checking these log files:
- `logs/langgraph.log`
- `logs/gateway.log`
- `logs/frontend.log`
- `logs/nginx.log`
---
### 4.2 Docker Mode Deployment (If Docker Is Selected)
#### 4.2.1 Initialize the Docker Environment
**Steps**:
1. Run `make docker-init`
**Description**: This command pulls the sandbox image if needed.
---
#### 4.2.2 Start Docker Services
**Steps**:
1. Run `make docker-start`
**Description**: This command builds and starts all required Docker containers.
---
#### 4.2.3 Wait for Services to Start
**Steps**:
1. Wait 60-90 seconds for all services to start completely
2. You can run `make docker-logs` to monitor startup progress
---
## Phase 5: Service Health Check
### 5.1 Local Mode Health Check
#### 5.1.1 Check Process Status
**Steps**:
1. Run the following command to check processes:
```bash
ps aux | grep -E "(langgraph|uvicorn|next|nginx)" | grep -v grep
```
**Success Criteria**: Confirm that the following processes are running:
- LangGraph (`langgraph dev`)
- Gateway (`uvicorn app.gateway.app:app`)
- Frontend (`next dev` or `next start`)
- Nginx (`nginx`)
---
#### 5.1.2 Check Frontend Service
**Steps**:
1. Use curl or a browser to visit `http://localhost:2026`
2. Verify that the page loads normally
**Example curl command**:
```bash
curl -I http://localhost:2026
```
**Success Criteria**: Returns an HTTP 200 status code.
---
#### 5.1.3 Check API Gateway
**Steps**:
1. Visit `http://localhost:2026/health`
**Example curl command**:
```bash
curl http://localhost:2026/health
```
**Success Criteria**: Returns health status JSON.
---
#### 5.1.4 Check LangGraph Service
**Steps**:
1. Visit relevant LangGraph endpoints to verify availability
---
### 5.2 Docker Mode Health Check (When Using Docker)
#### 5.2.1 Check Container Status
**Steps**:
1. Run `docker ps`
2. Confirm that the following containers are running:
- `deer-flow-nginx`
- `deer-flow-frontend`
- `deer-flow-gateway`
- `deer-flow-langgraph` (if not in gateway mode)
---
#### 5.2.2 Check Frontend Service
**Steps**:
1. Use curl or a browser to visit `http://localhost:2026`
2. Verify that the page loads normally
**Example curl command**:
```bash
curl -I http://localhost:2026
```
**Success Criteria**: Returns an HTTP 200 status code.
---
#### 5.2.3 Check API Gateway
**Steps**:
1. Visit `http://localhost:2026/health`
**Example curl command**:
```bash
curl http://localhost:2026/health
```
**Success Criteria**: Returns health status JSON.
---
#### 5.2.4 Check LangGraph Service
**Steps**:
1. Visit relevant LangGraph endpoints to verify availability
---
## Optional Functional Verification
### 6.1 List Available Models
**Steps**: Verify the model list through the API or UI.
---
### 6.2 List Available Skills
**Steps**: Verify the skill list through the API or UI.
---
### 6.3 Simple Chat Test
**Steps**: Send a simple message to test the complete workflow.
---
## Phase 6: Generate the Test Report
### 6.1 Collect Test Results
Summarize the execution status of each phase and record successful and failed items.
### 6.2 Record Issues
If anything fails, record detailed error information.
### 6.3 Generate the Report
Use the template to create a complete test report.
### 6.4 Provide Recommendations
Provide follow-up recommendations based on the test results.

612
deer-flow/.agent/skills/smoke-test/references/troubleshooting.md Normal file
View File

@@ -0,0 +1,612 @@
# Troubleshooting Guide
This document lists common issues encountered during DeerFlow smoke testing and how to resolve them.
## Code Update Issues
### Issue: `git pull` Fails with a Merge Conflict Warning
**Symptoms**:
```
error: Your local changes to the following files would be overwritten by merge
```
**Solutions**:
1. Option A: Commit local changes first
```bash
git add .
git commit -m "Save local changes"
git pull origin main
```
2. Option B: Stash local changes
```bash
git stash
git pull origin main
git stash pop # Restore changes later if needed
```
3. Option C: Discard local changes (use with caution)
```bash
git reset --hard HEAD
git pull origin main
```
---
## Local Mode Environment Issues
### Issue: Node.js Version Is Too Old
**Symptoms**:
```
Node.js version is too old. Requires 22+, got x.x.x
```
**Solutions**:
1. Install or upgrade Node.js with nvm:
```bash
nvm install 22
nvm use 22
```
2. Or download and install it from the official website: https://nodejs.org/
3. Verify the version:
```bash
node --version
```
---
### Issue: pnpm Is Not Installed
**Symptoms**:
```
command not found: pnpm
```
**Solutions**:
1. Install pnpm with npm:
```bash
npm install -g pnpm
```
2. Or use the official installation script:
```bash
curl -fsSL https://get.pnpm.io/install.sh | sh -
```
3. Verify the installation:
```bash
pnpm --version
```
---
### Issue: uv Is Not Installed
**Symptoms**:
```
command not found: uv
```
**Solutions**:
1. Use the official installation script:
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```
2. macOS users can also install it with Homebrew:
```bash
brew install uv
```
3. Verify the installation:
```bash
uv --version
```
---
### Issue: nginx Is Not Installed
**Symptoms**:
```
command not found: nginx
```
**Solutions**:
1. macOS (Homebrew):
```bash
brew install nginx
```
2. Ubuntu/Debian:
```bash
sudo apt update
sudo apt install nginx
```
3. CentOS/RHEL:
```bash
sudo yum install nginx
```
4. Verify the installation:
```bash
nginx -v
```
---
### Issue: Port Is Already in Use
**Symptoms**:
```
Error: listen EADDRINUSE: address already in use :::2026
```
**Solutions**:
1. Find the process using the port:
```bash
lsof -i :2026 # macOS/Linux
netstat -ano | findstr :2026 # Windows
```
2. Stop that process:
```bash
kill -9 <PID> # macOS/Linux
taskkill /PID <PID> /F # Windows
```
3. Or stop DeerFlow services first:
```bash
make stop
```
---
## Local Mode Dependency Installation Issues
### Issue: `make install` Fails Due to Network Timeout
**Symptoms**:
Network timeouts or connection failures occur during dependency installation.
**Solutions**:
1. Configure pnpm to use a mirror registry:
```bash
pnpm config set registry https://registry.npmmirror.com
```
2. Configure uv to use a mirror registry:
```bash
uv pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
```
3. Retry the installation:
```bash
make install
```
---
### Issue: Python Dependency Installation Fails
**Symptoms**:
Errors occur during `uv sync`.
**Solutions**:
1. Clean the uv cache:
```bash
cd backend
uv cache clean
```
2. Resync dependencies:
```bash
cd backend
uv sync
```
3. View detailed error logs:
```bash
cd backend
uv sync --verbose
```
---
### Issue: Frontend Dependency Installation Fails
**Symptoms**:
Errors occur during `pnpm install`.
**Solutions**:
1. Clean the pnpm cache:
```bash
cd frontend
pnpm store prune
```
2. Remove node_modules and the lock file:
```bash
cd frontend
rm -rf node_modules pnpm-lock.yaml
```
3. Reinstall:
```bash
cd frontend
pnpm install
```
---
## Local Mode Service Startup Issues
### Issue: Services Exit Immediately After Startup
**Symptoms**:
Processes exit quickly after running `make dev-daemon`.
**Solutions**:
1. Check log files:
```bash
tail -f logs/langgraph.log
tail -f logs/gateway.log
tail -f logs/frontend.log
tail -f logs/nginx.log
```
2. Check whether config.yaml is configured correctly
3. Check environment variables in the .env file
4. Confirm that required ports are not occupied
5. Stop all services and restart:
```bash
make stop
make dev-daemon
```
---
### Issue: Nginx Fails to Start Because Temp Directories Do Not Exist
**Symptoms**:
```
nginx: [emerg] mkdir() "/opt/homebrew/var/run/nginx/client_body_temp" failed (2: No such file or directory)
```
**Solutions**:
Add local temp directory configuration to `docker/nginx/nginx.local.conf` so nginx uses the repository's temp directory.
Add the following at the beginning of the `http` block:
```nginx
client_body_temp_path temp/client_body_temp;
proxy_temp_path temp/proxy_temp;
fastcgi_temp_path temp/fastcgi_temp;
uwsgi_temp_path temp/uwsgi_temp;
scgi_temp_path temp/scgi_temp;
```
Note: The `temp/` directory under the repository root is created automatically by `make dev` or `make dev-daemon`.
---
### Issue: Nginx Fails to Start (General)
**Symptoms**:
The nginx process fails to start or reports an error.
**Solutions**:
1. Check the nginx configuration:
```bash
nginx -t -c docker/nginx/nginx.local.conf -p .
```
2. Check nginx logs:
```bash
tail -f logs/nginx.log
```
3. Ensure no other nginx process is running:
```bash
ps aux | grep nginx
```
4. If needed, stop existing nginx processes:
```bash
pkill -9 nginx
```
---
### Issue: Frontend Compilation Fails
**Symptoms**:
Compilation errors appear in `frontend.log`.
**Solutions**:
1. Check frontend logs:
```bash
tail -f logs/frontend.log
```
2. Check whether Node.js version is 22+
3. Reinstall frontend dependencies:
```bash
cd frontend
rm -rf node_modules .next
pnpm install
```
4. Restart services:
```bash
make stop
make dev-daemon
```
---
### Issue: Gateway Fails to Start
**Symptoms**:
Errors appear in `gateway.log`.
**Solutions**:
1. Check gateway logs:
```bash
tail -f logs/gateway.log
```
2. Check whether config.yaml exists and has valid formatting
3. Check whether Python dependencies are complete:
```bash
cd backend
uv sync
```
4. Confirm that the LangGraph service is running normally (if not in gateway mode)
---
### Issue: LangGraph Fails to Start
**Symptoms**:
Errors appear in `langgraph.log`.
**Solutions**:
1. Check LangGraph logs:
```bash
tail -f logs/langgraph.log
```
2. Check config.yaml
3. Check whether Python dependencies are complete
4. Confirm that port 2024 is not occupied
---
## Docker-Related Issues
### Issue: Docker Commands Cannot Run
**Symptoms**:
```
Cannot connect to the Docker daemon
```
**Solutions**:
1. Confirm that Docker Desktop is running
2. macOS: check whether the Docker icon appears in the top menu bar
3. Linux: run `sudo systemctl start docker`
4. Run `docker info` again to verify
---
### Issue: `make docker-init` Fails to Pull the Image
**Symptoms**:
```
Error pulling image: connection refused
```
**Solutions**:
1. Check network connectivity
2. Configure a Docker image mirror if needed
3. Check whether a proxy is required
4. Switch to local installation mode if necessary (recommended)
---
## Configuration File Issues
### Issue: config.yaml Is Missing or Invalid
**Symptoms**:
```
Error: could not read config.yaml
```
**Solutions**:
1. Regenerate the configuration file:
```bash
make config
```
2. Check YAML syntax:
- Make sure indentation is correct (use 2 spaces)
- Make sure there are no tab characters
- Check that there is a space after each colon
3. Use a YAML validation tool to check the format
---
### Issue: Model API Key Is Not Configured
**Symptoms**:
After services start, API requests fail with authentication errors.
**Solutions**:
1. Edit the .env file and add the API key:
```bash
OPENAI_API_KEY=your-actual-api-key-here
```
2. Restart services (local mode):
```bash
make stop
make dev-daemon
```
3. Restart services (Docker mode):
```bash
make docker-stop
make docker-start
```
4. Confirm that the model configuration in config.yaml references the environment variable correctly
---
## Service Health Check Issues
### Issue: Frontend Page Is Not Accessible
**Symptoms**:
The browser shows a connection failure when visiting http://localhost:2026.
**Solutions** (local mode):
1. Confirm that the nginx process is running:
```bash
ps aux | grep nginx
```
2. Check nginx logs:
```bash
tail -f logs/nginx.log
```
3. Check firewall settings
**Solutions** (Docker mode):
1. Confirm that the nginx container is running:
```bash
docker ps | grep nginx
```
2. Check nginx logs:
```bash
cd docker && docker compose -p deer-flow-dev -f docker-compose-dev.yaml logs nginx
```
3. Check firewall settings
---
### Issue: API Gateway Health Check Fails
**Symptoms**:
Accessing `/health` returns an error or times out.
**Solutions** (local mode):
1. Check gateway logs:
```bash
tail -f logs/gateway.log
```
2. Confirm that config.yaml exists and has valid formatting
3. Check whether Python dependencies are complete
4. Confirm that the LangGraph service is running normally
**Solutions** (Docker mode):
1. Check gateway container logs:
```bash
make docker-logs-gateway
```
2. Confirm that config.yaml is mounted correctly
3. Check whether Python dependencies are complete
4. Confirm that the LangGraph service is running normally
---
## Common Diagnostic Commands
### Local Mode Diagnostics
#### View All Service Processes
```bash
ps aux | grep -E "(langgraph|uvicorn|next|nginx)" | grep -v grep
```
#### View Service Logs
```bash
# View all logs
tail -f logs/*.log
# View specific service logs
tail -f logs/langgraph.log
tail -f logs/gateway.log
tail -f logs/frontend.log
tail -f logs/nginx.log
```
#### Stop All Services
```bash
make stop
```
#### Fully Reset the Local Environment
```bash
make stop
make clean
make config
make install
make dev-daemon
```
---
### Docker Mode Diagnostics
#### View All Container Status
```bash
docker ps -a
```
#### View Container Resource Usage
```bash
docker stats
```
#### Enter a Container for Debugging
```bash
docker exec -it deer-flow-gateway sh
```
#### Clean Up All DeerFlow-Related Containers and Images
```bash
make docker-stop
cd docker && docker compose -p deer-flow-dev -f docker-compose-dev.yaml down -v
```
#### Fully Reset the Docker Environment
```bash
make docker-stop
make clean
make config
make docker-init
make docker-start
```
---
## Get More Help
If the solutions above do not resolve the issue:
1. Check the GitHub issues for the project: https://github.com/bytedance/deer-flow/issues
2. Review the project documentation: README.md and the `backend/docs/` directory
3. Open a new issue and include detailed error logs