API Development with AnythingLLM

High-Level Architecture

User
                  │
                  ▼
           REST / GraphQL API
                  │
                  ▼
          Authentication Layer
                  │
                  ▼
          AnythingLLM Agent
         (Planning & Reasoning)
                  │
      ┌───────────┼────────────┐
      ▼           ▼            ▼
   MCP Tools    APIs      Databases
      │           │            │
      ▼           ▼            ▼
 GitHub      CRM/ERP      PostgreSQL
 Slack       Internal     Redis
 Email        APIs

Typical Project Structure

anythingllm-agent-api/
│
├── app/
│   ├── api/
│   │    ├── routes.py
│   │    ├── auth.py
│   │    └── middleware.py
│   │
│   ├── services/
│   │    ├── anythingllm.py
│   │    ├── tools.py
│   │    └── workflow.py
│   │
│   ├── models/
│   │    ├── requests.py
│   │    └── responses.py
│   │
│   ├── config.py
│   └── main.py
│
├── prompts/
│      system.md
│      planner.md
│      api.md
│
├── tools/
│      github.py
│      jira.py
│      database.py
│
├── tests/
│
├── docker-compose.yml
├── Dockerfile
└── README.md

Development Workflow

Client Request
      │
      ▼
Authentication
      │
      ▼
Validate Request
      │
      ▼
AnythingLLM Agent
      │
      ├── Understand intent
      ├── Select tools
      ├── Plan actions
      └── Execute
      │
      ▼
Collect Results
      │
      ▼
Format Response
      │
      ▼
Return JSON

Example API Flow

1. User

POST /agent/chat

Body

{
  "message": "Create a GitHub issue and notify Slack."
}

2. API

request
↓
validate
↓
AnythingLLM
↓
Agent
↓
Tool Selection
↓
GitHub Tool
↓
Slack Tool
↓
Result

3. Response

{
  "success": true,
  "steps": [
    "Created GitHub issue #101",
    "Posted Slack notification"
  ]
}

Recommended Prompt Architecture

Instead of one huge prompt, split prompts by responsibility.

System Prompt
        │
        ▼
Developer Prompt
        │
        ▼
Workflow Prompt
        │
        ▼
Task Prompt
        │
        ▼
User Prompt

Example:

System

You are an API automation agent.

Get REIT monero’s stories in your inbox
Join Medium for free to get updates from this writer.
Remember me for faster sign in
Always:
- think before acting
- use tools only when required
- return structured JSON
- never hallucinate results

Workflow Prompt

For every request:

1. Understand intent
2. Identify required tools
3. Execute tools sequentially
4. Verify outputs
5. Summarize
6. Return JSON

Tool Prompt

Available Tools

GitHub
Slack
Postgres
REST API
Filesystem
Only call tools that are necessary.

REST API Design

POST   /agent/chat
POST   /agent/task
POST   /agent/workflow
GET    /agent/status/{id}
GET    /agent/history
DELETE /agent/session/{id}

Recommended JSON Response

{
  "success": true,
  "requestId": "...",
  "plan": [
    "Analyze request",
    "Create issue",
    "Notify Slack"
  ],
  "executed": [
    "GitHub.createIssue",
    "Slack.sendMessage"
  ],
  "result": {
    "issue": 123,
    "channel": "#dev"
  },
  "errors": [],
  "duration": 2.31
}

Development Lifecycle

Requirements
      │
      ▼
Prompt Design
      │
      ▼
Tool Definition
      │
      ▼
API Development
      │
      ▼
Integration Testing
      │
      ▼
Prompt Evaluation
      │
      ▼
Deployment
      │
      ▼
Monitoring

Best Practices

• Keep business logic in your API/service layer; let the agent focus on reasoning and orchestration.
• Use structured JSON for all tool inputs and outputs to simplify validation and debugging.
• Implement retries, timeouts, and error handling for external tool calls.
• Version prompts independently from application code so they can be iterated safely.
• Log agent plans, tool invocations, and outcomes for observability and auditability.
• Maintain conversation state only as long as needed, and summarize long histories to control context size.
• Secure all API endpoints with authentication, authorization, and rate limiting.
• Write automated tests for prompt behavior, tool integrations, and end-to-end workflows.