How this works

AI in a Box is a product that gives you access to a self-hosted large language model server (llm) and a vector database (vectordb).

It’s designed in a way that only the midsever included with it can access the llm and vectordb. This is to ensure that the data is secure and that the llm and vectordb are not used for any other purpose than the intended use case.

Simplified overview

Loading graph...

flowchart TB
    linkStyle default stroke-width:2px,fill:none,stroke:blue;
    RequestSummary[Something Requests Summary]
    AIinaBox[AI in a Box]
    MidSever[Mid Server]
    ServiceNowBackend[ServiceNow Backend]

    RequestSummary --Creates Inference Record--> ServiceNowBackend
    ServiceNowBackend --Calls--> MidSever
    MidSever --Calls--> AIinaBox
    AIinaBox ==Responds==> MidSever
    MidSever ==Updates Inference Record==> ServiceNowBackend

Technical overview

Loading graph...

flowchart TB


    subgraph AIinaBox[AI in a Box]
        APIServer[API Server - nodejs]
        LLMServer[LLM Server - ollama]
        VectorDBServer[VectorDB Server -chromadb]
    end
    subgraph MidSever[Mid Server]
        RestCall[REST Call]
        ECCQueue[ECC Queue Outbound]
        ECCQueueInbound[ECC Queue Inbound<br/>non-streaming only]
    end
    subgraph ServiceNowBackend[ServiceNow Backend]
        InferenceRecord[Inference Record]
        SCRAPI[Scripted REST API<br/>streaming only]
    end
    subgraph ServiceNowFrontend[ServiceNow Frontend]
        RequestSummary[Something Request Summary]
        AMBClient[Live Polling Inference Record]
        ClientScript[Show Summary from Inference Record]
    end

    RequestSummary --Create--> InferenceRecord
    InferenceRecord --Triggers--> RestCall
    RestCall --> ECCQueue
    ECCQueue --> APIServer
    APIServer --- LLMServer
    APIServer --- VectorDBServer
    APIServer -..-> SCRAPI
    APIServer ==> ECCQueueInbound
    ECCQueueInbound ==updates==> InferenceRecord
    SCRAPI -.updates.-> InferenceRecord
    AMBClient --- InferenceRecord
    ClientScript --> AMBClient

The Stack

AI in a Box: The main product that includes the llm and vectordb servers.
- Node.js: The API server is built using Node.js.
- Expressjs: The API server uses Expressjs.
- Ollama: The large language model server.
- Chromadb: The vector database server.
Mid Server: The server that sits between the AI in a Box and the ServiceNow backend.
ServiceNow Backend: The backend that interacts with the AI in a Box.