Conversational backend // Go + SSE agent service

Live

rony-chat-bot

Go HTTP service that wraps an LLM agent and streams responses to web clients over server-sent events. Configurable persona (Rony), persistent storage, MCP-style tool calling, and context-window management with on-demand compaction.

GoChromaDBOpenAI APIAnthropic APISSEMCP
Source ↗
  • OutputSSE service
  • RuntimeGo + Podman
  • SearchHybrid RAG
Chat backendReady
rony-chat-bot mark

Stream grounded answers, preserve conversations, and embed the same service in any web client.

Overview

Rony Chat Bot is the reusable backend behind chat experiences such as Portfolio AI Chat. It gives a web client a small HTTP contract while keeping provider selection, retrieval, persona, persistence, and context management on the service side.

The local stack runs as three Podman services: the Go bot on port 7331, a local llama.cpp chat model, and a llama.cpp embedder for semantic retrieval.

Rony Chat Widget running against the local bot

Runtime

The service accepts a conversation as JSON and can answer either as a complete response or as an SSE stream. The streaming path emits lifecycle events such as start, chunk, sources, compaction, done, and error, allowing a client to render progress without waiting for the full model output.

Rony Chat Widget showing a streamed answer and retrieved sources

The default local provider is qwen2.5-3b-instruct behind llama.cpp. The provider interface also supports OpenAI-compatible endpoints, Anthropic, Ollama, and other local models without changing the HTTP client contract.

Architecture

Boundary Responsibility
HTTP server Validates requests, exposes chat/health/reindex endpoints, and keeps the SSE stream flushable.
Agent runner Builds the system prompt, selects a provider, invokes tools, and manages the response lifecycle.
Hybrid RAG Combines SQLite FTS5 keyword search with multilingual embeddings and returns the source slugs used for an answer.
SQLite store Persists conversations, messages, indexed chunks, and retrieval metadata.
Web widget Provides a dependency-free drop-in client with bilingual controls, conversation restore, Markdown-lite rendering, and responsive layout.

The service is intentionally independent from the portfolio frontend. Projects and reference documents are fetched from JSON sources during reindexing, so the bot does not need a mirrored Markdown corpus.

Widget integration

The web/ directory contains a dependency-free widget that can be added to Astro, Next.js, or a plain HTML page with one stylesheet and one script tag. Configuration stays in data-* attributes:

<script
  src="/chat-widget.js"
  data-api-url="https://chat.example.com"
  data-title="Ask me anything"
  data-position="bottom-right"
  data-theme="auto"
  defer
></script>

The widget supports English and Spanish interface controls, stores the active conversation id in localStorage, restores known conversations after a reload, and switches to a full-screen panel on narrow screens.

Operations

  • Health — GET /api/health?deep=true verifies the LLM provider, SQLite store, and indexed chunk count. The local Podman stack reports a healthy bot, model, and store.
  • Reindex — POST /api/reindex refreshes project and document sources, including embeddings when the embedder is available.
  • Local runtime — Podman publishes the bot at http://localhost:7331; the rony-llm and rony-embedder services stay private to the compose network.
  • Configuration — provider, CORS origins, source URLs, RAG settings, and context limits live in configs/portfolio-bot.yaml and environment variables.

The service is self-hosted by default, but the same binary can point at a cloud provider when privacy, latency, or model quality requirements change.

Files of note

  • internal/server/ — HTTP handlers, SSE transport, health checks, conversations, and middleware.
  • internal/agent/ — provider-neutral agent loop, tool calls, and context compaction.
  • internal/portfolio/ — source fetching, chunking, hybrid retrieval, embeddings, and SQLite persistence.
  • web/chat-widget.js — dependency-free client with SSE parsing, language switching, and history restore.
  • web/chat-widget.css — scoped responsive styles and light/dark theme tokens.
  • configs/portfolio-bot.yaml — local provider, source, RAG, and runtime configuration.