# Vastu AI Chatbot - Backend

Node.js (ESM) + Fastify + MySQL + Qdrant + OpenAI. Strict RAG: the LLM only answers from retrieved, approved knowledge (admin Q&A and uploaded book content). Anything else gets a polite "not enough information" reply and is queued for admin review.

Docs: `../docs/` (architecture, schema, ER diagram, RAG workflow, API spec, deployment, mobile integration). `../requirement.txt` is the source of truth.

## Requirements
- Node.js 20+ (developed on 22)
- MySQL 8 (8.4 verified). MariaDB 10.4 is tolerated by the code (JSON columns are handled) but is not the target.
- Qdrant 1.10+ (1.19 verified)
- An OpenAI API key (server side only)

## Quick start (Docker)
```bash
cp .env.example .env            # set JWT_SECRET, OPENAI_API_KEY, SEED_ADMIN_*, DB_PASSWORD, MYSQL_ROOT_PASSWORD
docker compose up -d --build    # mysql + qdrant + api (migrations run on boot)
docker compose run --rm api npm run seed     # creates the default super_admin from SEED_ADMIN_*
curl localhost:3000/ready
```

## Quick start (local Node)
```bash
npm install
cp .env.example .env            # point DB_* and QDRANT_URL at your services
npm run migrate                 # creates the database (DB_NAME) + tables + seed rows; safe to re-run
npm run seed                    # default super_admin (idempotent)
npm run dev                     # or: npm start
```

## Scripts
| Script | What it does |
|---|---|
| `npm start` / `npm run dev` | API + in-process worker (`--watch` for dev) |
| `npm run migrate` | Runs `src/database/schema.sql` (idempotent) |
| `npm run seed` | Creates `SEED_ADMIN_EMAIL` as `super_admin` (bcrypt) if missing |
| `npm run reindex [-- --recreate] [-- --only=qa or docs]` | Rebuilds Qdrant from MySQL. `--recreate` drops the collection first (use after changing the embedding model or dimensions) |
| `npm test` | Unit and API tests (`node:test`, no external services needed) |
| `npm run check` | `node --check` over every JS file |
| `npm run mock:openai` | Dev-only OpenAI-compatible stub (see "End-to-end verification") |
| `npm run test:e2e` | Smoke test against a running stack |

## Environment
Everything is configured through environment variables; `.env.example` documents each one. The important ones:

| Variable | Notes |
|---|---|
| `JWT_SECRET` | required (>= 32 chars in production) |
| `OPENAI_API_KEY`, `OPENAI_CHAT_MODEL`, `OPENAI_EMBEDDING_MODEL` | models are never hard-coded. `OPENAI_EMBEDDING_DIMENSIONS` only if you need a non-default size. `OPENAI_TEMPERATURE=none` for models that reject it |
| `DB_*`, `QDRANT_URL`, `QDRANT_COLLECTION` | connections |
| `CORS_ORIGINS` | comma-separated allow-list for the admin panel. Mobile apps send no Origin and are unaffected |
| `WORKER_ENABLED` | run the worker on exactly one instance |
| `RATE_LIMIT_*` | per minute: global 100/IP, login 5/IP, chat 20/user, admin 300 |
| `UPLOAD_DIR`, `MAX_UPLOAD_MB` | document uploads (pdf/txt/docx, default 50 MB) |

Secrets are read from the environment only. They are never stored in the DB, returned by the API or written to logs (pino redaction on auth headers and password fields).

## Layout
```
src/
  server.js app.js container.js     entry, Fastify factory (no side effects), composition root
  routes/ controllers/              thin HTTP layer
  services/                         business logic (auth, questions, import, documents, chat, unanswered, settings, dashboard)
  repositories/                     all SQL (parameterised ? placeholders only)
  ai/                               OpenAI client (timeout, retry/backoff, usage logging), embeddings + LRU cache
  rag/                              queryUnderstanding, retriever, vectorStore (Qdrant), priorityMerge, promptBuilder,
                                    chatOrchestrator, chunker, textCleaner, extractors
  workers/                          in-process worker, documentProcessor, reindex script
  middleware/ validators/ models/ utils/ config/ database/
test/                               node:test suites (+ helpers with in-memory fakes)
scripts/                            check-syntax, mock-openai, integration (e2e smoke)
```
Layering is strict: routes -> controllers -> services -> repositories. AI and retrieval code lives only in `ai/` and `rag/`.

## How a chat message flows
1. JWT (device token), rate limit (20/min/user) and schema validation. A blank message returns `EMPTY_QUESTION`.
2. Load the conversation and the last N messages. One cheap LLM call rewrites the message into a standalone English query (skipped for plain single-turn English).
3. Embed (LRU-cached), search Qdrant top-k with an `is_active` filter, re-validate every hit against MySQL.
4. Drop hits below `min_similarity_score`, then priority merge (Q&A beats book on the same topic; priorities come from `knowledge_sources`).
5. If nothing usable is left: NO-ANSWER path (no LLM call) and an `unanswered_questions` upsert (dedupe by normalised-question hash, `times_asked++`).
6. Otherwise answer with the context prompt (temperature <= 0.2). A reply of `INSUFFICIENT_CONTEXT` also routes to the NO-ANSWER path.
7. Persist both messages (+ sources) and return `{answered, answer, conversation_id, message_id, sources}`. Sources are only included when `show_sources_to_mobile=true`.

## Documents and bulk embedding (no Redis)
`POST /api/admin/documents` stores the file. `POST /api/admin/documents/:id/process` marks it `processing` (202). The in-process worker polls MySQL, extracts (pdf-parse / mammoth / plain text), cleans, chunks (~600 tokens, 100 overlap), embeds in batches and upserts to Qdrant, then marks `processed` (or `failed` with a friendly `error_message`). Q&A rows are embedded the same way from `embedding_status='pending'`. Reprocessing wipes the old chunks and vectors first. Toggling `is_active` updates MySQL and the Qdrant payload. Deleting removes chunks, vectors and the file.

## Admin quick tour
```bash
TOKEN=$(curl -s -X POST localhost:3000/api/auth/login -H 'content-type: application/json' \
  -d '{"email":"admin@example.com","password":"..."}' | jq -r .data.token)
curl -X POST localhost:3000/api/admin/questions -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' \
  -d '{"category_id":3,"question":"Which direction is best for a kitchen?","answer":"The kitchen is best in the south-east."}'
curl -X POST localhost:3000/api/admin/questions/import -H "authorization: Bearer $TOKEN" -F file=@qa.csv      # csv, xlsx or json
curl -X POST localhost:3000/api/admin/documents -H "authorization: Bearer $TOKEN" -F name="Vastu Book" -F source_type=book -F file=@book.pdf
curl -X POST localhost:3000/api/admin/documents/1/process -H "authorization: Bearer $TOKEN"
```
Import columns: `category,question,answer` (case-insensitive; aliases `q` and `a`). Duplicates are detected by a normalised-question hash (case, punctuation and spacing are ignored; Tamil text is preserved). Unknown category names are created.

## Testing
`npm test` runs the suites with mocked OpenAI, Qdrant and MySQL: chunking and cleaning, priority merge, envelope and error handler (DB/AI/vector-DB failure mapping and no-leak checks), question-hash normalisation (Tamil/Tanglish), OpenAI client retry/timeout/usage logging, vector store, worker and document pipeline, import parsing and dedupe, and the full chat flow (also through HTTP with auth, validation, rate limiting and CORS).

### End-to-end verification (no OpenAI key needed)
`scripts/mock-openai.js` is an OpenAI-compatible stub (hash-based embeddings, extractive answers). With MySQL and Qdrant running:
```bash
npm run mock:openai &                                          # :4010
OPENAI_BASE_URL=http://127.0.0.1:4010/v1 OPENAI_API_KEY=mock OPENAI_TEMPERATURE=none \
  RATE_LIMIT_CHAT=1000 PORT=3055 QDRANT_COLLECTION=vastu_verify DB_NAME=vastu_ai_verify npm start &
npm run migrate && npm run seed
API_URL=http://127.0.0.1:3055 QDRANT_URL=http://127.0.0.1:6333 npm run test:e2e
```
The stub proves the plumbing, not answer quality. Always evaluate retrieval and answer quality with a real key, your real 250 Q&A and the book.

## Operations
- Health: `GET /health` (liveness), `GET /ready` (MySQL + Qdrant ping, 503 when degraded).
- Graceful shutdown on SIGINT/SIGTERM: stop accepting requests, let the worker finish the current job (up to `WORKER_SHUTDOWN_GRACE_MS`), close the MySQL pool. Unfinished documents stay `processing` and resume on the next start.
- Logs are JSON (pino). Every request is also summarised in `api_logs` (route pattern, status, duration, actor).
- See `../docs/06-deployment.md` for production guidance and `../docs/07-mobile-integration.md` for the mobile contract.