Back to Case Studies
Advisor-facing RAG MVPWorking MVP, synthetic data only

SIRE — Student Insight & Recommendation Engine

A permission-scoped, citation-backed RAG system that helps academic advisors turn SIS/LMS records into reviewable student insights.

Role: Full-stack / AI engineer (solo build)

Overview

SIRE retrieves a student's academic evidence, builds a cited strengths/weaknesses/engagement profile, and proposes recommendations. A guardrail layer checks every claim against retrieved evidence, and an advisor must approve or override each result before it counts.

Tech Stack

backend

Python 3.11+FastAPIPydanticSQLAlchemyAlembic

ai

LangGraphOpenAI-compatible chat and embeddings APIOllamavLLM / LocalAI compatibleHybrid vector + keyword retrieval

data

PostgreSQL 16pgvector (HNSW cosine index)SQLite fallback

frontend

ReactTypeScriptVite

quality

pytest + coverageRuffJSON golden-set regressionProduction build check

devops

DockerDocker Compose (API, frontend, PostgreSQL, optional Ollama profile)PowerShell start scripts

The Problem

Advisors work across fragmented SIS and LMS data (grades, attendance, submissions, study plans). Spotting at-risk students and choosing the next action is slow, and student data is highly sensitive.

  • Unify records from different source systems into one searchable form
  • Never expose a record the requesting user is not entitled to see
  • Keep AI output grounded in evidence instead of hallucinated
  • Avoid vendor lock-in to a single LLM provider
  • Keep a human accountable for every academic recommendation

The Solution

  1. 1Separate evidence retrieval, model reasoning, validation, guardrails and human approval into distinct layers, so each can be tested and replaced on its own.

Permission-scoped access

Advisors see assigned students; faculty and department heads see their department; students see only themselves; deans get aggregate-only access; admins have no academic detail access by default.

Canonical SIS/LMS records

Source records are normalized into one academic_records table with source provenance, structured JSON facts, scope metadata and an optional embedding. A unique source_ref makes ingestion idempotent.

Scope-first hybrid retrieval

Student and department/program filters are applied in the query before any ranking. Candidates are then scored as 0.7 × embedding cosine similarity + 0.3 × keyword overlap.

Evidence-grounded output

Every profile signal and recommendation carries evidence_record_ids. Citations are built from retrieved records, not generated after the fact.

Provider-neutral model gateway

The workflow depends on an analyze()/recommend() interface. A deterministic mock is the default; any OpenAI-compatible server (Ollama, vLLM, LocalAI, OpenAI) plugs in through configuration.

Guardrails and human review

Checks unknown evidence IDs, empty retrieval or recommendations, prompt-injection-like source text and SSN-shaped identifiers. Even passing results remain advisor-reviewable.

Advisor review dashboard

React UI showing the profile, recommendations, citations, confidence and flags, with approve/override actions.

Architecture

Layered web app with an agentic workflow and a model adapter boundary

Request lifecycle
  1. 1Advisor clicks Generate insight
  2. 2API resolves identity and authorizes the student
  3. 3LangGraph retrieves permission-scoped evidence
  4. 4Model gateway creates a profile
  5. 5Model gateway creates evidence-linked recommendations
  6. 6Guardrail checks grounding, flags and confidence
  7. 7InsightRun is saved
  8. 8Dashboard shows the result for advisor review
  9. 9Advisor approves or overrides

Components

Frontend

React + TypeScript + Vite advisor workspace with a typed fetch client.

API

FastAPI under /api/v1. Resolves the caller, authorizes the student, runs the workflow, returns validated responses.

Authorization

authorize_student() runs before the workflow and again when reading a saved run. The demo identity header X-User-Email is a development seam to be replaced by OIDC/Entra claims.

Workflow

LangGraph state machine over a typed InsightState: retrieve -> analyze -> recommend -> guardrail, then either awaiting_review or human_review (flagged_for_review).

Retrieval

ScopedRetriever filters by student, department and program, then ranks. On PostgreSQL an HNSW cosine index narrows the permission-filtered set; on SQLite scoring runs in Python.

Model gateway

MockModelGateway (deterministic, offline) or OpenAICompatibleGateway (temperature 0, JSON mode, output validated with Pydantic).

Persistence

User, Student, AcademicRecord, InsightRun and Review tables. Alembic migration for PostgreSQL + pgvector.

  • Security boundary: scope is enforced at the API and again inside the retrieval query, so ranking can never surface out-of-scope records.
  • Model boundary: the workflow does not know whether reasoning is deterministic code or a remote chat model.

Guardrail Logic

groundedness

1.0 only when claims exist and every claimed evidence ID was retrieved

confidence

min(1.0, 0.55 + 0.05 × retrieved_count) × groundedness

pass Condition

No flags and confidence ≥ 0.70; otherwise the run is flagged_for_review

API Surface

POST/api/v1/demo/seedIdempotently create the demo advisor, student and five synthetic records
GET/api/v1/studentsList students filtered by the caller's role and scope
POST/api/v1/students/{student_id}/insightsAuthorize, run the graph, persist an InsightRun
GET/api/v1/insights/{run_id}Fetch a saved run after re-checking authorization
POST/api/v1/insights/{run_id}/reviewRecord the advisor decision (approved or overridden)
GET/healthService health

Demo Workflow

Primary user: Academic advisor
  1. 1Seed a small synthetic dataset
  2. 2View students within the advisor's authorization scope
  3. 3Generate an insight for a selected student
  4. 4Inspect strengths, weaknesses, engagement signals, recommendations, citations, confidence and guardrail flags
  5. 5Approve or override with an optional note and 1–5 rating

Demo data

All data is synthetic. Student S1001.

  • CS201 grade 88%, improving
  • MA202 grade 64%, declining
  • MA202 engagement: 72% attendance, three late submissions, declining
  • MA202 study plan: tutoring approved, prerequisite for CS310
  • DB210 grade 84%, stable

Expected: At least one strength, one weakness, engagement evidence, recommendations, citations and groundedness of 1.0 with the mock provider.

Key Decisions

Scope filtering before ranking

Why: Relevance scoring must never be able to surface a record outside the caller's academic scope.

Trade-off: Slightly more complex queries in exchange for a hard security guarantee.

Provider-neutral gateway with a deterministic mock default

Why: Tests, golden-set evaluation and demos run with no API key, Docker or model server, and the provider can change through five env values.

Trade-off: The hash-based mock embedding is a test fallback, not a semantic model.

SQLite and PostgreSQL+pgvector side by side

Why: Zero-infrastructure local development, with a production-like vector path available.

Trade-off: Two code paths for scoring; the embedding dimension must be changed together with the migration.

No knowledge graph in the MVP

Why: Study-plan and prerequisite facts are small enough for relational rows plus JSON facts. A graph database would add operating cost before adding signal.

Trade-off: Revisit Neo4j or a Postgres graph extension only when multi-hop degree-plan queries show a measurable benefit.

Human review is always required

Why: The model drafts; guardrails check; an authorized advisor decides.

Trade-off: Less automation, more accountability for a sensitive domain.

Treat source text as untrusted data

Why: Prompt instructions tell the model record content is data, and prompt-injection-like text routes to review.

Trade-off: Heuristic detection can produce false positives that cost advisor time.

Outcomes

  • End-to-end working pipeline from authorization to retrieval, analysis, guardrails, persistence and advisor review
  • Runs fully offline with no key or model, and switches to a local open-weight model (for example Ollama with Qwen2.5) by configuration only
  • Every recommendation traces back to a cited source record
  • Authorization is enforced at three points: student access, retrieval query and saved-run access

Engineering Practices

  • Pydantic contracts validate both model output and API responses, so malformed model JSON fails loudly
  • Idempotent seeding and ingestion through a unique source_ref
  • temperature=0 and JSON mode for repeatable model output
  • API tests cover seed -> list -> generate -> review, plus unknown-identity rejection
  • Golden-set evaluation runner for regression checks
  • Alembic migration for the pgvector schema
  • Containerized with Docker Compose

Known Limitations

  • X-User-Email identity header is a development seam, not production authentication
  • Demo data is synthetic; real connectors need schema contracts, retry/outbox, secrets management, audit logging and retention policy
  • Confidence is a simple heuristic, not a calibrated score
  • Golden set is tiny; a pilot needs 50–200 anonymized, advisor-labeled cases

Next Steps

  • Verified OIDC / Microsoft Entra authentication
  • Precomputed embeddings with retries and provider-failure monitoring
  • Larger labeled evaluation set with retrieval recall, recommendation validity, fairness slices, latency and cost thresholds in CI
  • Real SIS/LMS connectors and audit logging
  • Knowledge graph only if multi-hop degree-plan queries justify it