Skip to content
Spectral Codex
GitHub
Customer

Org Llm Config

Generated

This page is generated from qa/customer/specs/org-llm-config.md — the source of truth. Edit the spec, not this page.

Last run: not yet recorded (run the replay suite to populate status).

Overview

An org admin configures their organization’s bring-your-own World-Agent LLM from the dashboard’s Organization model surface: an org-level (not domain-scoped) setting that selects the provider, model, and credential Spectral uses for their organization — preferred over the platform default for all the org’s workspaces. Configuration is versioned and append-only (each save activates a new version, deactivating the prior; rollback re-activates a prior version as a new one). The credential is write-only — entered here, stored encrypted, never shown again. A BYO credential runs on the customer’s own provider account, so the surface shows a terms-of-service responsibility notice and requires an explicit acknowledgement before saving.

The surface is org-admin gated: only a caller holding the admin:org scope sees the chrome’s Settings link and may load /org-settings. This spec drives that surface end to end against the running dashboard. It is credential-free: setting a model is a control-plane DB write (the entered key is stored, not exercised), so it replays without any live LLM call.

Preconditions

  • Signed in as the seeded customer (an org owner — carries the admin:org scope).
  • The Organization-model surface is reachable from the chrome’s “Settings” link (/org-settings).
  • If the customer dashboard session is blocked (the domain-claim carry-forward) or the signed-in customer is not an org admin, the org-admin scenarios skip-with-reason.

Scenarios

1. Reach the surface from the chrome (org admin only)

  • Sign in; click the chrome “Settings” link
  • Expected: The URL is /org-settings and the set-model form is shown.

2. Set the org model

  • Choose a provider (Anthropic), enter a model (claude-sonnet-4-6) and a write-only API key, acknowledge the terms-of-service notice, and save
  • Expected: The active-model card appears, naming the canonical anthropic/claude-sonnet-4-6 model id and reporting the credential as set.

3. The ToS acknowledgement gates the save

  • With provider/model/key filled but the ToS checkbox unchecked
  • Expected: The save button is disabled; checking the box enables it.

4. A second save appends a new active version, then rollback re-activates the prior

  • Set a different model (xAI grok-4.3) with its own key + ToS, then roll back to the Anthropic version
  • Expected: The second save is a higher version than the first (only the newest active); the rollback creates a new active version carrying the Anthropic config (append-only — history is never rewritten).

5. openai-compatible reveals the base-URL field

  • Choose the openai-compatible provider
  • Expected: A required base-URL field appears; choosing any other provider hides it again.

6. The OAuth-bundle credential type reveals a bundle field

  • Switch the credential type from “API key” to “Subscription (OAuth bundle)”
  • Expected: The API-key input is replaced by a bundle (JSON) textarea, with a hint to produce the bundle via the tools/dev/oauth_login.py PKCE helper.

7. Set an OAuth bundle

  • With the OAuth credential type selected, paste a token bundle JSON, acknowledge the ToS, and save
  • Expected: The active-model card appears naming the canonical model id and reporting the credential as set. (Credential-free in replay: the bundle is stored, not exercised.)

8. A non-admin customer cannot reach the surface

  • As a customer without admin:org (or with the dashboard session blocked), the “Settings” link is absent and a direct visit to /org-settings is redirected to the forbidden landing.
  • The replay harness seeds a single org-admin customer, so this scenario has no executable Playwright assertion here — when the signed-in customer lacks admin:org the admin scenarios skip-with-reason. The negative authorization (a non-owner is refused) is asserted server-side by the API integration test test_non_owner_is_403 (apps/api/tests/integration/test_org_llm_config_routes.py).

9. OAuth subscription credentials show a provider-specific experimental warning

  • Choose the openai provider and the “Subscription (OAuth bundle)” credential type
  • Expected: An experimental warning appears, and for OpenAI it adds the datacenter-block caveat (steering production users to an API key); the dev-tool hint names the OpenAI provider flag (oauth_login.py --provider openai). Switching to xai keeps an experimental warning but without the datacenter caveat (xAI’s block is unconfirmed). The warning is OAuth-specific — the supported API-key path shows none. (Server-driven copy via the LLM-config OAuth capability;. OpenAI’s subscription path is Cloudflare-blocked from datacenter origins;.)

Test Data

LabelValueNotes
Provider Aanthropic / claude-sonnet-4-6First saved version
Provider Bxai / grok-4.3Second saved version
API keya throwaway sk-… stringWrite-only; stored, never exercised in replay
ToSacknowledgement checkboxRequired before save (AC6b)