Visual representation
The BILN play-head, coordinate-derived 2D projection, and 3D residue focus render the same reader event.
One FoldSense event, three equivalent representations.
Human-readable text / ARIA
Waiting for the Molecular Reader event.
Machine-readable state
{"schema":"foldsense.event.v1","state":"WAITING_FOR_READER"}Device requests occur only after an explicit button press. Unsupported capabilities retain visible text and ARIA fallbacks.
Molecular Reader
BILN→pyPept / experimental context→3D play-head→sonification→speech→haptic
Frozen event packet supplied to sponsors
Waiting for reader…
Realtime conversational orchestration
Unique task: maintain the live voice turn and invoke bounded UI actions such as selecting a protein or starting the reader. Vapi never computes the molecular event.
Example: “Select insulin and start reading the receptor-binding window.”
Checking Vapi browser configuration…
Not run.
Speech modality
Unique task: Scribe v2 Realtime transcription and high-quality TTS of the same FoldSense event/explanation. No audio-only scientific information.
Example: say “Track residues 17 and 29 below five angstroms,” then compare the bounded parser output with the spoken response.
Voice session not started.
Partial
—
Committed
—
Command
—
Response
—
STT/TTS receipt
{"browser_e2e":"PENDING"}Not run.
Scientific evidence discovery
Unique task: fast semantic retrieval with highlights and a request ID when a user asks why the active feature matters.
Example: insulin PheB24–PheB26 → relevant receptor-binding literature.
PENDINGNot run.
Structured source ingestion
Unique task: execute a one-page Website Content Crawler Actor on a selected RCSB/UniProt/PubMed source, producing run ID, dataset ID and normalized output hash.
Example: snapshot the selected RCSB structure page after discovery.
PENDINGNot run.
Remote semantic inference
Unique task: classify priority and produce a concise accessible semantic summary from an already-frozen FoldSense event plus optional evidence.
Example: label the current event as receptor-binding context with high/medium/low priority. GMI does not calculate structure or distance.
PENDINGNot run.
Required tool-grounded explanation
Unique task: Gemini must call the search_evidence function; FoldSense executes that tool through Exa; only then may Gemini synthesize the answer.
Not run.
Provider readiness
No secret values are returned. This only reports whether the explicit provider/model/Actor locks needed by each test exist.
PENDINGChecking…
Manual browser acceptance
Server E2E cannot prove microphone permissions, audible playback, Vapi turn-taking, Vapi reader tool calls, VoiceOver/Narrator/TalkBack, or physical haptics. Those remain explicit review-device tests.
MANUAL DEVICE E2ERun Vapi and ElevenLabs controls on the target Mac/Windows/phone.
Passes, explicit blocks, and historical failures are different states.
The first five-protein receipt remains unchanged. Follow-up diagnostics explain causes without upgrading the historical evidence.
| Lane | Observed | Why | Judge interpretation |
|---|---|---|---|
| Core + pyPept | 5/5 PASS | Canonical event checks and generated PDB/HTML artifact hashes passed. | Local deterministic molecular spine works; pyPept remains generated conformer evidence only. |
| Vapi status | 5/5 PASS | Assistant/config readback succeeded. | Full microphone/tool-action interaction remains a manual device gate. |
| ElevenLabs realtime Scribe | 5/5 PASS | Single-use realtime token succeeded. | Realtime transcription path works independently from TTS. |
| ElevenLabs TTS | Historical FAIL · HTTP 401 Confirmed provider payment block | Account auth, realtime token and voice metadata returned 200; minimal TTS returned payment_issue / payment_required. Provider payment state was independently confirmed by the account owner. | Not a FoldSense code/auth/voice-ID defect. Successor state: EXPECTED_PROVIDER_PAYMENT_BLOCK. |
| Exa | 5/5 PASS | Protein-specific retrieval succeeded. | Evidence enrichment only; molecular state stays frozen. |
| Apify | 5/5 PASS | Actor start and dataset collection succeeded. | Bounded external ingestion path works. |
| GMI status | 5/5 PASS | Authentication/model discovery succeeded. | Status does not establish paid inference. |
| GMI inference | EXPECTED_BUDGET_BLOCK | USD 10 paid-call gate remains closed. | Intentional cost-control boundary, not a backend failure. |
| Direct Gemini + Exa | NOT_CONFIGURED_OR_NOT_LOCKED | GEMINI_NOT_CONFIGURED · HTTP 503. | Independent optional direct-Gemini route; no success claimed. |
| Integrated historical receipt | RECORDED_FAILURES | The original classifier counted the TTS payment response as generic FAIL. | Preserved historical result; a new exact-SHA run is needed before PASS_OR_EXPLICIT_BLOCKS can be claimed. |
Current integrated state
No tests run.
What sponsors cannot change
| Layer | Owner | Claim ceiling |
|---|---|---|
| BILN / play-head / pyPept mapping / experimental-PDB distinction | FoldSense deterministic code + cited sources | Molecular identity/representation state |
| Sound + haptic mapping | FoldSense | Deterministic interface rendering, not a physical spectrum |
| Voice turn orchestration | Vapi | Conversation/control layer |
| Speech recognition/rendering | ElevenLabs | Audio modality |
| Web evidence discovery | Exa | External retrieval |
| Source snapshot/dataset | Apify | External ingestion receipt |
| Semantic label/priority | GMI | Provider inference |
| Grounded explanation | Gemini + Exa | Tool-grounded synthesis |