Safety Protocol
English version
Companion Safety Protocol
Companion is an AI companion app for adults. This page describes how we detect and respond when a user expresses suicidal ideation or self-harm.
Where the checks run
Safety checks are not applied in one direction only. They run at all four of these points:
- Messages sent by the user
- Responses generated by the companion, before they reach the user
- Completed speech transcripts from voice calls
- Conversation summaries
How detection works
Two stages:
- Deterministic rules. Expressions indicating suicide or self-harm are checked first, in both Korean and English. This stage does not depend on an external service, so it runs regardless of network conditions.
- Classification model. OpenAI’s content classifier (
omni-moderation-latest) then runs, treating self-harm intent and requests for self-harm instructions as distinct categories.
How we respond
When suicide or self-harm is detected, we do not block the conversation. Cutting off someone in crisis is worse. Instead, the companion’s reply is replaced with crisis guidance containing:
- Contacts for immediate danger: in Korea, 119, 112, and the suicide prevention line 109. In the United States, 911 and 988.
- Encouragement to tell a trusted person who can stay with them.
- An explicit statement that the companion is an AI and cannot dispatch help.
We always include that last item because believing an AI can send help delays the moment a person reaches real help.
The guidance does not replace the conversation. Two things arrive together.
- A safety card — fixed text containing every item above. It does not vary with the situation, and the phone numbers are tappable. It is also the unit we count when we report how many referrals we issued.
- The companion’s reply — written for the situation, but only under constraints we set. It does not describe methods, does not judge how dangerous a dose is, does not tell anyone to wait and see, and never claims we contacted someone.
If a reply is missing the crisis numbers or the statement that the companion is an AI, we replace it with our own fixed text. The tone follows the situation; the referral never goes missing.
When other harmful categories are detected — sexual content involving minors, sexual coercion, sexually explicit description — the response is not generated and is replaced with a boundary message.
When the safety check cannot run
If the classification service is unreachable or returns a malformed result, Companion refuses to answer rather than answering unchecked.
What users can do
- Report any companion response directly in the app. Safety reviewers see only the reported response and its immediate context.
- Delete conversation content at any time.
- Request deletion of all data stored for the device.
Age
Companion is for adults only. Users identified as minors are blocked from AI features. Companion chatbots may not be suitable for some minors.
Contact
contact@stickyrice.kr
구현 대조 (게시본에 포함하지 않음)
이 게시본의 각 문장이 어느 코드에 대응하는지 — 구현이 바뀌면 게시본도 고쳐야 하므로 남긴다.
| 게시본 서술 | 근거 |
|---|---|
| 네 지점에 적용 | chat-respond/index.ts:165(입력), :275·:529(출력), voice-transcript/index.ts:118(전사), _shared/conversation_summary.ts:48(요약) |
| 결정론적 규칙 우선 | _shared/safety.ts의 crisisPattern이 분류 모델 호출 전에 검사됨 |
| 자해는 차단하지 않고 대체 | safety.ts가 { allowed: true, classification: "crisis", replacement: crisis(locale) } 반환 |
| self-harm 범주 분리 | safety.ts가 self-harm/intent·self-harm/instructions를 moderation_block이 아닌 crisis로 분기 |
| 안전 카드 | safety.ts의 crisisCard() — 119·112·109 / 911·988, “AI라 직접 구조를 보낼 수 없다” 포함. 상황에 따라 달라지지 않는다 |
| 답변 하한 | safety.ts의 meetsReferralFloor() — 연락처와 AI 고지가 없으면 crisisFallback() 으로 대체 |
| 위기 모드 지시 | safety.ts의 crisisModeInstruction() + 프롬프트 v7 — 방법 묘사·용량 판정·”기다려 봐” 금지 |
| 검사 불가 시 거부 | safety.ts의 unavailable(), allowed: false |
| 신고 | functions/reports/, iOS ChatView.swift의 응답 신고 |