> ## Documentation Index
> Fetch the complete documentation index at: https://social-b97141fb-feat-user-history-moderation-detail.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# AI User Profile Moderation

> AI-powered moderation for user profiles including display names, avatars, and descriptions, with field-level admin reset for any user

Extend your community safety beyond content with AI-powered user profile moderation. social.plus automatically scans user profiles — including display names, avatars, and descriptions — for policy violations, giving admins the tools to review and act on flagged profiles. Admins can also reset **any** user's profile — flagged or not — from the User History page, choosing exactly which fields to reset.

<CardGroup cols={2}>
  <Card title="Post-Moderation" icon="user-shield">
    AI reviews profile content after save — flag-only, no auto-delete
  </Card>

  <Card title="Admin Reset" icon="rotate-left">
    Reset any user's profile from User History — pick which fields (display name, avatar, description) to restore
  </Card>

  <Card title="Moderation Feed" icon="list-check">
    Unified Users tab to review AI-flagged and user-reported profiles
  </Card>

  <Card title="Profile Blocklist" icon="ban">
    Dedicated blocklist category for display names and descriptions
  </Card>
</CardGroup>

## Overview

AI User Profile Moderation is a **post-moderation** system — profiles are scanned after the user saves changes. Unlike content moderation (posts, comments, messages), user profile moderation is **flag-only**: the block confidence threshold that auto-deletes content does not apply to user profiles. Admins must explicitly decide to reset, clear the flag, or ban the user.

```mermaid theme={null}
graph LR
    A[User Updates Profile] --> B[Profile Saved]
    B --> C[AI Scans Text & Image]
    C -->|Violation Detected| D[Profile Flagged]
    C -->|No Violation| E[AI Mod: Passed]
    D --> F[Appears in Moderation Feed]
    F --> G{Admin Action}
    G -->|Reset Profile| H[Fields Reset to Defaults]
    G -->|Clear Flag| I[Flag Dismissed]
    G -->|Ban Globally| J[User Banned]
    H --> K[User Notified]
```

<Warning>
  **Flag-Only for Profiles**: Unlike posts and messages, the block confidence threshold does **not** auto-delete user profile content. All flagged profiles require manual admin review. This ensures that user accounts are never silently wiped by automated systems.
</Warning>

## What Gets Scanned

AI moderation scans three profile fields after each update:

| Field | Scan Type | Pre-Moderation | Post-Moderation |
| - | - | - | - |
| **Display Name** | Text | ❌ | ✅ Flagged if violation detected |
| **Description** | Text | ❌ | ✅ Flagged if violation detected |
| **Avatar** (uploaded file) | Image | ✅ Blocked before upload | ✅ Flagged after save |

<Info>
  **Avatar Pre-Moderation**: Uploaded avatar images are already scanned at upload time via. Post-moderation provides a second layer of review after the profile is saved.
</Info>

<Warning>
  **External avatar URLs**: Avatars set with `avatarCustomUrl` are not moderated. Only uploaded avatars (`avatarFileId`) are scanned, flagged, and reset.
</Warning>

## Detection Categories

User profile text is scanned against the same categories as posts, comments, and messages:

<AccordionGroup>
  <Accordion title="Text Detection Categories" icon="text">
    * **Harassment or Bullying**: Targeted abuse or intimidation in display names or descriptions
    * **Sexual Content or Nudity**: Explicit sexual references in profile text
    * **Violence or Threatening Content**: Violent threats or graphic descriptions
    * **Hate**: Hate speech targeting protected groups
    * **Fraudulent Intent and Scam Promotion**: Scam tactics, phishing links in descriptions
    * **Self Harm or Suicide**: Content related to self-harm or suicidal ideation
  </Accordion>

  <Accordion title="PII Detection" icon="fingerprint">
    * **URL** — Links and web addresses embedded in profile descriptions
    * **PersonType** — References to specific person types or identities
  </Accordion>

  <Accordion title="Image Detection (Avatar)" icon="image">
    Avatar images are scanned for the same visual categories as other media content:

    * Adult content, nudity, and suggestive imagery
    * Violence and harmful content
    * Hate symbols and extremist content
    * Substance-related content

    See [AI Content Moderation — Multimedia Content Detection](/analytics-and-moderation/console/ai-content-moderation#multimedia-content-detection) for the full list of image categories.
  </Accordion>
</AccordionGroup>

## Confidence Thresholds

User profile moderation reuses the same text and image confidence thresholds configured for content moderation. However, only the **flag confidence** threshold applies — there is no auto-block for user profiles.

| Threshold | Applies to Profiles? | Behavior |
| - | - | - |
| **Flag Confidence** (default: 40) | ✅ Yes | Profile flagged for admin review |
| **Block Confidence** (default: 80) | ❌ No | Does **not** auto-delete profile fields |

<Tip>
  Confidence thresholds are shared across content and user profiles. If you need different sensitivity for profiles, this would require a future enhancement for separate threshold configuration.
</Tip>

## Moderation Feed — Users Tab

Flagged user profiles appear in a dedicated **Users** tab within the Moderation Feed (**Moderation > Moderation feed**), alongside the existing Posts/Comments and Messages tabs.

<Tabs>
  <Tab title="To Review">
    The **To Review** list displays all user profiles that require moderator attention.

    Each flagged profile shows:

    * AI moderation labels with category occurrence counts (e.g., "Hate (2)", "Violence (1)")
    * Whether the display name or description triggered the flag — indicated by a **(Profile description is flagged)** label
    * User report counts from other community members (e.g., "4 users")
    * Last flagged timestamp

    **Finding profiles:**

    * **Search by user ID or name** (at least 3 characters)
    * **Report source**: All sources, User reports, or AI moderation
    * **Advanced filter**: Report type (Display name, Description, Profile photo, Behavior) and user Tags
    * **Sort**: Last flag date, Flag count, or Created date

    **Available actions:**

    * **Reset profile** — Reset flagged profile fields to safe defaults
    * **Ban globally** — Ban the user across all communities
    * **Clear flag** — Dismiss the flag and approve the profile

    <Info>
      To reset a profile that has **not** been flagged, use the [User History page](#admin-reset) — it opens the same reset modal for any user.
    </Info>
  </Tab>

  <Tab title="Reviewed">
    The **Reviewed** list shows profiles that have already been moderated.

    Each reviewed item displays the action taken and who performed it:

    * **Flag cleared by \[moderator]**
    * **Globally banned by \[moderator]**
    * **Profile reset by \[moderator]**

    Use the **Select reviewer** filter to view actions taken by specific team members, and the **Advanced filter** to narrow by user Tags.
  </Tab>
</Tabs>

## Admin Reset

Resetting a profile does not require the profile to be flagged. From the **User History** page, admins can reset **any** user's profile at any time — whether it was flagged by AI, reported by other members, or never flagged at all. Flagging simply surfaces the profiles that may be worth resetting.

Both entry points open the **same reset modal**, where the admin ticks which profile fields to restore:

| Where | Applies to |
| - | - |
| **Moderation feed > Users tab** | Profiles in the review queue (AI-flagged and user-reported) |
| **User History page** | Any user, flagged or not |

### Reset Behavior by Field

| Field | Reset Behavior |
| - | - |
| **Display Name** | Set to `{configuredPrefix}{UUID(12)}` (e.g., `User_a1b2c3d4e5f6`) |
| **Description** | Cleared (set to empty) |
| **Avatar** | Removed — reverts to default avatar. Original file is deleted from storage. |

### Resetting a Profile from User History

Reset is **field-level** — the admin ticks which of the three profile fields to restore. Fields that are not ticked are left untouched.

<Steps>
  <Step title="Open the User History Page">
    Find the user in the Console and open their [User History](/analytics-and-moderation/console/user-and-content-management/user-social-history) page. This works for any user, flagged or not.
  </Step>

  <Step title="Review the Profile">
    Check the user's profile details and moderation history — including any AI moderation labels and past flags — before deciding to reset.
  </Step>

  <Step title="Open Profile Moderation">
    Click the three-dot (**⋮**) menu and choose **Profile moderation**. The submenu offers **Clear flags** and **Reset profile**.
  </Step>

  <Step title="Select Reset Profile">
    Choose **Reset profile**. The reset modal opens — the same one used in the Moderation feed — listing the three profile fields as checkboxes.
  </Step>

  <Step title="Choose the Fields to Reset">
    Tick the fields you want restored to defaults: **Display name**, **Avatar**, **Description**, or any combination. At least one field must be ticked.
  </Step>

  <Step title="Confirm Reset">
    Confirm the reset. Only the ticked fields are changed, and the user is notified of which fields were reset.
  </Step>
</Steps>

<Warning>
  **Avatar Reset is Irreversible**: When an avatar is reset, the original file is deleted from storage. Only the fact that a reset occurred is logged — the original image cannot be recovered.
</Warning>

<Info>
  The display name reset prefix is configurable via the network setting `moderation.displayNameResetPrefix` (API-only).
</Info>

## User Notifications

When a profile is reset, the affected user receives a notification through the notification tray:

* **Notification type**: `USER_PROFILE_RESET`
* **Content**: Informs the user which fields were reset — only the fields actually reset are listed — and that they can update their profile again
* Delivered via the existing notification tray system

## Profile Blocklist

A dedicated blocklist category for user profiles allows admins to block specific words and phrases from appearing in display names and descriptions.

| Blocklist Category | Applies To | Description |
| - | - | - |
| **Content** (existing) | Posts, comments, messages | Standard content blocklist |
| **User Profile** (new) | Display names, descriptions | Blocks words/phrases in profile text |

<Info>
  The profile blocklist is independent from the content blocklist. You can maintain different blocked terms for user profiles versus post/comment content.
</Info>

## User History

Profile moderation events are recorded in the user's activity history, accessible from the User History page:

* **AI flag events**: When AI detects a violation in a profile field
* **Profile reset events**: Logged as `RESET_USER_PROFILE` activity with details on which fields were reset, the actor, and reason
* **Moderation actions**: Flag cleared, globally banned, etc.

## Best Practices

<AccordionGroup>
  <Accordion title="Review Strategy" icon="clipboard-check">
    * **Prioritize by severity**: Review profiles with multiple AI flag categories first
    * **Check context**: Some display names or descriptions may be flagged incorrectly — review before resetting
    * **Use reset judiciously**: Resetting a profile is disruptive to the user; prefer clearing the flag for borderline cases
    * **Reset only what's wrong**: Untick the fields that are fine — clearing a clean display name alongside an offending avatar is unnecessary churn for the user
    * **Monitor repeat offenders**: Users who are repeatedly flagged may warrant a global ban
  </Accordion>

  <Accordion title="Blocklist Management" icon="list">
    * **Separate concerns**: Maintain different blocklists for content vs. user profiles
    * **Common patterns**: Add known offensive display name patterns to the profile blocklist
    * **Regular updates**: Review and update the blocklist as new patterns emerge
    * **Test impact**: Check existing users before adding broad blocklist terms
  </Accordion>

  <Accordion title="Confidence Tuning" icon="sliders">
    * **Shared thresholds**: Remember that confidence thresholds are shared between content and profiles
    * **Monitor false positives**: Track how often legitimate profile content gets flagged
    * **Flag-only safety net**: Since profiles are flag-only, a lower flag threshold is safer than with content (no risk of auto-deletion)
  </Accordion>
</AccordionGroup>

## Limitations

<Warning>
  * **Flag-only**: AI cannot auto-delete user profile content — all flagged profiles require admin review
  * **External avatar URLs**: Avatars set with `avatarCustomUrl` are not moderated — use `avatarFileId` for avatars that need moderation
  * **Shared confidence thresholds**: Profile and content moderation share the same flag thresholds; separate configuration is not yet available
  * **Original avatar not recoverable**: Once an avatar is reset, the original file is permanently deleted from storage
</Warning>

## Related Topics

<CardGroup cols={3}>
  <Card title="AI Content Moderation" href="/analytics-and-moderation/console/ai-content-moderation" icon="shield-check">
    AI moderation for posts, comments, messages, images, and video content
  </Card>

  <Card title="Moderation Overview" href="/analytics-and-moderation/console/moderation/overview" icon="eye">
    General moderation tools, roles, and workflows
  </Card>

  <Card title="User Insights" href="/analytics-and-moderation/console/user-and-content-management/user-insights" icon="user">
    View detailed user information, history, and activity
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.