Skip to content

Annotations Main Page

The Annotations module provides access to two main tabs: Datasets and AI Agents. Use these tabs to create, organize, and annotate data while leveraging both human annotators and AI agents.

1. Datasets Tab

The Datasets tab lets you manage data for human and AI annotations.

1.1 Create & View Dataset

  • Create Dataset – Quickly set up data tailored to a specific annotation task.
  • View Existing Datasets – Access and manage previously created datasets. Selecting a dataframe name opens a dedicated page for advanced configuration.

Annotations List of Dataframes

1.2 Schema

Define and modify the structure of your dataset:

  • Field Management – Create/edit fields (e.g., prompt, response1, response2, etc.).
  • Question Creation – Attach questions for feedback (rating, ranking, labeling, multi-label, text).
  • Unification Strategy – Decide how multiple annotators’ answers are combined (label, multi-label, rating, ranking, text).
    • Label: Assigns a single category or class to each item.
    • Multi-label: Allows assigning multiple categories or classes to each item.
    • Rating: Assigns a numerical value to each item, indicating a score or level.
    • Ranking: Orders items relative to each other based on importance or relevance.
    • Text: Provides descriptive text annotations or comments for each item.

Add Fields
Add Questions
Unification Strategy

1.3 Records

Visualize and annotate individual records:

  • Populate from Dataframes – Import data that aligns with your defined fields.
  • Record Visualization – Inspect each record in detail.
  • Annotation Interface – Capture human or AI inputs, referencing existing fields/questions.

Records Tab

1.4 Annotators

Assign human annotators or AI agents:

  • Annotator Assignment – Manage who can annotate each record.
  • AI Agent Annotation – Enable automated annotation tasks by AI agents.

Annotators

1.5 Formatting

Convert data into different formats for NLP tasks:

  • Formatting Script – Customize code to output data for question answering, classification, etc.
  • Templates – Use built-in templates (DPO, QA) and tweak as needed.
  • Storage – The formatted output is saved as a new dataframe, ready for download or further usage.

Formatting

1.6 Dataframes

Preview or modify dataframes generated during annotation:

  • Data Preview – Quickly glance at the updated or newly created data.
  • Further Tasks – Use the newly formed data in advanced NLP pipelines.

1.7 Settings

Manage or delete dataframes in this tab:

  • Dataset Settings – Adjust configuration details.
  • Deletion – Remove datasets that are no longer needed.

2. AI Agents Tab

Use the AI Agents tab to manage automated annotators:

AI Agents Tab

2.1 Add New Agent

  • Select Provider – Choose OI’s native agent or external providers (OpenAI models, etc.).
  • Configure – Provide the correct API key and agent parameters.

Add AI Agent

2.2 View Existing Agents

  • Edit Config – Update existing agent settings.
  • Delete Agents – Remove agents no longer required.

Next Steps