diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..86ce0f3 --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +# Claude Code settings +.claude/ diff --git a/datasets/quick-start.mdx b/datasets/quick-start.mdx index 8f7ebf5..f8420db 100644 --- a/datasets/quick-start.mdx +++ b/datasets/quick-start.mdx @@ -2,6 +2,9 @@ title: "Quick Start" --- +Datasets are simple data tables that you can use to manage your data for experiments and evaluation of your AI applications. +Datasets are available in the SDK, and they enable you to create versioned snapshots for reproducible testing. + -Datasets are simple data tables that you can use to manage your data for experiments and evaluation of your AI applications. -Datasets are available in the SDK, and they enable you to create versioned snapshots for reproducible testing. - diff --git a/img/playground/play-action-dark.png b/img/playground/play-action-dark.png new file mode 100644 index 0000000..4fc39b6 Binary files /dev/null and b/img/playground/play-action-dark.png differ diff --git a/img/playground/play-action-light.png b/img/playground/play-action-light.png new file mode 100644 index 0000000..77a7f93 Binary files /dev/null and b/img/playground/play-action-light.png differ diff --git a/img/playground/play-column-list-dark.png b/img/playground/play-column-list-dark.png new file mode 100644 index 0000000..e486f4d Binary files /dev/null and b/img/playground/play-column-list-dark.png differ diff --git a/img/playground/play-column-list-light.png b/img/playground/play-column-list-light.png new file mode 100644 index 0000000..09b512a Binary files /dev/null and b/img/playground/play-column-list-light.png differ diff --git a/img/playground/play-column-options-dark.png b/img/playground/play-column-options-dark.png new file mode 100644 index 0000000..4ad127e Binary files /dev/null and b/img/playground/play-column-options-dark.png differ diff --git a/img/playground/play-column-options-light.png b/img/playground/play-column-options-light.png new file mode 100644 index 0000000..447f7dd Binary files /dev/null and b/img/playground/play-column-options-light.png differ diff --git a/img/playground/play-column-settings-dark.png b/img/playground/play-column-settings-dark.png new file mode 100644 index 0000000..fa869d1 Binary files /dev/null and b/img/playground/play-column-settings-dark.png differ diff --git a/img/playground/play-column-settings-light.png b/img/playground/play-column-settings-light.png new file mode 100644 index 0000000..5f1bea8 Binary files /dev/null and b/img/playground/play-column-settings-light.png differ diff --git a/img/playground/play-empty-table-dark.png b/img/playground/play-empty-table-dark.png new file mode 100644 index 0000000..18ea65c Binary files /dev/null and b/img/playground/play-empty-table-dark.png differ diff --git a/img/playground/play-empty-table-light.png b/img/playground/play-empty-table-light.png new file mode 100644 index 0000000..448f99d Binary files /dev/null and b/img/playground/play-empty-table-light.png differ diff --git a/img/playground/play-json-dark.png b/img/playground/play-json-dark.png new file mode 100644 index 0000000..3db6462 Binary files /dev/null and b/img/playground/play-json-dark.png differ diff --git a/img/playground/play-json-light.png b/img/playground/play-json-light.png new file mode 100644 index 0000000..aea5d72 Binary files /dev/null and b/img/playground/play-json-light.png differ diff --git a/img/playground/play-list-dark.png b/img/playground/play-list-dark.png new file mode 100644 index 0000000..a6a26cb Binary files /dev/null and b/img/playground/play-list-dark.png differ diff --git a/img/playground/play-list-light.png b/img/playground/play-list-light.png new file mode 100644 index 0000000..1235165 Binary files /dev/null and b/img/playground/play-list-light.png differ diff --git a/img/playground/play-multi-select-dark.png b/img/playground/play-multi-select-dark.png new file mode 100644 index 0000000..2a2dfd1 Binary files /dev/null and b/img/playground/play-multi-select-dark.png differ diff --git a/img/playground/play-multi-select-light.png b/img/playground/play-multi-select-light.png new file mode 100644 index 0000000..9012e0e Binary files /dev/null and b/img/playground/play-multi-select-light.png differ diff --git a/img/playground/play-number-col-dark.png b/img/playground/play-number-col-dark.png new file mode 100644 index 0000000..8c97e7b Binary files /dev/null and b/img/playground/play-number-col-dark.png differ diff --git a/img/playground/play-number-col-light.png b/img/playground/play-number-col-light.png new file mode 100644 index 0000000..17a503f Binary files /dev/null and b/img/playground/play-number-col-light.png differ diff --git a/img/playground/play-number-col-summary-dark.png b/img/playground/play-number-col-summary-dark.png new file mode 100644 index 0000000..cd6f58d Binary files /dev/null and b/img/playground/play-number-col-summary-dark.png differ diff --git a/img/playground/play-number-col-summary-light.png b/img/playground/play-number-col-summary-light.png new file mode 100644 index 0000000..9c27128 Binary files /dev/null and b/img/playground/play-number-col-summary-light.png differ diff --git a/img/playground/play-prompt-column-dark.png b/img/playground/play-prompt-column-dark.png new file mode 100644 index 0000000..5c9bd75 Binary files /dev/null and b/img/playground/play-prompt-column-dark.png differ diff --git a/img/playground/play-prompt-column-light.png b/img/playground/play-prompt-column-light.png new file mode 100644 index 0000000..991152f Binary files /dev/null and b/img/playground/play-prompt-column-light.png differ diff --git a/img/playground/play-prompt-write-dark.png b/img/playground/play-prompt-write-dark.png new file mode 100644 index 0000000..2f7ab7a Binary files /dev/null and b/img/playground/play-prompt-write-dark.png differ diff --git a/img/playground/play-prompt-write-light.png b/img/playground/play-prompt-write-light.png new file mode 100644 index 0000000..c06280f Binary files /dev/null and b/img/playground/play-prompt-write-light.png differ diff --git a/img/playground/play-single-select-creation-dark.png b/img/playground/play-single-select-creation-dark.png new file mode 100644 index 0000000..01e09e0 Binary files /dev/null and b/img/playground/play-single-select-creation-dark.png differ diff --git a/img/playground/play-single-select-creation-light.png b/img/playground/play-single-select-creation-light.png new file mode 100644 index 0000000..c140486 Binary files /dev/null and b/img/playground/play-single-select-creation-light.png differ diff --git a/img/playground/play-single-select-dark.png b/img/playground/play-single-select-dark.png new file mode 100644 index 0000000..8a4b4a3 Binary files /dev/null and b/img/playground/play-single-select-dark.png differ diff --git a/img/playground/play-single-select-light.png b/img/playground/play-single-select-light.png new file mode 100644 index 0000000..3630985 Binary files /dev/null and b/img/playground/play-single-select-light.png differ diff --git a/mint.json b/mint.json index 7992629..468b002 100644 --- a/mint.json +++ b/mint.json @@ -151,6 +151,20 @@ "group": "Prompt Management", "pages": ["prompts/quick-start", "prompts/registry", "prompts/sdk-usage"] }, + { + "group": "Playgrounds", + "pages": [ + "playgrounds/quick-start", + { + "group": "Columns", + "pages": [ + "playgrounds/columns/data-columns", + "playgrounds/columns/prompt", + "playgrounds/columns/column-management" + ] + } + ] + }, { "group": "Datasets", "pages": ["datasets/quick-start", "datasets/sdk-usage"] diff --git a/playgrounds/columns/column-management.mdx b/playgrounds/columns/column-management.mdx new file mode 100644 index 0000000..c6bfd20 --- /dev/null +++ b/playgrounds/columns/column-management.mdx @@ -0,0 +1,48 @@ +--- +title: "Column Management" +description: "Learn all columns general functionalities" +--- + +Columns in the Playground can be reordered, edited, or deleted at any time to adapt your workspace as your analysis evolve. Understanding how to manage columns effectively helps you maintain organized and efficient playgrounds. + +## Columns Settings +Column Settings lets you hide specific columns from the Playground and reorder them as needed. To open the settings, click the Playground Action button and select Column Settings + + + +To change the column order, use the six-dot handle on the right side of each column to simply drag the column into the desired position. + +To hide a column, toggle its switch in the menu. + + + + +## Columns Actions + +Each column has a menu that lets you manage and customize it. From this menu, you can: +- Rename the column directly by editing its title +- Edit the column configuration +- Duplicate the column to create a copy with the same settings +- Delete the column if it’s no longer needed + + + diff --git a/playgrounds/columns/data-columns.mdx b/playgrounds/columns/data-columns.mdx new file mode 100644 index 0000000..3f3c071 --- /dev/null +++ b/playgrounds/columns/data-columns.mdx @@ -0,0 +1,116 @@ +--- +title: "Data Columns" +--- + +Columns are the building blocks of playgrounds, defining what kind of data you can store, process, and analyze. + + + + + + + +**Need to reorder, edit, or delete columns?** + + Learn how to effectively manage your columns in the [Column Management](./column-management) guide. + + +## 📝 Data Input Columns +Store and manage static data entered manually or imported from external sources. + +### Text field +Free-form text input with multiline support + +### Numeric +Numbers, integers, and floating-point values + + + + +The last row allows you to choose a calculation method for the column, such as average, median, minimum, maximum, or sum. + + + + + + + + +### Single select +Single-choice columns let you define a set of predefined options and restrict each cell to one selection. +To create one, set the column name and add options in the Create Column drawer. +In the values box, type an option and press Enter to save it—once added, it will appear as a colored label. + +In the table, each cell will then allow you to select only one of the defined options. +This column type is especially useful for manual tagging with a single tag. + + + + + + + + + + + +### Multi select +Multi-select columns let you define a set of predefined options and allow each cell to contain multiple selections. The setup process is the same as for single-select columns: define the column name, add options in the Create Column drawer, and save them as labels. + +In the table, each cell can then include several of the defined options. This column type is especially useful for manual tagging with multiple tags. + + + + + +### JSON +A JSON column allows you to store and edit structured JSON objects directly in the Playground. Each cell can contain a JSON value, making it easy to work with complex data structures. + +When editing a cell, an Edit JSON panel opens with syntax highlighting and formatting support, so you can quickly add or update fields. + + + + diff --git a/playgrounds/columns/prompt.mdx b/playgrounds/columns/prompt.mdx new file mode 100644 index 0000000..b3ed6ae --- /dev/null +++ b/playgrounds/columns/prompt.mdx @@ -0,0 +1,75 @@ +--- +title: "Prompt Column" +description: "Execute LLM prompts with full model configuration" +--- + +### Prompt +A Prompt column allows you to define a custom prompt and run it directly on your Playground data. +You can compose prompts with messages (system, user, assistant or developer), insert playground variables, and configure which model to use. +Each row in your playground will be passed through the prompt, and the model’s response will be stored in the column. + + +Prompt columns make it easy to test different prompts against real data, compare model outputs side by side. + + + + + +## Prompt Writing + +Write your prompt messages by selecting a specific role—System, User, Assistant, or Developer. + +You can insert variables into the prompt using curly brackets (e.g., `{{variable_name}}`). These variables can then be mapped to existing column data, allowing your prompt to dynamically adapt to the playground + + + + + +## Configuration Options + +### Model Selection +Choose from supported LLM providers and models: +- **OpenAI**: GPT-5, GPT-5-Mini, GPT-5-Nano, GPT-4o, GPT-4o-mini and more +- **Anthropic**: Claude-3-Opus, Claude-3-5-Sonnet, Claude-3-5-Haiku and more +- **Gemini**: Gemini-2.5-Flash, Gemini-2.5-Pro and more + +### Structured Output +Structured output can be enabled for models that support it. You can define a schema either by writing a JSON structure in the JSON editor or by using the Visual editor to add parameters, specifying their names and types. + +## Tools +Tools let you extend prompts by allowing the model to call custom functions with structured arguments. Instead of plain text, the model can return a validated tool-call object that follows your schema. + +To create a tool, give it a name and description so the model knows when to use it. Then define its parameters with a name, description, type (string, number, boolean, etc.), and whether they are required. +### Advanced Settings +Fine-tune model behavior options: +- **Temperature** (0.0-1.0): Control randomness and creativity +- **Max Tokens**: Limit model output length (1-8000+ depending on model) +- **Top P**: Nucleus sampling parameter (0.0-1.0) +- **Frequency Penalty**: Reduce repetition (0.0 to 1.0) +- **Presence Penalty**: Encourage topic diversity (0.0 to 1.0) +- **Logprobs**: When enabled, returns the probability scores for generated tokens +- **Thinking Budget** (512-24576): Sets the number of tokens the model can use for internal reasoning before producing the final output +A higher budget allows more complex reasoning but increases cost and runtime +- **Exclude Reasoning from Response**: If enabled, the model hides its internal reasoning steps and only outputs the final response + +## Prompt Execution + +A prompt can be executed across all cells in a column or on a specific cell. + +Prompt outputs can be mapped to different columns by clicking a cell and selecting the mapping icon, or by double-clicking the cell \ No newline at end of file diff --git a/playgrounds/quick-start.mdx b/playgrounds/quick-start.mdx new file mode 100644 index 0000000..c10fe95 --- /dev/null +++ b/playgrounds/quick-start.mdx @@ -0,0 +1,54 @@ +--- +title: "Quick Start" +--- + +Playgrounds are interactive spreadsheet where you can organize your data and experiment with LLMs, evaluate outputs, and analyze data. +Think of them as powerful workbenches for AI development that combine the flexibility of a spreadsheet with the power of LLM evaluation and execution. +It’s designed for everyone, from product managers and analysts to QA, data engineers, and software developers. + + + + + + +## Playground Structure + +A playground is organized as a table-like structure with three fundamental components: **rows**, **columns**, and **cells**. Understanding how these work together is essential for effective playground usage. + +### Rows + +Rows represent individual **data points** or **test cases** in your playground. Each row is a complete record that spans across all columns. + +### Row Characteristics +Each row in the Playground is independent and can be executed on its own, maintains an order that can be rearranged as needed. + +### Row Operations +- **Add Row**: Create new rows manually or through bulk operations +- **Delete Row**: Remove unwanted rows individually or in bulk +- **Duplicate Row**: Copy existing rows to create variations +- **Reorder Rows**: Drag and drop to change row order +- **Generate Rows**: Use AI to generate multiple similar rows based on patterns + +## Create A Playground + +Data can be imported from different sources: + +1. CSV files +2. JSON file +3. From A Dataset +4. From production spans + +You can create a Playground from scratch and import data later. Simply set a name for the Playground and start adding columns, rows, and data. + + + + + + +Ready to build more sophisticated playgrounds? Dive into the [complete documentation](./index) or explore specific [column types](./columns/data-columns) to unlock the full power of Traceloop Playgrounds! \ No newline at end of file