Overview

Connect your agent to see action credit from Stash's shared evaluator, including tool calls. Contribute traces and corrections explicitly to improve future versions. You can also train a personal reward model and generate instructions with GEPA.

This experiment is enabled for accounts created after the rollout. Existing accounts keep their current experience and cannot access reward-model pages or APIs. Eligibility is per account, so a new account in an existing organization also enters the experiment.

The recordings below show an earlier interface; follow the current controls described here.

1. Connect Stash to your agent

Stash reads your agent's traces over OpenTelemetry. Install the instrumentation for your framework, set three variables, and every run shows up on the Traces page.

export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.joinstash.ai/api/v1/rm/otel
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20$STASH_API_KEY"

opentelemetry-instrument python agent.py

The instrumentation for an Anthropic agent, for example (the OpenInference packages cover OpenAI, the OpenAI Agents SDK, LangChain, LlamaIndex, CrewAI, DSPy, Bedrock, and more; the Vercel AI SDK emits OpenTelemetry on its own):

pip install opentelemetry-distro \
  opentelemetry-exporter-otlp-proto-http \
  openinference-instrumentation-anthropic

Frameworks group each run into one trace. If your agent calls a model SDK directly in a loop, wrap each run in a span so its calls land in the same trace:

with tracer.start_as_current_span("agent-run"):
    run_agent(task)

2. Review action credit

Once a shared evaluator is released, imported traces are scored automatically. Open a trace to see credit on responses and tool calls. No annotations are required. Use Comment on an action to correct its assessment. Before the first release, the viewer reports that it is waiting for an evaluator.

3. Help improve the shared evaluator

Enable the trace's training-contribution checkbox to let its feedback and independently reviewed comparisons contribute to future candidates. Scoring alone does not grant this permission. Stash compares candidates with the current evaluator on a separate reviewed benchmark before releasing an improvement. Model predictions never serve as their own labels.

Optional: train a personal reward model

Select traces and choose Create new reward model. The worker derives preferences from supported feedback and records the evidence for each pair. Training needs at least two usable pairs; ambiguous feedback is skipped. Choose View feedback on the model to inspect inferred labels and source quotes. These are classifier judgments, not human ratings. The trained model scores every trace you own, including unselected traces. You can also download its weights.

Optional: write a skill

On a trained reward model, choose View skill. It opens an existing run or starts one. GEPA uses the selected traces and their feedback to write and name a SKILL.md that teaches your agent to score well on that model. Your agent loads the skill next to its system prompt, which Stash leaves alone.

An excerpt from the skill in that recording:

---
name: refund-requests
description: Use this skill whenever a user requests a refund
  or reports a broken/damaged order. Always look up the order
  details and check the applicable refund policy before
  responding, and explain the next steps to the user rather
  than promising a refund outright.
---

## Hard rules

- **Never** promise a refund, replacement, or specific dollar
  amount as a done deal before eligibility is confirmed.
- **Never** claim to have looked something up that you did
  not actually look up.
- **Never** ask the user to repeat information already given
  in the conversation.