langchain Runnable 객체(chat model, retriever, chain 등)는 evaluate() / aevaluate()에 직접 전달할 수 있습니다.

​
Setup

평가할 간단한 chain을 정의해 보겠습니다. 먼저 필요한 모든 패키지를 설치합니다: 
pip install -U langsmith langchain[openai]
이제 chain을 정의합니다: 
from langchain.chat_models import init_chat_model
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

instructions = (
    "Please review the user query below and determine if it contains any form "
    "of toxic behavior, such as insults, threats, or highly negative comments. "
    "Respond with 'Toxic' if it does, and 'Not toxic' if it doesn't."
)

prompt = ChatPromptTemplate(
    [("system", instructions), ("user", "{text}")],
)

model = init_chat_model("gpt-4o")
chain = prompt | model | StrOutputParser()

​
Evaluate

chain을 평가하기 위해 evaluate() / aevaluate() 메서드에 직접 전달할 수 있습니다. chain의 입력 변수는 example input의 key와 일치해야 합니다. 이 경우 example input은 {"text": "..."} 형식이어야 합니다. 
from langsmith import aevaluate, Client

client = Client()

# Clone a dataset of texts with toxicity labels.
# Each example input has a "text" key and each output has a "label" key.
dataset = client.clone_public_dataset(
    "https://smith.langchain.com/public/3d6831e6-1680-4c88-94df-618c8e01fc55/d"
)

def correct(outputs: dict, reference_outputs: dict) -> bool:
    # Since our chain outputs a string not a dict, this string
    # gets stored under the default "output" key in the outputs dict:
    actual = outputs["output"]
    expected = reference_outputs["label"]
    return actual == expected

results = await aevaluate(
    chain,
    data=dataset,
    evaluators=[correct],
    experiment_prefix="gpt-4o, baseline",
)
runnable은 각 output에 대해 적절하게 추적됩니다. Runnable Evaluation
Edit the source of this page on GitHub.
Connect these docs programmatically to Claude, VSCode, and more via MCP for real-time answers.
⌘I