Start getting insights in just 2 minutes
LLUMO AI’s Custom Evaluation feature allows you to tailor the evaluation process to meet your unique requirements. Whether you’re analyzing customer service data, academic content, or any other dataset, this powerful tool enables users to assess AI-generated outputs against tailored metrics and criteria that align with their specific needs and objectives.
Unlike pre-defined evaluation methods, which may not fully capture the nuances of different use cases, Custom Evaluation allows users to define their own parameters, ensuring the evaluation process is both relevant and effective.
This guide will walk you through the steps of using Custom Evaluation to perform a detailed and personalized evaluation of your datasets.
Here, you can follow these steps:
Here, you can create your own Custom Evaluation by specifying a column name, defining your evaluation criteria instead of pre-defined criteria, and selecting the prompt or output you wish to assess using this custom evaluation.
In the screen below, we select Response Coherence as our evaluation metric from over 50+ available KPIs, which comes with predefined criteria. In custom evaluation, we replace the predefined criteria with our own custom definition to evaluate the output according to our specific niche.
Definition:
It refers to how logically consistent and well-structured the AI’s response is, ensuring that the content flows smoothly, aligns with the prompt, and makes sense within its context. A coherent response should present information in a way that is clear, unified, and free from contradictions, with all parts of the response connecting logically to one another.
Key Aspects of Response Coherence:
Imagine you’re evaluating customer service responses. A coherent response should:
Using Custom Evaluation, you can:
How do I choose the right KPIs for my evaluation?
Select KPIs based on your evaluation goals. For example, for customer service data, KPIs like Sentiment Analysis and Response Time are essential. For academic writing, Grammar Quality and Clarity may be more relevant.
Can I customize the KPIs for my evaluation?
While you can’t create completely custom KPIs, you can define KPIs specific to your niche by selecting and adjusting the existing ones. For example, you might adjust the Sentiment Analysis threshold to match customer service standards.
What happens if my outputs don’t meet the thresholds I set?
If an output fails to meet a threshold, it will be flagged as a failure. You can review the failed outputs and adjust the thresholds or improve the outputs to meet the required standards.
Can I export the evaluation results?
Yes, once the evaluation is complete, you can export the results in CSV, Excel, or PDF format for further analysis or reporting.
How long does the evaluation process take?
The time taken for the evaluation depends on the size of your dataset and the complexity of the model you’ve chosen. Typically, an evaluation of 100 prompts and outputs should take just a few minutes.
If you need additional assistance, don’t hesitate to reach out to our support team!
Start getting insights in just 2 minutes
LLUMO AI’s Custom Evaluation feature allows you to tailor the evaluation process to meet your unique requirements. Whether you’re analyzing customer service data, academic content, or any other dataset, this powerful tool enables users to assess AI-generated outputs against tailored metrics and criteria that align with their specific needs and objectives.
Unlike pre-defined evaluation methods, which may not fully capture the nuances of different use cases, Custom Evaluation allows users to define their own parameters, ensuring the evaluation process is both relevant and effective.
This guide will walk you through the steps of using Custom Evaluation to perform a detailed and personalized evaluation of your datasets.
Here, you can follow these steps:
Here, you can create your own Custom Evaluation by specifying a column name, defining your evaluation criteria instead of pre-defined criteria, and selecting the prompt or output you wish to assess using this custom evaluation.
In the screen below, we select Response Coherence as our evaluation metric from over 50+ available KPIs, which comes with predefined criteria. In custom evaluation, we replace the predefined criteria with our own custom definition to evaluate the output according to our specific niche.
Definition:
It refers to how logically consistent and well-structured the AI’s response is, ensuring that the content flows smoothly, aligns with the prompt, and makes sense within its context. A coherent response should present information in a way that is clear, unified, and free from contradictions, with all parts of the response connecting logically to one another.
Key Aspects of Response Coherence:
Imagine you’re evaluating customer service responses. A coherent response should:
Using Custom Evaluation, you can:
How do I choose the right KPIs for my evaluation?
Select KPIs based on your evaluation goals. For example, for customer service data, KPIs like Sentiment Analysis and Response Time are essential. For academic writing, Grammar Quality and Clarity may be more relevant.
Can I customize the KPIs for my evaluation?
While you can’t create completely custom KPIs, you can define KPIs specific to your niche by selecting and adjusting the existing ones. For example, you might adjust the Sentiment Analysis threshold to match customer service standards.
What happens if my outputs don’t meet the thresholds I set?
If an output fails to meet a threshold, it will be flagged as a failure. You can review the failed outputs and adjust the thresholds or improve the outputs to meet the required standards.
Can I export the evaluation results?
Yes, once the evaluation is complete, you can export the results in CSV, Excel, or PDF format for further analysis or reporting.
How long does the evaluation process take?
The time taken for the evaluation depends on the size of your dataset and the complexity of the model you’ve chosen. Typically, an evaluation of 100 prompts and outputs should take just a few minutes.
If you need additional assistance, don’t hesitate to reach out to our support team!