LLUMO AI’s Custom Evaluation Metrics feature lets you tailor the evaluation process to fit your unique needs, ensuring AI outputs align with your business goals. Whether you’re analyzing customer service, academic content, or any other data, it helps you gain more precise and relevant insights.
Custom metrics ensure your AI evaluation directly reflects your business objectives, whether it’s improving customer service, content quality, or any other area.
Granular feedback helps identify strengths and weaknesses, supporting informed decisions that drive AI and business growth.This guide will walk you through the steps of using Custom Eval Metrics to perform a detailed and personalized evaluation of your datasets.
Log into the LLUMO AI Platform and go to the “Evaluate Dataset” section.
Click “Upload File” and select your dataset. The file can be in CSV, JSON, or Excel format.
Review Your Data: A preview of the uploaded data will be displayed. Ensure that the file is structured correctly and the data looks accurate before proceeding.
Here, you can create custom evaluation metrics by specifying a column name, defining the evaluation criteria, and selecting the prompt or output to assess. You can also create multiple custom evaluation metrics to evaluate different aspects of your data.
Once you’ve entered all the details, it will appear like this. In this example, I’ve created a custom metric called Customer Intent Alignment (CIAS).Definition: CIAS measures how well the company’s response aligns with the customer’s intended goal or request, based on AI-driven analysis of both the customer’s inquiry and the company’s response.
Select your evaluation model and choose the output column in your dataset that you want to assess. You can experiment with different models and run evaluations on the same prompt to compare their performance, using your custom evaluation metrics to analyze the results.
Set Rule for KPIs:
Each KPI allows you to set a specific threshold to define a “pass” or “fail.”Example: For confidence, if the confidence score is more than 50, it passes.
How do I choose the right KPIs for my evaluation?
Select KPIs based on your evaluation goals. For example, for customer service data, KPIs like Sentiment Analysis and Response Time are essential. For academic writing, Grammar Quality and Clarity may be more relevant.What happens if my outputs don’t meet the thresholds I set?
If an output fails to meet a threshold, it will be flagged as a failure. You can review the failed outputs and adjust the thresholds or improve the outputs to meet the required standards.Can I export the evaluation results?
Yes, once the evaluation is complete, you can export the results in CSV, Excel, or PDF format for further analysis or reporting.How long does the evaluation process take?
The time taken for the evaluation depends on the size of your dataset and the complexity of the model you’ve chosen. Typically, an evaluation of 100 prompts and outputs should take just a few minutes.If you need additional assistance, don’t hesitate to reach out to our support team!