> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Luna-2 Overview
> Discover Splunk Agent Observability's Luna-2 Evaluation model, reducing the latency and cost for evaluations
Luna-2 is only available in the Enterprise tier. Fine-tuning Luna-2 models is only supported for on-premises, standalone, and custom deployments. [Contact us](https://www.splunk.com/en_us/about-splunk/contact-us.html) to learn more and get started.
**Luna-2** is the latest generation of our Luna small language models (SLMs), purpose built for scaling AI evaluations. Luna-2 models are fine tuned to provide low latency and reduced costs for evaluations. Luna-2 is designed to be further fine tuned for your specific use cases and custom evaluators with the goal of providing scalable, real-time, customizable evaluations for enterprises.
Luna-based evaluators offer highly accurate and efficient evaluations for AI applications, particularly those with agentic workflows.
Want to fine-tune a Luna-2 evaluator for your own use case without writing training code? See [Luna Studio](/luna-studio) — Splunk Agent Observability's web app for self-service custom evaluator fine-tuning.
Contact us to learn more about using Luna-2 in your evaluations
Learn how Splunk Agent Observability pushes the envelope on GenAI evaluation with our family of fine tuned small language models.
## Overview
LLMs are powerful judges for evaluations, but as your application scales up to from tens or hundreds of traces a day, to thousands or millions, they can fall short. Too often, organizations relying solely on LLMs to act as judges incur major inference costs and don't see the low-latency they need to enable real-time evaluations and runtime protection.
* LLMs are expensive
* LLMs don't provide the performance needed, especially for runtime protection
* LLMs are general purpose, and even leveraging [Autotune](/concepts/evaluators/autotune-llm-as-a-judge-evaluators) to enhance the evaluation prompts, can still be less effective for your specific needs.
The Luna-2 model mitigates these issues:
* Being an SLM, it is an **order of magnitude cheaper** to run than most LLMs
* SLMs run an **order of magnitude faster**, allowing for **runtime protection**
* Luna-2 is not only **fine-tuned for evaluations**, giving comparable performance out of the box with the top LLMs, but it can be **further fine-tuned using your data** to improve accuracy beyond any general purpose LLM.
The Luna-2 model works with most of the [out of the box evaluators](/sdk-api/evaluators/evaluators), or your [LLM-as-a-judge custom evaluators](/concepts/evaluators/custom-evaluators/custom-evaluators-ui-llm).
## Performance and cost comparison
### Comparison with different LLMs and content safety tools
| Model | Cost/1M token | Accuracy (F1 score) | Latency (avg) | Max tokens |
| :------------------- | ------------: | ------------------: | ------------: | ---------: |
| **Luna-2** | **\$0.02** | **0.95** | **152ms** | **128k** |
| GPT 4o | \$2.50 | 0.94 | 3,200ms | 128k |
| GPT 4o mini | \$0.60 | 0.90 | 2,600ms | 128k |
| Azure Content Safety | \$1.52 | 0.62 | 312ms | 3k |
### Latency vs compute requirements
These are the measured latencies for Luna-2 across a range of GPUs for different sized requests.
#### H100/H200 GPU
| Model | Small (500 tokens) | Medium (2K tokens) | Large (15K tokens) | Extra Large (100K tokens) |
| :-------- | -----------------: | -----------------: | -----------------: | ------------------------: |
| Luna-2 3B | 15ms | 15ms | 141ms | 2.8s |
| Luna-2 8B | 16ms | 30ms | 277ms | 4.71s |
#### RTX PRO 6000 GPU
| Model | Small (500 tokens) | Medium (2K tokens) | Large (15K tokens) | Extra Large (100K tokens) |
| :-------- | -----------------: | -----------------: | -----------------: | ------------------------: |
| Luna-2 3B | 17ms | 32ms | 245ms | 4.8s |
| Luna-2 8B | 28ms | 61ms | 514ms | 8.05s |
#### B200 GPU
| Model | Small (500 tokens) | Medium (2K tokens) | Large (15K tokens) | Extra Large (100K tokens) |
| :-------- | -----------------: | -----------------: | -----------------: | ------------------------: |
| Luna-2 3B | 15ms | 16ms | 81ms | 1.37s |
| Luna-2 8B | 15ms | 19ms | 146ms | 2.24s |
#### A100 GPU
| Model | Small (500 tokens) | Medium (2K tokens) | Large (15K tokens) | Extra Large (100K tokens) |
| :-------- | -----------------: | -----------------: | -----------------: | ------------------------: |
| Luna-2 3B | 27ms | 85ms | 750ms | 12.5s |
| Luna-2 8B | 51ms | 177ms | 1.51s | 21.2s |
#### L40S GPU
| Model | Small (500 tokens) | Medium (2K tokens) | Large (15K tokens) | Extra Large (100K tokens) |
| :-------- | -----------------: | -----------------: | -----------------: | ------------------------: |
| Luna-2 3B | 57ms | 91ms | 491ms | 8.06s |
| Luna-2 8B | 86ms | 163ms | 1.01s | 14.03s |
#### L4 GPU
L4 GPUs are only supported for calculating evaluators for Agent Streams and experiments. These GPUs are not supported for runtime protection.
| Model | Small (500 tokens) | Medium (2K tokens) | Large (15K tokens) | Extra Large (100K tokens) |
| :-------- | -----------------: | -----------------: | -----------------: | ------------------------: |
| Luna-2 3B | 51ms | 155ms | 1.66s | 29.45s |
| Luna-2 8B | 126ms | 364ms | 3.35s | 50.78s |
The actual latencies can vary a lot based upon the load on the system (Eg: QPS). This can be managed with more GPUs, but the cost will increase.
## Technical details
Splunk Agent Observability's Luna-2 evaluators utilize fine-tuned Llama models (3B and 8B variants) for generative AI evaluators. The technical process involves:
* **Fine-Tuning:** Base Llama models are fine-tuned with proprietary data for specific evaluator needs.
* **Classification:** Models output normalized log-probabilities of True/False tokens to determine evaluator accuracy.
* **Optimized Infrastructure:** Evaluators are hosted on Splunk Agent Observability's optimized inference engine with modern GPU hardware for low-latency and cost-effective evaluations. You can also self host on-prem or on your cloud infrastructure.
* **Adapters for Custom Evaluators:** Lightweight adapters on a shared base model enhance scalability and minimize infrastructure overhead for additional evaluators.
By leveraging fine-tuned Llama models, Luna-2 evaluators provide significant enhancements over traditional methods:
Luna evaluation models are fine-tuned on open-source base models, including but not limited to Llama and Mistral. Where applicable, third-party license terms apply — for example, Llama is licensed under the [Meta Llama Community
License](https://llama.meta.com/llama3/license), Copyright (c) Meta Platforms, Inc. All Rights Reserved.
* **Adaptability:** These models are most effective when fine tuned, requiring approximately 4,000 samples for fine-tuning to customer-specific use cases.
* **Efficiency and Cost-Effectiveness:** Luna-2 models enable simultaneous evaluation of multiple evaluators with low latency and reduced costs, ideal for real-time, high-scale deployments.
* **Enhanced Accuracy:** Luna-2 demonstrates at least a 10% accuracy increase compared to traditional BERT-based models, perfect for precise monitoring in production environments.
## Get started with Luna-2
If you are using the enterprise tier of Splunk Agent Observability, follow these steps to use Luna-based evaluators:
1. [Contact support](https://www.splunk.com/en_us/about-splunk/contact-us.html) to begin onboarding.
2. If you are using a Splunk Agent Observability-hosted instance, request L4 GPUs or higher, necessary for running Luna-2 models. Otherwise you can deploy to your own infrastructure, using L4 or higher GPUs.
3. Review the provided documentation and model cards for details on latency, accuracy, and comparisons to BERT-based evaluators.
4. Provide Splunk Agent Observability with relevant labelled sample data to fine tune the model. We can augment this with synthetic data if needed.
5. Splunk Agent Observability will fine tune your model for you, and deploy it.
6. Set up your experiments and Agent Streams to use [Luna-based evaluators](/sdk-api/evaluators/evaluators).
This is not a one-shot process. Your model can be tuned on a regular basis as required.
## Next steps
Contact us to learn more about using Luna-2 in your evaluations
Learn how to create evaluators cheaper and faster using the Luna-2 model
Learn how to use Luna-2 evaluators when running experiments in code
Use Luna Studio to fine-tune Luna-2 on your data, no code required.