LLM Hallucination Index RAG SpecialModel Insights
claude-3-haiku-20240307
Details
Developer
Anthropic
License
NA (private model)
Model parameters
NA (private model)
Supported context length
200k
Price for prompt token
$0.25/Million tokens
Price for response token
$1.25/Million tokens
Model Performance Across Task-Types
Chainpoll Score
Short Context
0.92
Medium Context
0.96
Long Context
0.7
Model Insights Across Task-Types
Digging deeper, here’s a look how claude-3-haiku-20240307 performed across specific datasets
Short Context RAG
Medium Context RAG
This heatmap indicates the model's success in recalling information at different locations in the context. Green signifies success, while red indicates failure.
Long Context RAG
This heatmap indicates the model's success in recalling information at different locations in the context. Green signifies success, while red indicates failure.
Performance Summary
| Tasks | Task insight | Cost insight | Dataset | Context adherence | Avg response length |
|---|---|---|---|---|---|
| Short context RAG | The model demonstrates exceptional reasoning and comprehension skills, excelling at short context RAG. It shows good mathematical proficiency, as evidenced by its performance on DROP and ConvFinQA benchmarks. It comes out as one of the best small closed source model. | One of the most affordable closed source model with best in class performance. It is nearly 10x cheaper than Sonnet 3.5 and 80% cheaper than Llama-3-70b. We recommend using this if cost is your priority. | Drop | 0.93 | 583 |
| Hotpot | 0.87 | 583 | |||
| MS Marco | 0.93 | 583 | |||
| ConvFinQA | 0.97 | 583 | |||
| Medium context RAG | Great powerformance overall with minor problems for context more than 10000 tokens. | Good performance but we recommed using 3x cheaper Gemini Flash for best results. | Medium context RAG | 0.96 | 583 |
| Long context RAG | Model shows issues at all context lengths and shows poor performance after 60000 making it unsitable for long context use. | We recommend using Gemini Flash 3x lesser price instead of this due to significantly better performance. | Long context RAG | 0.70 | 583 |