llm-sanity-checks

Questions & Answers

What is llm-sanity-checks?

llm-sanity-checks is a GitHub repository offering a practical guide and decision tree to evaluate if a Large Language Model (LLM) is genuinely required for an AI task. It helps users avoid over-engineering with frontier models when simpler methods are sufficient.

Who should use llm-sanity-checks?

This resource is for engineers, developers, and product managers who are evaluating AI solutions for their applications. It is particularly useful for those looking to optimize for cost, latency, and architectural simplicity by making informed choices about model selection.

How does llm-sanity-checks differentiate between AI solutions?

It guides users through a decision tree, starting with non-LLM solutions like regex, rules, or traditional machine learning models for structured data. It then progresses to smaller LLMs, RAG, and only as a last resort, frontier models, ensuring a methodical approach.

When should I consult llm-sanity-checks for my project?

You should consult llm-sanity-checks at the initial architectural design phase of any project considering an LLM. It helps validate whether your task genuinely demands a complex, often expensive, LLM or if a more efficient alternative can be utilized.

What is the 'JSON Tax' as described in llm-sanity-checks?

The 'JSON Tax' refers to the increased token count, latency, and cost associated with using JSON for structured output from LLMs compared to simpler delimiter-separated output. For basic extraction tasks, using delimiters can be significantly more efficient.