Fallback

With various Language Model APIs at your disposal, each with unique strengths and specialties, imagine the convenience of effortlessly transitioning between them based on their performance or availability. SimpliML's Fallback feature is crafted precisely for this purpose.

The Fallback functionality enables you to define a prioritized list of Language Model APIs (LLMs). Should the primary LLM encounter an error or fail to respond, SimpliML seamlessly switches to the next LLM in the specified list. This ensures the resilience and dependability of your application, offering a smooth and reliable user experience.

Enabling Fallback for Language Model APIs (LLMs)

To enable fallbacks, you need to add the config object to include in the Inference API request. Here's a quick example of a config to fallback to meta-llama/Llama-2-7b-chat-hf if mistralai/Mistral-7B-Instruct-v0.2 fails.

config object

    {
        "fallback": [
            {
                "provider": "simpliml",
                "model": "s7e2ca956beb6e87d7dae"
            }
        ],
    }

Option	Type	Description	Default	Required
provider	string	which provider to use for fallback	-	Yes
model	string	Model id to use for the fallback	-	Yes

Important Notes and Considerations:

While the Fallback on Language Model APIs (LLMs) feature significantly boosts the reliability and resilience of your application, it's crucial to keep a few key considerations in mind:

Verify the compatibility of the LLMs in your fallback list with your specific use case. Different LLMs may offer varied capabilities, so ensure they align with your application's requirements.
Monitor your usage patterns with each LLM. Depending on your fallback list, a single request might trigger multiple LLM invocations, affecting your overall usage metrics.
Recognize that each LLM comes with its unique latency and pricing structure. Shifting to a different LLM through fallback mechanisms may impact both the cost and response time of your application. Be mindful of these factors to make informed decisions.

Caching Retries