3.1.1. Criteria for Choosing Pre-trained Models

First Principle: Selecting the right pre-trained model is a critical design decision that requires a multi-faceted evaluation of the model's capabilities against the specific technical and business requirements of the application.

This extends the general selection factors from Phase 2 into more concrete design criteria.

Key Design Criteria:

• Modality: What kind of data does the application handle? If it's only text, a pure LLM is sufficient. If it needs to understand or generate images, a multi-modal model is required.

• Customization: Does the business problem require the model to learn company-specific jargon, styles, or information? If so, you need to choose a model that supports fine-tuning.

• Model Complexity & Size: Larger models generally have more knowledge and better reasoning capabilities but are slower and more expensive. A key design choice is finding the smallest model that can effectively perform the required task to optimize for cost and latency.

• Input/Output Length (Context Window): How much text can the model process at once? For tasks like summarizing long documents, a model with a large context window is essential. For simple Q&A, a smaller context window may be adequate.

• Latency: How quickly does the application need a response? For a real-time interactive chatbot, low latency is critical. For an offline report generation tool, latency is less of a concern.

• Cost: The cost per token (or cost for provisioned throughput) is a major design factor that must be weighed against the model's performance and the application's budget.

• Prompt caching support: If your application sends the same long content (a policy manual, a contract, fixed instructions) with many different questions, a model that supports prompt caching can reuse that prefix instead of reprocessing it, lowering input token cost and latency. Put static content at the beginning of the prompt and changing content at the end to make cache hits more likely.

Scenario: You are designing an AI application that must read a 50-page legal document and provide a detailed summary.

Reflection Question: Which design criterion is most critical for this task? Why would a model with a small context window be completely unsuitable, regardless of its other capabilities?

💡 Tip: There is often a direct trade-off between model capability, speed, and cost. Your job as a practitioner is to understand the business requirements well enough to make the right compromise.