Essential Chunking Techniques for Building Better LLM Applications

  Every large language model (LLM) application that retrieves information faces a simple problem: how do you break down a 50-page document into pieces that a model can actually use? So when you’re building a retrieval-augmented generation (RAG) app, before your vector database retrieves anything and your LLM generates responses, your documents need to be split into chunks.

source https://machinelearningmastery.com/essential-chunking-techniques-for-building-better-llm-applications/