Chunking by Markdown Headers Script

Snippet

Artificial Intelligence

Intermediate

Copy Snippet

Snippets are pieces of workflows. You can copy and paste them directly into any Tray workflow.

New to snippets? Watch this video

For more details please see our Snippet Documentation

About this Snippet

Copy

Chunking by markdown headers is a popular method for retrieval augmented generation pipelines. Use this script to easily setup your chunking logic. It allows you to easily update your chunk size so you can refine your chunking strategy in a way that optimizes for your use case and data sets.