Computer Science thesis · UMSA
LLM Optimal Segmentation
How to split a long text so it fits an LLM without breaking its meaning.
Splits text into segments that respect the model's context window, minimising computational cost and maximising semantic coherence. It uses dynamic programming with a global optimum, plus 2D, overlapping and sliding-window variants. It ships with a calibration module, a Streamlit interface, a CLI, a Python API and more than 173 tests.
