KV Cache Context Memory Estimator
Estimate inference KV cache from layers, KV heads, head dimension, context, batch, and data type.
Parameters
Results
Assumptions
- 采用标准自回归注意力 K/V 双缓存结构。
Important notes
- Pagination缓存、量化、滑动窗口、张量并行和实现开销会改变实际显存。
Estimate inference KV cache from layers, KV heads, head dimension, context, batch, and data type.