babylon.rag.context_window.token_counter

Token counting utilities for Context Window Management.

Functions

count_tokens(content)

Count the number of tokens in content of various types.

babylon.rag.context_window.token_counter.count_tokens(content)

Count the number of tokens in content of various types.

This is a simple implementation that estimates token counts. For production, this should be replaced with a proper tokenizer for the target model.

Parameters:

content (str | list[Any] | dict[str, Any] | Any) – Content to count tokens for. Can be string, list, dict, or other type.

Return type:

int

Returns:

Estimated token count