fix: isolate tokenizer truncation for concurrent encode calls - #385
tuanzirwar wants to merge 1 commit into
Conversation
|
Hello, we are aware of this and don't consider this an issue. |
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| ) -> list[np.ndarray]: | ||
| """Encode a batch of sentences as a sequence.""" | ||
| ids = self.tokenize(sentences=sentences) | ||
| ids = self._tokenize(sentences, self.tokenizer if tokenizer is None else tokenizer) |
There was a problem hiding this comment.
StaticModel subclass overrides the public tokenize() method to customize token IDs, encode_as_sequence() now calls _tokenize() instead. encode() makes the same change, so both methods silently ignore the override and can return different embeddings than before.
| if max_length != self.max_length: | ||
| # 默认长度复用只读 tokenizer，仅显式覆盖时复制一次供所有批次使用。 | ||
| tokenizer = copy.deepcopy(tokenizer) |
There was a problem hiding this comment.
encode_as_sequence() defaults to max_length=None, which differs from a model's usual non-None default. As a result, every such call deep-copies the tokenizer, even for a small request, adding repeated copying cost to sequence encoding.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
2 participants
Concurrent calls on one StaticModel can overwrite each other's tokenizer truncation settings. A controlled interleaving of max_length=1 and max_length=3 returns the longer embedding for the short request, even though sequential encoding returns the expected result.
Use a request-local tokenizer when max_length overrides the model default, pass it through serial and joblib batches, and keep the default path reusing the original tokenizer. This covers encode and encode_as_sequence and avoids restoring shared mutable state after a request. Concurrent changes to model configuration are outside this change.
Validation: the reproduction failed before the fix and now matches sequential results; 411 non-integration tests passed on Windows/Python 3.14, including 18 new tests covering mixed encoding modes, truncation removal, threaded batches, tokenizer reuse, and exceptions. Ruff and diff checks passed. Mypy reports the same pre-existing numpy dtype assignment error on both upstream and modified source; no new type errors were reported. No hosted/pretrained model integration suite was run.
Prepared with Codex; the reproduction and tests above were executed locally.