A non-validating SQL parser module for Python
-
Updated
Aug 13, 2026 - Python
#
tokenizer
A grammar describes the syntax of a programming language, and might be defined in Backus-Naur form (BNF). A lexer performs lexical analysis, turning text into tokens. A parser takes tokens and builds a data structure like an abstract syntax tree (AST). The parser is concerned with context: does the sequence of tokens fit the grammar? A compiler is a combined lexer and parser, built for a specific grammar.
Here are 628 public repositories matching this topic...
Persian NLP Toolkit
python nlp natural-language-processing tokenizer embeddings persian text-processing dependency-parser farsi pos-tagging persian-nlp normalization lemmatization
-
Updated
Apr 1, 2026 - Python
Solves basic Russian NLP tasks, API for lower level Natasha projects
-
Updated
Oct 2, 2026 - Python
한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.
-
Updated
Sep 29, 2026 - Python
An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation
translation tokenizer corpus linguistics tagger literature dependency-parser corpus-linguistics lemmatizer corpus-tools corpus-processing corpus-search corpus-statistics stopword corpus-analysis
-
Updated
Oct 30, 2025 - Python
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
nlp tokenizer text-processing semeval nlp-library word-segmentation spelling-correction tokenization text-segmentation spell-corrector word-normalization
-
Updated
Jun 2, 2025 - Python
[NeurIPS 2025 Spotlight] A Unified Tokenizer for Visual Generation and Understanding
tokenizer generative-model image-generation generative text-to-image autoregressive-models large-language-models generative-ai image-tokenizer
-
Updated
Nov 14, 2025 - Python
Python port of Moses tokenizer, truecaser and normalizer
-
Updated
Sep 7, 2026 - Python
A Japanese tokenizer based on recurrent neural networks
-
Updated
Sep 22, 2026 - Python
AAAI 2025: Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model
audio music semantic text-to-speech tokenizer speech sound codec audio-codec gpt language-model self-supervised-learning text-to-music vall-e text-to-sound speech-language-model
-
Updated
Oct 12, 2025 - Python
Text2Text Language Modeling Toolkit
search nlp information-retrieval translator tokenizer chatbot multi-lingual transformers embeddings levenshtein-distance tf-idf llama cross-lingual question-generation rag llm chatgpt
-
Updated
Jan 14, 2025 - Python
Bitextor generates translation memories from multilingual websites
crawler dictionaries tokenizer machine-translation wget apertium neural-machine-translation warc tmx statistical-machine-translation corpus-generator sentence-segmentation corpus-tools corpus-processing hunalign parallel-corpora document-aligner bicleaner bleualign bitextor
-
Updated
Nov 11, 2024 - Python
[CVPR 2026 Highlight] A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens
computer-vision deep-learning tokenizer pytorch generative-model semantic-segmentation depth-estimation video-compression world-models vision-transformer cvpr2026
-
Updated
Jul 17, 2026 - Python
Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.
-
Updated
Nov 7, 2022 - Python
A 1.6B causal Transformer audio tokenizer with streaming, variable bitrates, and semantic alignment across speech, sound, and music
audio music streaming tokenizer speech tts unified audio-codec speech-representation audio-tokenizer neural-codec
-
Updated
Jun 16, 2026 - Python
Full-stack open-source AI engine for building language models — tokenizer training, transformer architecture, cognitive reasoning and chat pipeline.
nlp open-source machine-learning deep-learning tokenizer transformer language-model ai-engine conversational-ai rag bpe turkish-nlp ai- llm llm-training llm-engine- cognitive-ai-
-
Updated
Sep 21, 2026 - Python
Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
nlp tokenizer language-detection wordlist lemmatizer morphological-analysis lemmatiser tokenization lemmatization corpus-tools language-identification low-resource-nlp
-
Updated
Sep 30, 2026 - Python
DadmaTools is a Persian NLP tools developed by Dadmatech Co.
natural-language-processing tokenizer embeddings spacy persian dependency-parser lemmatizer ner persian-nlp postagger chunker dataset-loader constituency-parser embedding-vectors nlptoolkit
-
Updated
Aug 14, 2025 - Python
Text tokenization and sentence segmentation (segtok v2)
-
Updated
Mar 12, 2022 - Python
Fast bare-bones BPE for modern tokenizer training
-
Updated
Jun 23, 2025 - Python
- Followers
- 10.9k followers
- Website
- github.com/topics/parsing
- Wikipedia
- Wikipedia