close

DEV Community

#tokenization

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
NLP #2: Unicode and UTF-8
gradientgym profile

NLP #2: Unicode and UTF-8

#nlp #machinelearning #tokenization #unicode
Comments
13 min read
What an Urdu sentence costs in tokens
abdul_raufghulamnabi_65 profile

What an Urdu sentence costs in tokens

#ai #nlp #tokenization #opensource
Comments 4
5 min read
Gemini 4 Argon has a new tokenizer: Korean and Japanese got 8% cheaper, Indian languages up to 19% dearer
jaehyun_cho_0dff271e0d2e5 profile

Gemini 4 Argon has a new tokenizer: Korean and Japanese got 8% cheaper, Indian languages up to 19% dearer

#gemini #ai #llm #tokenization
Comments
4 min read
The Rounding Error That Stops the Machine
nia_agent_d4ea108025639bc profile
NIA

The Rounding Error That Stops the Machine

#performance #llm #tokenization #inference
Comments
5 min read
Unlocking Institutional Finance on the XRP Ledger.
shotanate profile

Unlocking Institutional Finance on the XRP Ledger.

#xrpl #atomicity #collateral #tokenization
Comments
5 min read
Music Royalties Tokenization in Brazil: Monetize Your Work with Blockchain
andr_diasmoreiraprol_b profile

Music Royalties Tokenization in Brazil: Monetize Your Work with Blockchain

#tokenization #music #stellar #web3
Comments 1
3 min read
The Tokenization Supercycle
zenieverse profile

The Tokenization Supercycle

#tokenization #realestates #supercycle #solana
Comments 1
1 min read
Tokenizing Court Orders: How Brazil Uses Blockchain for Liquidity
andr_diasmoreiraprol_b profile

Tokenizing Court Orders: How Brazil Uses Blockchain for Liquidity

#tokenization #legal #blockchain #web3
BERJAYA 1
Comments
3 min read
Opening the Black Box of LLM Inference: Foundations of Generative AI
aridiosilva profile

Opening the Black Box of LLM Inference: Foundations of Generative AI

#ai #llm #sgaeia #tokenization
Comments
61 min read
Un token è un diritto, non il bene
omarbaruzzo profile

Un token è un diritto, non il bene

#blockchain #enterprise #tokenization
Comments
1 min read
Token je pravo, ne sama imovina
omarbaruzzo profile

Token je pravo, ne sama imovina

#blockchain #enterprise #tokenization
Comments
1 min read
A token is a claim, not the asset
omarbaruzzo profile

A token is a claim, not the asset

#blockchain #enterprise #tokenization
Comments
1 min read
Non-English Prompts Cost More Tokens, Context and Money in Most LLMs
khasky profile
Khasky

Non-English Prompts Cost More Tokens, Context and Money in Most LLMs

#llm #ai #tokenization #promptengineering
BERJAYA 1
Comments
6 min read
Why a Small Transformer Can't Copy a Word It Hasn't Seen
megapixel99 profile

Why a Small Transformer Can't Copy a Word It Hasn't Seen

#measurement #codegeneration #llm #tokenization
Comments
10 min read
Tokens: The Currency of AI
internals_decoded profile

Tokens: The Currency of AI

#ai #tokens #tokenization #llmpricing
Comments
10 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.