AI-NOMICS

Syllabus areas:

GS Paper III – Science & Technology, Economy, Internal Security

TOKENS – TOKENOMICS – TOKENISATION:

As humans, the smallest unit of information we use to represent language is a single character – i.e. a letter or a number or punctuation mark.
But the smallest unit of language that a Large Language Model (LLM) uses is called a Token.

A token can be a set of characters or part of word or syllable or whole word or even a short multi word phrase. The code that converts language to tokens is called Tokeniser.

We may say that a token is about 2/3 to 3/4 of a word. But – image/audio/video-based inputs, it is not possible. Hence Token is defined in terms of "syllablesete". An english word is represented by approximately 1–5 tokens.
If you input a prompt of 100 word long, the LLM will likely work with 150 tokens.

After the input is tokenised, the model then converts tokens into vectors which are long list of numbers.
These vectors form the basic computational unit on which the model performs calculations to generate output.

TOKENOMICS:
When it comes to LLMs, model providers such as OpenAI and the Anthropic bill customers by tokens, which directly relate to the amount of processing their GPUs (Graphic Processing Unit) perform for a given prompt.

Higher the tokens involved, higher the workload on GPU which in turn translate to higher cost for the model provider.

This billing by tokens is more true of enterprise customers who use APIs (Application Programming Interface) to connect their enterprise software to an LLM than of consumers like you and me.

Just as electricity bills are based on units consumed, LLM billing is based on tokens consumed. Electricity bills charge based on what kind of – industrial and residential. Similarly, pricing of LLMs depends on the complexity of the model. A larger, more robust model will cost more per token and vice-versa.

Typically, users are billed per million tokens and further, there is a difference based on whether token is part of the input or part of the output. Input tokens cost lower than output- tokens because the amount of processing required to generate a text output, for instance, is higher than reading a text input.

Over the past years, companies have pushed their workforce to adopt AI in their work flows, often without capping token consumption. Employees were even tracked and ranked based on token usage, as a proxy to measure productivity and to evaluate whether they have become “AI-Native”. This led to phenomenon known as Token maxxing.

Agentic AI:

Agentic AI is one of the most powerful applications of LLM. It refers to the ability of an AI model to spawn and deploy agents meant to carry out a complex task.

Here is how an Agent is defined:

“Agents are autonomous systems that can operate independently, use various tools to accomplish complex tasks, planned and executed step after step using the reasoning abilities of an LLM.”

Autonomy, ability to plan and reason, ability to handle complex tasks – all are defining characteristics of an agent. Another key feature of an agent is Ability to call the right tool to discharge the task.

Proprietary vs. Open-source models

A proprietary LLM is developed and owned by organisation that develops it. Everything is private and not available for access by outsiders.

Popular LLMs such as:

1.    Open AI’s GPT series                 

2. Anthropic’s Claude Sonnet              →          Proprietary models

3.     Google’s Gemini

Open source Model is available for commercial use by persons other than the developer

1.    Deepseek – R1

2.     Alibaba’s Qwen 3.6                       →      Open source models.

3.     Meta’s Llama 4       

In the traditional sense, open-source would mean that the source code is accessible.

Artificial General Intelligence AGI is when an AI system can match or surpass cognitive abilities of humans.

When that happens, AGI can autonomously learn, reason and transfer knowledge seamlessly from one unfamiliar task to another without human intervention.
∴ AGI is also referred to as strong AI

Artificial Narrow Intelligence ANI for only single and predefined task.

1. A chess bot

2. Facial Recognition software        

3. Classifying emails spam    

These cannot write essay or generate images or process medical scan.

∴ ANI is also referred to as weak AI

Today's LLMs are still technically ANI but just more.
 ∴ Between AGI and ANI, we have LLMs.

In Simple Terms:

Tokenisation is the process of breaking text into smaller pieces that an AI model can process.
Tokens are the resulting pieces.

For ex: consider the sentence - "Artificial Intelligence is amazing"
A tokeniser might split it into tokens like:
 "Artificial"
 "Intelligence"
 "is"
 "amazing"

or depending on the tokeniser, a long word may be split into subword tokens:

 "Art"
 "ificial"
 "intelligence"
 "is"
 "Amazing"

Modern AI models such as GPT models often use sub word tokenisation, where common words are single tokens but rare or long words are split into smaller parts. This allows the model to handle virtually any text efficiently.

Tokenisation:

∴ Tokenisation is the process and Tokens are the output of that process.
Without tokenisation, language models cannot convert human-readable text into a form they can process.

Tokenisation: 

Tokenisation is the creation, issuance or representation of assets on a digital token ledger or a programmable platform.

Tokenisation is linked to block chain technology. It came into prominence following the rise of cryptocurrencies.

All the transactions in the tokens are recorded on digital ledgers, which are transparent, secure and can be accessed by everyone.

Tokenisation is adopted in various countries.

The idea of converting all securities — equities, bonds, derivatives — into tokens and enabling trading and settlement through distributed ledger is good, since it is faster and lowers the intermediation cost, improves liquidity.

While many countries are weighing tokenisation, 91% of them have very limited or no tokenisation use cases, still in pilot phase. Only 11% of institutional investors have invested in tokenisation in 2025, 61% expecting to invest by 2026 end.

Real-World Assets (RWAs)

Real World Assets are physical or traditional financial assets such as real estate, commodities, bonds, stocks, and invoices that have intrinsic value in the real economy.

In this, tokenisation is the process of representing ownership or rights to these assets as digital tokens on a block chain.

While RWAs are the underlying assets, tokenisation is the technology that digitises them. This enables fractional ownership, improved liquidity, faster transactions and enhanced transparency.