Data tokenization replaces a sensitive value, such as a card number or a national ID number, with a substitute value called a token, so that most systems store and send the token instead of the real data. Asset tokenization is a different process with the same name: it records ownership of an asset as a token on a blockchain. This article covers data tokenization in full, then explains where asset tokenization differs.
After reading it, you can tell which kind of tokenization a vendor or a project means, how data tokens reduce PCI DSS scope, and when tokenization, encryption or masking fits a given dataset.
What is data tokenization?
The Payment Card Industry Security Standards Council (PCI SSC) defines tokenization as a process that replaces the primary account number (PAN) with a surrogate value called a token. De-tokenization is the reverse process, which exchanges a token for its PAN. The definition comes from the council's PCI DSS Tokenization Guidelines, an information supplement from August 2011 (called "the 2011 supplement" below), and the same idea applies to any sensitive field, not only card numbers.
A shop receives the card number 4111 1111 1111 1111 at checkout. The tokenization system stores that number in a protected place and returns a token such as 4111 8392 0475 1111. The shop's order database, analytics jobs and customer service tools keep only the token. The 2011 supplement also asks that a token be distinguishable from a real PAN, so that staff and assessors can tell which systems hold card data. When the shop needs to charge the card again, it sends the token to the tokenization system, which looks up or computes the real number and passes it to the payment processor.
According to the 2011 supplement, the security of a token depends on one property: nobody should be able to find the original PAN when they know only the token. Tokens should have no value if they are stolen, and a breach of a system that holds only tokens should give an attacker nothing usable.
How data tokenization works
The 2011 PCI SSC supplement names four common parts of a tokenization system:
- Token generation creates the token. The supplement lists three methods: a reversible cryptographic function with a strong key, a one-way function such as a hash with a strong secret salt, or an index, sequence number or random number that is not derived from the PAN.
- Token mapping links each token to its original value, so that an authorized request can retrieve one from the other.
- The card data vault is the central store for the original values and their tokens. The supplement calls it the most attractive target for attackers, because it holds the real data.
- Cryptographic key management covers the keys that encrypt the vault and any keys used to create or reverse tokens.
De-tokenization usually reverses the tokenization steps. The application sends a token, the tokenization system checks that the caller may make the request, and the system returns the original value if it finds one. The supplement adds that all access to the tokenization system should be logged and monitored, and that monitoring should detect unusual token-to-PAN requests.
Vault vs vaultless tokenization
The PCI SSC Tokenization Product Security Guidelines (version 1.0, April 2015) sort tokens into classes, and the two reversible classes match what vendors call vault and vaultless tokenization.
- Reversible non-cryptographic tokens are the vault model. The token has no mathematical link to the PAN, and the only way back is a lookup in the card data vault. For example, the system can assign each PAN a value from a pre-generated table of random numbers, so the only secret is the table that links them.
- Reversible cryptographic tokens are what vendors usually call vaultless tokenization. A keyed algorithm turns the PAN into the token and back again. The PAN itself is never stored, and only the key has to be kept secret.
- Irreversible tokens can never be turned back into the PAN. An authenticatable irreversible token comes from a one-way function and can confirm that a given PAN was used. A non-authenticatable irreversible token cannot be linked to any PAN at all, though a merchant can still link it to a customer account.
The 2015 guidelines also say that a product counts as reversible if de-tokenization is technically possible, even when the company producing the tokens does not plan to allow it.
| Vault (non-cryptographic) | Vaultless (cryptographic) | |
|---|---|---|
| How the token is made | Random or sequential value, stored next to the original | Keyed algorithm applied to the original value |
| What must be protected | The vault, which holds every original value | The key, which can reverse every token |
| How de-tokenization works | Lookup in the vault | Decryption with the key |
| Upkeep | The vault grows with each new value and must be backed up and kept in sync across sites | No growing table, but every token depends on the same key |
| PCI SSC view | Token has no mathematical relationship to the PAN | A token derived by encryption "is an encrypted PAN" and may face extra PCI DSS considerations (2011 supplement) |
The 2011 supplement also says that the council was still evaluating how encryption-based tokens affect PCI DSS scope.
Format-preserving tokens and FPE
Many tokens keep the format of the original value: the same length, the same character set and sometimes the same first and last digits. Legacy databases and forms then accept the token without schema changes. A vault can produce such tokens by drawing random digits, and a vaultless system usually produces them with format-preserving encryption (FPE).
NIST specifies FPE in Special Publication 800-38G, first released in March 2016. The publication explains the problem with a Social Security number: it has nine decimal digits, and ordinary encryption of that number can produce a longer value that no longer fits the field. FPE keeps the output in the same format, so an encrypted SSN is still nine digits.
The standard has changed since 2016. The second public draft of SP 800-38G Revision 1, published on February 3, 2025, removes the FF3 method after researchers found weaknesses in it and in its FF3-1 variant. The draft keeps only FF1 and requires that the set of possible input values (the domain) hold at least one million values. A four-digit field, with 10,000 possible values, falls below that limit. The revision was still a draft when NIST's page was checked on October 30, 2026, so check its status before you choose an algorithm.
Google Cloud's Sensitive Data Protection documentation says that FPE gives fewer security guarantees than other deterministic encryption methods, and it recommends AES-SIV unless the output must keep the original length.
Database tokenization in practice
Database tokenization usually means that values are tokenized before they reach the database or warehouse, and only selected users or services can see the real values. Cloud data platforms support this pattern in two ways:
- External tokenization. Snowflake's External Tokenization lets an account tokenize data before loading it. A masking policy then calls an external function at query time, and that function asks the tokenization provider for the real value when the user is allowed to see it. Snowflake requires Enterprise Edition or higher for this feature.
- Built-in transformations. Google Cloud Sensitive Data Protection offers three methods. Deterministic encryption with AES-SIV and format-preserving encryption can both be reversed with the original key. HMAC-SHA-256 hashing cannot be reversed. With the same key, each method turns the same input into the same output, so joins across tables still work.
A deterministic token lets analysts count customers and join tables without seeing any real ID. The AWS Security Blog's guide to tokenization notes that each attribute a token preserves, such as consistent mapping or sort order, can weaken the scheme. The same guide advises close monitoring of every de-tokenization request.
Tokenization and PCI DSS scope
PCI DSS applies to every system that stores, processes or transmits cardholder data, and to systems connected to them. A system that holds only tokens can sometimes be removed from that scope. The 2011 supplement states two principles. Tokenization does not remove the need to maintain and validate PCI DSS compliance, but it can reduce the number of system components the requirements apply to. And the merchant must confirm that no PAN can be retrieved from any component it removes from scope.
Under the supplement, these stay in scope:
- every part of the tokenization system, including de-tokenization and PAN storage;
- any system that can exchange a PAN for a token or a token for a PAN;
- any system with access to the tokenization system or its processes;
- any other system inside or connected to the cardholder data environment.
A system that stores only tokens may be out of scope when several conditions hold. Nobody can recover the PAN from the tokens, even after a breach of that system. The system is segmented from anything that can request de-tokenization or reach the vault or keys, and it holds no credentials for the cardholder data environment. It receives no card data through any other channel, and old card data on it has been securely deleted. If a merchant can retrieve the PAN, the supplement says the merchant's environment is in scope.
The current standard is PCI DSS v4.0.1. Its requirement 3.5.1 lists four accepted ways to make a stored PAN unreadable, according to a PCI SSC FAQ: encrypting the entire PAN, truncation, index tokens and one-way hashes. The same FAQ says that every hash generated after March 31, 2025 must be a keyed cryptographic hash. The 2011 and 2015 tokenization documents are guidance only. The council's FAQ states that no validation program is attached to the 2015 product guidelines.
Tokenization vs encryption vs masking
Each of the three methods leaves the data in a different state:
| Tokenization | Encryption | Masking | |
|---|---|---|---|
| What replaces the data | A substitute value with no use outside the token system | Ciphertext produced from the data with a key | A hidden or altered version, such as XXXX-1111 |
| Can the original come back? | Yes for reversible tokens, through de-tokenization; no for irreversible tokens | Yes, with the key | Dynamic masking: yes, the stored data is unchanged. Static masking: no |
| Where the secret lives | The vault, or the key for vaultless tokens | The key | The unmasked source data and its access rules |
| Format | Often kept | Changed, unless FPE is used | Usually kept |
| Typical use | Card numbers and IDs in applications, data warehouses and analytics | Data at rest and in transit, files, backups | Screens, support tools, test and development copies |
| Effect on PCI DSS scope | Can remove token-only systems from scope if the conditions are met | Systems that store encrypted PAN generally stay in scope, according to the AWS guide | Display masking is a separate requirement (3.4.1) and does not make stored PAN unreadable |
Tokenization vs encryption
Encryption transforms the data itself, so anyone with the key can read every record it protects. A vault token has no mathematical relationship to the original value, so stealing tokens gives an attacker nothing unless they also reach the vault. Vaultless tokenization is a form of encryption whose output looks like a token. The AWS guide says that encryption can suit large volumes of data, unstructured data and data that third parties must read in its original form.
Data masking vs tokenization
Masking has two forms. Dynamic masking hides values in query results while the stored data stays the same. Microsoft's documentation for SQL Server dynamic data masking says that the data in the database is not changed and that administrators can always see unmasked values. It also warns that users who can run ad hoc queries can infer masked values by guessing, for example with a range filter on a masked salary column. Static masking writes altered values into a copy of the data, usually for testing, and that copy cannot be restored to the original.
Tokenization changes the stored value and keeps a controlled path back to the original. Masking controls what a person sees, and it leaves the protection of the stored value to other controls.
Asset tokenization: a different meaning
In blockchain projects, tokenization means issuing a token on a ledger that represents a claim on an asset, such as a fund share, a bond, a property company stake or a piece of art. The token is the record of who owns the asset, or a claim on that record, and holders can see it and transfer it to each other.
| Data tokenization | Asset tokenization | |
|---|---|---|
| Goal | Hide a sensitive value from most systems | Record and transfer ownership of an asset |
| Where the token lives | Private databases and applications | A blockchain, often public |
| Value of a stolen token | None, by design | The token is the asset claim, so theft is a loss |
| Reversal | De-tokenization returns the original value | Redemption or transfer, governed by legal terms |
| Main rules | PCI DSS, privacy and data protection law | Securities, property and custody law |
In a statement of January 28, 2026, three SEC divisions said that a tokenized security is still a security and that its format or recordkeeping method does not change how federal securities laws apply. SEC staff described issuer-sponsored tokens and two third-party models, custodial and synthetic. Token standards for these assets, such as ERC-3643, check whether a wallet is allowed to hold a token before each transfer.
An asset tokenization platform can also use data tokenization. The platform holds investor names, ID documents and bank details for its identity checks, and personal data belongs off the chain. A data tokenization system can protect those records in the platform's own databases.
The site's tokenization guides cover the asset side in detail:
- Asset tokenization platform: the components, token standards, US and EU rules and build-or-buy costs.
- RWA tokenization by asset class: how offerings of funds, debt, real estate and other assets are structured.
- Real estate tokenization: property holding structures, law and platforms.
- Security token offering: how an STO is issued and sold.
Frequently asked questions
What is data tokenization?
Data tokenization replaces a sensitive value, such as a card number, with a substitute value called a token. Most systems then store only the token, and an authorized request to the tokenization system can exchange the token for the original value.
Is tokenization the same as encryption?
No. Encryption transforms the data with a key, so anyone with the key can read it. A vault token has no mathematical relationship to the original value, while vaultless tokenization uses a keyed algorithm and is a form of encryption.
Can tokenized data be reversed?
Reversible tokens can be turned back into the original value through de-tokenization, by a vault lookup or by decryption with a key. Irreversible tokens, made with a one-way function or not linked to the original value at all, cannot be reversed.
What is the difference between data masking and tokenization?
Tokenization replaces the stored value and keeps a controlled path back to the original. Dynamic masking hides values in query results while the stored data stays the same, and static masking writes altered values into a copy that cannot be restored.
Does tokenization take systems out of PCI DSS scope?
It can. A system that holds only tokens, cannot recover the card number and is segmented from the tokenization system and the cardholder data environment may be out of scope. The tokenization system itself always stays in scope.
Is data tokenization related to blockchain tokenization?
No. Data tokenization hides sensitive values inside private systems. Asset tokenization records ownership of an asset as a token on a blockchain, and a tokenized security is still a security under US law.