Skip to content

Splits a JSON string into an annotated list of tokens

License

Notifications You must be signed in to change notification settings

queckezz/json-tokenize

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

24 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

json-tokenize Build status NPM version Dependency Status License Js Standard Style

Splits a JSON string into an annotated list of tokens.

  • Keeps track of a tokens position
  • Whitespace significant
  • Doesn't validate the correctness of the syntax

Screenshot - formatted tokens outputted to stdout

View Example

Installation

npm install --save json-tokenize

Or even better

yarn add json-tokenize

Import and Usage Example

const tokenize = require('json-tokenize')

const obj = { "test": 1.0 }

tokenize(JSON.stringify(obj))
// ->
[
  { type: 'punctuation',
    position: { lineno: 1, column: 1 },
    raw: '{',
    value: '{' },
  { type: 'whitespace',
    position: { start: [Object], end: [Object] },
    raw: '\n  ',
    value: '\n  ' },
  { type: 'string',
    position: { start: [Object], end: [Object] },
    raw: '"test"',
    value: 'test' },
  { type: 'punctuation',
    position: { lineno: 2, column: 9 },
    raw: ':',
    value: ':' },
  { type: 'whitespace',
    position: { lineno: 2, column: 10 },
    raw: ' ',
    value: ' ' },
  { type: 'number',
    position: { lineno: 2, column: 11 },
    raw: '1',
    value: 1 },
  { type: 'whitespace',
    position: { start: [Object], end: [Object] },
    raw: '\n',
    value: '\n' },
  { type: 'punctuation',
    position: { lineno: 3, column: 1 },
    raw: '}',
    value: '}' }
]

Tokens

  • whitespace - Allowed white space between the actual relevant tokens.
  • punctuator - The characters sorrounding your data: {, }, [, ], : and ,
  • string - JSON String
  • number - JSON Number
  • literal - true, false or null

Author

json-tokenize © Fabian Eichenberger, Released under the MIT License.
Authored and maintained by Fabian Eichenberger with help from contributors (list).

GitHub @queckezz · Twitter @queckezz