TOON to JSON

1. What Is TOON Format?

TOON (Token-Oriented Object Notation) is a compact, deterministic, lossless representation of JSON. It was designed to reduce token usage inside LLM prompts and provide strict, self-validating structure that JSON does not offer.

JSON is familiar but verbose. In JSON:

• Every object repeats its keys.
• Strings require quotes.
• Structure relies heavily on punctuation.
• LLMs often hallucinate or corrupt brackets.

TOON fixes these issues using:

• Indentation-based objects
• Tabular arrays for uniform objects
• Explicit array length metadata
• Minimal quoting rules

Despite differences in form, TOON maintains full compatibility with the JSON data model.

2. How TOON Works (Engineer's Overview)

TOON has two core encoding strategies:

2.1 Indentation for Objects

Instead of braces, TOON uses indentation.

Example JSON:

{
  "user": { "id": 5, "name": "Ada" }
}

TOON:

user:
  id: 5
  name: Ada

This reduces tokens and improves readability.

2.2 Tabular Arrays for Uniform Objects

This is the main source of TOON's efficiency.

JSON:

[
  { "id": 1, "name": "Alice", "role": "admin" },
  { "id": 2, "name": "Bob",   "role": "user" }
]

TOON:

users[12]{id,name,role}:
  1,Alice,admin
  2,Bob,user

The [12] indicates the declared length, and {id,name,role} defines the schema.

2.3 Primitive Arrays

JSON:

["ana", "luis", "sam"]

TOON:

friends[3]: ana,luis,sam

2.4 Minimal Quoting

TOON avoids quotes unless needed, reducing tokens further.

3. JSON → TOON: How Conversion Works

A JSON to TOON converter reads JSON and transforms it into token-efficient TOON.

3.1 Conversion Steps

1. Parse JSON.

2. Detect uniform arrays.

3. Emit tabular arrays where possible.

4. Convert objects into indented form.

5. Apply TOON quoting rules.

6. Output deterministic TOON.

3.2 Example Conversion

JSON:

{
  "context": {
    "task": "Our favorite hikes",
    "location": "Boulder"
  },
  "friends": ["ana", "luis", "sam"],
  "hikes": [
    { "id": 1, "name": "Blue Lake", "km": 7.5, "sunny": true },
    { "id": 2, "name": "Overlook", "km": 9.2, "sunny": false }
  ]
}

TOON:

context:
  task: Our favorite hikes
  location: Boulder

friends[3]: ana,luis,sam

hikes[12]{id,name,km,sunny}:
  1,Blue Lake,7.5,true
  2,Overlook,9.2,false

3.3 When JSON → TOON Performs Best

• Large, uniform arrays
• Data tables
• Structured evaluation datasets
• Time-series logs
• Repetitive objects

3.4 When JSON → TOON Helps Less

• Deeply nested objects
• Non-uniform arrays
• Mixed data structures
• Very small payloads

4. Validating TOON Format

Before using TOON in LLM workflows, you should validate it. A TOON validator checks:

• row counts match [N]
• rows match declared field width
• indentation is consistent
• quoting rules are correct
• delimiters are consistent

This makes validate TOON format a critical part of pipelines.

4.1 Schema Embedded in the Format

TOON doesn't require separate schemas. The schema is declared inline:

items[4]{id,title,price}:

A TOON schema validator simply confirms:

• fields exist
• values match row width
• all rows follow the structure

4.2 Syntax Checking

A TOON syntax checker ensures:

• correct indentation
• valid minimal quoting
• legal delimiter characters
• valid header structure

5. TOON → JSON: How Decoding Works

Converting TOON back into JSON is deterministic because TOON contains explicit metadata.

5.1 Decoding Steps

1. Parse indentation into objects.

2. Parse primitive arrays with declared length.

3. Parse tabular arrays using schema {fields}.

4. Convert rows into JSON objects.

5. Produce proper JSON.

5.2 Example Conversion

TOON:

report:
  id: 52
  generatedAt: 2025-02-12

operators[3]: lee,jordan,mina

records[4]{id,status,latencyMs}:
  1,ok,14
  2,ok,16
  3,fail,220
  4,ok,13

JSON:

{
  "report": {
    "id": 52,
    "generatedAt": "2025-02-12"
  },
  "operators": ["lee", "jordan", "mina"],
  "records": [
    { "id": 1, "status": "ok", "latencyMs": 14 },
    { "id": 2, "status": "ok", "latencyMs": 16 },
    { "id": 3, "status": "fail", "latencyMs": 220 },
    { "id": 4, "status": "ok", "latencyMs": 13 }
  ]
}

5.3 Common Issues in TOON → JSON

• width mismatch
• wrong array length
• inconsistent delimiter use
• incorrect indentation
• missing rows

TOON validators catch these errors before decoding.

6. Practical Usage Patterns

6.1 Pipeline Example

1. JSON generated by backend

2. Convert JSON → TOON

3. Embed TOON in LLM prompt

4. LLM outputs TOON

5. Validate TOON

6. Convert TOON → JSON

7. Downstream tools consume JSON

6.2 Why LLMs Produce TOON More Reliably Than JSON

• strict length definitions
• strict field definitions
• fewer punctuation symbols
• fewer quoting requirements
• deterministic row structures

6.3 Why You Should Always Validate

LLMs often produce:

• truncated rows
• rows with missing fields
• inconsistent quoting
• slightly malformed indentation

TOON validators detect these immediately.

7. Full TOON Example (End-to-End)

export:
  version: 3
  source: monitor_alpha
  generatedAt: 2025-02-12

operators[3]: lee,jordan,mina

sensors[5]{id,type,value,timestamp,isCritical}:
  1,temp,21.4,2025-02-12T09:00Z,false
  2,pressure,13.8,2025-02-12T09:00Z,true
  3,temp,22.0,2025-02-12T09:10Z,false
  4,humidity,43,2025-02-12T09:10Z,false
  5,pressure,14.2,2025-02-12T09:10Z,true

This converts cleanly to JSON due to deterministic structure.