---
title: "Schema-validated output in LLMR"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{JSON output vs. schema-validated output in LLMR}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r}
knitr::opts_chunk$set(
  collapse = TRUE, comment = "#>",
  eval = identical(tolower(Sys.getenv("LLMR_RUN_VIGNETTES", "false")), "true") )
```  


## Overview

JSON mode asks the model for a JSON object; it is easy to request but gives
weak guarantees about shape. Schema output supplies a JSON Schema and requests
strict validation, which is more reliable when the provider enforces it.
Enforcement and request shapes differ across providers, so this vignette
relies on defensive parsing and local validation rather than trusting any one
provider's guarantee.

---

## What the major providers actually support

- **OpenAI-compatible (OpenAI, Groq, Together, x.ai, DeepSeek, Alibaba/Qwen, Zhipu, Moonshot, Xiaomi)**  
  Chat Completions accept a `response_format` (e.g., `{"type":"json_object"}` or a JSON-Schema payload). Enforcement varies by provider but the interface is OpenAI-shaped.  
  See [OpenAI Structured Outputs cookbook](https://cookbook.openai.com/examples/structured_outputs_intro), [Groq API (OpenAI-compatible)](https://console.groq.com/docs/structured-outputs), [Together: OpenAI compatibility](https://docs.together.ai/docs/json-mode), [x.ai: OpenAI API schema](https://docs.x.ai/docs/guides/structured-outputs), [DeepSeek: OpenAI-compatible endpoint](https://api-docs.deepseek.com/guides/json_mode)

- **Anthropic (Claude)**  
  No global "JSON mode." Instead, you **define a tool** with an **`input_schema`** (JSON Schema) and **force** it via `tool_choice`, so the model must return a JSON object that validates the schema.  
  See [Anthropic Messages API: tools & `input_schema`](https://platform.claude.com/docs/en/api/messages#tools)

- **Google Gemini (REST)**  
  Set `responseMimeType = "application/json"` in `generationConfig` to request JSON. Gemini 2.5+ models also accept **`responseJsonSchema`** (standard JSON Schema), which `enable_structured_output()` sends by default when you supply a schema; set `gemini_enable_response_schema = FALSE` in the config for an older model that rejects it.  
  See [Gemini documentation](https://ai.google.dev/gemini-api/docs/)
---

## Why prefer schema output?

- **Deterministic downstream code**: predictable keys/types enable typed transforms.  
- **Safer integrations**: strict mode avoids extra keys, missing fields, or textual preambles.  
- **Faster failure**: invalid generations fail early, where retry/backoff is easy to manage.

## Why JSON-only still matters

- **Broadest support** across models/providers/proxies.  
- **Low ceremony** for exploration, labeling, and quick prototypes.

---

## Quirks you will hit in practice

- Models often wrap JSON in **code fences** or add pre/post text.  
- Arrays/objects appear where you expected scalars; **ints vs doubles** vary by provider/sample.  
- **Safety/length caps** can truncate output; detect and handle "finish_reason = length/filter."  

### LLMR helpers for common parsing failures

- `llm_parse_structured()` strips fences and extracts the **largest balanced** `{...}` or `[...]` before parsing.  
- `llm_parse_structured_col()` hoists fields (supports dot/bracket paths and JSON Pointer) and keeps non-scalars as list-columns.  
- `llm_validate_structured_col()` validates locally via **jsonvalidate (AJV)**.  
- `enable_structured_output()` flips the right provider switch (OpenAI-compat `response_format`, Anthropic **tool + `input_schema`**, Gemini `responseMimeType` + `responseJsonSchema`).

---

## Minimal patterns (guarded code)

All chunks use a tiny helper so your document **knits even without API keys**.

```{r}
safe <- function(expr) tryCatch(expr, error = function(e) {message("ERROR: ", e$message); NULL})
```

### 1) JSON mode, no schema (works across OpenAI-compatible providers)

```{r}
safe({
  library(LLMR)
  cfg <- llm_config(
    provider = "deepseek",              
    model    = "deepseek-chat",
    temperature = 0
  )

  # Flip JSON mode on (OpenAI-compat shape)
  cfg_json <- enable_structured_output(cfg, schema = NULL)

  res    <- call_llm(cfg_json, 'Give me a JSON object {"ok": true, "n": 3}.')
  parsed <- llm_parse_structured(res)

  cat("Raw text:\n", as.character(res), "\n\n")
  str(parsed)
})
```

**What could still fail?** Proxies labeled "OpenAI-compatible" sometimes accept `response_format` but do not strictly enforce it; LLMR's parser recovers from fences or pre/post text.

---

### 2) **Schema-constrained output on Groq** (open model)

Groq serves `openai/gpt-oss-20b` with OpenAI-compatible APIs. Their
**Structured Outputs** feature enforces JSON Schema and (notably) expects
**all properties to be listed under `required`**.

```{r}
safe({
  library(LLMR); library(dplyr)

  # Schema: make every property required to satisfy Groq's stricter check
  schema <- list(
    type = "object",
    additionalProperties = FALSE,
    properties = list(
      title = list(type = "string"),
      year  = list(type = "integer"),
      tags  = list(type = "array", items = list(type = "string"))
    ),
    required = list("title","year","tags")
  )

  cfg <- llm_config(
    provider = "groq",
    model    = "llama-3.1-8b-instant",
    temperature = 0
  )
  cfg_strict <- enable_structured_output(cfg, schema = schema, strict = TRUE)

  df  <- tibble(x = c("BERT paper", "Vision Transformers"))
  out <- llm_fn_structured(
    df,
    prompt   = "Return JSON about '{x}' with fields title, year, tags.",
    .config  = cfg_strict,
    .schema  = schema,          # send schema to provider
    .fields  = c("title","year","tags"),
    .validate_local = TRUE
  )

  out |>
    select(structured_ok, structured_valid, title, year, tags) |>
    print(n = Inf)
})
```

If your key is set, you should see `structured_ok = TRUE`, `structured_valid = TRUE`, plus parsed columns.

**Common gotcha**: If Groq returns a 400 error complaining about `required`, ensure **all properties** are listed in the `required` array. Groq's structured output implementation is stricter than OpenAI's.

---

### 3) DeepSeek: JSON-object mode with local validation

```{r}
safe({
  library(LLMR)
  schema <- list(
    type="object",
    properties=list(answer=list(type="string"), confidence=list(type="number")),
    required=list("answer","confidence"),
    additionalProperties=FALSE
  )

  cfg <- llm_config("deepseek", "deepseek-chat")
  cfg <- enable_structured_output(cfg, schema = schema, name = "llmr_schema")

  res <- call_llm(cfg, c(
    system = "Return only the JSON object that matches the schema.",
    user   = "Answer: capital of Japan; include confidence in [0,1]."
  ))

  parsed <- llm_parse_structured(res)
  str(parsed)
})
```

---

### 4) Groq: another structured output example

```{r}
safe({
  library(LLMR)

  cfg <- llm_config(
    "groq", "openai/gpt-oss-20b"
  )

  schema <- list(
    type = "object",
    properties = list(name = list(type = "string"),
                      score = list(type = "number")),
    required = list("name", "score"),
    additionalProperties = FALSE
  )
  cfg_json <- enable_structured_output(cfg, schema = schema)

  res <- call_llm(cfg_json, c(
    system = "Reply as JSON only.",
    user   = "Produce fields name and score about 'MNIST'."
  ))
  str(llm_parse_structured(res))
})
```

---

## Defensive patterns (no API calls)

````{r}
safe({
  library(LLMR); library(tibble)

  messy <- c(
    '```json\n{"x": 1, "y": [1,2,3]}\n```',
    'Sure! Here is JSON: {"x":"1","y":"oops"} trailing words',
    '{"x":1, "y":[2,3,4]}'
  )

  tibble(response_text = messy) |>
    llm_parse_structured_col(
      fields = c(x = "x", y = "/y/0")   # dot/bracket or JSON Pointer
    ) |>
    print(n = Inf)
})
````

**Why this helps**
Works when outputs arrive fenced, with pre/post text, or when arrays sneak in. Non-scalars become list-columns (set `allow_list = FALSE` to force scalars only).

---

## Parallel execution with schema validation

For production ETL workflows, combine schema validation with parallelization:

```{r, eval=FALSE}
library(LLMR); library(dplyr)

cfg_with_schema = llm_config('deepseek','deepseek-chat')
  
setup_llm_parallel(workers = 10)

### Assuming there is a large data frame large_df

large_df |>
  llm_mutate_structured(
    result,
    prompt = "Extract: {text}",
    .config = cfg_with_schema,
    .schema = schema,
    .fields = c("label", "score"),
    tries = 3  # auto-retry failures
  )

reset_llm_parallel()
```

This processes thousands of rows efficiently with automatic retries and validation.

---

## Choosing the mode

* **Reporting / ETL / metrics:** Schema mode; fail fast and retry.
* **Exploration / ad-hoc:** JSON mode + recovery parser.
* **Lightweight field extraction:** Tag mode with `.tags`; useful when strict
  schema support is unavailable or unnecessarily heavy.
* **Cross-provider code:** Always wrap provider toggles with `enable_structured_output()` and run `llm_parse_structured()` + local validation.

---

## References

* OpenAI: Structured Outputs cookbook: [https://cookbook.openai.com/examples/structured_outputs_intro](https://cookbook.openai.com/examples/structured_outputs_intro)
* Groq: Structured Outputs: [https://console.groq.com/docs/structured-outputs](https://console.groq.com/docs/structured-outputs)
* Together: Structured Output: [https://docs.together.ai/docs/json-mode](https://docs.together.ai/docs/json-mode)
* x.ai: Structured Output: [https://docs.x.ai/docs/guides/structured-outputs](https://docs.x.ai/docs/guides/structured-outputs)
* DeepSeek: JSON Mode: [https://api-docs.deepseek.com/guides/json_mode](https://api-docs.deepseek.com/guides/json_mode)
* Anthropic: Messages API, tools & `input_schema`: [https://platform.claude.com/docs/en/api/messages#body-tool-choice](https://platform.claude.com/docs/en/api/messages#body-tool-choice)
* Google Gemini: Structured Output: [https://ai.google.dev/gemini-api/docs/structured-output](https://ai.google.dev/gemini-api/docs/structured-output)