Black-Box Watermarking

Distortion-free watermarking via best-of-\(m\) rejection sampling. The model’s sampling distribution is completely unmodified — the watermark is embedded by selecting the highest-scoring candidate from \(m\) independent generations.

Example code: 📓 View On GitHub

Author: Bahri & Wieting

Theory

The black-box watermark generates \(m\) candidate sequences from the unmodified model and selects the one with the highest score under a keyed pseudorandom function (PRF).

For each candidate sequence, the score is computed by:

1. Extracting all n-grams from the sequence

2. Hashing each n-gram with the secret key to produce a uniform PRF value \(u_i \in [0, 1]\)

3. Computing the Irwin-Hall CDF of the sum \(\sum_i u_i\)

The candidate with the highest score is returned. Under no watermark, scores are uniform on \([0, 1]\). With \(m\) candidates, the expected maximum score is \(\approx m/(m+1)\).

Detection recomputes the same PRF score on the text. The p-value is \(1 - \text{score}\).

The key property is zero distortion: since all \(m\) candidates are drawn from the original model distribution, the selected output is also a valid sample from the original distribution. The trade-off is computational cost — generation is \(m\) times more expensive.

Note

Detection power depends on n_candidates: \(m=16\) gives weak detection (score \(\approx 0.94\)), \(m=128\) gives strong detection (score \(\approx 0.992\)), \(m=256\) gives very strong detection (score \(\approx 0.996\)).

Paper reference

Bahri, D. & Wieting, J. (2026). A Watermark for Black-Box Language Models. TMLR (arXiv:2410.02099). https://arxiv.org/abs/2410.02099

Example code

import os
os.environ["VLLM_USE_V1"] = "1"
os.environ["VLLM_ENABLE_V1_MULTIPROCESSING"] = "0"

from vllm import LLM, SamplingParams
from vllm_watermark.core import (
    DetectionAlgorithm,
    WatermarkedLLMs,
    WatermarkingAlgorithm,
)
from vllm_watermark.watermark_detectors import WatermarkDetectors

llm = LLM(model="meta-llama/Llama-3.2-1B", enforce_eager=True, max_model_len=1024)

# n_candidates controls detection power vs. generation cost
wm_llm = WatermarkedLLMs.create(
    model=llm,
    algo=WatermarkingAlgorithm.BLACKBOX,
    hash_key=15485863,
    ngram=4,
    n_candidates=128,
)

detector = WatermarkDetectors.create(
    algo=DetectionAlgorithm.BLACKBOX,
    model=llm,
    ngram=4,
    hash_key=15485863,
    threshold=0.02,
)

prompts = ["Write about machine learning applications"]
sampling_params = SamplingParams(temperature=1.0, top_p=0.95, max_tokens=128)
outputs = wm_llm.generate(prompts, sampling_params)

for output in outputs:
    generated_text = output.outputs[0].text
    result = detector.detect(generated_text)
    print(f"Watermarked: {result['is_watermarked']}, P-value: {result['pvalue']:.6f}")