Quite interesting project done by Haoquan @Haoquan03. Raw pixel generation could be a useful proxy to assess LLM’s long-context reasoning capability, as generating a long sequence of pixels requires LLMs to be semantically and positionally aware of what they have already generated.
More broadly, this is also related to a series of work from SphereLab: SGP-Bench (sgp-bench.github.io) and SGP-Gen (spherelab.ai/SGP-Gen). PixelBench and these projects are all generating a textual format of visual content, except that pixels can be visualized without any external renderers. Like SVG codes, pixels are a natural verifiable format for images. This can be a useful source of data for improving LLMs.
More broadly, this is also related to a series of work from SphereLab: SGP-Bench (sgp-bench.github.io) and SGP-Gen (spherelab.ai/SGP-Gen). PixelBench and these projects are all generating a textual format of visual content, except that pixels can be visualized without any external renderers. Like SVG codes, pixels are a natural verifiable format for images. This can be a useful source of data for improving LLMs.
The era of evaluating multimodal models with QA is over.
Use pixels instead. (0/7)
QA only tests high-level text reasoning. SVG and CAD rely on symbolic shapes rendered by external engines. Neither tells us if a model can directly control or read raw pixels.
We introduce PixelBench: a direct text-to-pixel probe where LLMs and VLMs generate and understand images pixel by pixel.View quoted post on X
