ICVSS 2026 · Sicily, Italy

From Molmo
to Muse-Spark

A data perspective on building frontier
multimodal models
Ani Kembhavi
Meta Superintelligence Labs
Human detection Human–object interaction Scene labels
Recognition
People · Objects · Places · Locations · Actions
Shape matching via low-distortion correspondence SVM / scikit-learn
Human–object interaction data Human–object interaction data
Designing features & picking ML methods
Assembling a house of cards
Berg et al. — Shape Matching and Object Recognition using Low Distortion Correspondence
Gupta et al. — Observing Human-Object Interactions: Using Spatial and Functional Compatibility for Recognition
https://scikit-learn.org/stable/modules/svm.html
Caltech-101 MIT Indoor LabelMe
PASCAL VOC
Graduating to large datasets
Caltech-101, MIT-Indoor, LabelMe, PascalVOC
Data grew along many dimensions
Scale
1,000s of images Billions
Modalities
Single modalities Truly multimodal data
Complexity
Labels Reasoning Traces
Diversity
10s of classes Massive multilingual vocabularies
Reality
Real data & human annotation Synthetic data & annotations
Fast forward to todayModels can perceive images in great detail
Open-ended perception — natural images, charts, documents, the physical world — in natural language.
They can act in the real world and create new virtual ones
Native tool use during reasoning.
App generation from a prompt.
A data perspective on building frontier multimodal models
What matters and what doesn't.
Today's Roadmap
1
Training Multimodal Models
2
Molmo & PixMo
3
Muse Spark
4
Generation Models
5
The BitterBetter Data Lessons
01
Training Multimodal Models
LLM Training
1 Pre-training Next-token prediction over a vast web-scale corpus Determines the model's raw knowledge 2 Mid-training Upweight math, code & long context; refresh the mix Strengthen the reasoning prior 3 Annealing Decay the learning rate on a small, high-quality mix Polishing the base model 4 SFT Supervised fine-tuning on curated demonstrations From an LLM to an assistant: Tool use, multi-turn dialogue, refusal, persona 5 RL from preferences (RLHF) Train on human preference comparisons from verifiable rewards (RLVR) For math, code, and reasoning. Drives "reasoning model" progress.
Quantity shrinks, quality grows
Pre-training01
Quantity
Billions
Quality
Noisy web
Mid-training02
Quantity
M – B
Quality
Curated
Annealing03
Quantity
Millions
Quality
High-quality mix
SFT04
Quantity
100K – 10M
Quality
Human-verified
RL / RLHF05
Quantity
10K – 500K
Quality
Preference-judged
Models that
accept text + media &
produce text.
From a Language Model to a Multimodal Model
input tokens LLM Decoder output tokens Connector Vision Encoder Image Text Tokenizer “How many cats .. ?”
Connector architectures
  1. Linear / MLP (LLaVA)
    Simplest; remarkably effective.
  2. Cross-attention (Flamingo, Llama 3.2V)
    Adds cross-attention layers into the LLM.
  3. Q-Former (BLIP-2)
    Learned queries compress visual features to 32 tokens.
  4. Pixel Shuffle (InternVL)
    Spatially merge adjacent patches (4× fewer tokens).
Training Recipe for VLMs
  1. Start with a trained LLM and a trained Vision Encoder.
  2. Pre-train the VLM to teach it multimodal alignment.
  3. Post-train on downstream tasks.
Your data budget
is an architecture decision in disguise
  • More data → un-freeze moreLearn alignments that are more flexible for downstream tasks like VQA.
  • Less data → freeze moreDepend heavily on a well-aligned vision encoder.
  • MiniGPT-4 trains just 1 linear layerNeeds thousands of pairs.
  • Qwen2.5-VL trains everythingNeeds trillions of tokens.
  • Most labs lie in betweenBillions of data points.
Training the visual encoder
Use billions of pairs of images and ALT text captions on webpages
CLIP — image & text encoders, contrastive matrix
CLIP
Aligning image and text embeddings via a contrastive objective
SigLIP — pairwise sigmoid loss, large batches
SigLip
Scaling up via pairwise sigmoid loss
Radford et al. — Learning Transferable Visual Models From Natural Language Supervision
Zhai et al. — Sigmoid Loss for Language Image Pre-Training
MetaCLIP — 500K-concept vocabulary, sub-string matching, balancing
How you select data
matters more than
how much you have
Web data is notoriously noisy. Removing poor ALT text is a must.Concept diversity and balancing are key to learning head and tail concepts.MetaCLIP:
Start with a vocabulary of 500K,
retrieve image–text pairs via sub-string matching,
then balance.
Xu et al. — Demystifying CLIP Data
Multimodal Pre-Training
Next token prediction for multimodal data
  • Image–Text pairs (alt text, OCR text)Curated, higher quality than raw encoder pre-training data.
  • Image-Caption pairsTeach the model to recognise and describe visual content
  • Interleaved image–text documentsCrucial for in-context learning.
  • Text-only data mixed in (10–50%)Prevents degradation of the LLM's language ability.
Many Roads to Instruction Data
Visual
content
Human instructions & answers
Instruction
data
Food web diagram
Question
“What will lead to an increase in the population of deer?”
Human instruction data remains the gold standard
— but has challenges:
  • Scale
  • Expertise
  • Multilinguality
  • Calibration
Many Roads to Instruction Data
Visual content Page text & metadata alt-text, captions, surrounding HTML LLM reformats Instruction data Model-generated metadata captioner, grounder Human annotations people label & describe
Many Roads to Instruction Data
Visual content Human queries AI answer Calibration Instruction data Humans correct
Human + AI-in-the-loop.  Get instruction data that is just hard enough.
Many Roads to Instruction Data
Synthetic data generation via code / procedures.
LLM writes code LLM generates QA Instruction data Render visual content
Reinforcement Learning
Optimize for what we actually want
Image + Prompt task input VLM Policy model being trained Responses sampled per prompt Reward score each response RL update PPO / GRPO update the policy Reward signal comes from… Human preferences A vs B comparisons → RLHF reward model Absolute scores rubric / guideline ratings → regression reward model Verifiable reward checkers & simulators → no reward model
Reinforcement Learning
From RLHF to GRPO
  • RLHF aligned LLMs to human intentA reward model trained on human preferences, then PPO — the recipe behind ChatGPT. (Ouyang et al., 2022)
  • DPO dropped the reward modelOptimize the preference objective directly; simple and widely used for VLM alignment. (Rafailov et al., 2023)
  • GRPO is now the popular defaultGroup-relative baseline, no critic network — cheaper than PPO, more stable than DPO. (Shao et al. / DeepSeek, 2024)
  • Adopted across frontier models
Reinforcement Learning
Data for RL
Preference data
Set up annotators to mark A vs B preferences at large scale → train a reward model, or run DPO directly.
Absolute scores
Human annotators assign scores against guidelines (e.g. 1–5 rubrics) → a regression reward model. Calibration across raters is the challenge.
Automatic reward
Build simulators & checkers that score answers for free — no annotators in the loop.
  • Grounding models
  • Counting (exact match)
  • Math & Geometry verifiers
  • OCR string match
  • Code-rendered figures
Frontier models require frontier data. But frontier data doesn't always require deep pockets.
What it really takes: quality, design, and thinking outside the box.
02
Deep Dive: Molmo & PixMo
Molmo benchmarks Molmo benchmark scores
Image–Text Paired Data
Llama 3.1V 6 Billion Image–Text pairs PixMo 700k Image–Text pairs
Quality & Detail over Scale
Image–Text Paired Data on the Web
Web image — Japan, pink, aesthetic
“pink, japan, aesthetic”
Web image — winter
“love this winter picture by person”
Quality & Detail over Scale
Image–Text Data in PixMo
Organized work desk with a dual-monitor setup

This photograph captures a well-organized work desk set prominently in the middle of the frame. The desk is large and rectangular, made from a polished, rich wood that spans horizontally across the image. Its structure is supported by four distinctive A-shaped legs, adding an elegant touch. On the desk, a striking dual-monitor setup is noticeable: a tall, vertical screen placed behind and to the right of a wider, horizontal computer monitor.

To the right of these monitors, a black mouse rests on a mouse pad. Scattered around the mouse pad, some white papers or letters are strewn across the far right side. On the left side of the desk, a black desk lamp with an extended arm hangs down, illuminating the workspace. Nearby, a stack of books is neatly placed in the upper left corner of the table.

The background wall is painted a subtle beige-white, complementing the refined ambiance of the space. The floor below the desk features elegant pinkish marble tiles, enhancing the room’s sophisticated look. To the far right of the image, a large window or patio door allows natural light to pour in, with clear glass that offers a glimpse into the outside area. This exterior view includes part of a rustic brick wall and a metal pail, hinting at an adjacent patio.

Quality & Detail over Scale
Image–Text Data in PixMo
Tablet home screen screenshot

This image features a screenshot taken from a tablet device. At the top-right corner, the time is displayed as 11:48, alongside a battery icon indicating 67% power remaining. The device also shows a telephony signal strength of four out of five bars and roughly three-quarters Wi-Fi connectivity. Additionally, a clock alarm icon is present, as well as an icon resembling a white rectangle with a blue letter "N" in it, whose specific function is unclear.

The main portion of the screen, approximately 80% of it, is a solid medium blue color devoid of any content. At the bottom of the screen, there are several folders and icons representing various apps and functionalities:

1. The first folder contains three icons:

  • An icon likely for contacts.
  • An icon probably for telephone.
  • An icon that seems to represent a text messaging app.

2. The second folder houses four apps:

  • A Gmail app icon indicated by a red "M" on a white background.
  • A YouTube app icon characterized by a red play button.
  • A map app icon depicting a map.
  • An unidentified app icon represented by a blue folder with its top corner bent down on the right.

3. The third icon is a white oval with six dots, arranged in two rows of three, likely representing an app drawer or menu.

4. The fourth icon resembles a red, yellow, and green shutter with a blue dot in the middle, suggesting it might be for a camera or photo viewing app.

5. Next to it, there is a silver colored camera icon with a black lens and a blue spot in the center, hinting at a camera application.

At the very bottom of the screen, there are three navigational icons:

  • A left arrow triangle in white at the bottom-left corner.
  • A white oval in the center, indicative of a home button.
  • A white rectangle on the bottom-right corner, likely for accessing recent apps or multitasking.

Overall, this image captures the home screen of a tablet, providing an overview of available functionalities and connectivity status.

Meet people where they're strong
Humans don't like to type
as much as they like to speak
→ → →
PixMo asks annotators to speak for 60 to 90 seconds about an image.
Use LLMs to clean and convert to the desired instruction data format.
Meet people where they're strong
It's often better to let people pick what they are interested in.
Matching annotators to interests results in better data.
e.g. People who love gardening are really great at describing backyards/plants/flowers and introduce fewer incorrect details like species names, which results in fewer model hallucinations.
Human + AI can be more powerful than Human-only
Use LLMs with access to world knowledge to answer hard questions.
Ask Model Anything
PixMo-AskModelAnything
PixMo Ask Model Anything example — city bus

What is the white number on this bus?

The white number on the bus is 3260.

How many tires are visible on this bus?

There are three tires visible on this bus. Two of them are likely the front tires, which can be seen from this angle. Additionally, one rear tire is also visible, possibly on the side of the bus that's facing the camera. The other rear tires are likely obscured by the bus body or out of frame in this particular view. City buses typically have multiple tires, but only three are clearly visible from the perspective shown in this image.

What number is written on the bus on the yellow part above the front bumper?

The number written on the yellow part of the bus above the front bumper is 3260.

Where is this bus going?

The bus is heading to Highland Park. This is clearly visible on the destination sign at the front of the bus, which displays "71B HIGHLAND PARK".

Where is this bus stationed out of?

The bus is stationed out of Highland Park station in Pittsburgh. While the bus operates in the Pittsburgh area, as indicated by the reference to "The Pitt News" (the student newspaper of the University of Pittsburgh), it specifically originates from or is based at the Highland Park station. Highland Park is a neighborhood in the northeastern part of Pittsburgh, known for its large park of the same name and residential areas.

How many tires can we see on this bus?

In this image of a bus, we can see three tires. These are likely the front two tires and one of the rear tires visible from this particular angle. While buses typically have more tires, only three are visible in this specific view of the vehicle.

How many bags is the woman on the left holding?

The woman on the left is holding two bags. They appear to be purses or handbags that she's carrying as she walks along the sidewalk next to the bus.

Synthetic Data Generation
Topic
→
LLM
→
Generated code
→
Rendered figure
↓
LLM
→
QA
Synthetic Data Generation
PixMo-Docs
PixMo-Docs chart example

What is the most popular programming language in Tech Startups?

Python

Which industry sector prefers Ruby the most?

Tech Startups

Which programming languages have a similar popularity in Financial Services?

Java and C#

Which programming language has the highest variation in popularity across industry sectors?

Ruby

Think Outside The Box
PixMo-Points example
No more boxes!
No more categories!
Think Outside The Box
PixMo-Points example with dots
Open mouth
Guitar Pick
Electric guitar
Interstate sticker
Leather wristband
Arm band tattoo
Drum
Purple stage
Background
Engage with your annotators —
they're not a "resource," they're your community.
Molmo
Molmo caption example — Caribbean weather map

The image shows a detailed weather map of the Caribbean region, focusing on Tropical Storm Helene. The map displays a 1-day forecast precipitation with a color gradient ranging from light purple to dark purple, indicating varying levels of rainfall intensity.

Several cities are visible on the map:

Belize City, Belize
Belmopan, Belize
Guatemala City, Guatemala
San Pedro Sula, Honduras
Tegucigalpa, Honduras
Managua, Nicaragua
Puerto Cabezas, Nicaragua
Puerto Plata, Dominican Republic
Santo Domingo, Dominican Republic
Port-au-Prince, Haiti
Kingston, Jamaica
Nassau, Bahamas
Port Lucie, Florida (USA)
Miami, Florida (USA)
Havana, Cuba
Camaguey, Cuba
Holguín, Cuba
Guantanamo, Cuba
Port of Spain, Trinidad and Tobago

The map provides a comprehensive view of the potential impact of Tropical Storm Helene across multiple countries and islands in the Caribbean, as well as parts of the southeastern United States.

Molmo
Molmo QA example — album covers

What kind of music is this?

This collection appears to be primarily alternative and indie rock music. The album covers visible in the image showcase a variety of artists and bands that are typically associated with these genres. Some of the artists I can identify include Panda Bear, Tim Hecker, and Bad Luck, who are known for their experimental and ambient work within the indie and alternative music scenes.

The diverse range of artists and the visual style of the album covers suggest a focus on independent and alternative music, which often includes subgenres like indie rock, post-rock, experimental, and lo-fi. This collection likely represents a curated selection of music that appeals to fans of innovative and non-mainstream rock-based artists.

Molmo
Molmo count-by-pointing example — boats

Count the boats

boats

Counting the boats shows a total of 35.

Visual Chain of Thought via pointing improves performance on counting benchmarks.
Molmo + SAM
Molmo + SAM
These principles when scaled up produce remarkable models.
03
Muse Spark
Meta Superintelligence Labs · April 2026
A natively multimodal reasoning model
  • Muse Spark is built from the ground up to integrate visual information.Not bolted on post-training.
  • Visual chain of thought, native tool use, multi-agent orchestration
Muse Spark multimodal benchmark numbers
Domain specific training data
Health & Fitness
Scaling Axis 1
Pretraining efficiency
  • New pretraining stackArchitecture and optimization improvements
  • Detailed data curationAttention to data quality and diversity
  • >10× compute efficiency vs Llama 4 MaverickBetter data can substitute for raw compute at this scale
Scaling Axis 2
Reinforcement learning that scales
  • Large-scale RL after pretrainingLog-linear growth in pass@1 and pass@16 on training benchmarks.
  • Reliability without losing diversityThe model gets more reliable without collapsing its solution space.
  • Gains generalize to held-out evalsRL is "notoriously unstable" — Muse Spark shows stable, predictable RL scaling at frontier scale.
Muse Spark RL scaling
Scaling Axis 3 · Test-Time Reasoning
Maximize correctness subject to thinking time penalty
Muse Spark tool use
Muse Spark is agentic
You can ask it to leverage test-compute scaling methods to improve quality.
Birds — input image
Count how many birds are here, crop into 16 tiles, and count for each with parallel subagents, finally assemble the results and make a combined overlay visualization in python.
Birds — agentic overlay output
Muse Spark can convert Images To Code
Weather widget reference
Make this into a weather app in HTML that is interactive.
Muse Spark can convert Images To Code
04
Building Generation Models
Generated Instagram examples
Perceive and produce everything
AI Model Text Image Video Audio Text Image Video Audio
Unified-IO
Unified-IO: Lu et al. ICLR 2023
Unified-IO 2: Lu et al. CVPR 2024
Unified-IO model
Unified-IO pre-training data
Pre-training Data
Unified-IO post-training data
Post-training Data
Unified-IO examples
Merging Next Token Prediction with Diffusion
Transfusion
Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model
Zhou et al. — ICLR 2025
However in practice, Unified models are challenging to train at scale
Modalities compete with each other for capacity
Still hard to say if generation benefits perception
Yet Another Tool
Conversation User VLM Search Code Interpreter File System Calendar Scratchpad Media Generation
Diffusion based media generation
Progressively add noise until the image is pure noise
  • Forward adds noise over T steps; a neural network learns to denoise.Loss = MSE on the noise.
  • Latent DiffusionRun diffusion in a VAE-compressed space → Much cheaper. Add conditioning via cross-attention.
  • DistillationDistill the model to obtain a few step, efficient inference process.
How do you source training data?
A combination of Human Annotation and AI models (including Perception models)
Tropical sunset
DALL·E 3 showed very large improvements using this captioning trick

A photorealistic tropical sunset over a calm harbor inlet, golden hour. The sky is saturated warm amber, gold, and orange, with soft peach and pale blue breaking through the clouds in the upper left. Low, scattered clouds catch golden rim light, with a thick band of glowing clouds across the top third. The sun is low on the horizon, just right of center, a brilliant white-gold disc partially obscured by low clouds, casting intense golden light and a shimmering reflection across the water.

In the midground, right side, dark silhouetted rolling hills/mountains rise from the ocean horizon, layered with distant hazy islands to the left. The ocean beyond is calm with a soft glitter.

In the left-midground, three tall coconut palm trees in sharp black silhouette stand on a small vegetated point. Leftmost palm is tallest and most slender with a full fanned crown, the middle palm is slightly shorter and fuller, the right palm is similar height with distinct fronds. Below them is dense dark tropical shrubbery.

A long low rock breakwater/jetty runs horizontally across the middle of the image, separating the foreground calm inlet from the open sea behind. The rocks are dark gray in shadow. On the left side of the jetty, near the palms, a small dark blue boat is moored, barely visible in silhouette.

Foreground: a calm sheltered waterway filling the lower half of the frame, with gentle small ripples. The entire right half of the water is a brilliant mirror of liquid gold, reflecting the sunset sun, with warm orange-gold ripples fading to darker amber-brown on the left where the palm trees cast soft reflections. The water surface has fine horizontal ripples catching the light.

Color palette: intense golden yellow, burnt orange, amber, peach, warm coral, with silhouetted near-blacks for the land, and pale cool sky blue in the upper left cloud break.

Lighting: strong backlight, sun low and warm, everything in the foreground in silhouette, high dynamic range sunset glow, soft atmospheric haze.

Mood: tranquil, warm, serene tropical evening.

Composition: eye-level, wide landscape, horizon at the middle third, palm trees anchoring the left third, sun and mountain silhouette on the right, golden water reflection leading diagonally from right foreground to center horizon. Sharp focus throughout, natural photography, no people, no text, no filters.

↓
Tropical sunset harbor, photorealistic. Golden-orange sky with sun low right of center behind clouds, palm tree trio silhouetted on the left, distant dark mountains on the right horizon, calm water with a bright gold reflection in the foreground, rock breakwater mid-frame. Warm, serene, backlit, 16:9 landscape photo, no people.
↓
Generate an image of a beautiful sunset
LLM
Learns to expand user prompts and invoke the Media Generation tool
How do you source training data?
A combination of Human Annotation and AI models (including Perception models)
Tropical sunset
Perception Models
Sunset with detection boxes Sunset depth map
LLM
Generate hard prompts using perception metadata
What about data for media editing?
A combination of Human Annotation and AI models (including Perception and Generation models)
Learn To Holefill
Trees holefilled
Mask Objects and Holefill
Object removed
Training Data
Object removed
Add/Delete
Two dogs on a couch
Popularized by several papers including:
Paint by Inpaint: Learning to Add Image Objects by Removing Them First — Wasserman et al. — CVPR 2025
Media Generator Training
1 Pre-training Billions of multimodal points, 256px. Diversity is crucial Learn broad concepts 2 Mid-training Curated subsets to upweight domains of interest 512–1024px 3 Annealing Highly selective, heavily recaptioned. Teach Aesthetics / Style 4 SFT Supervised fine-tuning on curated demonstrations From an image generator to a multimodal assistant 5 RL from preferences (RLHF) Train on human preference comparisons from verifiable rewards (RLVR) Underexplored area. Open research questions.
Curation challenges
  • Perception models introduce noiseCleanup via humans is hard to scale
  • Mid-training/SFT sets are hard to sourceAesthetic scoring is challenging
  • RL is hard for generationVerifiable rewards are an open question
  • Video data gets much harderPerception models are just catching up
  • Everything is harder for videosDuplicate detection, motion quality, compression, scene cuts and more!
05
The Eight BitterBetter Data Lessons
Lesson 1
Quality beats Quantity
High quality data and annotations serve as a data and compute multiplier.
Lesson 2
Spend time on annotation design
Molmo's speech-based captioning hugely improved data quality.
Count-by-pointing turned annotation into chain-of-thought.
Lesson 3
Use AI models to push your data frontier
Human annotations are best used in conjunction with SOTA models.
Lesson 4
Synthetic data can shine
Particularly when human data is hard/impossible to obtain and scale.
But take care to ensure it's not OOD.
Lesson 5
Mixing and curriculum matter
One can often spend as much effort gathering SFT data as curating pre-training data in spite of it being orders of magnitude smaller.
Lesson 6
Architecture decisions are data decisions
Carefully consider the implications of your architecture and training choices on your data choices and vice versa.
Lesson 7
Filtering and curation are evolving arts
Seemingly simple tasks like filtering and deduplication can have outsized impacts on model performance.
Lesson 8
You cannot run away from
annotator dynamics
Treat your annotator pool with respect. Pay them well, make their work interesting and they will give you back returns on your investment.

Thank you

Ani Kembhavi
Meta Superintelligence Labs
ICVSS 2026 · Sicily, Italy