Introducing FigmaTrace: A comprehensive training dataset for Figma design workflows

Today, we introduce FigmaTrace, a novel dataset containing a total of 3,469 Figma design trajectories, capturing 200+ hours of human design work spanning 10 creative skills that generalize to several other domains. Our design nuanced tasks target both close-ended, such as accessibility fixes, design templating fixes, etc, and open-ended use-cases such as prototype-wiring, sketch to figma and platform adaptation. 

Vision Language Models have made significant progress on objective, verifiable tasks like document understanding, robotics, and computer use, yet they continue to underperform on subjective and creative design work, where nuances like design taste and figurative meaning matter. A major contributor to this gap is the lack of high quality human workflow data that captures the diverse preferences and decisions that make human experts good at design. Prior efforts worked backwards from finished Figma designs, converted existing HTML pages into Figma compatible JSONs, or generated reusable component JSONs, but these approaches focus on the final artifact rather than the sequence of decisions that produced it, and validating them is difficult and ambiguous. 

‍

With this dataset, our study covers three research questions:

1. Does training with realistic human captured design workflows teach VLMs to be better at agentic navigation and design? 
2. Does converting video data into trajectories benefit more from design-phase based trajectory curation as opposed to maximum context-length sharding for very long horizon tasks? 
3. What patterns in FigmaTrace influence qualitative performance improvements in models? 

What we found

On RQ1, training on FigmaTrace lets much smaller open models match frontier closed models like Claude Opus 5 and GPT-5.6-Sol on completely out of domain non-design tasks. As the figure shows, the fine-tuned Qwen3.8-27B outperforms Claude Opus 5 on GUI-Odyssey and AndroidControl by 6.4 and 11.8 absolute points, with gains of up to 46 absolute points on AndroidControl over the corresponding baselines and a 7.4 point gain on the in-domain ScreenSpot-Pro Creative split, showing the learned skills generalize to broader agentic and design tasks. On RQ2, our phase-based trajectory curation beats maximum context-length sharding by 7.3 absolute points on average (63.6 vs 56.3), because segmenting by design phase teaches skill-based grounding instead of cutting trajectories at inconsistent intervals where task intent becomes unclear. On RQ3, manual inspection traced the gains to three patterns: element selection accuracy (base median error of roughly 457 px reduced to within roughly 15 px), coordinate understanding (grid overflow errors on 10 of 150 GUI-Odyssey items eliminated entirely), and decisiveness (the fine-tuned model always answers where the base often emits no coordinates at all).

Key Takeaways

‍FigmaTrace demonstrates that realistic, intent-segmented expert workflow data can teach models both Figma design skills and agentic navigation abilities that transfer well beyond the domain they were captured in, and that how you convert long recordings into trajectories matters as much as the data itself. We are open sourcing the dataset, with 2883 training and 586 evaluation trajectories, along with our best performing Qwen3.8-27B model on Hugging Face for the community to build on.

Paper: https://cdn.patronus.ai/FigmaTrace.pdf

Dataset: https://huggingface.co/datasets/PatronusAI/figmatrace