content type paperpublished August 2025
Misty: UI Prototyping Through Interactive Conceptual Blending
AuthorsYuwen Lu, Alan Leung, Amanda Swearngin, Jeffrey Nichols, Titus Barik
UI prototyping often involves iterating and blending elements from examples such as screenshots and sketches, but current tools offer limited support for incorporating these examples. Inspired by the cognitive process of conceptual blending, we introduce a novel UI workflow that allows developers to rapidly incorporate diverse aspects from design examples into work-in-progress UIs. We prototyped this workflow as Misty. Through an exploratory first-use study with 14 frontend developers, we assessed Misty’s effectiveness and gathered feedback on this workflow. Our findings suggest that Misty’s conceptual blending workflow helps developers kickstart creative explorations, flexibly specify intent in different stages of prototyping, and inspires developers through serendipitous UI blends. Misty demonstrates the potential for tools that blur the boundaries between developers and designers.
Related readings and updates.
SQUIRE: Interactive UI Authoring via Slot QUery Intermediate REpresentations
April 6, 2026research area Human-Computer Interactionconference UIST
Frontend developers create UI prototypes to evaluate alternatives, which is a time-consuming process of repeated iteration and refinement. Generative AI code assistants enable rapid prototyping simply by prompting through a chat interface rather than writing code. However, while this interaction gives developers flexibility since they can write any prompt they wish, it makes it challenging to control what is generated. First, natural language on…
ILuvUI: Instruction-Tuned Language-Vision Modeling of UIs from Machine Conversations
July 14, 2025research area Human-Computer Interactionconference IUI
Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused understanding of images and language, but many perform poorly on UI tasks due to the lack of UI training data. In this paper, we adapt a recipe for generating paired text-image training data for VLMs to the UI domain by combining existing pixel-based methods with a Large Language Model (LLM). Unlike prior art, our method requires no human-provided annotations,…
Discover opportunities in Machine Learning.
Our research in machine learning breaks new ground every day.