The Visual Language of Fabrics
Siggraph 2023

Abstract

We introduce text2fabric, a novel dataset that links free-text descriptions to various fabric materials. The dataset comprises 15,000 natural language descriptions associated to 3,000 corresponding images of fabric materials. Traditionally, material descriptions come in the form of tags/keywords, which limits their expressivity, induces pre-existing knowledge of the appropriate vocabulary, and ultimately leads to a chopped description system. Therefore, we study the use of free-text as a more appropriate way to describe material appearance, taking the use case of fabrics as a common item that non-experts may often deal with. Based on the analysis of the dataset, we identify a compact lexicon, set of attributes and key structure that emerge from the descriptions. This allows us to accurately understand how people describe fabrics and draw directions for generalization to other types of materials. We also show that our dataset enables specializing large vision-language models such as CLIP, creating a meaningful latent space for fabric appearance, and significantly improving applications such as fine-grained material retrieval and automatic captioning.

Downloads

BibTeX

@article{deschaintre23_visual_fabric,
author = {Deschaintre, Valentin and Guerrero-Viu, Julia and Gutierrez, Diego and Boubekeur, Tamy and Masia, Belen},
title = {The Visual Language of Fabrics},
year = {2023},
issue_date = {August 2023},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
volume = {42},
number = {4},
issn = {0730-0301},
doi = {10.1145/3592391},
journal = {ACM Trans. Graph.},
month = {jul},
articleno = {50},
numpages = {15},
keywords = {material appearance, descriptions, perception}
}

Acknowledgements

We thank Kushal Kafle for his help in navigating the NLP domain; Yulia Gryaditskaya and Zoya Bylinskii for advice about data collection and user studies; J Eisenmann for his help in data generation; Jorge Gracia del Rio for NLP pointers; and Sergio Izquierdo and the members of the Graphics and Imaging Lab for discussions and help preparing the final figures. We would also like to thank the Substance 3D Assets team for the creation of the materials, all the participants in the user studies, and the anonymous reviewers. This work has received funding from the European Research Council (ERC) within the EU's Horizon 2020 research and innovation programme (project CHAMELEON, Grant No. 682080) and under the Marie Skłodowska-Curie grant agreement No. 956585 (project PRIME), as well as a generous donation from Adobe. Julia Guerrero-Viu was supported by the FPU20/02340 predoctoral grant. We borrow this Template from Michaël Gharbi.