EUVIMEDEUVIMED entdecken
Crossref

DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models

Yuqing Zhang, Yuan Liu, Zhiyu Xie, Lei Yang, Zhongyuan Liu, Mengzhou Yang, Runze Zhang, Qilong Kou, Cheng Lin, Wenping Wang, Xiaogang Jin

ACM Transactions on Graphics · 19.07.2024 · 43 (4) · 1-18

DOI: 10.1145/3658170 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Graphics, online erschienen am 19.07.2024. Verfasst von Yuqing Zhang und 10 weiteren Autor:innen. Die Arbeit wurde bisher 40-mal zitiert (Crossref) und nennt 113 Literaturangaben. Gefördert von: Key R&D Program of Zhejiang, FDCT.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recent advancements in 2D diffusion models allow appearance generation on untextured raw meshes. These methods create RGB textures by distilling a 2D diffusion model, which often contains unwanted baked-in shading effects and results in unrealistic rendering effects in the downstream applications. Generating Physically Based Rendering (PBR) materials instead of just RGB textures would be a promising solution. However, directly distilling the PBR material parameters from 2D diffusion models still suffers from incorrect material decomposition, such as baked-in shading effects in albedo. We introduce DreamMat , an innovative approach to resolve the aforementioned problem, to generate high-quality PBR materials from text descriptions. We find out that the main reason for the incorrect material distillation is that large-scale 2D diffusion models are only trained to generate final shading colors, resulting in insufficient constraints on material decomposition during distillation. To tackle this problem, we first finetune a new light-aware 2D diffusion model to condition on a given lighting environment and generate the shading results on this specific lighting condition. Then, by applying the same environment lights in the material distillation, DreamMat can generate high-quality PBR materials that are not only consistent with the given geometry but also free from any baked-in shading effects in albedo. Extensive experiments demonstrate that the materials produced through our methods exhibit greater visual appeal to users and achieve significantly superior rendering quality compared to baseline methods, which are preferable for downstream tasks such as game and film production.

Förderung

  • Key R&D Program of Zhejiang
  • FDCT

Zitieren

APA

Zhang, Y., Liu, Y., Xie, Z., Yang, L., Liu, Z., Yang, M., Zhang, R., Kou, Q., Lin, C., Wang, W., & Jin, X. (2024). DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models. ACM Transactions on Graphics, 43(4), 1-18. https://doi.org/10.1145/3658170

Vancouver

Zhang Y, Liu Y, Xie Z, Yang L, Liu Z, Yang M, et al. DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models. ACM Transactions on Graphics. 2024;43(4):1-18. doi:10.1145/3658170

BibTeX
@article{Zhang2024,
  author = {Zhang, Yuqing and Liu, Yuan and Xie, Zhiyu and Yang, Lei and Liu, Zhongyuan and Yang, Mengzhou and Zhang, Runze and Kou, Qilong and Lin, Cheng and Wang, Wenping and Jin, Xiaogang},
  title = {DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models},
  journal = {ACM Transactions on Graphics},
  year = {2024},
  volume = {43},
  number = {4},
  pages = {1--18},
  doi = {10.1145/3658170}
}
Kostenlos starten →