DragTex: Generative point-based texture editing on 3D mesh

Creating 3D textured meshes using generative artificial intelligence has garnered significant attention recently. While existing methods support text-based texture generation and editing 3D meshes, they often struggle to precisely control pixels of texture images through intuitive interaction. Meanwhile, editing texture images by direct image deformation with precise point-to-point control does not necessarily guarantee that the generated mesh textures are a good match to the interactive intent. Existing work supports generative editing of 2D images via dragging interactions, but applying such methods directly to 3D mesh textures still leads to problems such as lack of local consistency between multiple views, error accumulation, and long training times. To address these challenges, we propose a generative point-based 3D mesh texture editing method, DragTex. It utilizes a diffusion model to blend locally inconsistent textures in the region near the deformed silhouette between different views, enabling locally consistent texture editing. We further fine-tune a LoRA decoder to reduce reconstruction errors in the non-dragged region, thereby mitigating overall error accumulation. Moreover, we train LoRA using multi-view images instead of training each view individually, which significantly shortens the training time. Our experimental results show that our method can effectively drag textures on 3D meshes and generate plausible textures that meet the user’s intent.