Read the full analysis: Understanding The Process Of Fine-Tuning 3D Generation Models on ThorstenMeyerAI.com
TL;DR
Hugging Face published scripts and container images for fine-tuning its TRELLIS.2 3D-generation model on custom asset datasets, locally or through Hugging Face Jobs. An example uses LoRA adapters and 188 footwear models to investigate inaccurate shoe soles, but the supplied material reports no completed results or measured improvement.
Hugging Face has published scripts and containers for fine-tuning its TRELLIS.2 3D-generation model on custom asset collections, offering a route to adapt a general-purpose model to specific object categories, as detailed in the original fine-tuning guide. The accompanying example targets shoe geometry with LoRA adapters and 188 footwear objects, but the supplied account does not report whether training improved the model’s results.
The example is aimed at a limitation observed in initial tests: generated shoes generally looked like shoes, but their soles were inaccurate. The stated experiment uses 188 footwear objects from Google Scanned Objects (GSO) to test whether category-specific fine-tuning can address that problem. The source describes the plan and setup, not a completed comparison.
The training scripts use low-rank adaptation (LoRA), a method that adds small trainable adapter matrices while leaving the original model weights frozen. The post says this reduces the number of parameters trained and the memory needed for gradients and optimizer state; the scripts use the PEFT library. These are explanations of the method, not reported measurements of savings for this particular run.
Users can run preprocessing and training on a local machine with a GPU or through Hugging Face Jobs, which runs containerized workloads on cloud hardware. The described setup requires uploading a mesh dataset to the Hugging Face Hub, a Hugging Face access token, and account access to gated DINOv3, used to encode input images. The source says the scripts can be adapted through arguments, but does not give a hardware configuration, cost, or training duration.
Fine-tuning TRELLIS.2 for custom 3D assets
Hugging Face released scripts and containers to adapt TRELLIS.2 to specific object categories. The footwear example outlines an experiment, but reports no completed results.
What the workflow targets
The example addresses shoes that look plausible but have inaccurate soles.
Coarse shape
LoRA adapters target sparse structure to adapt the model to footwear.
Detailed shape
The shape stage generates geometry; the example aims to improve shoe details such as soles.
Stage left unchanged
The dataset lacks separate material maps, and the authors say base colors and textures are satisfactory.
How TRELLIS.2 builds a mesh
The described pipeline turns an input image into a textured mesh.
BiRefNet removes the background, then DINOv3-L encodes the image.
Sparse structure guides the model’s detailed geometry generation.
A separate texture stage generates materials before the outputs are combined into a GLB mesh.
Ways to run the experiment
Teams with suitable assets can choose a local or cloud workflow.
Use the provided Docker commands with your own GPU.Run preprocessing and training scripts on your machine.
Run containerized workloads on cloud hardware.Upload the mesh dataset to the Hugging Face Hub.
What teams need
Access and evaluation matter alongside the training setup.
A Hugging Face token and access to gated DINOv3.A comparison of pretrained and fine-tuned outputs, including held-out footwear.Clear geometry criteria and reported training settings, hardware, and costs.
From General 3D Output to Shoe Geometry
The release gives developers a packaged way to test whether a broad 3D generator can be adapted to a narrow category or feature. That distinction matters for uses such as robotics, simulation and media production, where a plausible-looking asset may still be unsuitable if a specific part of its geometry is wrong. For footwear, the example focuses on soles rather than trying to retrain every part of the output pipeline.
The approach could be relevant to teams that already have suitable 3D assets and want more control over a category. However, the available account does not show that the footwear experiment succeeded, quantify any improvement, or establish that the resulting model works on shoes beyond the examples used in training. Its immediate value is a reusable experiment workflow, not demonstrated gains in generation quality.
The release also illustrates a practical choice in model adaptation: selecting which stages to train. TRELLIS.2 separates structural shape, detailed geometry and materials. The example adds adapters to the sparse-structure and shape stages, while leaving texture generation unchanged. That choice is tied to the stated footwear task and available data; the source provides no comparison showing that it is better than training other components.
How TRELLIS.2 Builds a Mesh
The supplied account describes TRELLIS.2 as using a unified latent representation intended to support multiple 3D output formats through decoders. Its pipeline begins by removing an image background with BiRefNet and encoding the image with DINOv3-L. A sparse-structure flow model predicts a coarse shape, which guides a shape flow model that generates detailed geometry. A separate texture flow stage produces surface materials; the outputs are combined into a textured mesh that can be exported as a GLB file.
For the shoe example, the adapters are applied to the sparse-structure and shape flow diffusion transformers, the stages the authors associate with shoe form. They leave texture generation unchanged because the dataset has diffuse color maps but does not include the separate material maps needed to train that stage. The post also says the base model already produces satisfactory colors and textures for this task. These details explain the experiment’s design, but do not prove that the selected training targets improve geometry.
The workflow is presented as a way to adapt a model that can generate 3D assets to a specific dataset, rather than as a new model architecture or a reported benchmark result. The source does not provide a full evaluation protocol or a comparison with alternative fine-tuning choices.
“The example targets shoe geometry, including inaccurate soles, using 188 footwear objects from Google Scanned Objects.”
— Hugging Face, as described in the supplied post
The Shoe Experiment Has No Results Yet
The supplied material does not say whether the fine-tuning run was completed or whether it produced more accurate soles. It provides no before-and-after examples, quantitative benchmark or independent evaluation. The account also does not specify the test set, scoring criteria, or how sole accuracy was judged in the initial observations.
Practical requirements remain unreported, including the compute needed, training time, cloud costs and the amount of dataset preparation. The source does not say how well a fine-tuned model performs on footwear outside the 188-object subset, or whether the results hold for different input images. Those points matter when judging whether the workflow is useful beyond a demonstration.
Run the Workflow and Test Generalization
Developers can use the published repository to prepare a mesh dataset on the Hugging Face Hub and run preprocessing and training locally or through Hugging Face Jobs. They will need a suitable GPU for local runs, a Hugging Face token and access to gated DINOv3. The source also refers to a walkthrough, but the supplied material does not provide its full details.
The next evidence needed is a report of the completed experiment: generated shoe examples, a clear evaluation method and comparisons with the base model. Results on footwear not used for training, along with compute and cost details, would help establish whether the approach improves the targeted geometry and how practical it is for other teams.
Key Questions
What did Hugging Face publish?
It published scripts and containers for fine-tuning TRELLIS.2 on custom 3D assets, with options to run the workflow locally or through Hugging Face Jobs.
What is the footwear example testing?
The example investigates whether fine-tuning can improve shoe geometry, particularly soles, using 188 footwear objects from Google Scanned Objects. The supplied account does not report the experiment’s outcome.
Why does the workflow use LoRA?
LoRA adds small trainable adapter matrices while keeping the original model weights frozen. The post says this reduces the number of parameters trained and the memory needed for gradients and optimizer state; it gives no measured savings for this experiment.
What do users need to run the example?
Users need a prepared mesh dataset uploaded to the Hugging Face Hub, a Hugging Face token, and access to gated DINOv3. Local runs require a GPU; the alternative is to use Hugging Face Jobs. The source does not specify hardware, cost or training duration.
Has fine-tuning been shown to improve the generated shoes?
No result is reported in the supplied material. It describes the training plan but includes no completed comparison, benchmark, or evaluation of shoe soles.
Primary source: Hugging Face · via ThorstenMeyerAI.com