AI image generation can still be very hit and miss. Even with style and structure references, it can take a lot of guesswork with text prompts and many attempts to get the structure you want for an image. The GPU giant Nvidia thinks it has a solution – and it uses Blender, one of the best 3D modelling software options.
The new Nvidia AI Blueprint for 3D-guided generative AI (a catchy moniker, I know) aims to provide more precise control for AI image generation by allowing uses to create scenes in 3D first. It then takes the 3D scenes into Black Forest Lab’s Flux.1 image generator.
The process behind the Nvidia AI blueprint (Image credit: Nvidia)
The idea is that you can create a draft layout of an image using Blender, placing things like buildings, plants, vehicles and other objects where you want them. Using Blender means you can adjust the camera position etc. The draft 3D scene will then be used as a reference for the AI model to generate a 2D image.
The process provides more control than using text prompts alone because the reference Blender scene can be adjusted visually, reducing the guesswork involved in having to try out multiple prompts to get somewhere near the composition you want. And there’s no need for the Blender scene to have a lot of detail or polish because it’s only there to provide the composition.
Nvidia says the AI Blueprint for 3D-guided generative AI is a ready-to-use workflow that provides an automatic pipeline for intuitive control over image structure, layout, and perspective directly from Blender.
The blueprint comes with step-by-step instructions, sample files, and a preconfigured environment, including customisable source code and a demo showing how to integrate ComfyUI workflows. Performance is optimised for RTX AI PCs via Nvidia NIM microservices and accelerated by TensorRT and FP4.
Nvidia notes that it’s suitable for non-commercial work only.
Daily design news, reviews, how-tos and more, as picked by the editors.
What are NVIDIA NIM Microservices and AI Blueprints for RTX AI PCs? – YouTube
