Generative Design Workflows

Overview

The field of AI is evolving daily. The tools are changing, new models or versions are constantly being updated, and if your feed looks anything like mine, it’s flooded with experimentation of how to implement or integrate AI in every nook and cranny possible.

I recently gave a lecture at the University of Toronto for an architecture class focused on Generative Design Thinking & Workflows. This presentation / article is an excerpt from the lecture that suggests methods for combining currently available AI tools to develop design processes or exercises that may help you become more familiar working with AI in Architecture. So let’s jump right in:

I categorized these experimental approaches into 5 workflows, but you’ll see there’s a lot of overlap in the use of synthetic image generation and ChatGPT since those are the primary AI tools available right now. So we have:

• Componentized Digital Fabrication
• Scripted Installation
• Automated Attribute Application
• Reverse Prompt Engineering
• Virtual Environmental Design

With all of these workflows, my goal was either to develop a process that may eventually be used to fabricate architecture, or to assist the architectural documentation process.

Componentized Digital Fabrication

Starting with Componentized Digital Fabrication, this is a 2D IMG to 3D Model to 3D Print workflow, and as the title suggests, this workflow leans heavily on 3D Printing, CNC Milling, or any other form of digital fabrication to produce the final product.

Everyone’s familiar with the traditional architectural project phases, which include concept design, engineering, construction documents, and then traditional construction techniques. But this proposed emergent workflow attempts to improve the relationship between concept design and final product, by extracting AI generated imagery as a digital 3D model, which can then be analyzed structurally and immediately 3D printed. This workflow also offers a unique opportunity to reimagine how architectural components may be synthesized or merged together to create unibody parts that eliminate unwanted joints or seams between dissimilar building systems. For instance, the lower right hand corner of this diagram shows an enlarged view of a component that functions as part of a composite structural system, and also acts as a storefront jamb detail. I call this Synthitecture, but we’ll leave that for another presentation.

Let’s walk through an example of this workflow. This is a prompt for an architectural pavilion, and remember it's important to use natural language very deliberately. Keep in mind, these image generators, as amazing as they seem, are still simply models or programs that produce content based on data it was trained on.

This prompt was refined through producing hundreds of images, but once you develop something you like, the next challenge is to create a 3D model from that 2D image.

One way to approach this is by using depth maps. These allow you to communicate 3-Dimensional information by mapping lighter tones to objects that are closer to the viewer and darker tones for objects that are farther away. Fortunately, there are also AI-driven tools online that create depth maps for you, and allow you to view the final product, and here’s an example of that.

So this gives us an idea of how the pavilion is starting to feel spatially, but we still don’t have any digital geometry to fabricate from. There are some companies working on AI tools that can generate 3D models from 2D images, or create 3D models directly from natural language text prompting. But, again, these programs are only as good as the data it’s trained on, so as these tools become more available they are likely to be better at producing known forms such as furniture pieces, vehicles or other entourage items rather than unique architectural components.

Here’s an example of Text-to-3D by DreamFusion and you can see the fidelity of the model is already fairly accurate, so you can imagine what we’ll be producing in 5 or 10 years from now, or sooner.

I’ve also seen examples of people converting depth maps into digital meshes using Grasshopper to interpolate the tones in the image as coordinates for a mesh. You can then 3D print that terrain, or you can also use this approach on a CNC milling machine by setting the routing depths to match the image tones based on their value.

Scripted Installation

Next we have, Scripted Installation, and this one is a bit more conceptual taking a 2D IMG and running it through an IMG Recognition program, or using ChatGPT to then write a script for Robotic Installation

So we start with an image prompt, and once we get a result we like, we can again use the depth map generator to produce a displacement image

And then jump into ChatGPT. So, just as a reminder, these workflows are still very theoretical and most of them don’t have proven case studies, but the idea here is to encourage you to think about these processes and how to string them together to eventually create built forms. I asked ChatGPT to “Write a python script for a Kuka robotic arm to pick up bricks from a stack of 10x10x10 modular units and place them according to a depth map image of an architectural wall design”

And it did. Now, I haven’t tested any of this, but it looks like it’s trying to identify all the appropriate steps, and the image is a little blurry on the screen so I’ll abbreviate ChatGPT’s response which was:

• Import necessary libraries and assign task variables
• Initialize the environment and load necessary models or images
• Load the depth map image and convert it to a 3D point cloud
• Generate a movement path for the robotic arm based on the position and orientation of the bricks in the stack, and the desired position and orientation of the bricks in the wall design.
• Initiate the robotic arm build process, and verify movement path to the first brick in the stack.
• Use the robotic arm's gripper to pick up the brick and move it to the first position in the wall design. Repeat until wall design is complete

Just to be clear this video is for reference only - it’s not operating based on this ChatGPT script. But it’s pretty easy to see how digital fabrication or robotic installation can start to merge earlier into the concept design phase of any project. Another approach is to jump straight to toolpath generation in ChatGPT for 3D printing, but when I asked it to generate “Write a gcode toolpath for a 3D printed dome structure with a radius of 10 feet and a height of 15 feet” it basically told me, "too soon buddy".

Automated Attribute Application

Next we have Automated Attribute Application, and this is another conceptual approach with some wish list items that I hope software developers are currently working on. The idea here is to work with synthetic image generators in our actual 3D models, which already exists, but then have the ability to apply the changes directly to our model.

This is an example from Revit using an add-in called Veras by EvolveLAB, which uses a ControlNet based process to maintain the image composition while offering text-to-image generation. And if you’re not familiar with ControlNet, it’s a behind-the-scenes extra step that image generators can use to recognize patterns or geometry in an image either by detecting Canny edges, automating depth maps, or using a number of other recognition methods to lock the image composition before generating iterations.

What’s missing here is the next step, which is applying content from  the 2D generated image directly to the 3D model with parameter controls that allow you to further refine the model's geometry. But when combined with some of the earlier methods I’ve outlined, you can see how that might be possible in the very near future. Incase you're interested in seeing Veras in action, here's a Video Tutorial.

Reverse Prompt Engineering

Next is Reverse Prompt Engineering, and this isn’t as much of a workflow as it is an exercise that I think is critical for architects and designers to practice moving forward while collaborating with AI. This one also leans heavily on the idea of communicating precisely with AI to produce more accurate results.

This can be done manually through trial and error by literally plugging in single words or short phrases into Midjourney and cataloging the results. Or you can train a closed dataset with your own images and descriptions so that the model can produce images specific to your language. 

In case you didn’t know, the datasets used to train the current image generators is available to browse, and you can search specific words to see how the AI models were trained to respond.This is LAION, a large-scale AI Open Network used for machine learning and is available to the general public.

But again, something to think about is how to use available image-recognition and object-detection tools to identify scenes, or unique geometries so that results can be more reliable. This is a prompt I wrote to try and identify the scene, to see if I could recreate the image in Midjourney, and it was actually pretty difficult.

So here’s my recreation, but I did end up using the original image as a reference in combination with the prompt to recreate it.

Virtual Environmental Design

Last but not least we have Virtual Environmental Design, and this is also not necessarily a workflow, but could be used as a tool to advance architectural design and documentation. It’s essentially, synthetic image generation for 360 virtual reality views, but also with the ability to modify specific attributes of the environment.

There’s already some amazing programs out there that allow you to generate seamless 360 panoramic views with only a text prompt. So you can start with a prompt in Skybox Lab, and it will immediately generate something like this:

But then to take it a step further, you can export the equirectangular panoramic image, plug it into stable diffusion using ControlNet, and generate design options that hold true to the image geometry.

We'll pause here for now. I encourage everyone to zoom out, take a step back, and evaluate what’s happening, in the world, in your social circle, and with your academic, personal or professional pursuits. It’s ok to stop scrolling, and sit silently with your thoughts. Because those thoughts will allow you to approach this AI revolution with your personal experience, and your interests, which will produce passion-driven results that hopefully benefit your industry in a meaningful way.

I also want to emphasize the importance of systems thinking, or this idea about making connections across platforms, activities, or concepts. Systems thinking is the process of synthesizing or merging ideas at their point of commonality, and this ability to link distinct concepts is what will allow you to develop emergent processes or workflows that support your goals.

If you've made it this far, you might be interested in my YouTube channel Architecture for Thought which offers a deeper look into various emergent technologies that are shifting the way we practice, document and build architecture. Thanks for your interest and feel free to connect with me on LinkedIn to continue this conversation.

Sincerely,

Stephen Coorlas