Decoding GPT's Temperature Setting

In GPT models, temperature is an important but complex setting that affects how the AI generates text. It helps control how creative or random the text is. For developers, understanding this temperature setting is key to making the most of GPT's abilities. In this article, will give a brief history, explain the math behind temperature, show examples, and give advice on how to use temperature in different use cases.

Temperature control three behaviours of GPT model as follows:

• Creativity in the context of GPT models refers to the generation of novel and original ideas or combinations of ideas in the model's output.
• Cohesiveness in text generation denotes the logical and consistent flow of ideas, ensuring the output is understandable and well-structured.
• Randomness in generative models refers to the degree of unpredictability and variation in the output.

The concept of temperature in AI stems from statistical mechanics, repurposed to control output randomness in probabilistic models. Claude Shannon's 1948 research laid the groundwork, evolving with AI advancements to modulate language model creativity.

The mathematics behind temperature in GPT models involves adjusting the softmax function, which turns logits (model output scores for each possible next token) into probabilities. By dividing the logits by the temperature value, we can influence the probability distribution. At low temperatures, the model favors more probable tokens, reducing randomness. High temperatures flatten the probability distribution, allowing for less likely tokens to be chosen, increasing randomness and potential creativity in the generated text. This modulation is key in tailoring GPT outputs for various applications.

For an example of a temperature experiment, I used a single prompt to test the model's creativity. This experiment showed how varying the temperature setting affected the titles generated by the model, as follows:

Model used : GPT 3.5

The observation from the temperature experiment indicates that at low settings, the model's creativity is quite constrained, with little variation in repeated outputs. At a setting of 1, the results become significantly more creative and vary with each generation. At the highest setting of 2, the model's creativity reaches its peak, producing highly imaginative and novel concepts, such as the idea of cyborg sentience and unique names like 'Andruval'.

The given prompt serves as a quick measure of the model's creative capacity. For a more comprehensive assessment, the following prompts are designed to evaluate the model's ability to maintain cohesiveness and to introduce randomness in its responses.

The prompts that can effectively test the cohesiveness of a model's response could be:

"Explain the steps involved in converting sunlight into electrical energy using solar panels."

"Describe the journey of a water droplet as it cycles from a cloud to a river and back into the atmosphere."

These prompt requires the model to provide a clear and logically ordered explanation, which would demonstrate its ability to maintain cohesiveness in the text.

The prompts designed to test the randomness in a model's responses could be:

"Imagine a futuristic world where animals have evolved to possess human-like abilities. Describe a day in this world."

"Create a story about a mysterious object discovered in an ancient, hidden city. Each time it's touched, the object grants a different superpower.

This open-ended prompt encourages a wide range of unpredictable responses, allowing the model's randomness to be effectively evaluated.

Suggestions for application

As per OpenAI Forum:

Lower values for temperature result in more consistent outputs (e.g. 0.2), while higher values generate more diverse and creative results (e.g. 1.0). Select a temperature value based on the desired trade-off between coherence and creativity for your specific application. The temperature can range is from 0 to 2.

Here are use cases and suggested temperatures.

In summary, higher temperatures usually result in increased creativity and randomness in the generated text, as the model becomes less likely to choose the most probable next word and instead explores more varied options. However, this also means that at higher temperatures, the cohesiveness or logical flow of the text can decrease, as the model may generate more unusual or unexpected phrases that don't always fit well together. Conversely, at lower temperatures, the model's outputs are more predictable and coherent, but less creative and varied.