Machine Learning in Unity 🤖

You are curious about Reinforcement Learning or Machine Learning in general and interested in building custom environments? Then, I think Unity’s ML-Agents framework is the tool for you. In the last year, it has become a pretty stable and extensive platform to do experiments on.

Machine Learning is fun! 🎾

What is Unity3D?

Unity is a game engine that tries to democratize 3D and 2D game development. It is an extremely powerful tool, which allows you to create complex games with cutting edge graphics for many platforms (from PC to mobile to consoles). In the last years, the capabilities of the engine grew and grew to a point where it is hard to keep track. But for now, you don’t have to worry about that. When using ML-Agents we can ignore many of those features without feeling bad.

As developing a custom engine is a very costly endeavor, it is not surprising many game studios are using Unity to develop their games like Cuphead, Gwent, Monument Valley and many more.

Why combine Unity and Machine Learning?

Unity investing heavily in Machine Learning is a pretty smart idea in my mind. As a game engine, it already provides all the tools you need to build environments. Furthermore, AI in games has mostly been using techniques like pathfinding and decision trees, which are not able to dynamically adapt to player behavior as well as machine learning is. Unity is envisioning a future where NPCs are not strictly hardcoded anymore.

Wanna get a taste of Machine Learning? Check this out:

Where to start?

If you think: “Okay you got me. This sounds interesting but where do I even begin?”. This is the question we are going to tackle now. As said before, Unity is a beast and it can be very overwhelming at first. Before jumping into Machine Learning I would recommend learning the basics. There are many good tutorial series on this, like this one.

A few tips for you to keep in mind when learning the basics of Unity:

• Don’t bother with lighting, shaders, and graphics. Making your scene look great and feel realistic is not necessary for Machine Learning. The AI doesn’t care if the environment is pretty.
• Focus on logic and scripting. You will be spending a lot of time here. The good news is: It isn’t complicated. Unity does most of the work for you.

When you got the basics down, you can jump into Machine Learning. You can find the ML-Agents toolkit here. Don’t worry if you still feel confused about many areas of the engine. This is to be expected!!

ML Agents — a few tips ☝️

First of all the documentation is your best friend. Make sure to read it carefully, the devil really is in the detail. The goal of this article is not a step by step breakdown. But I want to give you a few tips that I would have appreciated before starting out.

• Check out the examples first. And I mean really check them out. Look into the code and try to understand every detail. You can find many best practices here and it helps you to build your own environments.
• Start as simple as possible. If you think: “Oh this environment is too easy, that’s boring”. Perfect, only then your environment is simple enough. When your agents have trained successfully, you can always increase complexity afterward. But it minimizes your chance of making mistakes. Little errors can lead to disaster in reinforcement learning.
• Work carefully. In programming, we are used to pressing play first and fixing errors later. This is a bad idea with machine learning because iteration time is dependent on training time. And training time can be anything from a few hours to a few days.
• Work with human input until everything works. By using the so-called “Player Brains” you can control the agents. It allows for faster testing and iteration and you can be sure that the fundamentals of your environment are solid before training an agent.
• Switch to a standalone version if everything works. After using your input to test all the core functionality, you can switch to actual training. Start off by doing this in the editor. If everything seems to work fine, switch to a standalone version. The editor has a lot of overhead that unnecessarily slows down your training time.
• Human demonstrations are your friend. When I was starting, I overestimated the abilities of the agents. Seemingly simple environments can be overly challenging. Recording your gameplay can speed up the learning process immensely. Use it! 🎉
• Reward shaping is dangerous, use in moderation. Many times, rewards are only given at the end of a level. So, a complex sequence of steps is required to receive any rewards. When you are watching the agents struggle to get any rewards even after hours of training in can be very tempting to give some intermediary rewards. This can easily lead to exploitation or shape the gameplay in a way you may not like. Try to use curiosity or human demonstrations before you shape rewards.

I hope this helped! I am running a youtube channel with content on reinforcement learning if you are curious. Check it out here!