Tensorflow Unity

I’ve been thinking for a while about how best to combine machine learning knowledge I’ve built up and my other hobby - making video games. To this end I’ve been looking into using Tensorflow with Unity3d. I forsee a lot of issues around performance at runtime, along with cross platform issues issues down the road.

In this session, we’ll build a little smartphone game, train a bot to play it using reinforcement learning, Python, and TensorFlow, and deploy it to a smartp. I have a trained model built in Tensorflow and I want to use it in Unity.I've searched a lot and TensorflowSharp is the only plugin I could find that can be used.However,TensorflowSharp plugin haven't been updated and I cant use a model trained on recent Tensorflow release.My inquiry,how can we load a pre-trained model in Unity and use the latest Tensroflow framework.I've checked ML-agent.

As a start here is a quick rundown of compiling Tensorflow to run a trained graph from a C# Unity script using the C++ api.

1. Getting a Graph to Use

The idea here is to keep it to a bare minimum for what you need. The goal here is to make sure we can get TF will run at all rather than spending time making it do something useful. For these first couple of steps I borrowed heavily from Jim Flemming’s excellent Medium post on the basics of using the C++ api, which we’ll need. Drivers fuzhou rockchip. Using a simple python script you can generate a protobuf file to store your graph, I used:

which does the very impressive task of finding the max of two numbers and storing it in a variable c. Google modems driver.

2. Compiling Tensorflow to a shared libaray (.so)

We can use the C++ api for TF to read and excetue graphs we’ve already saved pretty easily. In addition the Google documentation for using their build tool, Bazel is pretty good so compiling the shared library isn’t really too much of an issue. Here I used Jim Flemming’s build script with a few changes.

Debugging

When we start using this with Unity it’s not the same as running a script internally, we have to recompile to add debug messages. There’s not a simple method of accessing the Unity editor console to print useful debug messages anyway since we’re constrained by the return type of our function. This is a problem even with this very simple example as we’ve hard coded the path to the protobuf file which will cause problems down the road. Unity isn’t handling the paths so it won’t put the files in sensible places when building for instance. To give us some useful output on errors I’ve added logging to a file instead of standard out and known return values so it’s obvious where in the code we hit an error.

Shared library

The function is incased in an extern 'C' {}. Since we’re not building an exceutable we can’t just have a main function that returns 0, we want to return our max number. The C++ compiler however doesn’t preserve function names so calling our function after compilation won’t work, we have to tell the compiler we want to declare our funtion with C linkage. The Unity documentation does a fine job of explaining why we need to do this.

Here’s the code put inside the Tensorflow repo at /tensorflow/loader/loader.cc

We’ve hard coded here that it should output 3, which is the max of 3.0 and 2.0 cast to an int

We can compile using Bazel to either an exceutable or a shared library. The former is good to make sure the above code works, you can just throw the function into a main function and compile it. Once that’s working we can make our shared library. The Bazel BUILD file for that looks like:

If you keep this inside the loader folder with the cc file we can build from that folder using bazel build :loader.so. The actual so we need will end up

3: Running in the Unity Editor

You can access the shared library using the [DllImport ..] statement for C#, it doesn’t matter this isn’t a dll! This works the same as any Unity plugin, I have it in assets/plugins Here’s my very simple unity script:

All this is doing is:

• Loading our library
• On scene start running the run function from the library
• Putting the resulting integer as the string on an attached UI element.

Because of the way we hardcoded the model loading in the C++ and how the Unity editor does paths the protobuf needs to be in the root of the Unity project itself, not the plugins folder, not /assets. Now we can run this from the editor…

… Very impressive Unity, 3 is the right answer…

4: Running in a build

The shared libaray we’ve made as far as I know will only work for Linux. We need to compile different plugins for other platforms. Within the Unity editor we can set plugins to be included in different builds from the inspector for the asset. By default everything gets included which is fine if we’re only bulding for Linux but our project size will get pretty out of control if we have 3 or 4 TF libraries always being included in each build.

The default settings will work for the build; however this is where our lazy hard coding of paths is a problem. In this case the root path is going to be whereever your excutable is run from, so the /models folder needs to go there, not anywhere within the /data folder!

It’s not particularly impressive but it works - we can access Tensorflow from a built Unity3d project! Now to run some more exciting graphs…

Porting of “TensorFlow Lite Examples” to Unity. and some utilities for Unity.

Tested on

• iOS / Android / macOS / Windows
• Unity 2019.4.24f1
• TensorFlow 2.4.0

Samples

• TensorFlow
  • MNIST
  • SSD Object Detection
  • DeepLab
  • PoseNet
  • Style Transfer
  • Text Classification
  • Bert Question and Answer
• MediaPipe
  • Hand Tracking
  • Blaze Face
  • Face Mesh
  • Blaze Pose (Upper body)
• MLKit
  • Blaze Pose (Full body)
• Meet Segmentation

Included prebuilt libraries

• All libraries except iOS are targeted 64bit platform: arm64 or x86_64.

Install TensorFlow Lite for Unity

• Clone this repository with examples
  • Need Git-LFS to build for iOS
• The TFLite core library is available on:
  • OpenUPM
    Run openupm add com.github.asus4.tflite from the command line.
  • Or add git URL from the Package Maneger UI: https://github.com/asus4/tf-lite-unity-sample.git?path=/Packages/com.github.asus4.tflite

Tensorflow Unity Object Detection

Build TensorFlow Lite libraries

Pre-built libraries are included. If you want to build the latest TFLite,

Tensorflow Unity 2020

1. Clone TensorFlow library
2. Run ./configure in the TensorFlow library
3. Run ./build_tflite.py (Python3) to build for each platform

• To build macOS Metal Delegate on TensorFlow v2.3.0 or later, You need to apply following changes the issue

TIPS

[Android] You can see logs from tflite by filtering with “tflite”

Show Cases

MNIST

SSD Object Detection

DeepLab Semantic Segmentation

Tensorflow Unity Tutorial

Global wireless modems driver download for windows 10. Style Transfer

Hand Tracking

BERT

License

Samples folder Assets/Samples/* is licensed under MIT

Unity Tensorflow Plugin Download

Other Licenses

• TensorFlow: Apache License 2.0
• MediaPipe: Apache License 2.0
  • Some MediaPipe C# codes are based on terryky/tflite_gles_app

Model Licenses

📌 : Each TensorFlow Lite model might have a different license. Please check the license of the model you use.

Tensorflow Unity Point

• Official TFlite Models
  • Bert
  • SSD
  • DeepLab Lab
  • MNIST
  • Style Transfer
  • PoseNet
  • Text classification
  • Smart Reply
• MediaPipe Models
  • Blaze Pose
  • Face Mesh
  • Hand Tracking
• Meet Segmentation Model
  • Using the modified model from PINTO_model_zoo to remove the custom post-process.