BabylonJS with React-Native Tutorial
Getting started with BabylonJS in react-native mobile app
Implementing BabylonJS in a react-native mobile app. Create sims like avatar creating mobile app with BabylonJS react native.
We’ll be doing the following three things in this article:
- Loading a basic 3D model in our react-native app
This will set the base and will help you get the very basics of BabylonJS with React-Native. - Play pause animations on button press
Using a horse 3D model we’ll play with animation where we can make the horse walk and jump. - Changing shirt and hairstyles of a 3D Model
Just like how in Sims game or main avatar creating apps we’ll have a quick look on how to do it.
Installations:
My react is v18.2.0, and react-native is v0.74.2. For which the follow library versions are supported so install the following libraries:
yarn add @babylonjs/core@7.12.0
yarn add @babylonjs/react-native@1.8.5
yarn add @babylonjs/loaders@7.12.0
yarn add @babylonjs/react-native-iosandroid-0-71@1.8.5
yarn add react-native-permissions@4.1.5The
babylonjs/react-native-iosandroid-0–71package is needed forbabylonjs/react-nativeas it’s Babylon React Native iOS and Android Runtime. My react-native is v0.74.2, hence I’m installing the0-71.You can check which one you need to install by going to https://github.com/babylonjs/BabylonReactNative?tab=readme-ov-file#supported-versions.
Perquisites:
Basic understanding of 3D from both designing and development POV is good to have.
For example: I had basic idea of things like sphere, cylinder, squares etc are the shapes which are used to bend and create a shape in 3D say I’m making a rugby ball I’ll take a sphere and modify it by transforming it’s shape and other properties. If I have to give it the same look as a read rugby ball I have to add some material to it. This was just what was in my mind about 3D and I have seen people working on softwares like Blender etc. So, no in-depth knowledge is required.
BabylonJS basics is a must as I’ll not be getting into the details of say what BabylonJS engine is etc, because this tutorial is focused on implementing a 3D model with react-native. I recommend watching this https://www.youtube.com/watch?v=e6EkrLr8g_o as it’ll cover some basics before starting this tutorial.
BabylonJS React Native
If you’re not a 3D designer, you’ll need to work with a 3D designer so that they can export the correct file format of the 3D file which you’re going to then import in your react-native project. There is a section at the end titled Finding free 3D models:, where you can find links to websites for free .gltf files.
We’ll be using the file format .gltf which is widely used as there are others like .glb or .obj etc. The .gltf is a JSON structured file which is very easy to read or find information like what materials, animations etc are available.
If you have a .gltf file you can preview it online too using platforms like: https://gltf-viewer.donmccurdy.com/.
We’ll upload the .gltf file to the cloud or any CDN and get a URL that ends in .gltf (Recommended way), I’ve uploaded on cloud and below are the links of the two main GLTF files I’ll be using in this tutorial:
const basicGLTFURL = "https://raw.githubusercontent.com/thechaudharysab/babylonjspoc/main/src/assets/Client.gltf";
const horseGLTFURL = "https://raw.githubusercontent.com/thechaudharysab/babylonjspoc/main/src/assets/Horse.gltf";If I view all of the 3D models .gltf files I’m going to use in this project on the gltf-viewer, I get these and each starts with the default first animation they are assigned.
App Structure:
I’ve a single screen app, which I’m going to use App.tsx as my main screen. I’ve created src/assets folder to store all assets files and below is how my App.tsx looks initially:
//... all required imports
function App(): React.JSX.Element {
const basicGLTFURL = "https://raw.githubusercontent.com/thechaudharysab/babylonjspoc/main/src/assets/Client.gltf";
return (
<View style={styles.container}>
{/* ... UI Code Coming Soon ... */}
</View>
);
}
const styles = StyleSheet.create({
container: {
flex: 1,
},
});
export default App;Loading a basic 3D model in our react-native app
Let’s first load our basicGLTFURL. Add the following variables and imports in App.tsx screen:
//... Other imports
import { useEngine } from '@babylonjs/react-native';
import { Camera, Scene } from "@babylonjs/core";
import '@babylonjs/loaders/glTF';
function App(): React.JSX.Element {
const engine = useEngine();
const [scene, setScene] = useState<Scene>();
const [camera, setCamera] = useState<Camera>();
//... Remaining codeIn the code, we are importing a hook named useEngine, that’ll allow us to utilize the Babylon.js engine. Then from BabylonJS we have imported a Scene and a Camera, both of which are required to view a 3D object in BabylonJS where scene is the scene in which our 3D model will be loaded and camera will be the main camera that’ll display a part of that scene.
Next replace the {/* … UI Code Coming Soon … */} code comment with the <EngineView… line below:
import { useEngine, EngineView } from '@babylonjs/react-native';
//... Code before
return (
<View style={styles.container}>
<EngineView camera={camera} />
</View>
);
//... Remaining codeThe EngineView with camera prop will load a view with BabylonJS engine inside.
Next we’ll create a function where we’ll load the .gltf file.
//... Other imports
import { ArcRotateCamera, Camera, Scene, SceneLoader, HemisphericLight, Vector3 } from "@babylonjs/core";
//... Code before
const renderbasicGLTF = () => {
SceneLoader.LoadAsync(basicGLTFURL, undefined, engine).then((loadedScene) => {
if (loadedScene) {
setScene(loadedScene);
// Light
const light = new HemisphericLight("light", new Vector3(0, 1, 0), loadedScene);
light.intensity = 0.7;
// Camera
const camera = new ArcRotateCamera("camera", -Math.PI / 2, Math.PI / 2, -6, new Vector3(0, 2, 0), loadedScene, true);
camera.attachControl(true);
setCamera(camera);
} else {
console.error("Error loading loadedScene.");
}
}).catch((error) => {
console.error("Error loading scene: ", error);
});
};
//... Remaining codeIn the code above, we’ve use SceneLoader from @babylonjs/core to LoadAsync the .gltf file. This returns a promise with the loadedScene. Which means now we have the scene with our 3D character inside a babylon engine. So, we added a Light and a Camera (You can see more about adjusting light and camera in a scene and what the properties are with them from BabylonJS docs and adjust for your model accordingly). The camera.attachControl(true); will enable gesture control on the 3d Model, means you can rotate zoom in/out on the 3D character.
As I need the. 3D model to be loaded as soon as I open the screen, so I’ll call the above function in a useEffect in App.tsx:
useEffect(() => {
if (engine) {
renderbasicGLTF();
}
}, [engine]);Below is the screenshot of my App.tsx code as of now:
Let’s run it and see. I’ll be running it on Android emulator, but before we do that, add Renderer maximum under OpenGL ES API under Emulator Settings > Advanced as show below:
Running the android app and I see this as output:
You can see a 3D character, with an animation and you can rotate and zoom in and out as well to some extent. This is how you can load a basic model in BabylonJS react-native.
Adding a horse 3D model and playing with animations
We’ll first add a horse 3D model and the file format we need if .gltf. We’ll do the following:
- Import Horse 3D model (
.gltffile) - Give horse a default Idle animation
- Create buttons to trigger different animations
- and some more…
Open App.tsx or any screen file where you want to render this 3D model, and add the following code in it:
//... Other imports
import { useEngine, EngineView } from '@babylonjs/react-native';
import { ArcRotateCamera, Camera, Scene, SceneLoader, Color4, AnimationGroup, Nullable } from "@babylonjs/core";
import '@babylonjs/loaders/glTF';
function App(): React.JSX.Element {
//... Other code like engine, scene and camera
// (1.)
const renderHorseGLTF = () => {
SceneLoader.LoadAsync(horseGLTFURL, undefined, engine).then((loadedScene: Scene) => {
if (loadedScene) {
setScene(loadedScene);
// (1.1) Camera & Light
loadedScene.createDefaultCameraOrLight(true, undefined, true);
(loadedScene.activeCamera as ArcRotateCamera).alpha += Math.PI;
setCamera(loadedScene.activeCamera!);
// (1.2) Getting an animation and starting it
const idleAnim = loadedScene.getAnimationGroupByName('Idle');
if (idleAnim) {
idleAnim.start(true, 1.0);
}
} else {
console.error("Error loading loadScene.");
}
}).catch((error) => {
console.error("Error loading scene: ", error);
});
}
useEffect(() => {
if (engine) {
renderHorseExperiments();
}
}, [engine]);
// (2.)
const startWalkAnimation = () => {
//... coming soon
};
const stopWalkAnimation = () => {
//... coming soon
};
const doJumpAnimation = () => {
//... coming soon
};
const randomizeBGColor = () => {
//... coming soon
};
return (
<View style={styles.container}>
<EngineView camera={camera} displayFrameRate={true} />
{/* (3.) */}
<View style={styles.absoluteView}>
<TouchableOpacity style={styles.buttonContainer} onPress={startWalkAnimation}>
<Text>Start Walking</Text>
</TouchableOpacity>
<TouchableOpacity style={styles.buttonContainer} onPress={stopWalkAnimation}>
<Text>Stop Walking</Text>
</TouchableOpacity>
<TouchableOpacity style={styles.buttonContainer} onPress={doJumpAnimation}>
<Text>Jump</Text>
</TouchableOpacity>
<TouchableOpacity style={styles.buttonContainer} onPress={randomizeBGColor}>
<Text>Random BG Color</Text>
</TouchableOpacity>
</View>
</View>
);
}
const styles = StyleSheet.create({
container: {
flex: 1,
},
absoluteView: {
position: 'absolute',
bottom: '10%',
flexDirection: 'row',
alignItems: 'center',
alignSelf: 'center',
},
buttonContainer: {
backgroundColor: '#61dafb',
borderWidth: 1,
padding: 4,
}
});
export default App;The above code is the same as what we did in the basic example. I’ve added numbering in comments and below is each code element quick description:
(1.) Same as previously, we created a function named renderHorseGLTF to which we have provided a horseGLTFURL and the engine.
(1.1) Then instead of creating light and camera separately, I’ve used the createDefaultCameraOrLight which is good enough for my use-case. It’ll depend on your model and how you want to set it to check for what camera angle, ration and light it looks the best in.
(1.2) I’ve fetched an animation named Idle, from the horse .gltf file. Started it with loop true and speed 1.0.
The question of “How do I know there is an animation named Idle in the .gltf file?” Well there can be two ways to find that; one is opening a GLTF file in any editor and you can see it’s all JSON. There will be a lot of arrays in it like meshes, materials, and animations etc. If you see everything has a "name" property and for animations there is Idle, Idle_2, Idle_Headlow and many more as show below:
You can either find name like this or can add this function below:
const logAnimationNames = (animationGroups: AnimationGroup[]) => {
console.log("---------------ANIMATIONS--------------------");
animationGroups.forEach(animationGroup => {
console.log(animationGroup.name);
});
}Which you can call on loadedScene as logAnimationNames(loadedScene.animationGroups); as show in the commented line in the code. Also the output on the right shows all the animations the horse has.
(2.) I’ve added some empty functions which as clear by the name will do their job. The startWalkAnimation will change animation from Idle to Walk and stopWalkAnimation will set the animation back to Idle again. Just for fun we’ll have a doJumpAnimation which will make the horse do a jump and then go back to Idle or Walk animation whichever was active before pressing the Jump button. I’ve also added a randomizeBGColor function which will give our scene a random RGB color. You can also overwrite to have just one color. We’ll be adding code to them shortly.
(3.) Added the buttons UI to the screen.
With all the above if you run it then it should be able to see the horse rendered with the Idle animation wiggling his tail and you can rotate and/or zoom in and out (Check out the bonus section some more bits on the scene):
Now let’s add code to the buttons we have in our UI.
//.. All Other imports
import { AnimationGroup, Nullable } from "@babylonjs/core"; // Just added these two types for this step
//.. All Other imports
//.. Other Code
const [currentAnimation, setCurrentAnimation] = useState<Nullable<AnimationGroup>>(null);
//.. Other Code
const startWalkAnimation = () => {
if (scene) {
const walkAnimation = scene.getAnimationGroupByName("Walk");
if (walkAnimation) {
if (currentAnimation) {
currentAnimation.stop();
}
walkAnimation.play(true);
setCurrentAnimation(walkAnimation);
} else {
console.warn("Animation not found:", walkAnimation);
}
}
};
const stopWalkAnimation = () => {
if (scene) {
const idleAnimation = scene.getAnimationGroupByName("Idle");
if (idleAnimation) {
if (currentAnimation) {
currentAnimation.stop();
}
idleAnimation.play(true);
setCurrentAnimation(idleAnimation);
} else {
console.warn("Animation not found:", idleAnimation);
}
}
};
const doJumpAnimation = () => {
if (scene) {
var jumpAnimation = scene.getAnimationGroupByName("Jump_toIdle");
if (jumpAnimation) {
jumpAnimation.play();
} else {
console.warn("Animation not found:", jumpAnimation);
}
}
};
const randomizeBGColor = () => {
if (scene) {
// Generate random color values (0-255)
const red = Math.floor(Math.random() * 256);
const green = Math.floor(Math.random() * 256);
const blue = Math.floor(Math.random() * 256);
const randomColor = new Color4(red / 255, green / 255, blue / 255, 1.0);
scene.clearColor = randomColor;
}
};I added AnimationGroup andNullable just so that I can create a useState in typescript with the right type to store which animation is currently active. As I know the names of the animations I have with this Horse.gltf file I can pick which ones I want. So in the startWalkAnimation it’ll fetch the animation by name Walk. At first we stop() any currentAnimation and then play the new animation we want to i.e. walkAnimation. The main difference between play and start an animation as we did with Idle is we get some more parameters, but as we need to keep the default speed of the animation so I just used play with true on the param loop because I want the Walk animation to be on loop.
For stopWalkAnimation same thing as startWalkAnimation but instead the animation given is now Idle. This will loop as well.
For doJumpAnimation everything is the same but the animation I choose is Jump_toIdle. Notice how in play() there is no true which means just jump once and don’t loop. Meaning the animation will be run once and then it’ll be back to whatever animation there was last which in my case can be Walk or Idle.
Finally the randomizeBGColor is creating a random RGB color and then setting it as the scene color. scene is the loadedScene.
Let’s test it out!
So you can see how by just making animation loop I’ve created the above. This was just the way I’m handling animation for this particular example.
You may need more than just playing with loops as may be you need to run it frame-by-frame or other. Plus the horse animations are not very smooth, meaning if I’ve Walking animation and I press Jump it’ll not seems realistic as the horse takes a different position before jumping (Because I took horse model from internet so not a lot of room for me to fix that). It’ll really depend on your communication with the 3D designer who is putting together your whole 3D business in how to start and end animation so that each animation is kinds in a sync’d starting and ending-point. Like below are just some of the things you can do with animations from which we have only used start, play and stop.
🏆 Challenge: As I took the horse model from this free resource https://poly.pizza/bundle/Animated-Animal-Pack-ILAPXeUYiS where you can find other animals like Cow, Donkey, Bull, Wolf etc. Using the knowledge you have just obtained in the above example, create an app named MyPets that allows a user to select an animal and then create a virtual pet experience where they have to feed the pet, say my pet is Horse, and it’s time to feed the horse I can press the button and Eating animation of horse is played just like that add a many functions as you can for horse and if a user fails to do so play the Death animation available for horse.
Changing shirt and hairstyles of a 3D Model
Creating a sims like avatar creation experience can be done in many ways. The one I used is the very simple one (not ideal) but it works. I asked a 3D designer friend of mine to create a 3D model just for this example and give me the gltf file of it, so it’s not perfect but you’ll get the gist.
This is what I see when I use the gltf-viewer to view it:
This weird looking 3D model is because it is showing two hairstyles and two shirts that are overlapping each-other. Now we’ll use BabylonJS to hide and show one shirt and one hair at a time.
Let’s begin by importing the 3D model in our react-native as we did in the basic example:
//... Imports and other code
const simsCharacterGLTF = "https://raw.githubusercontent.com/thechaudharysab/babylonjspoc/main/src/assets/ibjects_test.gltf";
//.. Other code like engine, scene, camera states
const renderSimsGLTF = () => {
SceneLoader.LoadAsync(simsCharacterGLTF, undefined, engine).then((loadedScene) => {
if (loadedScene) {
setScene(loadedScene);
// Camera & Light
loadedScene.createDefaultCameraOrLight(true, undefined, true);
(loadedScene.activeCamera as ArcRotateCamera).alpha += Math.PI;
setCamera(loadedScene.activeCamera!);
// Scene Bg Color
const lightBlueColor = new Color4(208 / 255, 236 / 255, 255 / 255, 1.0);
loadedScene.clearColor = lightBlueColor;
// Adding more code here shortly...
} else {
console.error("Error loading loadedScene.");
}
}).catch((error) => {
console.error("Error loading scene: ", error);
});
}
useEffect(() => {
if (engine) {
renderSimsGLTF();
}
}, [engine]);
return (
<View style={styles.container}>
<EngineView
camera={camera}
displayFrameRate={false}
/>
</View>
);
}
//.. Remaining codeIn the above code we haven’t done anything new but just simply loaded the model with a fixed background color. You can run and se the model being rendered:
Each shape be in a shirt, arm, shoe, hair, eye is all a Mesh. Just like how we did with animation, we can also see all the meshes that a .gltf ontains. So, let me extend logAnimationNames to logMeshesAndAnimationNames as below:
const logMeshesAndAnimationNames = (meshes: AbstractMesh[], animationGroups: AnimationGroup[]) => {
console.log("-------------------MESHES--------------------");
meshes.forEach(mesh => {
console.log(mesh.name);
});
console.log("---------------ANIMATIONS--------------------");
animationGroups.forEach(animationGroup => {
console.log(animationGroup.name);
});
}Now doing logMeshesAndAnimationNames(loadedScene.meshes, loadedScene.animationGroups); on the above gltf file we’ll get:
As you can see in this .gltf file there is only one animation but many meshes. I don’t care about all of them but as I know this information that Hair1, Hair2, Shirt1 and Shirt2 are the meshes I’ll be working with. I’m going to do these two things:
- Set
Hair1andShirt2as default - Create a UI to change Hair or Shirt accordingly to what user presses
Where I added the comment // Adding more code here shortly… add the code below:
loadedScene.meshes.forEach(mesh => {
if (mesh.name === 'Shirt1' || mesh.name === 'Hair2') {
mesh.isVisible = false; // Hide this mesh
}
});I’ve looped through the meshes and where I could fine the name of mesh to be Shirt1 or Hair2 I’m hiding that mesh and the result will be a much cleaner 3D Model:
Now on to the UI. Same as I did with horse I’m going to add a few buttons and actions:
//... All code before
const onAvatarEditOptionSelect = (hair1: boolean, hair2: boolean, shirt1: boolean, shirt2: boolean) => {
if (scene) {
scene.meshes.forEach(mesh => {
if (mesh.name === 'Shirt1') {
mesh.isVisible = shirt1;
} else if (mesh.name === 'Shirt2') {
mesh.isVisible = shirt2;
} else if (mesh.name === 'Hair1') {
mesh.isVisible = hair1;
} else if (mesh.name === 'Hair2') {
mesh.isVisible = hair2;
}
});
}
};
const renderSimsUI = () => {
if (scene) {
return (
<View style={{ position: 'absolute', bottom: 0, backgroundColor: 'white', padding: 20 }}>
<Text style={{ fontWeight: 'bold' }}>Select Hair</Text>
<TouchableOpacity onPress={() => onAvatarEditOptionSelect(true, false, scene.getMeshByName('Shirt1')?.isVisible as boolean, scene.getMeshByName('Shirt2')?.isVisible as boolean)} style={{ marginVertical: 4 }}>
<Text>Hair 1</Text>
</TouchableOpacity>
<TouchableOpacity onPress={() => onAvatarEditOptionSelect(false, true, scene.getMeshByName('Shirt1')?.isVisible as boolean, scene.getMeshByName('Shirt2')?.isVisible as boolean)} style={{ marginVertical: 4 }}>
<Text>Hair 2</Text>
</TouchableOpacity>
<TouchableOpacity onPress={() => onAvatarEditOptionSelect(false, false, scene.getMeshByName('Shirt1')?.isVisible as boolean, scene.getMeshByName('Shirt2')?.isVisible as boolean)} style={{ marginVertical: 4 }}>
<Text>No Hair</Text>
</TouchableOpacity>
<Text style={{ fontWeight: 'bold' }}>Select Shirt</Text>
<TouchableOpacity onPress={() => onAvatarEditOptionSelect(scene.getMeshByName('Hair1')?.isVisible as boolean, scene.getMeshByName('Hair2')?.isVisible as boolean, true, false)} style={{ marginVertical: 4 }}>
<Text>Shirt 1</Text>
</TouchableOpacity>
<TouchableOpacity onPress={() => onAvatarEditOptionSelect(scene.getMeshByName('Hair1')?.isVisible as boolean, scene.getMeshByName('Hair2')?.isVisible as boolean, false, true)} style={{ marginVertical: 4 }}>
<Text>Shirt 2</Text>
</TouchableOpacity>
</View>
)
}
}
return (
<View style={styles.container}>
<EngineView
camera={camera}
displayFrameRate={false}
/>
{renderSimsUI()}
//... Remaining codeSame as before I added all the required buttons and created a function named onAvatarEditOptionSelect. Which takes four boolean values and this is how I’ll tell my code what to show or hide. Using a forEach loop we loop through the meshes and hide/show them accordingly.
Let’s test it…
This worked as expected. The avatar has a very smooth breathing Idle animation and it’s very easy to hide/show the meshes. So there you have it, this is a technique of how you can create a similar experience to what they do in avatar creation of the sims game.
Here is the github project that include code for all the three examples we did:
If you have any questions or are stuck on a step feel free to connect on LinkedIn and ask.
Like to support? Scan the QR to go to my BuyMeACoffee profile:
What you can do next?
BabylonJS is a huge library and there is a lot of learning left. We’ve barely scratched the surface yet. Below are some of the ideas for you to practice:
- Find the challenge I posted after the horse example, and create that app
- Enhance the sims example, and move the camera when say a user is trying to change the hairstyles, zoom in on face if they are changing the shirt zoom out and show full body.
Extras
Something the model is not just simple one .gltf file, it can have things in separate folders like this walking man file I downloaded form the internet has these in the folder:
Make sure all the files are inside a folder as it is downloaded or given to you by the 3D designer. If you open the gltf file in a text editor you can see it reference to Scene.bin and the jpeg in the textures folder. Hence it is important to have the files and folder as it is.
Then you can load the gltf file as you have been doing i.e. SceneLoader.LoadAsync(walkingManGLTFURL, undefined, engine) and it’ll automatically know where to look for scene.bin and textures folder. Just to test, try to move say scene.bin to some other place and you’ll see the model will fail to load.
Resources: Finding free 3D models
Below are some of the many websites to find .gltf or .glb files for free:
