Convai

Real-time AI Agents in Games and Simulations.

[Convai @convaitech]

Introducing Scene Tagger from Convai for @UnrealEngine

Large 3D environments can contain hundreds or even thousands of objects.

For an AI system to meaningfully understand that environment, those objects need more than mesh names and transforms. They need useful names, descriptions, context, and structure.

That is what Convai Scene Tagger is designed to help with.

In our latest Unreal Engine tutorial, we take an entire museum environment and use Scene Tagger to analyze the level, identify significant objects, generate names and descriptions, review and refine the results, and apply that structured information back into the project.

The workflow also allows developers to:

1. Use a Convai character’s AI configuration during analysis

2. Add Knowledge Bank content for additional context

3. Generate an environment description from the current view

4. Analyze selected actors or an entire level

5. Review and correct generated annotations

6. Inspect scene information through the debug view

The museum is simply the example environment here. The same approach can be useful anywhere an AI character or system needs structured information about a 3D world, including games, training environments, guided experiences, simulations, and other interactive spaces.

The result is a faster way to turn a raw 3D environment into structured scene information that can support AI-based development and runtime execution.

#Convai #UnrealEngine #SceneTagger

[Convai @convaitech]

One of the coolest applications of Convai we’ve seen in a legacy game like chess.

Chessbuddy uses real-time board awareness, dynamic context, and conversational AI to let players actually talk through positions, mistakes, and ideas with an AI coach as the game unfolds.

Huge shoutout to @AlystriaAI for building this.

#Chess #AICharacters #Convai

[Alystria AI @AlystriaAI]

Meet Chessbuddy! Four AI chess coaches you can play with and talk to in real time. They follow the board as you play, so you can ask about the position, talk through ideas, understand mistakes, and review important moments after the game. It’s free to use.

Try it here: https://t.co/7W6L2cxxlc

Powered by Convai.

[Convai @convaitech]

@UnrealEngine Watch the full tutorial on our YouTube channel: https://t.co/DcDg2DiDHx

Also, Check out the detailed Scene Auto-Tagger docs for Unreal Engine: https://t.co/WHqfyseneD

[Convai @convaitech]

Introducing Scene Tagger from Convai for @UnrealEngine

Large 3D environments can contain hundreds or even thousands of objects.

For an AI system to meaningfully understand that environment, those objects need more than mesh names and transforms. They need useful names, descriptions, context, and structure.

That is what Convai Scene Tagger is designed to help with.

In our latest Unreal Engine tutorial, we take an entire museum environment and use Scene Tagger to analyze the level, identify significant objects, generate names and descriptions, review and refine the results, and apply that structured information back into the project.

The workflow also allows developers to:

1. Use a Convai character’s AI configuration during analysis

2. Add Knowledge Bank content for additional context

3. Generate an environment description from the current view

4. Analyze selected actors or an entire level

5. Review and correct generated annotations

6. Inspect scene information through the debug view

The museum is simply the example environment here. The same approach can be useful anywhere an AI character or system needs structured information about a 3D world, including games, training environments, guided experiences, simulations, and other interactive spaces.

The result is a faster way to turn a raw 3D environment into structured scene information that can support AI-based development and runtime execution.

#Convai #UnrealEngine #SceneTagger

[Convai @convaitech]

Watch the full tutorial on our YouTube Channel: https://t.co/haiFEo2iqf

Multi-Character Sessions documentation: https://t.co/bfGI7Uq94t

[Convai @convaitech]

Introducing multiple AI characters in a single environment!

Most 3D experiences are not one-to-one conversations. Games can have multiple characters with different roles. Training simulations can have instructors, assistants, supervisors, and trainees sharing the same environment. Virtual worlds can have several characters the user needs to interact with naturally.

Once those characters are powered by conversational AI, there is one important problem to solve: How does the system know which character the player is actually speaking to?

In Part 1 of our Multi-Character Sessions tutorial for @unity, we show how to handle this using Convai.

We place three different AI characters in the same scene, each with its own Character ID, then configure how Convai determines which character should receive the player's interaction.

There are three selection methods:

1. Look At - selects the character based on where the player is looking.

2. Proximity - selects the character based on its distance from the player.

3. Manual - gives the developer direct control over character selection.

For the demo, we use Look At, allowing the player to speak with Sophia, turn toward Ethan for his perspective, then continue with Marcus naturally within the same scene.

In Part 2, we’ll expand this further with all-to-all multi-character awareness, where players and AI characters can stay aware of nearby conversations and events, and characters can react or engage when something is relevant to them.

#Convai #Unity #AICharacters #EmbodiedAI #MultiCharacterSessions

[Convai @convaitech]

Watch the full tutorial here: https://t.co/cmAD1s9Xto

In case you missed the previous part, watch it here: https://t.co/yBOvv8xM1I

[Convai @convaitech]

In Part 1, we showed how Embodied AI Characters can use Dynamic Context to understand live game state, tagged objects, pressure plates, moving platforms, and built-in actions inside Unreal Engine.

In Part 2, we take the next step: connecting AI characters to custom gameplay logic.

In this @UnrealEngine tutorial, the human player and a Convai-powered AI teammate return to the Stack-O-Bot puzzle. But this time, movement and scene awareness are not enough.

The AI needs to interact with a crate, pick it up, carry it to a destination, and place it on a pressure plate to help solve the puzzle.

This is where custom actions become essential.

Instead of limiting the AI character to generic behaviors, we define project-specific actions like Pick Up and Drop At, pass scene objects as parameters, and connect the AI's intent to existing Blueprint logic.

The tutorial covers custom Convai Actions, Actor Reference parameters, Action Handlers, movement before action execution, success and failure reporting, and integration with the Stack-O-Bot interaction system.

The result is an AI teammate that can reason, act, fail, recover, and coordinate with the player.

For teams building games, simulations, XR experiences, training applications, or interactive 3D worlds, this is a practical look at how AI characters can move beyond conversation and become active participants in real-time environments.

#Convai #UnrealEngine #AI #GameDev

[Convai @convaitech]

Watch the full tutorial here: https://t.co/0sosED1gEm

And check out the detailed actions documentation for @unity: https://t.co/yWBHiHpXMc

#Convai #Unity #AICharacters

[Convai @convaitech]

Player: "It's quite dark in here. Could you turn on the lights?"

AI Character: "There. The workshop lights are on."

The player didn't click anything. No menu, no hotkey. They asked in a sentence and the character did it.

That gap is bigger than it looks. A traditional NPC is an information kiosk with a face. It can tell you where the assembly station is. An AI character, on the other hand, guides you there, turns the lights on when you say it's dark, and resets the bench before it explains anything. One describes, the other takes actions.

Convai Actions in Unity comes down to three pieces. Add the Actions component and pick from the built-in set: Follow the Player, Walk To, Look At, Show or Hide Object, Play Gesture. Register the scene objects your character is allowed to touch, and give each one a name and a description. Write that description the way you'd describe the object to someone over the phone, since it's what your character reasons with when you refer to something.

That widens what you can build. An onboarding character stops narrating a map and walks new users through the space, showing them each thing instead of pointing at it. A training instructor sets the equipment up, demonstrates a step, then resets the bench when someone wants another attempt, which turns a walkthrough into something closer to practice. A companion in a game answers "follow me" or "wait here" without a command wheel in the way.

Custom actions are in the next tutorial. Which one are you hoping is on the list?

[Convai @convaitech]

@BangMingYong It’s online. Convai’s AI processing runs in the cloud, while your Unreal project can feed live game context, such as Blueprint events and changing state values, to the character so it can reason and respond based on what’s happening in the scene.

[Convai @convaitech]

Our AI shooting instructor called out the target the player kept avoiding.

"Four hits out of five shots. Not bad, but you’re leaving that 15m target untouched."

We didn’t write that line anywhere. The character read a Blueprint variable and drew its own conclusion.

Three layers of world awareness in Unreal:

MANUAL. You send the fact yourself.

Add Context Event hands the character something that happened instead of a line to perform. "Player picked up the gun." Three flags control what it does with that: whether it answers, whether the event persists, whether it interrupts. For running values like shot count we used Set Context State with Should Respond on Never, so the number updates without the character narrating every trigger pull.

AUTOMATIC. It works the rest out on its own.

You drop a ConvaiObject component on the actor and give it a name and a description, and there’s nothing else to configure. The character now knows where that object sits relative to itself, to the player and to every other tagged object, plus whether it’s moving and which way. Tick Gaze Attention on the player and it tracks what you’re looking at.

"The 20 meter target is the one on the move. It is drifting to your right. Try to lead your shots a bit."

We never flagged that target as moving. There’s no isMoving bool anywhere in the project. It’s reading the transform.

SEMI-AUTOMATIC. Point it at a variable.

Add a Tracked Property, pick a Blueprint variable, done. Ours points at HitCount on each target. We left Should Respond on Never for the static targets and switched it to Auto on the moving one, which hands the character the call on whether a hit is worth mentioning.

That’s where the line at the top came from. A hit count sitting at zero on the 15m target was worth flagging, and it said so without being asked.

Hit Ctrl+Alt+K and you see what the agent receives:

20m target: "some distance away, in front of you, moving to your right, and in front of the 15m target"

Player: "close by, in front of you, facing away from you"

Those sentences come out of the transforms, live. We didn’t author them.

A shooting range is the cleanest way to demo it, but the same three layers cover any character that has to respond to what’s happening around it. Training sims that check whether the trainee followed procedure, digital twins where the equipment state moves underneath the conversation.

Anyone tracking NPC world state a different way? Blackboard, custom event bus, something homegrown?

[Convai @convaitech]

Full Unreal Engine walkthrough, all three levels, ~15 min:
https://t.co/iH8iU6DY7A

[Convai @convaitech]

For teams building games, simulations, XR experiences, training applications, or interactive 3D worlds, this is a practical look at what AI characters can become when they are connected to real environment state and action systems. (6/6)

#Convai #UnrealEngine #AI #GameDev

[Convai @convaitech]

Convai is enabling Embodied AI Characters with Dynamic Context with game state input, which enables these characters to understand their environment and execute complex actions while interacting with other players. (1/6)

[Convai @convaitech]

This is a step-by-step teaching tutorial designed so developers can follow along from scratch with the provided project files, Delta files, setup guide, and supporting resources (check out the comments to learn more). (5/6)

Watch the full tutorial here: https://t.co/yBOvv8xM1I

[Convai @convaitech]

Watch it on our YouTube Channel: https://t.co/XWfCmoi4GR

And consider subscribing to see AI characters in action across @unity, @UnrealEngine and Web Projects!

#Convai #WebSDK #ReactThreeFiber #ThreeJS #AICharacters #WebDev

[Convai @convaitech]

Bring 3D conversational AI characters directly to the browser.

Our new tutorial shows how to use the Convai Web SDK with @claudeai and @reactjs to create a browser-based AI avatar with real-time conversation, custom GLB avatars, and lip sync using blendshape data.