AIPlayer Spinning[Solved]

Hi all. I'm nearing an alpha stage on a game I've been working on but there is one looming bug that just continues to elude me. This particular bug has popped up from time to time for the past at least 10 years in Torque history, and it still(apparently) exists. I was hoping some of the hard hitters around here might be able to help with squashing this guy once and for all.

The Goal

Alright, so first what I'm trying to do. I want the mouse cursor visible on the screen and the AIPlayer to follow it. I have this currently working, check!

The Problem

Stock Torque3D 3.10.1 with default Soldier model and datablock:

The bug arises randomly where moving the mouse will result in the AIPlayer's forward vector 'spinning' visually. If the mouse cursor is still, the AIPlayer runs along to it just as it should because there is no rotation being calculated for the AIPlayer. As soon as the mouse cursor deviates from it's straight ahead forward vector course, something causes the AIPlayer to visually 'twitch' or 'spin'. As soon as the rotation is complete and the AIPlayer is once again following a straight ahead path to the cursor, all is well.

This is all intermittent, as in it occurs sometimes and sometimes it does not. I have noticed that when this happens there is a visual 'hitch'(similar to what you'd see when a new material appears for the first time in a scene) and immediately the rotation is whack. It will remain in this 'state' of visual ugliness until I hit the space bar to make the AIPlayer stop. Once the AIPlayer starts moving again by the space bar toggle, the rotation is working fine again until it decides once again it is time to fail.

This occurs whether I am calling a schedule to move the AIPlayer or if I only call the movement command 'per click'.

Sources

Here are some examples of how this problem has been around and continues to be around since(probably) Torque's conception:

Bare in mind the old GG site is throwing up certificate errors for awhile now on some of their old forum posts.

From 2005: Here the problem seems to have been caused by the 'Eye' node in the player art.

http://www.garagegames.com/community/forums/viewthread/34046

From 2010: Not sure if it's helpful but this is an early case of the problem. It does seem similar to what I'm seeing now.

https://www.garagegames.com/community/forums/viewthread/122291

From 2011: (In this case the bug 'disappeared' yet a fix was never found.)

https://www.garagegames.com/community/forums/viewthread/124932

From 2011:This one is particularly interesting since it's marked as resolved yet I have implemented the code here to try to fix it and it doesn't fix it today.

http://www.garagegames.com/community/forums/viewthread/123597

From 2011: This is a resource that was made by @GuyA . I did implement this to try it, and it still works to this day but sadly doesn't stop the 'jitter'.

http://www.garagegames.com/community/resources/view/21188

From 2013: This one is by @Nils

http://www.garagegames.com/community/forums/viewthread/133374

As you can see this has occurred as far back as 2005. Of course, we all have our own implementations and what have you but this can be replicated with current Torque3D 3.10.1 without altering any code. Just make an overhead cam, spawn an AIPlayer, set the camera to orbit that player, and hook in some standard raycast code to tell the AIPlayer where to go. The problem will show itself eventually. I can provide any and all scripts if necessary :geek:

In the world editor you can spawn an AI test player and have him move around by clicking where he should go, that works just fine, what is so different in your implementation?

My AI players also do not spin, some rare occasion where I had them spin is when the point they were walking to was very high or very low from their position, seems like they don't like vertical so much.

Do you use the NavMesh for navigation or do you just send them blind?

PS: A video of them spinning would be helpful or at least funny.

In the world editor you can spawn an AI test player and have him move around by clicking where he should go, that works just fine, what is so different in your implementation?

Actually not much at all. As a matter of fact, I directly copied Daniel Buckmaster's code from his NavEditor to replicate his exact method of movement through the source code via a custom GuiControl in an attempt to eliminate the problem. It still persisted.

Do you use the NavMesh for navigation or do you just send them blind?

This is on a built NavMesh that does test properly in the NavEditor.

PS: A video of them spinning would be helpful or at least funny.

Perhaps one of the only statements you've ever made on these forums I agree with...

This issue has cropped up for experienced dev teams and renowned T3D developers for as long as I've used this engine. It is most certainly a closet case scenario that occurs infrequently only under certain conditions. What I am attempting to find, in the name of science, is the true underlying issue. Blanket statements like, "My AIPlayers work" aren't helpful in ultimately tracking this down. It is obviously in my implementation - but simply pointing a finger at it and calling it names likely won't make it go away.

Anyhow, moving forward, upon closer inspection of the movement code...the math does seem to point to a vector being rotated by extremely precise(i.e. 2PI) floating point numbers...or imprecise depending how you look at it lol. This is almost certainly the culprit, and I have to question why this was used in favor of a matrix rotation. Given a bit more time on the issue, it's more likely than not I'll end up scrapping all of that and just rewriting the rotation to suit. The problem with that is, no one else will know how I did it and the problem will remain unresolved in the future for other users of the engine. I am a strong supporter of sharing solutions around things like this, hence the entire 'open-source' engine idea in the first place. That's why I'm here on the boards with the issue providing sources for the problem and attempting to solve this as a community. It's this type of response that discourages 'community'.

TLDR - If you don't have something helpful to contribute, why speak up at all?

TLDR - If you don't have something helpful to contribute, why speak up at all?

Because I'm curious, I never saw this issue, so I wanted to see it.

My AI players also do not spin, some rare occasion where I had them spin is when the point they were walking to was very high or very low from their position, seems like they don't like vertical so much.

Because I'm curious, I never saw this issue, so I wanted to see it.

You have somehow managed to contradict yourself in the scope of two posts. Just mark this as solved, I'd rather just keep pecking at this and solving it on my own. I'm hot on the heels of it now anyway.

are you sanitizing your ray cast? It seems like the ray is still being cast after the mouse press or at least still being received by the aiplayer, check out richards rts tutorial series, moving ai position with mouse clicks is covered quite extensively in his tutorial series.

i think this is the link but i thought there was a later version somewhere

http://docs.garagegames.com/torque-3d/official/content/documentation/Scripting/Advanced/RTSPrototype.html

you did make it sound like it was constantly chasing the mouse which would likely cause confusion, but if its click to move there was a sort of fix some time ago that involved some form of deceleration at a certain distance from the destination point. Another alternative is to add a 'dead-zone' to the algorithm so that very near destination is ignored.

Allow me to contradict myself in the scope of two posts as well. The truth is I don't want to solve this on my own because I likely won't and one bad apple shouldn't spoil the bunch. That being said, @Duion did actually ask some valid questions. It doesn't stop me from equipping my Trollslayer +2 greatsword when I respond to his posts, but I'd hope we have a mutual understanding at this point. I acknowledge that I do have a tendency to proclaim things as fact prior to them being proven as such, and that just doesn't mix well with troubleshooting. I know that there are many minds far brighter than my own that have been doing amazing things with this amazing engine and I don't want that to be discredited by all of this being cast in a negative light.

With that all out of the way, I am going to need to post up some specific steps here for this issue to be replicated. Maybe try to get a Journal of the whole thing going on and post that as well. Maybe then it could be possible that community troubleshooting can actually occur. I mean, I am on here seeking guidance in the first place and I can't expect everyone to just write out all the scripts and what-not. I'll follow up here when I go through all the motions once again on a fresh build of 3.10.1.

Alright, I have managed to isolate this issue and reproduce it 100% of the time. Here is the relevant information:

After Respawning The Bug Is Gone

I noticed that if the AIPlayer dies, once it respawns the issue never appears again. I'm reading this as, "the respawn code is good, the initial load in spawn might not be". So I went through all of the initial spawn code with a fine-toothed comb and it actually mirrors the respawn code fairly accurately aside from the normal initialization of the new client stuff.

Only Occurs When Load-in Hitch Happens While AIPlayer is Rotating

So what I've noticed is this will ONLY occur within the first 5-10 seconds of a new game. We all know that Torque will hitch visibly when new objects or materials or what-not are loaded in for the first time. Basically what I'm observing here is a single hitch within that first 5-10 seconds, and within that tiny frame of time it's enough to bork the interpolation of the AIPlayer rotation.

Edited to add: Actually once that initial AIPlayer has been corrupted, it can potentially spin again even after stopping movement and resuming movement. Stopping will temporarily resolve it, but it will occur if instigated by things like impulses etc...UNTIL a respawn. After that, boom, no problems the bug disappears.

Here is why it's so hard for others to notice it and it's not evident in the NavEditor, or at least my hypothesis. If you click only, the AIPlayer generally is just going to walk forward with no rotation happening. Hence the AIPlayer is not attempting to calculate that yaw at the precise moment the initial load hitch occurs. However, if you click off to the right side of the screen and then click off to the left in quick succession...and the AIPlayer happens to be rotating to turn around at the precise moment an intial load hitch occurs...this can still happen when just directing the AIPlayer by way of a click. Additionally, if you are within the editor and working with the NavEditor gui, by that time all of the intial load hitching has already happened.

What Next?

So I'm not entirely sure what to do in order to combat this. I remember an old datablock preloading resource; I wasn't certain if that might help to alleviate the hitching but I doubt it. As far as I can remember, Torque has always had those little hitches when loading stuff in for the first time and I believe up-front datablock loading is just cutting back on load times for the level.

I had the notion of perhaps running a separate thread to handle the passing of 'AiMove' commands to the server, but I hadn't the slightest idea on how to get that rolling in Torque. I mean, the movement script is straightforward as it could be - Every 30ms or so grab the position under the cursor and send it to the server like so:

schedule(30, commandToServer('AiMove', %xyz)); // I have tested this schedule speed as slow as 500ms and same result

The move code feels solid, always smooth after respawning. I'm not entirely sure if this is an initial ControlObject issue or AIPlayer rotation...The initial ControlObject is the camera, and it remains that way during runtime.

Any suggestions are welcomed. I'm still open to posting up some scripts but tbh with the understanding of how/when it occurs it should be fairly easy to replicate.

Why don't you use the respawn function also for the initial spawn? I only have one respawn function that is used for everything, anything else would not make sense in my case anyway.

hmmm if it is only happening on the initial spawn, is it possible the initial spawn is loading some fps controls on that initial character?

Are you loading the same datablock on the initial spawn as you are on the respawn? Sounds like the initial spawn is loading in a player controlled character and the respawn is loading in a purely ai character.

Check the initial spawns datablock in the editor and see if there are a few dynamic fields

My guess is that this is going to be an issue deeper in the engine, you were probably on the right track looking at the yaw code. The fact that it happens during the initial spawn makes me think it's an issue with the way yaw data is initially networked & interpolated.

Is there any change if you disable warping?

// Set these to 1 in aiPlayer.cpp
static S32 sMaxWarpTicks = 1;          // Max warp duration in ticks
static S32 sMaxPredictionTicks = 1;   // Number of ticks to predict

@Duion : You are right to suggest this. I totally did try that as well yet the problem persisted.

@marauder2k9 : It's a good idea, and something I hadn't thought to check, but looking over it all I've got the global variables...

$Game::defaultPlayerClass = "AIPlayer";
$Game::defaultPlayerDatablock = "AIPlayer";

...and spawning in using the same Sim::spawnObject() consolefunction:

%player = spawnObject($Game::defaultPlayerClass, $Game::defaultPlayerDatablock);

@Happenstance : I found the static variables enherited from Player, so ended up changing those in player.cpp to 1. The visual effect of this is the spin occurs immediately and is much more noticeable. If someone's trying to replicate this that'll likely do the trick =). Setting these values to 1 I don't have to wait for the AIPlayer to be rotating while an early loading hitch occurs to produce the bug, the AIPlayer will display the unwanted behavior immediately. However, it does allow for a good look at it. With the warping disabled, it is easy to see that the issue isn't happening if the AIPlayer is walking on a straight forward vector but interpolates incorrectly as soon as a rotation occurs.

Speaking of looking at it, I will get a capture of this and post it up here as soon as I can so folks can see it.

Thanks for the help on this everyone, it's been a rough haul and since this particular bug has persisted since early in the game's development I figured it did warrant a bit of scrutiny.

This is probably a stupid solution, but at game launch you could kill them all and instantly respawn them.

Naw, it's the closest thing I've currently got to a solution without attempting to repair some confusing interpolation code and netcode that inherits from another class. I feel like, "If it looks good it is good". I guess when you really boil it down to its essence, any game is just smoke and mirrors =)

Here I've got a colleague that 'might' have a handle on this, but if he doesn't I've just been planning to do something similar to what you are proposing. Perhaps the initial spawn in and then respawn might not trigger the load hitch though, so I'm brewing up a few other plans around that.

At the end of the day, I'd love to see this closet case hammered down for future users of the engine regardless. I couldn't grab a vid of it today, just too much other stuff going on, but should be able to get to it this weekend.

Tinker with those warp settings as well - try setting them to 64 or even 128 and see how that changes things.

My suspicion is that the AIPlayer's yaw handling is breaking in situations where the client & server desync. The server says the AIPlayer's rotation is one thing and it should turn to this angle but the client thinks it's a slightly different angle. Once an update packet is received by the client the rotation is synced back to the server's version and the change in rotation is interpolated over several ticks. The difference in angles could potentially be enough to result in a wildly different return value in that bit of code that calculates the shortest turning distance (see comment in aiPlayer about "shortest way around a circle"). You'd see the AIPlayer spin from one extreme to the other as it corrects the rotation if the difference was large enough.

That's my shot in the dark theory, anyway.

@Happenstance Yep, you are right

I think we have this solved. My colleague was able to discover my shortcoming around this issue. What happened here was when I implemented the resource by @GuyA posted above, I hadn't implemented the adjusted warping code from the other resource by Ivan. I tried each resource separately, while Guy explicitly had said that Ivan's code would be necessary for his resource to work. I was wondering why I had used that resource numerous times in the past and it worked. This is why I could still get the turn rate working but the warp was still broken.

When I revisited this all earlier this morning, rather than create a new AIPlayer datablock as in the resource I added the maxTurnRate stuff to the player.h and player.cpp code as suggested in the comments of Guy's resource. Basically adding only the relevant warp change to AIPlayer in getAIMove(). So far, this all equates to a fix here for me. I hadn't been able to replicate the issue since.

To Note

Stock Torque presently does suffer from this closet case. Without modification of the warp code and that limit on turn rate, these issues can still crop up. I noticed that some of the existing code does include similar changes to what Ivan had originally proposed. However, if it isn't used in conjunction with Guy's maxTurnRate limiting warping can be broken in certain cases.

Damn, it popped up again. I believe existing AIPlayer rotation warping is not providing the level of functionality that it should.

I fixed this, but it wasn't pretty...

I ended up having to completely replace the AIPlayer class with an older version from around 2015/2016. There have been several pose, navigation, and AFX-animation additions since that time. Something broke the rotation warping, unfortunately, so I was forced to revert.

No guarantees, since I haven't been hitting that specific issue, but a couple things of note:

https://github.com/GarageGames/Torque3D/blob/development/Engine/source/T3D/player.cpp#L6371

+

https://github.com/GarageGames/Torque3D/blob/development/Engine/source/T3D/player.cpp#L6477 really should probly read signed there. beating that one up a bit before a PR to fix it.

and if it is indeed a wrap issue, did add a pair of https://github.com/GarageGames/Torque3D/blob/561f010f2e6411d8253d23f0cfcff794e81f60bf/Engine/source/math/mMathFn.h#L240-L251 you can give a shot, see if that helps.

@Azaezel

So nice of you to post this up on Christmas. I had suspected something getting lost over the wire, but I took it for granted the Player class was already passing along a rotation value. Chances are if that didn't exist already, the older versions I've reverted to didn't pass it along either.

As of yet I hadn't been able to replicate the issue using the older versions of AIPlayer/Player, but I will definitely give this a shot using the current versions as well. If I had to pick between the two, I'd prefer the newer version of AIPlayer because all updates have been useful. Still can't believe I hadn't thought to check if Player was passing a rotation variable across the wire, as essential as rotation is to my specific implementation. Perhaps it wasn't as necessary prior to the AIPlayer datablock and additional poses being added.

In any case, with this bug stomped out we've reached alpha with our current project so still plenty of time for updates/improvements(internally we term 'alpha' as bug-free, concept fully functional, working on visual polish and added features).

I'll post back here when I get around to plugging this into the newer AIPlayer class and running it through the gauntlet. For science!

@Azaezel

I finally found the issue. Bear in mind this is using T3D v3.10.1. Actually the pack/unpack methods already included the rotation.

Since I had deduced that the older version worked, I went through the motions of plugging in the updates one at a time to find the culprit. It ended up being this:

In aiPlayer.cpp:

void AIPlayer::updateMove(const Move* move)
{
   if (!getControllingClient() && isGhost())
      return;
 
   Parent::updateMove(move);
}

Removed that method from the source and issue is gone. It appears all this really does is make a check for controlling client and ghost, then calls the original Player class' updateMove(). In my case the camera is the control object, so this here breaks the move or some such shenanigans for me I guess. Anyhow, issue resolved, and I didn't have to revert after all. Glad you posted something about it, since I woulda just carried on using the older version had I not took another peek :lol: At least now we've got it cornered.