Work Blog - JeffR

Re: Work Blog - JeffR

Duion
Posts: 1401
Joined: Sun Feb 08, 2015 1:51 am

Torque3D has been fully functional for over a decade now, no need to wait for 4.0, just saying. The only major visible difference will be PBR and you can use the development branch, if you want that. 4.0 as I understood it has some major redesigns, so it will likely take even longer after it is released until everything works stable.

PS: There are no open source alternative engines to Torque3D, except Godot maybe, but it is only designed for small scale 2D/3D mobile game types.

Re: Work Blog - JeffR

JeffR
Steering Committee
Posts: 939
Joined: Tue Feb 03, 2015 9:49 pm

Mildly delayed workblog is a go!

So, as you all know, we kicked off a pre-alpha tester build of 4.0, and a ton of you guys have been testing and giving excellent feedback. Not only has this lead to the pointing out of bugs, as was expected, but also general design/implementation weaknesses/faults that hadn't really come up until you guys started actually laying into it.

So July was a bit of a funky month so I didn't get as much done as normal, but there was some progress made and August should go much, much faster progress wise. So below lets get into what all was worked on, and then what all is left to do!

Preview Build July work
So, the short version is, that since the last workblog post, we've had 28 commits fixing quite a bunch of stuff. Various PBR issues like probe indicies not updating when probes are deleted, re-org'ing the cubemap loading code on probes for consistency, and fixing up and making the forward pass version of it all have almost complete parity. Mainly for translucent materials, but we'll be able to support full PBR in forward rendering going forward too, which is always handy.

Outside of that, I've been keeping a list of all the stuff everyone's been noting problem-wise and been checking it off as it's been cropping up. I'll get the list posted over into the dedicated preview build thread for reference later, but it's definitely been helpful in pegging down the problem spots.

I've been trying to finish out the themeing work in the editor, and while by and large most of it is converted over now:

There's still some bits to fix up, a few improvements to be made from feedback, and get the verve editor updated to hook into the theme settings as well.
Still, big progress on that end.

I also fixed up a lot of asset path initialization, so they use 'loose files' now, so assets only need the name of the file in the same dir as the asset def, and it figures out all the pathing automagically. This makes moving assets around much, much easier.

From there, we started hammering on the module/asset stuffs more in earnest. I started with shifting the importer config settings stuff to use the already existing Settings class. This is nearly done, and will make it much more consistent and much less code duplication, so that'll be much nicer to work with.
I also shifted new code asset creation to utilize template files, instead of generating the text for new script files via code in the asset browser scripts. So it's much more standardized and streamlined.
We also realized with Lukas' c# stuff that we could have an editor setting that would pick between TS and C# template script files, so that'd make using either mode much more consistent and straightforward.

Most recently, me and Az realized that while the existing module loading code worked, it was fairly cludgy, especially with the networking considerations. Having clear, comprehensible separation between client loading code and server loading code wasn't really a thing, so it was easy to mush it all together and have it break.

So, I implemented a system for separation of how modules load in each case:

We now have split calls that happen when:
The server is initialized
The client is initialized
The server is created or destroyed as part of the game-server starting up/shutting down
The client is created or destroyed as part of a client connecting or disconnecting from a server

This allows us to exec scripts just during setup, but stuff that may need to be ensured up to date, such as datablocks, can be loaded only when the game actually starts.

It worked like a treat, and I had no problems getting a local multiplayer game working:

So, just have to do cleanup and make the other modules properly utilize this approach.

Tying off of this is the new code for handling Game Modes.

In the full template, it handled gamemodes via some awful, cthulhu-esque mess of overriding packages, namespace overrides and inheritance voodoo and the like to create an indirection/rerouting nightmare so you could end up in the right namespace for running of gamemode functions.

Since that's objectively awful at almost every level, I wanted to do a much more clear approach, which is what I hammered out this weekend.

The new way is much more clean, less bloat, more direct, but also much, much, much more flexible, because it allows you to utilize Scenes and subscenes to invoke gamemodes in a seamless way, and gamemodes can exist via modules in a simple, drop-in sort of way.

Game Modes and You
So how does this new setup work?
Lets start with how gamemodes are actually run. We have, effectively, 5 functions of interest
• onMissionStart - This is called when the server finishes loading the level, and activates the Gamemode(s)
• onMissionEnd - This is called when the server concludes the game, either due to gamemode logic, or it's shut down
• onMissionReset - This is called when the level is told to reload, it effectively cleans up then kicks off onMissionStart again
• onClientEnterGame - This is called when a client enters the server's game
• onClientLeaveGame - This is called when a client leaves the server's game
Pretty self explanatory, really. In the gamemode code, these are used to set up gameplay rules like kill counts, game timers, player spawns and loadouts, and basically everything you'd actually use to implement gameplay beyond 'clients have connected to the server, which has loaded a level'

As said, before now it was a gigantic mush of hard to track code, so the new approach is much simpler.

When the server finishes loading a level, the level naturally contains a Scene and potentially SubScenes. I added a new field to Scene: GameModeName.
If this is filled in, it'll do a eval check and try and invoke the onMissionStart call for the gamemodes for any of the scenes that have it defined.

This is useful, because it lets you, say, have a common base level Scene, but then have subScene level assets that you can load for each given gamemode.
the CTF subscene can have the flag locations and modified spawns, versus the deathmatch subscene won't have those flags, and may have more weapons and randomized spawn locations.

So if we load our Scenes in, we roll through and find any scenes with a gamemodename defined and call the above functions to kick off said gamemode behavior.
If no scenes have any gamemodenames defined, then we defer to the Project Settings, which has a DefaultGameMode field. If that is also empty, then and only then will it just spawn the clients in as spectator cameras, with no gamemode logic in effect.

This is much cleaner and simpler, but also much more flexible and powerful. It would even let you combine gamemodes if you wanted, which could get pretty wild!
I already successfully tested this in the FPSGameplay module, where one level(as in the above screenshot) spawn in players with full kits for basic deathmatch gameplay, and other levels have nothing defined, so you just spawn as spectators.

The plan is to have FPSGameplay offer a few different gamemodes so you have a solid example of how to lean on this, but it's all looking incredibly more streamlined and easier to get than the old way, so I'd call that a win already!

So what's next?
Outside the aforementioned bugfixes and corrections people have spotted. I still have to finish the overhaul of the BaseGameUI, finish breaking down the FPSGameplay into it's properly separated parts, finish the Das Boot test level so we have a proper stress test environment again, and flesh out the component/game objects stuff.

I'd hoped to get most of this done in July, but as said, circumstances arose to make that untennable. But progress, as you can see, hasn't exactly been bad, either so I've very high hopes for getting most everything wrangled in August.
I'll have a build up soon with the FPSGameplay module effective in it so multiplayer testing can be leaned on while I try and wrap on the UI overhaul so it's overall an easier build to work with, but I believe that's the bulk of the update for the moment(as usual, I feel like I've forgotten some bits, so if I remember anything else, I'll definitely update )

Until next time!
-JeffR

Re: Work Blog - JeffR

Chelaru
Posts: 217
Joined: Wed Jul 01, 2015 10:33 am
Hey JeffR, I know you are busy but i would love to see some new shine images.

Re: Work Blog - JeffR

JeffR
Steering Committee
Posts: 939
Joined: Tue Feb 03, 2015 9:49 pm

Well, great news! Because it's happening!

In reality, it was supposed to happen monday or tuesday, but Dad's birthday I totally spaced on + some shadergen issues shenaingans put a small delay on that. But I attended the former and fixed the latter, so here.
We.
Gooo.

So, unsurprisingly, a lot of changes, updates and fixes have been going into the preview build. A number of solid contributions from the fine testers putting forward their own improvements as well, so I'll go ahead and walk through most of the changes. It total I've got commits/notes for the following:
• Adjusted init'ing logic to defer to project settings for the name of the play gui and mainMenu gui instead of hardcoding them
• Added D3D error code interpreter
• Added sanity check for findMatch
• Finished most of asset importer logic to utilize settings system
• Cleaned up code for finding associated files
• Added General importer settings category and integrated logic for those settings fields
• Updated logic in variableGroup to support callbacks in custom fields
• Updated logic in variableInspector to better handle callbacks, as well as being able to manually update when manipulating fields
• Updated scripts to utilize project settings values for playGUI and mainMenuGUI names
• Added util function for populating custom fonts
• Force proper update of box probe when attenuation changes
• Fixed multiple case sensitivity issues
• Adds GL ARB check for cubemap arrays for certain cases where the driver only supports the extension, not as core
• Fixes undeeded namespace declaration in an inline operator function
• Cleaned up/reordered some modules default in the data dir
• WIP of Das Boot test map
• Begun refactoring visualizer tools
• Added default cloud and water textures to core/rendering so water and cloud objects work correctly
• Added default Fog_Cube mesh so volumetric fog works correctly
• Update to SDL2.0.10
• from @rextimmy: appends vcruntime as a release dependency when using visual studio versions exceeding mark 1920
• linux followup work by tim
• leftovers from when we added explicit 16 bit greyscale texture support
• from @rextimmy - linearcolor set and = operators
• Changed findModule default version value to 1 to match default module version number
• Added running of onCreateGameServer and onDestoyGameServer callOnModules for core modules as well to enable default datablock stuffs
• Added default datablocks, materials and shaders for some base required objects such as spawn markers, ribbon particles and the like.
• Includes ExampleModule to act as a template/documentation case for how modules should be set up
• Includes GameUI module so there's a PlayGUI gui object by default.
• Makes the graphics menu in the stock UI mostly functional again
• Adds sanity check to editing of gameasset script action in asset browser
• Fixes checking of popup menu items
• Caught the main modules up to new module script file paradigm
• Have initServer only get called when a server is actually started for the first time
• Added utility method callGamemodeFunction to streamline exec'ing of gamemode function calls
• Added onClientConnect gamemode function, called when the client establishes the connection
• Added onInitialControlSet gamemode function, called when the client actively gains control in the game, to allow gamemodes to special override GUI settings or input commands
• Cleaned up init'ng of the engine so stock UI module isn't required to trip configuration of the canvas
• Added fallback so if nothing set the main content control for the UI after boot, we attempt to set the defined mainMenuGUI, on the chance nothing explicitly sets it on load
• ForcedMaterialMeshMgr tweaked to allow proper setting of override material
• Ongoing WIP of update/fixing of options menus
• WIP of expanded visualizers, including material complexity viz
• Adds no-pie flag when compiling on linux with non-clang compilers
• from user Ewyncat: adds invcameratrans, cameratoscrren, and screentocamera postfx and shader matrix interfaces.
• also a bit of backend work to normalize those later once we've proof-of-concepted the calcs
• forgot the initializers and shader variable to source hooks
• Implemented proper ScriptAsset execution on load
• Implemented script dependency handling
• Added test-case of script dependency handling in ExampleModule
• Cleanup of redundant getSceneCount calls
• Properly get scene count in callGamemodeFunction
• Remove unneeded TODO comment in shaders
• Converted onMissionEnded gamemode func call to use callGameModeFunction function
• Convert ExampleGameMode to be container-object based, and updated callGamemodeFunction to handle that
• Correct import settings typoe so image suffixes are read correctly
• Largely fixed companion image scanning when importing images and streamlined image-material interop during import preprocessing
• Added handling for reading in PBR maps and creating a composite image + asset
• Added WIP of Cubemap asset, and editing integration with a standalone cubemap editor
• Added ability to create new Cubemap asset in Asset Browser
• removed and moved files for baseGame
• removed templates/baseGame/game/data/art as it's a copy of /images
• removed templates/baseGame/game/data/scripts/guis as it's a copy of ../gui
• moved fileDialog.cs to /ui/gui
• remove duplicate scripts under /ui/scripts that are in /ui/gui already
• moved guis to where they're called in script
• kill duplicate gui as a source of correction confusion
• pad the internals of the GuiScrollCtrl encapsulated within the EWCreatorWindow/[CreatorWindow] to prevent overhang issues due to the parent rescaling it's position/extent
• Utilizes old method to clean up tree items in GuiTreeViewCtrl to resolve guiEditor crashes
• Updates probeManager to correctly utilize skylight idx to infer if a skylight is active, and set the SKYLIGHT_ONLY macro correctly, resolving erroneous behavior
• Cleaned up some of the guiEditor initialization script to remove redundancy
• looks like the with the latest translucnency work, dynamic shadows are no longer triggering malformed values in forward lit materials, so flipped that back on
• Moved unneeded modules to Templates/Modules
• Added templated getObjectsByClass to Scene for easier engine-side polling of objects, including nested checks for subscenes
• Proper init'ing of mGamemodeName in LevelAsset, as well as proper fieldType for mIsSubLevel
• D3D11 added logic to handle scaling down of textures in cubemap arrays for lower texture resolution preferences
• Added ability to collapse groups programmatically to GuiVariableInspector
• Upped PSSM shadowmap max size to 4096
• Caught GL deferred lighting/probes up to D3D11
• Temporarily disabled forward lighting/probes on GL materials until conversion finished
• Upped smMaxInstancingVerts to 2000 from 200 to support slightly more detailed meshes being instanced
• Reordered project settings so they load ahead of core modules, so that core modules can actually use project settings
• Established current preset file for PostFXManager to use for reverting
• WIP logic for forcing probes to update as part of level lighting load step in loading process
• Streamlined PostFXManager code, removing unnecessary/redundant files
• Coverted HDR, Lightrays and SSAO and ExamplePostEffect to use new PostFX Manager/Editor paradigm
• PostFX manager now enacts callbacks so that postFXs' can process their own settings as well as provide editor fields
• Changed PostFX editor to work with new callbacks via using VariableInspector
• Updated PostEffectAsset's template file so new PostFX's will now automatically register with the PostFXManager and have the needed new callbacks for integration
• Made HDR on by default, removed enable field from editing
• Made probe bake resolution a project setting
• Updated many GL postFX shaders to have proper case for PostFx.glsl
• Example module now loads ExampleGUI and ExamplePostEffect during init'ing
• Removed unneeded autoload definitions from ExampleModule's module file
• Fixed Graphics Adapter settings field to properly display as well as apply setting
• Updated many referenced profiles in tools folder to use the Tools specific gui profiles to make theming more consistent
• Fixed coloration of tools button bitmap to make theming more consistent
• Updated a few theme settings for improved visibility with theme, particularly selected/highlighted text
• Moved AssetBrowser field types to separated folder/files
• Updated new module creation to properly utilize template file instead of overriding it with a programmatic script generation.
• Removed unneded default autoload definitions from new modules
• Added WIP for editing Module/Asset dependencies
• Updated the PostEffectAsset to properly generate glsl and hlsl files from templates
• Updated module editor window to display only necessary fields
• Updated ToolsGuiTextPadProile to comply to editor theme
• Further tweaks/reorg to UI module
• Implemented cubemap array scaling on GL so it doesn't crash when texture quality is lowered
• Re-enabled ExampleModule's keybinds
• Added check for when getting the Detail feature's output, to validate if we have the foliage feature, which was causing a stomp on the required float4 texCoord for the foliage featuer to work right
• Temporarily disabled wsNormal addition when we have no defined normal map until it's finished being integrated
• Fixed the structure for the lighting/probe shadergen logic in GL so it generates correctly, resolving a crash on unix machines.
So....yeah, a little bit of work has been done since the last workblog(honestly, probably didn't need the full list, but hot diggity it's worth taking note of!)

So what's the highlights? Where's the shineys?
Good question. A lot, and interspersed with my ramblings about the highlights of course!

Now,a number of things are code related and not super simply to convey(which is why we have an example module to better document and convey all the nitty-gritty stuff. You can have a peep at it here: https://github.com/Areloch/Torque3D/tre ... mpleModule)

The adjustments to module and gamemode logic are particular standouts though.

I'd gone over the initial take on it last update, but in the interest of more consistent behavior with module order of operations, we realized we needed to flesh out the execution stack beyond 'created' and 'destroyed'. So we end out with a number of new functions:
• initServer
• onCreateGameServer
• onDestroyGameServer
• initClient
• onCreateClientConnection
• onDestroyClientConnection
Now, largely, the names of those should inferr a few things, and the writing in the ExampleModule.cs file in the above link goes into each pretty in-depth, but the important thing to take away is a better separation for the various stages of setup(and teardown) for module loading, and when servers/client connections are created or destroyed.

Most notably, while the example is all in a single file, this makes it *miles* easier to have projects completely separate the client a server by just having a client and server module with the appropriate assets, scripts and use the appropos above functions for handling. Then when packaging your game up, just ditch the module you don't need for the given build.

This will make it much, much, much easier for those who want to have separate builds for clients and servers without needing to manually chop-shop the template scripts to make it happen.

The explicit stages also makes loading/cleanup order much more deliberate, which we found got really important really fast when trying to ensure dependencies are handled properly.

Following on that, is changes to how GameModes are executed. In the full template, there was quite a quagmire with Packages, overridden functions and the like for how gamemodes are init'ed and run. This lead to a good bit of difficulty tracking code execution, and quite a lot of duplicated/bloat code doing the same thing multiple times for the sake of redundancy.

So, instead, we shifted to a much more clean, but flexible system for gamemodes. It even technically allows you to layer multiple gamemodes at once if you want to get hilariously over the top with it.

So, in your levels' Scene, there's now a field: GameModeName which - unimaginatively - infers the gamemode that scene wants to load up.
The level loading code iterates over all active scenes(hence how you could layer gamemodes if you inclined) and if the scene has a gamemode defined, kicks off the function calls to do the setup and other behaviors as defined by said gamemodes, if any.

If no gamemodes are defined by the scenes, it looks up the project settings for a default mode. And if that fails, it just farts you into the level as a spectator.

The example gamemode(exampled here: https://github.com/Areloch/Torque3D/blo ... eScript.cs

The reason for the scene integration by default is because it opens up a lot of flexibility. Because scenes let you load subscenes, this allows for layering scenestuffs for specific gamemode purposes.

For example, you could have your main level's objects and geometry, the stuff that doesn't change like, say, Chinatown in the primary level Scene.
But then you want gamemode stuffs for 2 different gamemodes: Deathmatch, which only requires random spawn locations, but then also CTF, which requires team-based spawns and also the flags and capture zones.

In the original setup, you'd basically have either two levels, or do some show-hide trickery with simgroups. But here, you could pick the gamemode in question, load the related subscene which contains JUST those gamemode specific objects, which would layer on over the regular level's stuff, and it has your Deathmatch or CTF gamemodeName defined, and thus causes the gamemode logic to be initialized and run.

In both the case with the modules work and the gamemodes, a number of utility scripts have been added to help easily kick stuff off in a clean way.

An example of this would be a call like this here:
if(\$missionRunning)
{
%hasGameMode = callGamemodeFunction("onClientLeaveGame", %client);
}

We're leaving the game, so we want to have the gamemode logic know it. the utility function smoothly does the iteration over the scenes for any gamemodes in effect and does the callthrough. Clean and consistent while still being really easy to expand on.

What else? I want pictures!
Fair enough. Well, begun putting together a new test map to replace Outpost - Das Boot. Still very WIP, as bugfixing has the priority, but ultimately, it should have basically every main gameplay object class in it and just about every main rendering effect we have in our stable. So it'll be really easy to test just about everything by firing that bad boy up.

You may also be able to note ongoing work to clean up/standardize the dark theme, which is getting pretty sexy now.

I also updated the visualizers. Instead of one big ol' dump list of the viz modes, which was hard to navigate visually, it now uses a normal popup window, with some expanded options, overall much more clean and compacted code to handle the viz behaviors, and space for further additions to the vizualizer modes:

It's not ready yet, but an WIP example of my testing with expanding the visualizers on offer was an initial foray into a Material Complexity Visualizer:

This was a very, very simple take, but the idea would be it'd take a 'complexity' value from each shader feature in shadergen, and then add 'em together. The more features and the more expensive a given feature, the more the total value, and the more dark and red the material gets displayed as in the complexity Viz. This would allow for an at-a-glance estimation of how expensive your scene is to compute in terms of shader/material complexity, which is where a LOT of rendering overhead goes. So it'll be pretty important to the optimization phase.
Another suggested addition down the line would be a UV/Text density Viz so you can see if there's anything in your scene with an out of place or irregular texel density, which can ruin the visual cohesiveness of the rendering.

Ok, this is getting pretty long, so I'll put the breaks on this here, and post PART 2 tomorrow! So much to cover!

And as a note, I'll have the next preview build for all 3 platforms uploaded with part 2 tomorrow, so you all can look forward to that in the meantime

Re: Work Blog - JeffR

Chelaru
Posts: 217
Joined: Wed Jul 01, 2015 10:33 am
It looks nice.I do have a rtx 2080ti if you want to test some stuff just ask.

Re: Work Blog - JeffR

Chelaru
Posts: 217
Joined: Wed Jul 01, 2015 10:33 am
Hmm raytrace torque 3D

Re: Work Blog - JeffR

marauder2k9
Posts: 318
Joined: Wed Feb 18, 2015 7:36 am
amazing work as always jeff!! looking forward to testing out the next preview

Re: Work Blog - JeffR

Steve_Yorkshire
Posts: 332
Joined: Tue Feb 03, 2015 10:30 pm

And if that fails, it just farts you into the level as a spectator.
Imagine the smell

Re: Work Blog - JeffR

JeffR
Steering Committee
Posts: 939
Joined: Tue Feb 03, 2015 9:49 pm

Part Two!

Now, I'm sure a skim of the changelog above will hint to the fact that a looooot of the changes are fixes, tweaks and corrections that aren't easy to show, but are important to making the experience as smooth as possible.

That said, still a few nice shiney bits to show off.

Updated PostFX Editor/Manager
@ Ewyncat was trying to get work done on getting a number of the PostFX's updated, and kept running into a number of annoying hitch-ups that were stonewalling progress. So, did a quick pass to streamline the PostFX Manager code, and separated out the editor, and we end out with this:

This is nice because it chops down a lot of the redundancy in the old code, which is good for maintenance, and the editor works in a more efficient way. Instead of the old editor, which was a bunch of special snowflake hand-made pages that means a lot of finnegaling by hand to add/modify any of it, let alone adding editing for a new postFX, we instead have a number of callbacks that let the postFX report it's own fields to be exposed to the editor.

This also makes postFX's working with modules miles easier. Dropping in the module that contains a postFX now just lets it show up in the editor, ready for....well, editing. No manual integration required.

OpenGL Probestuffs caught up
While the forward shading stuff still needs a few fixes, the main implementation of PBR lighting and probes/IBL has been caught up on the GL side to the D3D side, which means the Linux and Mac builds are back in action properly. These are part of the reason why the original plan for a monday/tuesday update was delayed - some shadergen issues on the GL side(and one on the D3D side) meant that the linux and mac builds would crash on release. So that obviously needed fixing before the build could be done up. But, it's at parity now, so that means pretty well smooth sailing again with platform builds.

Semi-related, but I added a fix so that when lowering the texture quality, it properly updates the cubemap arrays' levels as well. This means more potato-y machines can still get proper IBL lighting with much less performance cost. And it actually manages to still look pretty respectable:

The metal still looks decently metal-y even though the cubemaps are tiiiiiiiiiny at the lowest texture quality.

Theming
Ongoing theming catching for the editor is ongoing as well. Hard to really show the full spread of tweaks, but the number of controls not at least decently matched to the theme is pretty low now. Blood's hatched a plan to better group and sort the GuiProfiles for the editor as well, which should trim down on redundancy, as well as make it easier to get the remaining bits themed properly.

Asset Importing/Config stuff
Also not easy to show, but a number of improvements to the asset import process as well. A number of fixes, improvements and standardizations. One handy new bit is a config setting to auto-generate Composite files for PBR materials on import. If the importer with this setting on detects a metalness, AO and roughness map, it'll just automatically generate the combined composite map and save that out instead of the less efficient 3 separate files.
Blood's also been doing a beautiful job of utterly beating up the importer, and he's spotted a number of problems or hangups, so at this rate, the importer should be a very smooth little diddy to use very soon indeed

That probably wraps the big highlights that are easy to show. We'll keep smashing all the bugs you fantastic people make note of with these preview builds, and once it's nice and refined, can push out the final release. Getting close now, we just need to make it all polished to a mirror sheen.

So for the updated binaries:
http://ghc-games.com/SC/Preview4_0_bld5_win.zip
http://ghc-games.com/SC/Preview4_0_bld5_lin.zip
http://ghc-games.com/SC/Preview4_0_bld5_mac.zip