Work Blog - JeffR

Re: Work Blog - JeffR

Monkeychops
Posts: 27
Joined: Wed Feb 18, 2015 11:53 am
Looks good, but it also looks like a lot of work.
How feasible would it be to get the PBR and ECS tasks done in the space of say a month or so and make a release? As others have said, it is important to have regular releases to keep the community alive and engaged and right now it is tricky for people to start new projects since switching materials and lighting to PBR and scripting to ECS are two quite big changes from a workflow point of view... so it would be nice to have something bedded down and released in that regard.

Threading and the GFX tasks look like they could be a time sink and take months to get right, and are largely going to be under the hood changes which most users won't need to worry about - I am assuming existing single threaded game logic scripts will continue to work as before, and that most users just import their assets and assign textures in the editor, so it should be possible for any materials saved using the old format to be automatically converted to the new format by and editor script that assigns the old textures and values to the new system?
That stuff could maybe be pushed back towards a release in several months time?

Re: Work Blog - JeffR

Azaezel
Posts: 442
Joined: Tue Feb 03, 2015 9:50 pm

Perhaps I simply misread last weeks convo. I was under the impression folks were asking for stuff they could hop onto and assist with speeding up via active contributions. Which was why there's several things spread over the spectrum.

Re: Work Blog - JeffR

Razer
Posts: 53
Joined: Tue Jan 10, 2017 11:29 am

Threading
• Verify via threading the GroundCover class(specifically, item placement of cells being populated) that the system operates cleanly and can accept regular "WorkItem" jobs
[**] If it works: Expand usage to other parts of the engine
[**] If it fails, fall back to the WIP conversion work of the threading stuff provided by the Life is Feudal guys.
• Threaded Asset loading/preloading
• Threaded stock physics subsystem(Bullet and PhysX already thread themselves)
PBR[I'm already working on this]
• Complete probe projection via zone clustering/filtering
[**] If it works, no leaks or major flaws visible, do next major bulletpoint subtask
[**] If it fails to work, due to leaking or undesirable clipping, shift to embedding the distance to the captured pixel in the alpha channel of the specular cubemap, compare there for projection cutoffs
• Finalize probe-to-probe (and skylight) blending to ensure no overbrightening from overlaps
• refix editors after removal of global cubemap
• complete terrain composite map blending via the same methods as other materials
Entity/Components
• Review networking to ensure entities created on a server properly ghost all network-tagged components and component-states to connected clients
• Review rendering of editing-related elements (SoundComponent emission radius sphere when hitting preview, for example)
• Create ParticleEmissionComponent based on the SoundComponent as a template (ie, replicates the ParticleEmitterNode values), 'play' bool for simple start/stop handling, preview button to make it fire off in-editor for previewing/testing
• Create finite state machine system(a StateMachine component already exists, but we may want to utilize the above suggested play/preview aspects)
• Convert core gameplay classes(Player, vehicle, etc) to internally use Components(or make variants and flag the existing classes as deprecated)
GFX
• Physical Material vs Rendered Material separation(break out the non-visual aspects like sound, footprints, particles, etc into their own class which can be looked up, as opposed to being integrated into a single massive class)
[**] If it works, (prototyped within at least 2 weeks from start. anything more is likely to reflect too large a shift for folks to deal with when converting): proceed with split features
[**] If it doesn't, (not done within a reasonable ammount of time): deal with it as-is, and expand any additional features in the singular class
• Use the visual-only portion as a hook-in base for the node-based shader system
• Further vulkan/moltenVK future feasability research

This is my take on the roadmap :

Threading
This can be a reasearch task instead of something that should be absolutely included in v4.
I htink it's a long term optimisation tasks with other aspects like culling and objects instancing.

PBR[I'm already working on this]
I think this should be the main priority task for 4.0, most people will expect rendering as it is done in other 3D engines around.
I would add :
- Remove lightmaps, add some voxel global illumination approximation on the roadmap, because Torque 3D is about large outdoor games.
- Add pbr multi texturing to terrain
- Improve post effects quality, with cinematic like shaders, this will play a major role on the overall game look, making it look modern.
- Add extra shaders like anisotropic, sub surface scattering, refraction, displacement mapping.

Entity/Components
- Make scene objects as stack components system, able to stack as many components as needed , for example a rigidbody object could stack many particle components, many sound components, many collision components, and one or many scripts components (when one object could support many scripts components).
- Let people able to create prefabs and nested prefabs
- Let physics drive the gameplay, a physic capsule should be driven by script and would be used to make fps or tps player object.
Keep old vehicle or character classes conversion for later, people should be able to create their own classes and gameplay using physics.
Get RigidBody, KinematicBody and Static Body physics components working with scripts.

GFX
No physics Vs rendered. Forward ++ renderer instead of a deffered renderer is lot faster and what some last 3D engines are using.
Everything is a shader with parameters.

Footprints are a script not a component, don't keep Torque 3D old system, make it more adaptable instead.
People should be able using an animation graph to create animation events calling a script function, the script will create a foot decal , play a sound or do anything else.
Let people do what they want on animation events

Particles is a component, not a GFX, it uses it's own particle shader, and it can be staked as a component into an object components stack.

Classes
Let down most Torque 3D gameplay classes, users should be able to create gameplay on their own with physics and using scripts with components.
Keep under the wood all that is network related, someone making a game should never have to deal with server or network in scripts or objects until he is doing a multiplayer game.
Create some scripts to work with components that will be fps or tps, ball physics and vehicle gameplay.

Long term roadmap
Node based shader system and Vulkan can wait and be pushed further in the roadmap, they are not essential to get a good V4 version first.
Instead extending the animation editor with states machines , blending and animation events creation is lot more important.

Adopt a conventional scripting language
C# is vastly used and very known, lot of users already creating games know C#.
Torque 3D would get lot more people involved with C#, it would benefit excellent Visual Studio editor
I don't like Torque script, perhaps it has been a reason why people scripting games didn't choose Torque 3D

My take on the first changes to make a V4 :
- get a good objects component stack system, each scene object be a stack system to store other components (particles, collision, physics, scripts , or any other component)
Get script functionality to get components from some objects
- get all physics objects and collision working with collision layers, collision detection functions, raycasting functions
- Make parenting possible in objects, a physics parent object could have many physic child object
- pbr for most materials, custom pbr shaders for terrain, shader for particles, shader for water
- improved post effects shaders
- Extend the animation editor with states machines , blending and animation events creation
- start working on a approximation global illumination lighting , fast enough for big outdoor games.
- start a migration to a popular scripting language like C#

Once that's done, people should be able to start coding their games from scratch with components without having to use old Torque 3D classes.

It's a long roadmap to get all that ready for V4, while there is competitors among 3D engines, what advantages will it be to choose Torque 3D instead of another ?

Re: Work Blog - JeffR

Azaezel
Posts: 442
Joined: Tue Feb 03, 2015 9:50 pm

Is there a single line of that set of demands that you are willing to contribute to forwarding. Yes or No.

Re: Work Blog - JeffR

Happenstance
Posts: 76
Joined: Sat Apr 11, 2015 9:08 pm
Yeah, that's a pretty hefty list.

I think we need to be realistic about what the engine is and can become given the size of the community. Torque's strength at this point is being opensource with a permissive license, having best-in-class multiplayer support, and having some built-in features (navmesh, terrain, etc.) that would cost real money to implement in other engines. When you only have a handful of people contributing finding ways to double down and improve on those strengths makes more sense to me than trying to grasp at every whizbang feature ThatOtherEngine™ has.

That line of thinking is why I'm going to do what I can to support:

1). Cleaning up the codebase (in whatever form that takes, whether it be in-place refactors or leveraging T2D in some way). A cleaner, leaner engine makes the opensource license all the more attractive.

2). Moving to PBR and some type of E/C model, along with all of the editor improvements that are sure to follow. It makes sense given the direction the industry is taking and removes some of the workflow hurdles people run into right now.

Re: Work Blog - JeffR

Azaezel
Posts: 442
Joined: Tue Feb 03, 2015 9:50 pm

So to be perfectly blunt this is a collaborative endeavor. Not a sweatshop for folks to expliot one anothers time for free with. If you can't handle that, then this simply isn't the engine for you.

That being said, lets get into the brass tacks of what made the list and why.

Threading:
A few months back, the LiF folks were kind enough to offer us a revamped threadpool system. There's some contention that it's needed, hence the test.
If it's found that it's needed, a stumbling block would be the incorporation of several portions of the boost library that folks asked roundly rejected.
https://github.com/Azaezel/Torque3D/tree/LIFshare contains partial conversions first to the STL, then when that ground out to SDL2. Either way would presumably work, just need a hand there to dial that on in one way or the other.

PBR:
to clarify the
"Complete probe projection via zone clustering/filtering
[**] If it works, no leaks or major flaws visible, do next major bulletpoint subtask
[**] If it fails to work, due to leaking or undesirable clipping, shift to embedding the distance to the captured pixel in the alpha channel of the specular cubemap, compare there for projection cutoffs
Finalize probe-to-probe (and skylight) blending to ensure no overbrightening from overlaps"
bits, the math for single probe and skylight application is pretty much knocked out to folks satisfaction so far as I'm aware, which leaves the probe end. PBR isn't just slaping in new textures and new math, it requires an underlying system to pick up and project context.

for
"refix editors after removal of global cubemap"
as a temp-measure, for a while there to get cubemap application going on in the editing suite, we leveraged the older 'assign a cubemap directly to this material' code. that'll be going away due to the shift of effectively using the equivalent of light cookies to project onto the scene rather than wrap arround each individul object.

for
"complete terrain composite map blending via the same methods as other materials"
https://github.com/GarageGames/Torque3D/pull/2049 could use some cleanup prior to roll-in, as well as opengl conversion, and composite to composite blend should follow the same end blending methodology.
folks will find the latest fork on that subject https://github.com/Azaezel/Torque3D/tree/PBR_math_wip

Entity/Components:
for
"Review networking to ensure entities created on a server properly ghost all network-tagged components and component-states to connected clients"
Theres currently a bug with the implementation where you're not seeing meshcomponents when you connect, nor hearing soundcomponents play when connecting from a non-host. That'll need fixing prior to anything else.

for
"Review rendering of editing-related elements (SoundComponent emission radius sphere when hitting preview, for example)"
That busted a while back with a retool to meshcomponent rendering for speed. need to revisit that end for better(tm) built-in tool support. After all, if we're going to make current users rewrite thier code, it better be a step up.

should be self-evident, though somethin that should probably be knocked out after the other two above.
"Create ParticleEmissionComponent based on the SoundComponent as a template (ie, replicates the ParticleEmitterNode values), 'play' bool for simple start/stop handling, preview button to make it fire off in-editor for previewing/testing"

** not listed, should probably also go ahead and toss in the capacity to set any component to use a given object-node instead of object-origin.

for
"Create finite state machine system(a StateMachine component already exists, but we may want to utilize the above suggested play/preview aspects)"
this ones 'simply' expanding that end with more options.

for
"Convert core gameplay classes(Player, vehicle, etc) to internally use Components(or make variants and flag the existing classes as deprecated)"
this makse the list for two reasons. The first is simply that forcing ourselves to use that as a pass/fail measure ensures that E/C does in fact have everything folks will really be needing at a basic level. The second is that it gives existing users a well documented set of conversions for thier own customised codebases.

GFX:
for
"Physical Material vs Rendered Material separation(break out the non-visual aspects like sound, footprints, particles, etc into their own class which can be looked up, as opposed to being integrated into a single massive class)
[**] If it works, (prototyped within at least 2 weeks from start. anything more is likely to reflect too large a shift for folks to deal with when converting): proceed with split features
[**] If it doesn't, (not done within a reasonable ammount of time): deal with it as-is, and expand any additional features in the singular class"
There is at present two paradigms for nonvisual material aspects. the former, used throughout the lions share of classes, uses a singular material definition that includes sound effects to be played when walking or jumping on things with that mat, and wether those take footstep decals when walked uppon, while the latter in terrain breaks that on up into two classes.
A fairly simple example of expanding on the former system would be https://github.com/Azaezel/Torque3D/commits/FX where we tack on a tracking ID for explosions (they have sounds, particle effects, and decals built in, so seemed worth an eyeball form a simplification standpoint)
Other uses would be the physics end, like adding friction and restitution entries for more physics-based games.
It was expressed that the terrain model of material creation was preferable in terms of aspect-sorting, so figured float that out there as a notion, though obviouly from the listed time-frame for prototyping, not wedded to it.
Since alot of 4.0 is about retooling the frontend experience for folks, best we lock that in ne way or the other, instead of breaking things more in a 4.X.

for
"Use the visual-only portion as a hook-in base for the node-based shader system"
frankly, this one deserves it's own thread.

for
"Further vulkan/moltenVK future feasability research"
While one of our contributors is pretty confident that they could just toss in a GFX plugin, it'd be nice if we could go beyond that end and see how much we could leverage for gains.

General Ongoing Tasks:
for
"Warning cleanups"
https://gist.github.com/Azaezel/f34201c ... 601a45c88d
should be one anybody can jump into. Pick a warn number, and have at it. Makes for a simultaneous code-review, and just generally means less to wade through to see if folks introduce new problems accidentally.

for
"Further incorporation of SDL as the core platform layer, further deprecation of per-platform code"
we're now up to platform, platformMac, platformPOSIX, platformSDL, platformWin32, platformX86UNIX. that's alot of potential redundancy, and sdl2 handles those and more platforms. probably best to leverage that further and kill off extra crap to wade though. this made the list, just like warnings, because it serves as a component that isn't necessarily reliant on any of the more.. shall we say, arcane... ways the engine handles things.

Again, if there's anything on the list you'd want to help with, pick one or two. If there's anything not on the list you would want to contribute, there's at least a few folks willing to work ***with*** you to see that happen as well. And if there's a particular point you find contentious, by all means, explain why you that specific item should be put off for a 4.X, not a 4.0 release.

Re: Work Blog - JeffR

JeffR
Steering Committee
Posts: 908
Joined: Tue Feb 03, 2015 9:49 pm

When you only have a handful of people contributing finding ways to double down and improve on those strengths makes more sense to me than trying to grasp at every whizbang feature ThatOtherEngine™ has.
Quite so, which is why we've actually been pretty selective with what work we're putting real effort towards. Some of the stuff I talk about spending a little time on in these blogs is just neat R&D or convenience feature bits, but stuff like whatever comes out of the clean core notions, the refactored material and render pipeline stuff, etc actually do a LOT to reducing the workload involved in development and maintenence.

Think of it along the lines of spending $100 for a nice pair of boots once instead of$10 for a pair of boots each year for the next 20 years. It takes an investment (of time, in this case) up front, but it goes an incredible a way to deal with the concerns of time and effort allocation when people don't get hung up on the squirrely old bits of code.

But yeah, @ Happenstance if the E/C stuff sounds particularly keen for the part you want to jump in on, then by all means
You can hit me up in the discord at almost any time and I'd be more than happen to help you get rolling on stuff like that. Otherwise if you wanna help on the cleanup end, me and az can definitely steer you there as well
How feasible would it be to get the PBR and ECS tasks done in the space of say a month or so and make a release?
Well, if people could put up for me to work on dev full time, I could move you a couple of mountains, really

For realisms sake, If we JUST did PBR and E/C(which I'm not quite i agreement with, though those absolutely are the thrust of 4.0 either way) PBR's like 90% of the way done, and E/C is like, 80-90% of the way done for just the core entity/component stuff, though we're going to want to deprecate the current game classes and remake them in the e/c system which hasn't really been started on yet, so that'd take time.

If you're curious about that last part and want to help with it, I can definitely get you pointed and you can take a stab at converting a simple class like Item or static shapes as a start.

Re: Work Blog - JeffR

Happenstance
Posts: 76
Joined: Sat Apr 11, 2015 9:08 pm
@
JeffR
I'll try to jump on discord this weekend and poke around with the E/C stuff if possible. It's been a while since I've looked at any of it so I'll need a refresher and probably a better idea of what needs to be tested to move it along.

I'd like to tackle the ridiculous amount of warnings in the codebase at some point as well. I worked on that a few years ago and got stuck in a quagmire of "is this a -real- S32 or should it be U32... wait, a float!?!" I have no doubt there's some nasty bugs hiding amongst those warnings but getting to them is scary.

In the near term I'm going through the issues list on github and @'ing people. Not sure if that's helpful but people have a habit of submitting a bug report and then vanishing into the ether.

Re: Work Blog - JeffR

Razer
Posts: 53
Joined: Tue Jan 10, 2017 11:29 am

Again, if there's anything on the list you'd want to help with, pick one or two. If there's anything not on the list you would want to contribute, there's at least a few folks willing to work ***with*** you to see that happen as well. And if there's a particular point you find contentious, by all means, explain why you that specific item should be put off for a 4.X, not a 4.0 release.
Sorry but i don't know C++ or Torque 3D specific code to be able to contribute.

Do you want only contributors to use Torque 3D and make a wish list , while scripters and users are not important ?
When scripters make a great game i think it would give lot of prize for Torque 3D demonstrating how good it could be, or all users making tutorials, i think those users are as important as people contributing.

It's important to get more and more people that are scripters not necessary C++ contributors, more people is using Torque 3D more the community will grow.
CryEngine have a great 3D engine, but their community is empty

I have a game going on in a 3D engine that has all features i listed, until Torque 3D is able to catch up i will only follow Torque 3D progress from time to time, but starting a game in Torque 3D is a no go. There is too many things that will change and we don't know how stable it will be or how good it will perform or not.
It's like the next Torque 3D development is just beginning.

Re: Work Blog - JeffR

JeffR
Steering Committee
Posts: 908
Joined: Tue Feb 03, 2015 9:49 pm

@ Happenstance Sounds like a plan to me

@ Razer Obviously if people can contribute to fixes and core development, that's most pertinent to the current specific discussion happening.

That absolutely does not, however, mean that people just working on games and stuff script-side are unimportant. Very much the opposite, actually. A lot of the work going into things is to better facilitate the people just trying to make stuff to get things done.

I'm not quite sure why you feel that development is just beginning though. Just because some things are being discussed on if they should/can be done now as opposed to a follow up release doesn't mean that the work isn't already being worked on. I'd hope that my posts in this workblog show that a lot of work has been done in most things we've been talking about, it's largely a matter of 'ok, what do we prioritize now to get stuff locked down'.

Who is online

Users browsing this forum: No registered users and 1 guest