The Engine-sideObviously, the guts of Torque3D lie with the C++ engine code. Without that obviously there’d be no engine. So it’s not a surprise that a huge amount of the work will be focused here.
Of the main work targeting the engine for 4.0, you have the Entity and Component stuffs, Rendering improvements, and cleanup and refactoring of a lot of the old code we have sitting around.(Obviously this isn't everything slated work-wise engineside, but they are the big chunks)
Entities and ComponentsI won’t go into extreme detail here for this post, partially because I’ve gone into it more in-depth in my
work blog, so if you’re completely unfamiliar with the idea, I’d say hop over to that thread and give a perusal to get up to speed on the general concept.
The crux of it is breaking up the monolithic game classes we’ve had in T3D for a long time into much more maintainable, customizable, easier to understand pieces. Monsters like ShapeBase, Player, Vehicle. These and others stand as obelisks of code. Rather robust, fairly stable code, but obelisks nonetheless.
And with that property means that any gameplay changes you may want to do for your project require fussing with those monstrosities. That wouldn’t be so bad on it’s own, but the problems multiply as changes occur.
Issues caused by changes can cascade and take more time to track down, old, delicately balanced code may crumble with the changes, and even if everything goes off without a hitch, the steady march of updates and changes to the main branch of the engine can rapidly get away from you and you need to make a call on if you keep things up to date, or stop constantly back-peddling on merges and just get on with it.
This is a huge issue when it comes to working on gameplay classes on the engine side. The good news is to all these problems, is shifting over to the Entity/Component stuff does a huge service in combating this problem.
Because it’s no longer and issue of “I want to have X gameplay mechanic, so I have to edit it into ShapeBase or Player or Vehicle, and hope nothing conflicts and nothing gets updated with unrelated parts of those classes”, and becomes “I create a self-contained component to implement that bit of gameplay, and outside of MAJOR functionality shifts to the backend, no updates to any other components will affect it and requires no additional work to maintain”.
In other words, once the standard and API are established, unless huge sweeping, breaking changes come, such as API rewrites, the components you write should remain unaffected, or require minimal updates to comply. If you write a component to render a custom shape, updates to physics components or collision components in the engine have no real bearing on your render component, and that means keeping things up to date becomes far, FAR easier.
In fact, if you don’t do any engine-side components, and only implement script-side components, that means all you need to do is utilize up to date binaries and you’re golden.
So while this approach is obviously far more flexible, and easier to maintain, obviously if you’ve already got a project underway, switching from utilizing Shapebase, Player, Vehicle, Item or any of the other gameplay classes doesn’t sound appealing because it would obviously necessitate a total rework, right?
Well, not really unless your changes are deep and extensive. For 4.0, out of the gate we’ll be replicating all the gameplay classes’ functionality with built-in components and Game Objects. The rendering, collisions, physics, sound, particles that classes like Shapebase, Player and all those are known for will be replicated in the new format, so it should be easy to port anything you’ve worked on so far over to the new system. And once you do, maintaining and expanding it will be miles easier than before.
RenderingAnyone that keeps up to date with all the frankly crazy updates that have occurred in the realtime rendering world over the past several years will know that while T3D’s rendering is pretty stable, it’s not as fast as it should be, and definitely not able to do the new, pretty stuff.
So, obviously that’s dumb and it’s going to change.
BUT HOW? You may ask, in a weirdly over dramatic way. Allow me to delve into the sweet, delicious details.
First and foremost, we talked about it with 3.9, but didn’t do it for 3.10 since it was largely intended for bugfixes, but the first thing that’s really going to happen is D3D9 is gunna be kaput. Ripped out, thrown into the bin. Maybe a few people will get a bit of pointing and laughing in at it.
D3D9’s been an incredible, reliable old girl, but with new rendering methodologies, it’s very literally holding things back. So it’s gone.
After that, we’ll be switching over quite a few subsystems to more modern configurations. The first step is switching internal texture format handling for images and render targets to sRGB. If you’re unfamiliar with what that means, a quick google should find you plenty of resources to read up on it, but the basic gist is we skip out on doing math to make sure images look correct under lighting conditions, so things are more efficient under the hood, things look more correct as an end result, and everything, top to bottom, is more consistent which means a lot less code lying around to do all that maintaining work, so things are easier to maintain and cleaner.
Then, the biggie: PBR. Physically Based Rendering. You’ve no doubt seen us talking a lot about it, and no doubt seen it used danged near everywhere at this point, but what it means at it’s core is modern realistic rendering, but more importantly a industry-consistent art pipeline, ensuring assets are more consistent to create, more consistent to use, and can be ported around between tools and platforms without having to fuss with it at each step.
And, you know, it’ll look freaking awesome too.
Other important updates aside from that, will see the HDR post processing effect see a pretty big rework, in part with the sRGB textures standardizing the linearized light math(which helps PBR a huge deal as well), so HDR will look nicer, and use a modern tonemapping technique to behave consistently.
We’ll also see an update to shadows, with a shift over to Hardware-accelerated PCF(Percentage Closer Filtering), so not only will they perform better, but they’ll look nicer. A lot nicer. No more ‘lazy spraypaint’ soft edges on shadows. Improvements to how the render APIs do their calls via updates to constant buffers is also a hot point which should help render performance a good deal.
I’ve also been doing a good deal of research into bringing baked static lightmapping back, as well as a Global Illumination implementation. So the end result should see several orders of magnitude improvement in the graphical fidelity that T3D can put out, and all with far more industry-consistent workflow so it’ll actually be easier to do.
Another bit that’s less ‘in your face’ but all the same very important will be a fairly big update to the ShaderGen system, which is how materials generate shaders the graphics API uses for rendering. T3D’s current shadergen is actually very powerful, but expanding it is quite an undertaking and generally not that intuitive. The new shadergen will streamline this approach, allowing us to keep flexible, procedural generation of shaders from materials, but without being a nearly impenetrable wall of cross-talking elements. More on this in the tools section.
The other big thing to rendering will be shunting it into it’s own render thread. As-is, rendering happens on the main thread, which while easy to code, is also not in any way efficient. We’ve been doing a good deal of reading and testing on this front and will have the rendering pushed into a separate render thread, which should see a massive performance boost to rendering. Long-term goals will look at async render support and eventually Vulkan/D3D12, but for the near-term, we’re looking at one main render thread to capitalize on the performance D3D11 and OpenGL can get us.
Code Cleanup and RefactorObviously, for a codebase as old as Torque, there’s going to be lots of parts that don’t meet the code standard/guidelines, have extra cruft that no one wants and other junk that just generally makes things kinda annoying to go rooting around in. So for 4.0, we’ll be going through and using some neat utilities to run through and do some spring cleaning, as well as looking at updating some of the older parts to more modern conventions where reasonable.
Stuff like space/tab compliance, bracket indentations, and extraneous includes slowing compiling down are all things that will be targeted by this, as well as a general pass to make the code compliant to common things - initialization order, crushing more warnings lying around, etc.
Even if it doesn’t make the code run faster, it’ll make the code cleaner and easier to work with and navigate.
So what’s there to do?So given all the above, what’s done, in progress, and needs to be done yet?
Well here’s the good news! For all that is in that gigantic wall of words up there, a huge amount of it is either done, or already had work started on it.
The Entity/Component stuff is a very long way along. Most of the remaining work involves making sure networking is as lean as physically possible, and replicating the current gameplay classes.
The rendering stuff likewise is very far along. The removal of D3D9 has been waiting in the wings for quite a while and is ready to go, likewise with the sRGB work. The PBR stuff is nearly complete, only really needing finalization on reflection probes. The HDR, Shadows and other lighting overhauls are either pretty much done, or a good ways into work being done.
For cleanup, most of that is done through several tools we have, and should be able to be hammered through in rapid succession without much fuss.
