Jump to content

Torque 3D 3.10 Released!


JeffR

Recommended Posts

Hey everyone!


This post announces excellent news, and that news is the release of Torque3D 3.10!


This was originally plotted to just be a simple bugfix release, but some new features snuck in and a lot of fixes and improvements followed along for the right, so 3.10 is actually quite the doozy of a release. To wit, 3.10 represents 144 pull requests, totaling to a rather monstrous 423 commits, done by 14 various participants - before even mentioning of all those who tested and provided bug reports and feedback so these PRs could be made.


While the sheer number of commits was lower, it ended up being 2 more PRs than 3.9, and more people directly contributed than the last.


Some major additions for this update include Mac OSX support(!), Ipv6 support(!), OpenAL-Soft for audio, Hardware Skinning support, and OpenVR(Vive) support, as well as updates to various libraries such as PhysX, Bullet, libVorbis, libOGG, libPNG, SDL, and recast.


I've got a lot to write down for the changelog, so expect that tomorrow, but I can readily make mention of all the fine people that put in PRs and file issues!

  • elfprince13
  • dud3
  • Azaezel
  • Lopuska
  • rextimmy
  • jamesu
  • JeffProgrammer
  • John3
  • irei1as
  • pacomont
  • rasteron
  • CouleeApps
  • lukaspj
  • Duion
  • petrifiedroadkill
  • HeadClot
  • CEV-EricLiu
  • hsdxpro
  • RichardsGameStudio
  • huhorier
  • Bloodknight

 

EDIT: And here is the changelog!


Alright, where can I get it?

As always, you can nab it from the wiki's download page, found here.


With the closing of 3.10, we now move onwards towards 4.0, as we’ve talked about many a time here. I’ve articulated on what this means to various degrees here, and elsewhere, but allow me to form a more consolidated thesis to properly explain the various plans that are nearly done, in progress, or slated for 4.0.


The Big Four-Oh

We’ve been talking about 4.0 for quite some time, and how it’ll bring a lot of changes for Torque 3D. I can assure you that though a few of the changes may take some getting used to, the end result will see a huge improvement in flexibility, ease of use top to bottom, and a huge improvement to development and iteration speed.


To that end, there are 3 major branches to the changes, which I’ll go into below, and articulate the particulars on each - both where things currently stand, and what’s left to do.

Link to comment
Share on other sites

The Engine-side

Obviously, the guts of Torque3D lie with the C++ engine code. Without that obviously there’d be no engine. So it’s not a surprise that a huge amount of the work will be focused here.


Of the main work targeting the engine for 4.0, you have the Entity and Component stuffs, Rendering improvements, and cleanup and refactoring of a lot of the old code we have sitting around.(Obviously this isn't everything slated work-wise engineside, but they are the big chunks)


Entities and Components

I won’t go into extreme detail here for this post, partially because I’ve gone into it more in-depth in my work blog, so if you’re completely unfamiliar with the idea, I’d say hop over to that thread and give a perusal to get up to speed on the general concept.


The crux of it is breaking up the monolithic game classes we’ve had in T3D for a long time into much more maintainable, customizable, easier to understand pieces. Monsters like ShapeBase, Player, Vehicle. These and others stand as obelisks of code. Rather robust, fairly stable code, but obelisks nonetheless.

And with that property means that any gameplay changes you may want to do for your project require fussing with those monstrosities. That wouldn’t be so bad on it’s own, but the problems multiply as changes occur.


Issues caused by changes can cascade and take more time to track down, old, delicately balanced code may crumble with the changes, and even if everything goes off without a hitch, the steady march of updates and changes to the main branch of the engine can rapidly get away from you and you need to make a call on if you keep things up to date, or stop constantly back-peddling on merges and just get on with it.


This is a huge issue when it comes to working on gameplay classes on the engine side. The good news is to all these problems, is shifting over to the Entity/Component stuff does a huge service in combating this problem.


Because it’s no longer and issue of “I want to have X gameplay mechanic, so I have to edit it into ShapeBase or Player or Vehicle, and hope nothing conflicts and nothing gets updated with unrelated parts of those classes”, and becomes “I create a self-contained component to implement that bit of gameplay, and outside of MAJOR functionality shifts to the backend, no updates to any other components will affect it and requires no additional work to maintain”.

In other words, once the standard and API are established, unless huge sweeping, breaking changes come, such as API rewrites, the components you write should remain unaffected, or require minimal updates to comply. If you write a component to render a custom shape, updates to physics components or collision components in the engine have no real bearing on your render component, and that means keeping things up to date becomes far, FAR easier.


In fact, if you don’t do any engine-side components, and only implement script-side components, that means all you need to do is utilize up to date binaries and you’re golden.


So while this approach is obviously far more flexible, and easier to maintain, obviously if you’ve already got a project underway, switching from utilizing Shapebase, Player, Vehicle, Item or any of the other gameplay classes doesn’t sound appealing because it would obviously necessitate a total rework, right?


Well, not really unless your changes are deep and extensive. For 4.0, out of the gate we’ll be replicating all the gameplay classes’ functionality with built-in components and Game Objects. The rendering, collisions, physics, sound, particles that classes like Shapebase, Player and all those are known for will be replicated in the new format, so it should be easy to port anything you’ve worked on so far over to the new system. And once you do, maintaining and expanding it will be miles easier than before.


Rendering

Anyone that keeps up to date with all the frankly crazy updates that have occurred in the realtime rendering world over the past several years will know that while T3D’s rendering is pretty stable, it’s not as fast as it should be, and definitely not able to do the new, pretty stuff.


So, obviously that’s dumb and it’s going to change.


BUT HOW? You may ask, in a weirdly over dramatic way. Allow me to delve into the sweet, delicious details.


First and foremost, we talked about it with 3.9, but didn’t do it for 3.10 since it was largely intended for bugfixes, but the first thing that’s really going to happen is D3D9 is gunna be kaput. Ripped out, thrown into the bin. Maybe a few people will get a bit of pointing and laughing in at it.

D3D9’s been an incredible, reliable old girl, but with new rendering methodologies, it’s very literally holding things back. So it’s gone.


After that, we’ll be switching over quite a few subsystems to more modern configurations. The first step is switching internal texture format handling for images and render targets to sRGB. If you’re unfamiliar with what that means, a quick google should find you plenty of resources to read up on it, but the basic gist is we skip out on doing math to make sure images look correct under lighting conditions, so things are more efficient under the hood, things look more correct as an end result, and everything, top to bottom, is more consistent which means a lot less code lying around to do all that maintaining work, so things are easier to maintain and cleaner.


Then, the biggie: PBR. Physically Based Rendering. You’ve no doubt seen us talking a lot about it, and no doubt seen it used danged near everywhere at this point, but what it means at it’s core is modern realistic rendering, but more importantly a industry-consistent art pipeline, ensuring assets are more consistent to create, more consistent to use, and can be ported around between tools and platforms without having to fuss with it at each step.

And, you know, it’ll look freaking awesome too.


Other important updates aside from that, will see the HDR post processing effect see a pretty big rework, in part with the sRGB textures standardizing the linearized light math(which helps PBR a huge deal as well), so HDR will look nicer, and use a modern tonemapping technique to behave consistently.

We’ll also see an update to shadows, with a shift over to Hardware-accelerated PCF(Percentage Closer Filtering), so not only will they perform better, but they’ll look nicer. A lot nicer. No more ‘lazy spraypaint’ soft edges on shadows. Improvements to how the render APIs do their calls via updates to constant buffers is also a hot point which should help render performance a good deal.


I’ve also been doing a good deal of research into bringing baked static lightmapping back, as well as a Global Illumination implementation. So the end result should see several orders of magnitude improvement in the graphical fidelity that T3D can put out, and all with far more industry-consistent workflow so it’ll actually be easier to do.


Another bit that’s less ‘in your face’ but all the same very important will be a fairly big update to the ShaderGen system, which is how materials generate shaders the graphics API uses for rendering. T3D’s current shadergen is actually very powerful, but expanding it is quite an undertaking and generally not that intuitive. The new shadergen will streamline this approach, allowing us to keep flexible, procedural generation of shaders from materials, but without being a nearly impenetrable wall of cross-talking elements. More on this in the tools section.


The other big thing to rendering will be shunting it into it’s own render thread. As-is, rendering happens on the main thread, which while easy to code, is also not in any way efficient. We’ve been doing a good deal of reading and testing on this front and will have the rendering pushed into a separate render thread, which should see a massive performance boost to rendering. Long-term goals will look at async render support and eventually Vulkan/D3D12, but for the near-term, we’re looking at one main render thread to capitalize on the performance D3D11 and OpenGL can get us.


Code Cleanup and Refactor

Obviously, for a codebase as old as Torque, there’s going to be lots of parts that don’t meet the code standard/guidelines, have extra cruft that no one wants and other junk that just generally makes things kinda annoying to go rooting around in. So for 4.0, we’ll be going through and using some neat utilities to run through and do some spring cleaning, as well as looking at updating some of the older parts to more modern conventions where reasonable.

Stuff like space/tab compliance, bracket indentations, and extraneous includes slowing compiling down are all things that will be targeted by this, as well as a general pass to make the code compliant to common things - initialization order, crushing more warnings lying around, etc.

Even if it doesn’t make the code run faster, it’ll make the code cleaner and easier to work with and navigate.


So what’s there to do?

So given all the above, what’s done, in progress, and needs to be done yet?

Well here’s the good news! For all that is in that gigantic wall of words up there, a huge amount of it is either done, or already had work started on it.

The Entity/Component stuff is a very long way along. Most of the remaining work involves making sure networking is as lean as physically possible, and replicating the current gameplay classes.


The rendering stuff likewise is very far along. The removal of D3D9 has been waiting in the wings for quite a while and is ready to go, likewise with the sRGB work. The PBR stuff is nearly complete, only really needing finalization on reflection probes. The HDR, Shadows and other lighting overhauls are either pretty much done, or a good ways into work being done.


For cleanup, most of that is done through several tools we have, and should be able to be hammered through in rapid succession without much fuss.

Link to comment
Share on other sites

New Template

So all that engine work is great, but if there’s no updates to the old, aging starter templates, then it’s still a pain to work through things, even if the engine is made squeaky clean and up to date. And you know what? You’re right. The good news is, I’ve been working on a brand new template that does a huge amount of streamlining, cleanup and consolidating.


Originally based on buckmaster’s most excellent t3d-bones project, I expanded it out as a proper replacement to the existing templates. It offers the same functionality, but without any of the baggage, and the code is cleaned up and streamlined to remove all the confusing hops back and forth and it being a single base template means we're not pointlessly duplicating files.

It also heavily utilizes the new Modules system, so dropping in packages is stupid easy, and making your own is likewise(more on this in the next section). But unlike the existing templates, you can strip the template down to the absolute bare minimum with no effort at all, even ditching the client/server scripts and doing everything purely local if you want. At it’s most bare, it comes in at a bit over 10mb, and basically just includes what’s needed to initialize the engine and display a blank window. From there, modules can be dropped in and you can get client/Server action, UI, gameplay, etc very quickly. Because they’re segmented and more self-contained, it’s easier to understand what each part does and making your own is far easier than trying to jury rig around the swaths of code in the old template.


The new template’s option menu also sees a huge expansion of options, and the editor will be shifted over to utilizing the Dark Theme to give it that sleek, modern feel.

There will also be a swath of example modules that will be added as we go to demonstrate various - if simple - gameplay concepts to help you jumpstart development. With the entity/component stuff, it should be easy to drop in several starter modules, mix and match components and rapidly hammer out gameplay concepts so you can spend more time designing and less time hemming and hawing over ‘wtf does this stuff even do?’


So what’s there to do?

Most all the core of the new template is done. All that really remains is the various starter example modules. I’ll be posting a list of ones we’d looked at to have in there, but any suggestions would be great. The more examples to offer people as a jumpstart, the easier a time people will have starting a new project, or even expanding/developing an existing one.

Link to comment
Share on other sites

Editors, Assets and Pipelines

This is another big one. T3D’s general workflow pipeline isn’t bad, a big advantage is it allows you some flexibility your average asset-driven pipeline doesn’t offer, but it also has quite a number of things that drag things down.

Asset tracking, management, updating, dependency tracking, formats, all of these sorts of things can be perfectly functional, but not optimal, and thus cause a drag on efficiency.


Most things in this section are things I’ve touched on in my workblog as well, so feel free to hit that for more in-depth info, but we can definitely cover some of the major highlights here.


Assets and Modules

The new template will utilize these heavily, but the main advantage is not having to keep tabs on stuff yourself, and not needing to use explicit paths all over the place. This means if something gets reorganized, deleted or renamed, you don’t have a ton of different things exploding left and right because the image or model being referenced no longer exists. It also means it’s easier to use them, as you don’t need to remember the path of an asset, just the module name and asset name if you’re doing stuff manually, and also really easy to install assets and just have it automagically work. The Asset Browser is being introduced to consolidate and standardize how assets are browsed, created and used, so everything should be far more consistent and easy to understand.


Drop a pre-configured package into your directory and refresh the module database and those assets are good to go to be used by everything. No more needing to do a bunch of manual exec’ing of script files and the like to get stuff to appear.


Assets will also cut out a lot of the format finnegaling you need to do now. We’ll be shifting to use Assimp in the backend, so we’ll support a much more broad range of model files that can be imported, not just collada, which is great for the art pipeline and the asset system will deal with the importing and conversion to be used. Same deal for textures. Only have PNGs images? No problem, import them in and it’ll convert them to DDS for you so you don’t have to fuss.


The end goal is to have a very sleek, efficient art pipeline that removes the fussing and just lets you use and keep tabs on the content you introduce.


Editors and Tools

Obviously, one thing that’s really nice about T3D is the breadth of editors and tools it has on offer, but some aspects of the editor suite are feeling dated. Firstly is the look. I mentioned in in the New Template section, but the editor will be switched over to a more modern Dark Theme, which should be easier on people’s eyes when they’re working at 4am. In the long-term, I’m looking to introduce a proper themes system so it can be more readily customized, but the dark theme is generally a good starting step forward.


From there, there’s the layout of things. I’d done some experiments into it prior(again, workblog) but I plan to implement windows as being dockable into tabbooks, as some other engine editors use, and as some tools like visual studio do, as well as the ability to use multiple windows with various editor tools on different windows. This will allow for a much deeper customization of the editor workspace and let people build an arrangement that helps with their workflow.


Another thing is looking into standardizing and streamlining how to add new editors. Sometimes you just need to implement a new editor tool to do a particular, custom job, but right now that’s pretty vague and not well covered outside of using existing tools as an example. Going forward, there will be a common workflow for setting up these sorts of things very easily, and can be done in scripts attached to modules. So you drop in an AI module, and when you load the editor, it automatically loads a tool built for it, etc.


Other advancements to the toolset will be several things heavily associated to some of the things mentioned prior, but a few big ones is a Web Graph editor(workblog) for very quickly editing state machines with a visual interface, and an overhauled Shadergen, as mentioned prior. The new shadergen will utilize a visual node editor, similar to other engines and art tools, not only allowing artists to make what they want, but allow everything from simple, pre-set materials like what T3D’s mats offer now, to 100% custom materials/shaders through the same, consistent interface.

This will allow art-types to focus on art, and now fussing about the particulars of how the materials work, and if they need something added, not having to throw it at the poor coders to add in for them at some point.


So what’s there to do?

Compared to the other sections, this one is still the most WIP comparatively. But that’s not to say progress hasn’t already been made preemptively!

I’ve been using the Asset/Module stuff in my personal RnD build(which is what’s been used in the workblog posts) for a while now, so it’s getting decently hammered on already. Similarly for some of the Editor/Tools stuff like the web and node graphs.


The todo is finalizing the various asset types we need, as well as cleanup on the Asset Browser to make it ironclad in terms of stability. From there, that standardization method for adding tools needs to be implemented, as well as the dockable windows deal. So a fair bit to do, but solid progress is already made and I’ve no doubt we’ll be able to hammer it all out.

Link to comment
Share on other sites

Anything else?

The only real remaining bit to discuss is documentation, tutorials, and learning the engine. Torque certainly doesn’t have the worst documentation out there, but it’s definitely spotty, with some parts being pretty out of date, and not as many examples as it should. The examples will largely be resolved with the starter example modules I mentioned before, but documentation/tutorial wise would still be lacking.


My personal goal is to ensure that the core scripts in the new template, as well as the entity/component stuff gets thoroughly, aggressively documented at minimum. Those will be the cornerstone files in which most all other work you guys do will stem from, so if nothing else, they should be documented out the face so you can know what those parts do intimately if nothing else. The advantage of well documented components is it makes for a good lead-in on how the root functionality on other parts of the engine work, so even if everything isn’t 100% deep documented, it’ll provide a very solid foundation of understanding the engine as a whole.


The other thing I’d been looking at were tutorials. There’s some good tutorials out there, but not as many as we’d like. So one thing I’d been looking into is a “Tutorial Mode” that can be activated in the editor suite. These would be choreographed tutorials that walk you step-by-step through utilizing various parts of the toolset. Similar to game tutorials that prompt you to press ‘C’ to crouch, and the tutorial doesn’t advance further until you prove you can find and press the button, this mode would, conceptually, describe what the current step is trying to do, such as say ‘activating the terrain editor’, and will outline the terrain editor button and/or prompt on-screen the hotkey. Once either of these are pressed, the tutorial will advance to the next step of the tutorial, walking you through whatever the tutorial is trying to teach.


If this approach can be standardized for a lot of the editor suite, I think it’ll provide a rock solid entry point for newcomers trying to acquaint themselves with the engine.


So that would be the breakdown of our plans for 4.0 and some even further looking stuff. More details and particulars will obviously be explained further as we go and I fully plan to do a lot more workblogging(and maybe even some streaming of devwork), but this post is intended to give you an idea of what the plan is, where we’re already currently at, and where we’re going with things.


We look forward to your continued ideas, suggestions, and contributions to make things even more awesome going forward!


-JeffR

Link to comment
Share on other sites

  • 3 weeks later...

Alright, did a post on my workblog(well, 2 posts) going over some of the deets for the new template and the module/assets stuff. Check on it here:

http://forums.torque3d.org/viewtopic.php?f=8&p=7637#p7636 and also the thread for discussing the PR/branch here if you wanted to give some feedback stuffs with testing on the new template: http://forums.torque3d.org/viewtopic.php?f=40&p=7638#p7638

Link to comment
Share on other sites

  • 1 month later...

Quick follow-up on this: but it looks like the binaries are being bit by something in the upload/download process and MacOS is deciding to quarantine the files, which prevents them from working properly. I remember @Azaezel mentioning one of their guys running into this as well in retrospect, but I'm not sure where the tangle-up would be coming from as the archive is uploaded right off the mac.

Best I can think off-hand is when downloading, because it's not signed and you have to manually OK it with the OS security settings, MacOS is auto-flagging the files for quarantine.


I'll keep poking around to see how we can deal with that so it's a non-issue going forward and then get the binaries re-uploaded once I do.


For the moment, you can either compile it yourself, or you can remove the quarantine attribute by opening terminal, navigating to where the binaries folder is, and typing in:

 

xattr -dr com.apple.quarantine Torque3D-310-MacBinaries

 

Assuming that the extracted folder name from the archive hadn't changed. That should remove the quarantine and let it run normally.

Link to comment
Share on other sites

Pretty sure our end was due to moving stuff over to the win box then tossing on our private SVN. Still, for the record, what was cropping up (and the apropriate workaround) was:

https://superuser.com/questions/898124/the-application-someapp-app-can-t-be-opened not it being unable to find the working directory, which is what that bit with it executing but not able to find main.cs seems to be, unless I'm misinterpreting the report.

Link to comment
Share on other sites

×
×
  • Create New...