Page 1 of 1

WorkBlog - July 2020, Part 2

Posted: Sat Aug 01, 2020 8:44 am
by JeffR
Minor delay because I was feeling under the weather.

Anyways! Part twooooo!

Engine Work
Predictably, a bunch of various improvement and fixes have gone in on the engine side of things. Various crash fixes, out of bounds fixes, etc.
Beyond that, though some other nice stuff has happened. A big one was moving the asset importing logic up from the tools scripts into the engine. This was done for a few reasons.
One was processing speed. The native C++ is obviously going to run faster than TS, so in general, the processing time for import has improved. But bigger than that is the ability to do in-place auto-importing.
Similar to how, say, a TSStatic would reference a DAE file. If no cached.dts file was found, it would automatically run the import of the DAE file, and generate the cached.dts file. The assets system can now operate in the same way. In the event a non-asset loose file is utilized, we can try a multi-step process to handle that.
  1. We would first try and find any asset that utilizes that particular file, and if found, we just utilize that asset and update the TSStatic's fields to reflect it.
  2. If that fails, we then try and run an in-place auto-import. If the tools folder exists we even try and look up if a default import config is set in the editor preferences, otherwise we just use the default import settings. If the import is successful, we like above return the new assetId and that is updated in the TSStatic's fields.
  3. If all that fails, we can return back a placeholder assetId(in TSStatics' case, it's a noShape mesh) to indicate 'hey something went wrong with loading' to the level editor, while still retaining the original fields so debugging can be done to find out what went wrong without contaminating the setup.
We've been beating the system up on TSStatics and a few remaning issues are there, but it's proven to be rather fast and robust. And because it can defer to an importConfig, it's customizable to the project's needs, as well as utilizing Assimp for multiple other file formats like FBX.
Once the last bits are plugged, we can then move on to implementing it for all the 'regular' types, shapes, images, materials, etc.
This means that legacy projects being imported in will auto-process a ton of content into assets without needing to do any manual work, which is a huge timesaver, and non-legacy projects have it as a handy fallback for workflow efficiency.

There was also various improvements to the PBR and IBL stuff to make the math more correct. It'll continue to get refined as we find assets and scenes that throw odd results, but it's pretty robust at this point, both for new, PBR-friendly assets:
Image

And even non-PBR legacy assets can end out looking pretty solid by just being tossed in with no additional work(with some bonus footage of the asset import process doing it's thang)


Beyond that, some other various improvements/expansions on the engine side in no specific order:
  • Shifted glow over to a full separate RBG image map, allowing a lot more artistic control
  • Added the ability to in-place insert inspector fields and groups, allowing arbitrary editor hook ins(an example of this would be if ECS was a module, it could be dropped in and just inject the inspector group for components into an entity in the inspector without having to hard-code the fields. This allows flexible and dynamic inspector and editor behavior and opens up a LOT of cool options for the future
  • Added ability for text edit fields to have placeholder text rather than needing to do that janky voodoo of setting it's text to some placeholder value and hoping nothing goes funky when typing
  • Fixed CPU detection and reporting on 64bit windows(courtesy of Hutch)
  • Functionality expansions to groundcover spawning
  • Corrected some erroneous behavior with terrainAssets and terrainMaterialAssets so everything works together more predictably
  • Added a guiVariableInspector field type 'Range' for slider support in inspectors
  • Fixed OGL handling of probe baking - very important for linux and mac for obvious reasons
  • Whole lotta cleanup of uninitialized vars which should help with stability in the engine by preventing unexpected default values
  • Fixed an issue with Assimp loading where shape scales were off(courtesy of Mars)
  • Added ability for shadows to work with translucent materials utilizing alpha test
And a sprinkling of other stuff like crashfixes, as mentioned prior with several contributors helping out on that end like Chaigler, Bloodknight, Tony, Hutch and Mars.

There's a number of other stuff cooking at the moment getting worked on, but a few highlight bits that I'll actually be making PRs for over the weekend are performance oriented.
Namely, the ability to, via preference variables(as sometimes you don't want it, or want to leave it up to the end user) to force on static object fading, as well as fading of local light(point and spot) and limiting how many local lights can be rendered at a time.
These are pretty handy for lower-end machines because whatever standard tricks to help performance in rendering - reducing shadow resolution, not rendering shadows at all, LODs, etc - ultimately the fastest possible thing to render is absolutely nothing at all ;)

So in action, that all looks like this:

In the first part you can see that in the current setting, I have a maximum of 3 local lights allowed, so the others in the grid are culled, based on score which is a calculation of the lights' size, brightness and distance from the camera.
Then as the camera pulls out you can see that the lights smoothly fade out based on the light fade setting. There's a start and end fade distance. So, say, fading doesn't start until the light is 50m away, and will fully fade out by 75.
Then I flip on the object fade setting and we can see that now the static objects behave similarly. Based on their size, they can begin fading out sooner, so small clutter objects can be quickly culled from rendering without negatively impacting the fidelity of the scene up close, while larger objects are slower to fade out.
Additionally, there's a setting that allows the defining of an object size considered large enough it shouldn't ever be culled like this, which is useful for landmark objects and the like.
The idea being that these can be adjustable for the end user via some options menu settings, so people with weaker computers can have this stuff kick in fairly aggressively(in combination with other options to lower render cost) giving more control and options for making the game run better without negatively impacting those with powerful enough rigs to render everything at full blast.
When I get these added in, I can then add options to the graphical options menu to facilitate this. The dev can set the desired baseline behavior, and the preference would permute it to be more aggressive as needed(or disabled entirely for an 'ultra' setting.

Next, tools stuff!

Tools Stuff
The big one would be updates to the Asset Browser.
Image
Image

There's a LOT to unpack here, so lets get into it.
From the notations in the first image, we can see...well, a lot of stuff, so lets walk through it.
First, you can note a modified layout from what was there before, which is, hopefully, overall clearer and easier to understand most of it at a glance. Refinements will continue, of course, but this should(especially the utility buttons at the top) be much more immediately legible.
So, per the notations:

New Asset
Slap this to create a new asset in the current path, via a popup listing of all acceptable types. Can also just RMB in the main browser window.
Image

Import File
Opens a 'open file' dialog to go rooting around for valid file types to import in as new assets into the current path. Can also do the same via Drag-n-Drop.

Filter Options
A popup that lets you easily toggle various filter options, like filtering by asset type.
Image

Auto-Import in Current Directory
This one is a cool convenience feature. It'll run through the current directory and run an in-place auto-import on any loose files(files that don't have an associated asset that owns it). Another handy feature for importing of legacy projects. This button doesn't display if there are no non-asset loose files in the directory.

Complex Search
Ok, now we get into the FUN stuff.
Before, search was pretty basic, conceptually. You typed something in, like say 'wood' and any asset with a matching name would be filtered for display in the main view.
Now, however, we have complex search phrasing logic.
Take a search string like this:
type:materialAsset;tag:wood;old
What this would do is search in the current directory(and any child folders, of course) for any asset that is:
  • A materialAsset
  • Has a tag of 'wood'
  • Has 'old' in it's name
So the end result would be any asset that is, effectively, a material asset of a wood type with old in the name, like a hypothetical 'Old_Ship_Wood_Material' asset.
Neat, right?
This feeds back into the filter options above, so flipping on/off the type filters automatically updates the search phrase. Poking any of the 'tag' section entries from the second Asset Browser image likewise toggles in/out tags.
It also can support multiple words per 'term', via comma separation.
So if we did:
type:materialAsset;tag:wood;old,new
It'd find any materialAsset with a tag of wood that has old Or new in the name.

Pretty powerful, to be sure! But it gets niftier. Because as neat as complex searches like that are, having to flip the various tag and filter values on, and then type a relevent name to always look up, say a common set of materials you use is a pain every time.
If only there was some way to save them...

Save As Collection Set
Oh wait, we totally can. If you slap that little button there, you get a name prompt, and then it saves the current search phase into a collection set of that name, which then shows up in the Collections section in the Asset Browser.
You can then hit that item and it automatically fills out the search term for you with that saved one. So just like that we click a single item and we are back to seeing only materialAssets with the tag wood and old or new in the name.
This can be pretty huge if you have collections of associated assets and can save a lot of time from having to constantly dig around to find them again.
Basically, the general idea is it doesn't really matter where or how you ultimately organized your assets(or maybe you didn't organize them yourself at all like if you got an art pack), you should be able to find what you're looking for quickly and easily.

Asset Preview Size Slider
Not quite as cool as that, but still important, is the slider for the Asset Preview size. It...uh, adjusts the size of the Asset Previews. In fact, if you crank it all the way down to the smallest, it turns into a list for maximum browsability!
Image

Given that the plan is for the asset browser to replace the old Creator, Meshes and Library tabs in the Scene Window(with the ability to make the AB dock into that window so it doesn't always take up a big ol' chunk of your screen) this is very useful for being able to actually see what the heck you're working with in the current directory.

Topically, but in the earlier updated AB walkthrough image, you can see not only the Tags section, but the Creator section. The ability to create new objects like how you do in the creator tab will all end up over there, so the Asset Browser will be a one-stop-shop for all your content browsing needs. You can even browse and spawn 'scripted' objects via their datablocks!

So spawning behavior is consistent, and it'll work with dang-bloody-near everything. So yeah, pretty slick.

Also, you may have noticed some colored outlining in the screenshots around the previews. The idea there is that different asset types would get different outline colors so even in list mode you can at a glance determine if it's a material asset, or shape asset or whatnot. Minimize confusion, maximize quick parsing of what content you're working with, you know.

Topical to looks, there's some hard-to-read text and other funkiness in those shots. The editor theme will be cleaned up and more or less finalized based on what was learned with the BaseGame UI's theme, so it'll all be much more legible and visually consistent when we're done.

For one last shot, A quick gander at the current Asset Import Window:
Image
(Remember, the hard to read text'll be fixed ;) )

Under normal circumstances, you won't likely even see this window unless you intentionally disable the 'Auto Import with Default Import Config' setting in the editor preferences, but just to highlight some cool logic the importer can do:
  • I dragged and dropped just the shape file into the Asset Browser window.
  • It then parsed the materials from the mesh, and added it to the importing list of assets, namely the material GothicCabinet_01, and automatically childed to the shape.
  • Then, it noted that the material had an image assigned to it in the shape file, so it grabbed that, and childed it to the material. The image asset is that GothicCabinet_01_8-bit_Diffuse (sweet naming scheme, am I right?). It recognized this as an albedo image via the 'diffuse' suffix on the end, so when everything is imported, it'll automatically be slotted into the materials' albedo image slot.
  • It then proceeded to scan the original file's directory for potentially related images based on naming convention. It looked for GothicCabinet_01_8-bit with a bunch of different, common suffxes on the end based on various types. It found a file that matched the name but had _NORMAL on the end, so it also brought that into the import and childed it to the material as well. Like our diffuse, it recognized the suffix so it knows it's a normal map, and will ultimately fill in that image into the materials' normalMap slot.
All of that is driven by the settings of the import configuration, and ultimately meant that when I said 'import this model' it automagically found and processed in a bunch of associated assets saving me from having to manually import in the images and assign them to the material myself, which over the course of development can save quite a lot of busywork time.

I've shown off the general thrust of that before, but The UI is in a better space now to convey how things associate, and I thought it was worth re-iterating how 'smart' the import tries to be.
In fact, you may note from the image that it considered the Shape file a 'duplicate asset'. This is because the naming convention of the shape and material collide as they both have the same name, which could cause reference problems. I can either let it auto-resolve the problem by, say, renaming the offending asset, or having it auto-append _Mat to the material name, etc based on import config options.

And the entire import processing is logged now, so if anything happens that is unexpected, you can go back through the logs and see exactly what happened, where, and why.

No system is infallible, but I'm definitely shooting for as robust as feasibly possible at least ;)

Ok, wow! This part actually ended up being longer than I anticipated, so we'll be doing a part three where I cover the remainder bits, and then a whats-what of whats left!
Exciting!
See you tomorrow!

Re: WorkBlog - July 2020, Part 2

Posted: Sat Aug 01, 2020 6:30 pm
by LoLJester
Great post, Jeff! You guys are doing a great job. So many nice new features. :)
Looking forward to a stable 4.0!

Re: WorkBlog - July 2020, Part 2

Posted: Sat Aug 01, 2020 7:03 pm
by Marson
Thank you Jeff, very cool!