### Replacing TorqueScript with Lua

Scripting questions, discussions, etc

#### Re: Replacing TorqueScript with Lua

rlranft
Posts: 298
Joined: Thu Feb 05, 2015 3:11 pm

Well, that seems to be the only logical reason to have it there. Only I doubt that's very efficient. How hard is it to reconfigure the project, really? In CMake all you need to do is set some variables and then execute two commands. And that's it, you're set: you can now do what you need to do in the script while also having an engine that is custom-fitted to your needs. No redundant code sitting in the memory and never being used, faster loading times. I see only benefits in this approach.
I think you missed my point again. If you initialize the subsystems you want at run-time via script you only load the subsystems you want and so there is no "redundant code sitting in memory never being used." It just doesn't load the stuff you don't ask for. How will you allow your end-user to select the subsystems they want? Provide a special version of your game executable for each variant? Like Az said - T3D supports DirectSound and OpenAL for sound subsystems, so you provide the user the ability to select between them (or the null device, which initializes nothing). The same for renderers (though at the moment this is still mostly a moot point in T3D - Windows just uses Direct3D and Linux just uses OpenGL). 3D Game Engine Programming (while not the greatest book out there) shows a perfectly viable (and not uncommon) approach to creating your subsystems as .dlls that can be loaded, swapped, and unloaded at runtime. T3D doesn't use this particular approach, but the internal subsystem constructed in memory is the one you ask for and no other is created.

#### Re: Replacing TorqueScript with Lua

eugene2k
Posts: 6
Joined: Wed Aug 19, 2015 10:41 am
Why would the end-user need to select a different subsystem, though? The days when OpenGL handled stuff differently than Direct3D are pretty much dead and gone. Both APIs nowadays support essentially the same features. I think it's the same with sound APIs. And further, I said it in the previous post already, but if you want to offer the user a way of switching between two subsystems, it can be done with an in-engine function that is geared specifically for that, instead of a collection of functions.

#### Re: Replacing TorqueScript with Lua

rlranft
Posts: 298
Joined: Thu Feb 05, 2015 3:11 pm

Yes, OpenGL and Direct3D are achieving feature parity - that does not mean you can just throw the same data structures at both and get the same results. Likewise, DirectSound and OpenAL have wildly different APIs to achieve the same results.

Look man, I'm not making this up - go check out UnrealEngine, or CryEngine. This is industry standard practice. I'm not trying to justify it - you asked why, I was just trying to answer the question.

But do it however you like. You obviously have something in mind, so I'll stand aside.

Keep us apprised of your progress - I would love to use Lua in T3D, so I eagerly await your success.

#### Re: Replacing TorqueScript with Lua

eugene2k
Posts: 6
Joined: Wed Aug 19, 2015 10:41 am
Ehh... Not saying I will be able to get Lua integrated. I set my goals high, so that I could achieve more by the time I give up on the goal

And thanks for responding. I'm not trying to argue either, just trying to figure out if I'm missing something in my line of thinking.

How should I say this... I'm primarily studying Torque's architecture, since there's no design manual and no comments in the code as to why something was done one way and not the other. Just by looking at the code of a shipped commercial product one can't tell whether a certain part of the code was a conscious design decision or just someone being lazy or if this was a design decision born out of some concrete need instead of an imaginary one.

#### Re: Replacing TorqueScript with Lua

JeffR
Steering Committee
Posts: 847
Joined: Tue Feb 03, 2015 9:49 pm

My recommendation would probably be to try and implement lua alongside TS as a starter.

The DefineEngineMethod macros and the like are designed to act as the interop between whatever language you want, they're just currently geared towards TS. In theory you could modify them to work with any language you want.

Rather than stripping out what's there and does the same job just with a different language, I'd probably suggest trying to copy them and just make a permutation that works with lua alongside the existing one.

This gives you the learning benefit of implementing stuff yourself, so you learn it as you go, but you don't lose your frame of reference - in this case a working script interop. So if you hit a part you can't get working, you can just walk through the TS side of things to see how it works there and compare why yours isn't.

#### Re: Replacing TorqueScript with Lua

LukasPJ
Posts: 388
Joined: Tue Feb 03, 2015 7:25 pm

I've (almost) implemented C# into Torque6. Torque6 =/= Torque3D, but the base architecture is very alike.

I haven't read most of this discussion, but as JeffR said, I implemented it alongside TorqueScript, prioritizing C# so they'd be faster.

The branch is kind of big: https://github.com/lukaspj/Torque6/tree/C%23-Bridge but the general idea was to simply make a C-Interface for outgoing calls, and then for ingoing (i.e. C#->C++) I wrote C functions for these methods and simply DLL-imported them in C#. It'd be a lot different in Lua, but perhaps you could use some of it (such as the call C# or TS code).

I think these two commits might be of interest, they are where I make the engine call C# instead of TorqueScript:
https://github.com/lukaspj/Torque6/comm ... ff4bcd27fd
https://github.com/lukaspj/Torque6/comm ... a22ab8db6b

#### Re: Replacing TorqueScript with Lua

rlranft
Posts: 298
Joined: Thu Feb 05, 2015 3:11 pm

With Lua you could use Luna to make the task easier - it's a short template-based header chunk that is pretty easy to use:
/*
* File:   luna.h
*
* Created on November 15, 2013, 9:58 AM
*/
#pragma once

#include "lua/lua.hpp"
#include <string.h> // For strlen

template < class T > class Luna {
public:

struct PropertyType {
const char     *name;
int             (T::*getter) (lua_State *);
int             (T::*setter) (lua_State *);
};

struct FunctionType {
const char     *name;
int             (T::*func) (lua_State *);
};

/*
@ check
Arguments:
* L - Lua State
* narg - Position to check

Description:
Retrieves a wrapped class from the arguments passed to the func, specified by narg (position).
This func will raise an exception if the argument is not of the correct type.
*/
static T* check(lua_State * L, int narg)
{
T** obj = static_cast <T **>(luaL_checkudata(L, narg, T::className));
if (!obj)
return *obj;		// pointer to T object
}

/*
@ lightcheck
Arguments:
* L - Lua State
* narg - Position to check

Description:
Retrieves a wrapped class from the arguments passed to the func, specified by narg (position).
This func will return NULL if the argument is not of the correct type.  Useful for supporting
multiple types of arguments passed to the func
*/
static T* lightcheck(lua_State * L, int narg) {
T** obj = static_cast <T **>(luaL_testudata(L, narg, T::className));
if (!obj)
return *obj;		// pointer to T object
}

/*
@ Register
Arguments:
* L - Lua State
* namespac - Namespace to load into

Description:
Registers your class with Lua.  Leave namespac "" if you want to load it into the global space.
*/
// REGISTER CLASS AS A GLOBAL TABLE
static void Register(lua_State * L, const char *namespac = NULL) {

if (namespac && strlen(namespac))
{
lua_getglobal(L, namespac);
if (lua_isnil(L, -1)) // Create namespace if not present
{
lua_newtable(L);
lua_pushvalue(L, -1); // Duplicate table pointer since setglobal pops the value
lua_setglobal(L, namespac);
}
lua_pushcfunction(L, &Luna < T >::constructor);
lua_setfield(L, -2, T::className);
lua_pop(L, 1);
}
else {
lua_pushcfunction(L, &Luna < T >::constructor);
lua_setglobal(L, T::className);
}

luaL_newmetatable(L, T::className);
int             metatable = lua_gettop(L);

lua_pushstring(L, "__gc");
lua_pushcfunction(L, &Luna < T >::gc_obj);
lua_settable(L, metatable);

lua_pushstring(L, "__tostring");
lua_pushcfunction(L, &Luna < T >::to_string);
lua_settable(L, metatable);

lua_pushstring(L, "__eq");		// To be able to compare two Luna objects (not natively possible with full userdata)
lua_pushcfunction(L, &Luna < T >::equals);
lua_settable(L, metatable);

lua_pushstring(L, "__index");
lua_pushcfunction(L, &Luna < T >::property_getter);
lua_settable(L, metatable);

lua_pushstring(L, "__newindex");
lua_pushcfunction(L, &Luna < T >::property_setter);
lua_settable(L, metatable);

for (int i = 0; T::properties.name; i++) { 				// Register some properties in it
lua_pushstring(L, T::properties.name);				// Having some string associated with them
lua_pushnumber(L, i); 									// And a number indexing which property it is
lua_settable(L, metatable);
}

for (int i = 0; T::methods.name; i++) {
lua_pushstring(L, T::methods.name); 					// Register some functions in it
lua_pushnumber(L, i | (1 << 8));						// Add a number indexing which func it is
lua_settable(L, metatable);								//
}
}

/*
@ constructor (internal)
Arguments:
* L - Lua State
*/
static int constructor(lua_State * L)
{
T*  ap = new T(L);
T** a = static_cast<T**>(lua_newuserdata(L, sizeof(T *))); // Push value = userdata
*a = ap;

luaL_getmetatable(L, T::className); 		// Fetch global metatable T::classname
lua_setmetatable(L, -2);
return 1;
}

/*
@ createNew
Arguments:
* L - Lua State
T*	- Instance to push

Description:
Loads an instance of the class into the Lua stack, and provides you a pointer so you can modify it.
*/
static void push(lua_State * L, T* instance)
{
T **a = (T **)lua_newuserdata(L, sizeof(T *)); // Create userdata
*a = instance;

luaL_getmetatable(L, T::className);

lua_setmetatable(L, -2);
}

/*
@ property_getter (internal)
Arguments:
* L - Lua State
*/
static int property_getter(lua_State * L)
{
lua_getmetatable(L, 1); // Look up the index of a name
lua_pushvalue(L, 2);	// Push the name
lua_rawget(L, -2);		// Get the index

if (lua_isnumber(L, -1)) { // Check if we got a valid index

int _index = (int)lua_tonumber(L, -1);

T** obj = static_cast<T**>(lua_touserdata(L, 1));

lua_pushvalue(L, 3);

if (_index & (1 << 8)) // A func
{
lua_pushnumber(L, _index ^ (1 << 8)); // Push the right func index
lua_pushlightuserdata(L, obj);
lua_pushcclosure(L, &Luna < T >::function_dispatch, 2);
return 1; // Return a func
}

lua_pop(L, 2);    // Pop metatable and _index
lua_remove(L, 1); // Remove userdata
lua_remove(L, 1); // Remove [key]

return ((*obj)->*(T::properties[_index].getter)) (L);
}

return 1;
}

/*
@ property_setter (internal)
Arguments:
* L - Lua State
*/
static int property_setter(lua_State * L)
{

lua_getmetatable(L, 1); // Look up the index from name
lua_pushvalue(L, 2);	//
lua_rawget(L, -2);		//

if (lua_isnumber(L, -1)) // Check if we got a valid index
{

int _index = (int)lua_tonumber(L, -1);

T** obj = static_cast<T**>(lua_touserdata(L, 1));

if (!obj || !*obj)
{
luaL_error(L, "Internal error, no object given!");
return 0;
}

if (_index >> 8) // Try to set a func
{
char c[128];
sprintf_s(c, "Trying to set the method [%s] of class [%s]", (*obj)->T::methods[_index ^ (1 << 8)].name, T::className);
luaL_error(L, c);
return 0;
}

lua_pop(L, 2);    // Pop metatable and _index
lua_remove(L, 1); // Remove userdata
lua_remove(L, 1); // Remove [key]

return ((*obj)->*(T::properties[_index].setter)) (L);
}

return 0;
}

/*
@ function_dispatch (internal)
Arguments:
* L - Lua State
*/
static int function_dispatch(lua_State * L)
{
int i = (int)lua_tonumber(L, lua_upvalueindex(1));
T** obj = static_cast < T ** >(lua_touserdata(L, lua_upvalueindex(2)));

return ((*obj)->*(T::methods.func)) (L);
}

/*
@ gc_obj (internal)
Arguments:
* L - Lua State
*/
static int gc_obj(lua_State * L)
{
T** obj = static_cast < T ** >(lua_touserdata(L, -1));
T* instance = *obj;
if ((*obj)->isPrecious)
return 0;

if (obj && *obj)
delete(*obj);

return 0;
}

static int to_string(lua_State* L)
{
T** obj = static_cast<T**>(lua_touserdata(L, -1));

if (obj)
lua_pushfstring(L, "%s (%p)", T::className, (void*)*obj);
else
lua_pushstring(L, "Empty object");

return 1;
}

/*
* Method which compares two Luna objects.
* The full userdatas (as opposed to light userdata) can't be natively compared one to other, we have to had this to do it.
*/
static int equals(lua_State* L)
{
T** obj1 = static_cast<T**>(lua_touserdata(L, -1));
T** obj2 = static_cast<T**>(lua_touserdata(L, 1));

lua_pushboolean(L, *obj1 == *obj2);

return 1;
}
};

I did not write Luna - it's available from http://www.lua.org somewhere in all of the binding discussion links...
For examples of it's usage, see https://github.com/RichardRanft/LuaVMTool

#### Re: Replacing TorqueScript with Lua

buckmaster
Steering Committee
Posts: 321
Joined: Thu Feb 05, 2015 1:02 am
Since we were talking about engine startup I'm going to mention t3d-bones, which contains a very minimal example of starting up the engine. My 'minimal' I mean it still has 1MB of scripts in the sys/ directory, but at least it's a bit more approachable. Also, since credit is due: Michael Hall is responsible for pretty much all of sys/; I figured out most of the stuff outside it.
No redundant code sitting in the memory and never being used, faster loading times.
@ eugene2k while I tend to agree with making the engine more configurable at compile time, this strikes me as wishful thinking. T3D is pretty tiny at ~15MB, and unused code will be sitting in RAM, not in your processor's cache; compared to other game assets I think the code is probably the smallest memory worry. As for faster startup times, I'm very skeptical that you'll gain much. Bigger gains are to be had optimising the code that does run rather than removing code that doesn't.
The days when OpenGL handled stuff differently than Direct3D are pretty much dead and gone. Both APIs nowadays support essentially the same features.
That may be true, but driver support still isn't ubiquitous. Though if we're talking about D3D maybe it's a moot point; are there any known cases where OpenGL works better on Windows?

#### Re: Replacing TorqueScript with Lua

eugene2k
Posts: 6
Joined: Wed Aug 19, 2015 10:41 am
and unused code will be sitting in RAM, not in your processor's cache; compared to other game assets I think the code is probably the smallest memory worry. As for faster startup times, I'm very skeptical that you'll gain much. Bigger gains are to be had optimising the code that does run rather than removing code that doesn't.
I agree with all that. I suppose I was a bit unclear, though. There's a lot of stuff done in TS that should be implemented inside the engine. And while it would make the engine a tiny fraction more efficient, the biggest gain would be that that particular bit of code will be easier to maintain than the whole TS + Needlessly exported functions mess. Case in point: adding a new scripting language would've been a lot easier if a bare-bones torque app didn't have 1MB worth of scripts Thanks for the t3d-bones branch, by the way. Now I don't have to go the long way trying to figure out what is actually needed in a bare-bones Torque app.

#### Re: Replacing TorqueScript with Lua

buckmaster
Steering Committee
Posts: 321
Joined: Thu Feb 05, 2015 1:02 am
Yeah, I definitely agree that having more stuff in the engine would make it easier to ditch TS - and something we were chatting idly about within the SC around the start of the year was starting to port the editor suite into C++ for just that reason. But the effort involved would be monumental.

#### Who is online

Users browsing this forum: No registered users and 2 guests