Sorry guys! Halloween was busy for me!
Ok, so yeah the just of what everyone is saying is correct.
If you want to add a new library and then have it be accessable in script, you have a few steps to do:
1) add it to the engine via a library in CMAKE, and then compile it into the engine. As Jason said, you can look at the existing libraries for how to do that(though a wiki page should get made about it). But you'll have a file for the library in cmake, and then add a reference to that via addLibrary(myLibrary) in the torque3d.cmake file so it gets actually included on generation.
2) Once it's in and compiling, you'll want to have a way to actually call it to do things. This is where what @ irei1as said comes in.
You have 2 main macro functions you can add to a file in the engine so that you can call stuff from script:
DefineEngineMethod and DefineConsoleFunction.
DefineEngineMethod basically attaches a script function to the namespace/object class in question.
So if you made a new object derived from ScriptObject to do whatever your library does, you could then add something like:
DefineEngineMethod(MyNewObject, myFunction, void, (Point3F position), (Point3F::Zero), "This is a test function that calls a function in myNewObject")
As you can see from above, if we have our new ScriptObject derived class in the engine - MyNewObject in this case, this will add/expose a script-side function 'myFunction' to any object of MyNewObject type.
So in script, you could then do:
%myObject = new MyNewObject();
And it will go 'ok, so %myObject is of the class MyNewObject, and yep, there's a script function myFunction, so we'll pass %aPosition in and that gets turned into the 'position' argument.
This then calls object->myEngineSideFunction (object in this case being the object you're calling this script function on, and myEngineSideFunction being whatever function in that class you're trying to call) and we pass our position argument into it.
If you're not making a new object class and just want to call something, like say, a math function, you do DefineConsoleFunction, which would look like this:
DefineConsoleFunction(myFunction, Point3F, (Point3F position), (Point3F::Zero), "This is a test function")
Point3F magicNumber = Point3F(10,9,100);
return magicNumber + position;
As you can see, it's similar, but not identical. We don't have the MyNewObject bit in there, because this doesn't associate to any given class or namespace, it exists in the global namespace.
We also have a different return type after the function name, instead of void like in the prior example, this one will return a Point3F vector. We similarly pass in a vector argument.
So in script, you would do:
%result = myFunction(%aPosition);
We don't have a namespace, so we call the function directly, passing in our argument and it'll return a result.
In practice, you would have your function there call into your library's functions to do whatever you're trying to accomplish, but the setup is the same.
Hopefully that explains it! Let me know if you have any other questions or something doesn't make sense