I'm very bad at math as well, but I always try to find an out of the box solution or a workaround.
In may game players just run into each other, which is not a problem, since they usually kill each other before and if they hit each other they get pushed away through the physics engine.
I think the engine can already do what you need with the navMesh functionality, you only need to re-build the navmesh where the NPC is standing and the area the NPC is on will be cut out from the accessible area and therefore anyone trying to move onto the NPC position will stop before him, depending on the buffer radius you set to use in the navMesh.
I have no idea how to re generate the navMesh at runtime, but I saw @ Azaezel
doing it in his game, so AI players do not run into each other.
So maybe that solution will work, you can try it out manually first before implementing it fully, if the NPCs are static and do not move it is even simpler, then you can regularly exclude the area they are standing on out of the navMesh, so the player will always stop at an exact distance before it.