If you get the error "(-12) Invalid returned data from license server system":
you are probably trying to use Mathcad Prime 6.0 with the old FLEXlm that you can install from the Mathcad 15.0 installation media:
To get floating licenses for Prime 6.0 to work you need to upgrade to a newer Flex. Please refer to the article Installing Flexnet.