Locked, the problem with your analysis is you've chosen an a and b such that your n turns out to be less than your x. Because your x is greater than the modulus it is impossible to revert back to it from the ciphertext because the only unique values are the class representatives of the modulus. Everything else is simply x + n*k and there's no way to know what k is, hence why the modulus NEEDS to be greater than any x you apply it to.

On to the algorithm...
This resembles RSA almost identically, obviously the d, e, and p/q are chosen differently. The fact that your public exponent is static is troublesome though, this means any a & b that give that modulus n could be used to decrypt your message assuming they meet your 5 mod 12 requirements. I'm not in a position to prove how hard that would be, I can't think of a proof off the top of my head and I'm running short on time at the moment. However the strength in RSA is that the exponent is generated from phi(p*q) and the decryption exponent relies on the encryption exponent through this value which is only computable if you know p & q which should be near impossible to factor from the public n if you've chosen them correctly. You make no statement on the size of your n and you also make no proof that a & b are primes, meaning n may be easily factorable which in turn means you could find the d that would work for your n by finding the inverse of 3 mod phi(n) if no little catches show up from the a & b chosen.

This is also unbelievably vunerable to a broacast attack. Send your message to 3 different people using their public keys, assuming they're each unique, and this is broken. You can simply build the message back from the ciphertext using the same technique used on the toy RSA model that first brought about the idea.

You could argue that this isn't meant for public/private key systems and it might be acceptable depending on the size of your a & b but unless you can prove that the 5 mod 12 actually ensures the decryption method works ONLY for those two numbers then your system is even weaker.

For example.. somehow you get a mod of 26 like Locked came up with. If I can come up with numbers such as 56 and 30 that give me the same n and they work, then you have a problem. I'm not saying those numbers work, I doubt they do, I just added 30 to 26. But if a situation like this exists then it might be quite feasible to just solve for a smaller a & b than were actually used, making brute forcing the key even easier.