Even with my admittedly limited mathematic skills, this just doesn't work AT ALL. Okay so say the two numbers chosen are... 221 and 65.
a= 221
b=65
so n is 221%65(don't use 17 with 221, produces a modulo of 0) is 26
n=26
and lets say that we encrypt the value of 'A' which is 65.
x=65
since c is (x^3)%12 so c will be: 5
c=5

now lets try decoding it:
since x = (c^((a OR b +1)/6))%12
so x will be: (c^11)%12 OR (c^37)%12
x=5

is it me or is this not working?