after you've read Computer Architecture by A.S. tanenbaum you might wanna take a look at this (free) tool called
IDaSS (http://www.xs4all.nl/~averschu/idass/). It's a simulator for digital circuits, wich includes a basic processor. with this your will be able to write your own instruction set and see it actually work in a simulation. It includes writing instructions for the state controller, ALU and memory operations.

PM me if you want a working schematic for an (8-bit) processor, that works in IDaSS