From Application Debugging by Robert Binder (Prentice-Hall, Inc., 1985)

The term debugging was apparently coined in 1945 byGrace Murray Hopper , who was later to be instrumental in the committee which defined the first industry standards for COBOL. The machine involved was the MARK II , the second of four electromechanical machines built by the Harvard Computation Labratory under the descretion of Howard Aiken for the US Navy and Air Force. The Mark series was a conceptual precursor to the vacuum tube machines designed and built by Eckert, Mauchly, and von Neumann. The MARK II was designed towards the end of 1944 and put to use by the Navy's Dahlgreen Proving Ground in Virginia for calculating ballistic trajectory tables. It was programmed by punching instruction code on paper tape. It had storage for 100 ten digit decimal numbers ( ) It could multiply two such digits in roughly 0.4 seconds. Hopper describes how debugging came to be used in this enviroment:

"In the summer of 1945 we were building the MARK II; we had to build it in an awful rush--it was wartime--out of components we could get our hands on. We were working in a World War I temporary building. It was a hot summer and there was no air-conditioning, so all the windows were open. MARK II stopped, and we were trying to get her going. We finally found the relay that had failed. Inside the relay--and these were large relays--was a moth that had been beaten to death by the relay. We got a pair of tweezers. Very carefully we took the moth out of the relay , put it in the logbook, and put scotch tape over it.

Now commander Howard Aiken had a habit of coming into the room and saying "Are you making any numbers?" We had to have an excuse when we weren't making any numbers. From then on if we weren't making any numbers, we would tell him we were debugging the computer. To the best of my knowledge that's where it started."
Heh, here is the pic of the bug taped to the logbook...