Thoughtcrime is a subtle and many-faceted concept. I am surprised that only the actual crime aspect of it has been noted here.
In order to understand thoughtcrime a few points about newspeak must be noted. First, newspeak as a language is narrowing the range of thought by cutting down the number of words available to describe things and not introducing unnecessary concepts. For example, 'bad' is replaced by 'ungood'. Also, the purpose of this narrowing of the range of thought is for the control of the people in Oceania. By not providing the words necessary to describe unacceptable concepts, the Party plans to limit the range of thought to only those things that are 'good'. Good means 'in the interests of the Party', basically, and has strong connotations of 'orthodox'. Thus, 'ungood' really means 'unorthodox'.
As can be seen, simple translations of newspeak do not suffice for an examination of its meaning. I think that's where the confusion here is - many people think of newspeak as a kind of grammar for English. It's not. It's a complete other language in which the words have different loadings and connotations to what they have to us today. Thoughtcrime is not just any crime committed in your thoughts (btw, boytext, it's newspeak that is the thought control device not thoughtcrime itself). We can't think of 'crime' in this sense as being the crime we have today, the breaking of laws. There are no laws in 1984. Thoughtcrime, if you will, basically means 'unorthodox thought', or 'independent thought'. It thus serves the purpose of ensuring that there is no positively connotated word for independent thought in newspeak. People who thus think counter to the orthodoxy are thoughtcriminals, not 'free thinkers' (no such word as free in the final version of newspeak), and it is in this way that you should apply the word (unless, as Shanoyu has done, you're drawing comparisons). They feel guilty purely because of the concepts attached to the act they perform, rather than the act itself. This is a powerful deterrrent to free thought.
I have a thing or two to say about this. It's doubleplusungood (yeah, I know, I'm applying current meaning to newspeak words). Thoughtcrime is not a crime. Thinking of it as such is precisely what the word was created to ensure. When summer comes I'm getting the following design printed in black on a white t-shirt:
>>thoughtcriminal<<
Thoughtcrime, in the newspeak sense, is nothing to be ashamed of. I reject the rule of the Party. Down with Big Brother.