Is your answer yes or no?

The answer for Yes/No questions sometimes puzzle me a lot because the answer for a negative question is opposite to Japanese one. I mean Japanese answer is like this "Yes, I didn't do that." I always try to avoid mistaking, but I have answered in Japanese style many times. Therefore, I often confuse other people and in some case, I'm misunderstood. Recently, I stay away saying just "yes" or "no", instead, I make long answer. Probably, this leads less misunderstanding.