No, this is not crazy. What you may be experiencing is a compulsion to jump. So when you have this thought that you may jump when you are on top of a tall building, it scares you as you don't really want to hurt yourself. If you really wanted to hurt yourself, then you would not be scared. But because you don't really want to jump, but have this thought anyways, you avoid going to the top of tall building and bridges. So it scares you to go on top of tall buildings and bridges because you can't trust yourself with that seemingly 'crazy' thought, as you don't really want to hurt yourself.
If you ever have that thought of jumping, but you don't have the fear, then you should be concerned as you may being having suicidal thoughts. If you do have suicidal thoughts, then you should go to your nearest emergency room immediately, for an emergency psychiatric assessment.
