One panic attack does not an anxiety disorder make. An isolated panic attack (or anxiety attack) can occur from multiple triggers, and does not neccessarily mean that you have an anxiety disorder such as panic disorder. A panic attack can be triggered by being exposed to a highly feared object (a mountain lion) or situation (standing over the ledge of a skyscraper). People who experience panic attacks from these feared stimuli do not need to have an anxiety disorder.
Having an anxiety disorder is different. When you have an anxiety disorder, you have anxiety symptoms frequently, intensely, and chronically, and this in turn severely impacts your functioning in relationships, school, job, and self-care. When you have an anxiety disorder, you don't just have one isolated panic attack- you have them frequently, intensely, and chronically.
Only when you have multiple panic attacks should you be worried- at that point, a visit with your doctor is warranted for an assessment.